video gets stuck when its seeked back (C++) (QT) (libvlc)

Question

I am using libvlc to play video on my GUI Application using QT 4. When I seek forward the video it works fine, but when I seek back the video, it gets stuck I mean frame doesn't change and my callback function which is videopostrender doesn't get any call.

    void videoPrerender(void *p_video_data,
                    uint8_t **pp_pixel_buffer,
                    int size)
{
    // Locking
    //printf("cbVideoPrerender %i\n",size);
    //printf("vtest: %lld\n",(long long int)p_video_data);
    VLCPlayer* player = reinterpret_cast<VLCPlayer*>(p_video_data);
    player->m_mutex.lock();
    if ((size > player->m_videoBufferSize) || !player->m_videoBuffer)
    {
        printf("Reallocate raw video buffer\n");
        if(player->m_videoBuffer) free(player->m_videoBuffer);
        player->m_videoBuffer = (uint8_t *)malloc(size);
        player->m_videoBufferSize = size;
    }
    *pp_pixel_buffer = player->m_videoBuffer;
}

void videoPostrender(void *p_video_data,
                     uint8_t *p_pixel_buffer,
                     int width,
                     int height,
                     int pixel_pitch,
                     int size,
                     int64_t pts)
{
    Q_UNUSED(p_pixel_buffer)
    Q_UNUSED(size)
    //printf("cbVideoPostrender %i\n",size);
    // Unlocking
    VLCPlayer* player = reinterpret_cast<VLCPlayer*>(p_video_data);
    player->m_mutex.unlock();
    player->m_frameWidth = width;
    player->m_frameHeight = height;
    player->m_framePixelPitch = pixel_pitch;
    player->m_framePts = pts;
    if(player)
    {            
        player->m_frameImage = QImage((uchar*)player->m_videoBuffer,
                         width, height,
                         QImage::Format_ARGB32_Premultiplied);
        if(player->isLocalFile())
        {
            // Need not use timer based playing mechanism for local files.
            int interval = 100 / player->videoFPS();
            while(!player->m_elapsedTimer.hasExpired(interval))
                qApp->processEvents();

            player->m_elapsedTimer.restart();
        }

        if(player->record_state() == VLCPlayer::Playing)
        {
            player->emitRecorderTimeChanged();
        }

        qDebug()<<player->m_framePts;
        emit player->frameReady(player->m_frameImage);
    }
}

EDIT 1 - Adding more code for better understanding

    VLCPlayer::VLCPlayer(const QStringList &args, QObject *parent)
    : QObject(parent),
      m_frameWidth(0),
      m_frameHeight(0),
      m_framePixelPitch(0),
      m_framePts(0),
      m_videoBuffer(0),
      m_audioBuffer(0),
      m_videoBufferSize(0),
      m_audioBufferSize(0)
{
    d = new VLCPlayerData;

    qRegisterMetaType<VLCPlayer::VLCPlayerState>("VLCPlayer::VLCPlayerState");

    char **argv = (char **)malloc(sizeof(char **) * args.count());
    for (int i = 0; i < args.count(); ++i)
        argv[i] = (char *)qstrdup(args.at(i).toUtf8().data());

    // VLC options
    char smem_options[1000];
    sprintf_s(smem_options
            , "#transcode{vcodec=RV32,acodec=s16l}:smem{"
            "video-prerender-callback=%lld,"
            "video-postrender-callback=%lld,"
            "audio-prerender-callback=%lld,"
            "audio-postrender-callback=%lld,"
            "audio-data=%lld,"
            "video-data=%lld},"
            , (long long int)(intptr_t)(void*)&videoPrerender
            , (long long int)(intptr_t)(void*)&videoPostrender
            , (long long int)(intptr_t)(void*)&audioPrerender
            , (long long int)(intptr_t)(void*)&audioPostrender
            , (long long int)(intptr_t)(void*)this //This would normally be useful data, 100 is just test data
            , (long long int)(intptr_t)(void*)this); //Test data

    const char * const vlc_args[] = {
        "-I", "dummy", // Don't use any interface
        "--sout-x264-preset=ultrafast",
        "--file-caching=100",
        "--disc-caching=100",
        "--live-caching=100",
        "--network-caching=300",
        "ffmpeg-hw",
        //        "--ignore-config", // Don't use VLC's config
        //        "--extraintf=logger", // Log anything
        //        "--verbose=1", // Be verbose
        //        argv,
        "--sout", smem_options // Stream to memory
    };

    d->libvlcInstance = libvlc_new(sizeof(vlc_args) / sizeof(vlc_args[0]), vlc_args);

    d->libvlcMediaPlayer = libvlc_media_player_new(d->libvlcInstance);

    d->libvlcEventManager = libvlc_media_player_event_manager(d->libvlcMediaPlayer);




void VLCPlayer::setPosition(float position)
{
    if(!d->libvlcMediaPlayer)
            return;

        libvlc_media_player_set_position(d->libvlcMediaPlayer, position);
    }

    void VLCPlayer::updatePlayingState()
    {
        qDebug()<<"updatePlayingState";
        this->setState(VLCPlayer::Playing);
        QMetaObject::invokeMethod(this, "updateStateTimer", Qt::QueuedConnection);
    }
void VLCPlayer::emitPositionChanged()
{
    if(!d->libvlcMediaPlayer)
        return;

    qDebug()<<"emitPositionChanged";
    emit positionChanged(libvlc_media_player_get_position(d->libvlcMediaPlayer));
}


void VLCPlayer::libvlc_callback(const libvlc_event_t *event,
                                void *data)
{
    VLCPlayer* player = (VLCPlayer*) data;
    if(!player)
        return;

    switch(event->type)
    {
    case libvlc_MediaPlayerMediaChanged:
        break;
    case libvlc_MediaPlayerNothingSpecial:
        break;
    case libvlc_MediaPlayerOpening:
        player->setState(VLCPlayer::Opening);
        break;
    case libvlc_MediaPlayerBuffering:
        player->setState(VLCPlayer::Buffering);
        break;
    case libvlc_MediaPlayerPlaying:
        player->updatePlayingState();
        break;
    case libvlc_MediaPlayerPaused:
        player->setState(VLCPlayer::Paused);
        break;
    case libvlc_MediaPlayerStopped:
        player->setState(VLCPlayer::Stopped);
        break;
    case libvlc_MediaPlayerForward:
        break;
    case libvlc_MediaPlayerBackward:
        break;
    case libvlc_MediaPlayerEndReached:
        player->setState(VLCPlayer::EndReached);
        break;
    case libvlc_MediaPlayerEncounteredError:
        player->setState(VLCPlayer::Stopped);
        break;
    case libvlc_MediaPlayerTimeChanged:
        player->updatePlayingState();
        player->emitTimeChanged();
        break;
    case libvlc_MediaPlayerPositionChanged:
        player->updatePlayingState();
        player->emitPositionChanged();
        break;
    case libvlc_MediaPlayerSeekableChanged:
        break;
    case libvlc_MediaPlayerPausableChanged:
        break;
    case libvlc_MediaPlayerTitleChanged:
        break;
    case libvlc_MediaPlayerSnapshotTaken:
        break;
    case libvlc_MediaPlayerLengthChanged:
        player->emitLengthChanged();
        break;
    case libvlc_MediaPlayerVout:
        break;
    default:
        player->setState(VLCPlayer::Unknown);
        break;
    }
}

Gillead

You should probably try to create a minimal reproducible example to have a better chance of someone being able to help. Now you don't even show the "seek back" call, that causes it to get stuck, as far as I can tell. – Dail

One thing to check, is it guaranteed that videoPostrender is always called after videoPrerenderer. Is that kind of mutex use from the docs/examples? It seems rather risky design, it's very easy to leave mutex accidentally locked, unless there are strong guarantees, that this is how library expects your code to do it. – Dail

One thing that might help is printing thread IDs and locks/unlocks of that mutex everywhere in your code. That way you can see if you try to double-lock the mutex in the same thread, and also if you need the mutex at all (if it is always locked in the same thread, then it actually does nothing at best, leads to deadlock at worst). – Dail

Also, this is nonsensical code: player->m_framePts = pts; if(player) { ..... If player is nullptr, then it crashes before your if tests for it. – Dail

Also, calling qApp->processEvents(); yourself is often asking for trouble, as it is hard to control what code ends up called by it, code which might do something funny (in this case it could for example call code which resets your timer, when you "seek back"). – Dail

hey @Dail thanks for your reviews, I have added more code as you suggested. – Gillead

Yeah, I have checked the usage of the mutex , it's not creating any kind of a deadlock. – Gillead

i can't remember the details but i stopped using smem because of problems like these and followed this example: wiki.videolan.org/LibVLC_SampleCode_SDL – Blastomere

in addition to @hyde, make sure the callbacks and stop request are in different threads. otherwise it will be a deadlock. make sure the vlc callbacks are firing non-blocking qt signals – Blastomere