wxScrollWindow and Video Stream

[renatone]

Hello everybody, nice to join the forum!
I am trying to adapt the following code to my needs (many thanks to PBfordev, the Author!) https://github.com/PBfordev/wxopencvtest
The code consists (very basically) in a worker thread that polls every few ms an OpenCV Videocapture instance and if data is available emits a wxEVT_CAMERA_FRAME event. The (main thread) event handler converts the openCV image in a wxBitmap, and calls the setBipmap routine of the following class, derived from wxScrolledCanvas, which in turn calls the paint handler OnPaint:

Code: Select all

class wxBitmapFromOpenCVPanel : public wxScrolledCanvas
{
public:
    wxBitmapFromOpenCVPanel(wxWindow* parent);  
    bool SetBitmap(const wxBitmap& bitmap, const long timeGet, const long timeConvert);
private:
    wxBitmap m_bitmap;
    .....
    virtual void OnPaint(wxPaintEvent&);
};

The OnPaint routine finally paints the bitmap on the screen, together with some overlays.


The original code works flawlessly. But I would like to add the possibility to zoom in and out. So I modified the OnPaint event of the original code as follows.

Code: Select all

    
    void wxBitmapFromOpenCVPanel::OnPaint(wxPaintEvent&)
   {
    wxAutoBufferedPaintDC dc(this);
    dc.Clear();
    if ( !m_bitmap.IsOk() ); //m_bitmap is the wxBitmap instance object already converted 
        return;
    DoPrepareDC(dc);   		//as far as I understood this routine should (among other things) keep track of the current scrolled position
   
    dc.SetUserScale(ZoomFactor, ZoomFactor);  //<-- FIRST MODIFICATION: ZoomFactor is a global variable set by the main Frame class
	
    SetVirtualSize(m_bitmap.GetSize()*ZoomFactor);//<--SECOND MODIFICATION; necessary to correctly scroll the zoomed image
  
    
    dc.DrawBitmap(m_bitmap, 0, 0, false);
    
    ....overlays are drawn
    
    

The problem is that when I zoom in and the image get larger than the window, the scrollbars appears (correctly) but scrolling doesn´t work. Any new frame is shown with origin in (0,0). The scrollbars don´t move (apparently, maybe that due to the high frame rate they move and get back at each new frame). So it looks as at each new frame the position of the scrollwindow is reset and the window doesn´t keep track of the scrolled position.

If instead of a series of frames from a webcam I load a static image I can scroll normally.
Now if for example I pause the camera, I can scroll the static image that remains on the screen. But wherever I scroll, as soon as I restart the camera the frame are again shown with the origin in the pixel (0,0).

I even tried to manage the scrolling in a "manual" fashion, saving the scrolled position at each scrolling event by wxScrolled::GetViewStart() and setting this saved position again in the OnPaint handler by wxScrolled::Scroll(). It resulted in a miserable failure.

I hope I could make my problem clear, and someone has some suggestion, how to solve this issue!
thank you in advance

[doublemax]

Hello and welcome to the forum!

Code: Select all

SetVirtualSize(m_bitmap.GetSize()*ZoomFactor);

This must not be in the paint event handler. Call it "outside" and only when the zoom factor or the size of the bitmap changes.

[PB]

FWIW, for wxBitmapFromOpenCVPanel I added a member variable declaration

Code: Select all

    double   m_bitmapScale{1.0};

and a method

Code: Select all

void wxBitmapFromOpenCVPanel::SetScale(double scale) 
{
    if ( scale == m_bitmapScale )
        return;

    m_bitmapScale = scale;
    SetBitmap(m_bitmap, m_timeGetCVBitmap, m_timeConvertBitmap); // update sizes and refresh
}

In wxBitmapFromOpenCVPanel::SetBitmap() replaced

Code: Select all

    if ( m_bitmap.IsOk() )
    {
        if ( m_bitmap.GetSize() != GetVirtualSize() )
        {
            InvalidateBestSize();
            SetVirtualSize(m_bitmap.GetSize());
        }
    }

with

Code: Select all

   if ( m_bitmap.IsOk() )
    {
        wxSize bitmapScaledSize(m_bitmap.GetWidth() * m_bitmapScale, m_bitmap.GetHeight() * m_bitmapScale);

        if ( bitmapScaledSize != GetVirtualSize() )
        {
            InvalidateBestSize();
            SetVirtualSize(bitmapScaledSize);
        }
    }

In wxBitmapFromOpenCVPanel::OnPaint() (it's not a virtual method, BTW) added two variables

Code: Select all

double       oldScaleX, oldScaleY;

and replaced

Code: Select all

    dc.DrawBitmap(m_bitmap, 0, 0, false);

with

Code: Select all

    dc.GetUserScale(&oldScaleX, &oldScaleY);
    dc.SetUserScale(m_bitmapScale, m_bitmapScale);
    dc.DrawBitmap(m_bitmap, 0, 0, false);
    dc.SetUserScale(oldScaleX, oldScaleY);

I then in OpenCVFrame::OnIPCamera() called

Code: Select all

    m_bitmapPanel->SetScale(3);

right before starting camera capture to test the scaling.

The updated wxBitmapFromOpenCVPanel header and source are attached.

bmpfromocvpanel+zoom.zip

(2.25 KiB) Downloaded 2 times

This seems to work but I don't think its the best solution for the real code. Resizing the bitmap with an OpenCV function or wxImage (only once in SetScale(), not during each repaint) could result into less ugly upscaled bitmap, not sure which of the two would be actually faster.

[renatone]

Hi Doublemax and PB,
thanks for your help!! I have managed to fix the issue.
I had put the call to SetVirtualSize elsewhere as Doublemax recommended, and that was in the routine SetBitmap (as per PB's advice)
but also in the routine that handles the wxSlidebar event by which the user changes the zoom factor.

PB,while you're here, I would like to thank you once more for the code and also ask you a last question. In the working thread polling the cv::Videocapture instance, before calling wxMilliSleep(), you have written the following comment:

// In a real code, the duration to sleep would normally
// be computed based on the camera framerate, time taken
// to process the image, and system clock tick resolution.

What are the shortcomings of this approach, that you say is not the optimal one?

Renato

[PB]

PB,while you're here, I would like to thank you once more for the code and also ask you a last question. In the working thread polling the cv::Videocapture instance, before calling wxMilliSleep(), you have written the following comment:

// In a real code, the duration to sleep would normally
// be computed based on the camera framerate, time taken
// to process the image, and system clock tick resolution.

What are the shortcomings of this approach, that you say is not the optimal one?

One should normally always avoid polling. But whether you need it or not depends on video source. When you remove the sleep, what fps do you get from camera, i.e., does the camera OpenCV capture driver itself waits for the next frame to maintain certain fps or do the frames just come as quickly as possible? Based on this, one had to decide what to do.