Virtual PTZ camera via physical fisheye camera and Open CV

Question

I'm trying to realize a virtual Pan-Tilt-Zoom (PTZ) camera, based on data from physical fisheye camera (180 degrees FOV).

In my opinion I have to realize the next sequence.

1. Get the coordinates of center of fisheye circle in coordinates of fisheye sensor matrix.
2. Get radius of fisheye circle in the same coordinate system.
3. Generate a sphere equation, which has the same center and radius as flat fisheye image on flat camera sensor.
4. Project all colored points from flat image to upper hemisphere.
5. Choose angles in X Y plane and X Z plane to describe the direction of view of virtual PTZ.
6. Choose view angle and mark it with circle around virtual PTZ view vector which will be painted on the surface of hemisphere.
7. Generate the plane equation which intersection with hemisphere will be a circle around the direction of view.
8. Move all colored points from the circle to the plane of circle around the direction of view, using direction from hemisphere edge to hemisphere center for projection.
9. Paint all unpainted points inside circle of projection, using interpolation (realized in cv::remap).

In my opinion the most important step is to rise colored points from flat image to 3-D hemisphere.

My question is: Will it be correct to just set Z-coordinate to all colored points of flat image in accordance with hemisphere equation, to rise points from image plane to hemisphere surface?

Dric

This sounds far too complex. I've looked at this question a few times, and I still don't get exactly what you are proposing. But the one part I do think you are planning is to copy some pixels exactly, and interpolate others. – Glazer

Thank you for reply. To make my question more clear, I added a picture with an example of desirable result. In top-left corner you can see image from fisheye camera with red, blue and green regions. This circle in top-left corner is a projection of hemisphere to plane of camera sensor. Red, green and blue regions are projections of rectangular regions, wrapped around of 3D hemisphere. In top-right, bottom-left, and bottom-right corners you can see this regions in dewarped form. This is my final task – Dric

The final result is pretty obvious for someone familiar with FishEye camera's. It's your proposed process that's not at all obvious. Did you google "OpenCV fisheye" ? – Glazer

Of course I did :) There are a lot of solutions for "Virtual PTZ via fisheye", but they all are commercial products :( – Dric

And yes, I used cv::fisheye module. Unfortunately cv::fisheye can't undistort 100% of initial fisheye image surface if camera has 180 degrees FOV or more. More detailed explanation I placed HERE. As you can see on my last picture some commercial products make possible to undistort 100% of fisheye image surface. – Dric

Hi! Have you achieved some results for this time? – Aparejo