What is output from OpenCV's Dense optical flow (Farneback) function? How can this be used to build an optical flow map in Python?

Question

I am trying to use the output of Opencv's dense optical flow function to draw a quiver plot of the motion vectors but have not been able to find what the function actually outputs. Here is the code:

import cv2
import numpy as np

cap = cv2.VideoCapture('GOPR1745.avi')

ret, frame1 = cap.read()
prvs = cv2.cvtColor(frame1,cv2.COLOR_BGR2GRAY)
hsv = np.zeros_like(frame1)

hsv[...,1] = 255
count=0

while(1):
    ret, frame2 = cap.read()
    next = cv2.cvtColor(frame2,cv2.COLOR_BGR2GRAY)
    flow = cv2.calcOpticalFlowFarneback(prvs,next,None, 0.5, 3, 15, 3, 10, 1.2, 0)
    mag, ang = cv2.cartToPolar(flow[...,0], flow[...,1])

    hsv[...,0] = ang*180/np.pi/2
    hsv[...,2] = cv2.normalize(mag,None,0,255,cv2.NORM_MINMAX)
    rgb = cv2.cvtColor(hsv,cv2.COLOR_HSV2BGR)
    if count==10:
        count=0

        print "flow",flow

    cv2.imshow('frame2',rgb)
    count=count+1
    k = cv2.waitKey(30) & 0xff
    if k == 27:
        break
    elif k == ord('s'):
    prvs = next

cap.release()
cv2.destroyAllWindows()

This is effectively the same code as given in the OpenCv tutorial on dense optical flow. I receive the following output from the print function:

flow [[[  0.00000000e+00   0.00000000e+00]
  [  0.00000000e+00   0.00000000e+00]
  [  0.00000000e+00   0.00000000e+00]
  ..., 
  [  0.00000000e+00   0.00000000e+00]
  [  0.00000000e+00   0.00000000e+00]
  [  0.00000000e+00   0.00000000e+00]]

 ..., 
 [[ -3.54891084e-14  -1.38642463e-14]
  [ -2.58058853e-14  -1.54020863e-14]
  [ -5.56561768e-14  -1.88019359e-14]
  ..., 
  [ -7.59403916e-15   1.16633225e-13]
  [  7.22156371e-14  -1.61951507e-13]
  [ -4.30715618e-15  -4.39530987e-14]]

 [[ -3.54891084e-14  -1.38642463e-14]
  [ -2.58058853e-14  -1.54020863e-14]
  [ -5.56561768e-14  -1.88019359e-14]
  ..., 
  [ -7.59403916e-15   1.16633225e-13]
  [  7.22156371e-14  -1.61951507e-13]
  [ -4.30715618e-15  -4.39530987e-14]]

I would like to know what exactly these values are? Original X,Y coordinates? Final X,Y coordinates? Distance moved?

I plan to try and find the initial and final coordinates to make a quiver plot using code from the following page: https://www.getdatajoy.com/examples/python-plots/vector-fields This is because in python there is no function that i am aware of that plots an optical flow map for you.

Thank you in advance!

Kissel

Answer 1

You were almost there. Lets first take a look at the calcOpticalFlowFarneback Documentation it says there:

flow – computed flow image that has the same size as prev and type CV_32FC2.

So what you are actually getting is a matrix that has the same size as your input frame.
Each element in that flow matrix is a point that represents the displacement of that pixel from the prev frame. Meaning that you get a point with x and y values (in pixel units) that gives you the delta x and delta y from the last frame.

Moller

Okay awesome, Thank you. So the flow matrix would be a matrix the same size as the original image, and the X and Y values stored in it at certain positions would be the X and Y distance traveled to get where they currently are. So if i were to find out where they moved from I would simply use their current pixel location in the flow matrix and subtract the X and Y distances they traveled? – Kissel

do all optical flow algorithms give output in pixel units ? – Delainedelainey

Yes. Since it captures the pixel motion – Gutty

Answer 2

I'm going to hijack this because it is the same topic.

If units are pixels as stated by @shravya, why this code does not show maximum flow equal to one?

I really dont get the units

Code

import numpy as np
import cv2
import seaborn as sns

# Generating img
img = np.zeros(shape=(3,50,50)) # 3 time frames, 50x50

center0 = np.array((10,10))
center1 = np.array((30,30))
for each_time, each_x, each_y in itertools.product(range(img.shape[0]), range(img.shape[1]), range(img.shape[2])): 
    img[each_time, each_x, each_y] =  img[each_time, each_x, each_y] + 1000 *  1/(  0.1* ((center0[0]+each_time*displacement_x - each_x)**2 + 1*(center0[1]+each_time*displacement_y - each_y)**2)**0.5 + 1)    
    img[each_time, each_x, each_y] =  img[each_time, each_x, each_y] + 1000 * 1/(  0.1* ((center1[0]+each_time*displacement_x - each_x)**2 + 1*(center1[1]+each_time*displacement_y - each_y)**2)**0.5 + 1)


img = (img - img.min())/(img.max()-img.min()) # Normalizing

## Ploting
fig, axs = plt.subplots(ncols=3, squeeze=True, figsize=(20,5))
for i in range(3):
    im = sns.heatmap(img[i,:,:], ax = axs[i], vmin=0, vmax=np.max(img))

fig.suptitle('Image') 

def calc_flow(img):

    ## Optical flow
    img = img.astype(np.int16)

    prev = np.zeros(img[0, :, :].shape).astype(np.int16)
    flows = np.zeros(shape=(img.shape[0], img.shape[1], img.shape[2], 2))

    for i, each_frame in enumerate(img):
        if i > img.shape[0]:
            break

        next_ = each_frame
        flow = cv2.calcOpticalFlowFarneback(prev, next_, None,
                                           pyr_scale = 0.5,
                                            levels = 3,
                                            winsize = 12,
                                            iterations = 5,
                                            poly_n = 5,
                                            poly_sigma = 1.2,
                                            flags = 0) 

        flows[i, :, :, 0] = flow[..., 0]
        flows[i, :, :, 1] = flow[..., 1]

        prev = next_
    
    return flows

flow = calc_flow(img)

fig, axs = plt.subplots(ncols=3, nrows=2, squeeze=True, figsize=(20,10))
for i in range(3):
    im = sns.heatmap(flow[i,:,:, 0] ,ax = axs[0,i], vmin=0, vmax = np.max(flow)) 
    im = sns.heatmap(flow[i,:,:, 1] ,ax = axs[1,i], vmin=0, vmax = np.max(flow)) 
    
fig.suptitle('Flow x and y plots')

mag_img, pha_img = cv2.cartToPolar(flow[..., 0], flow[..., 1]) 

fig, axs = plt.subplots(ncols=3, nrows=2, squeeze=True, figsize=(20,10))
for i in range(3):
    im = sns.heatmap(mag_img[i,:,:], ax=axs[0,i], vmin=0, vmax = np.max(mag_img)) 
    im = sns.heatmap(pha_img[i,:,:], ax=axs[1,i], vmin=0, vmax = np.max(pha_img))  
    
fig.suptitle('Magnitude and phase plots')

## Output
print(flow.max()) # This should be equal to displacement!
print(np.abs(flow).min()) # this should be zero


print(mag_img.max()) # This should be equal to displacement!
print(mag_img.min()) # this should be zero

Jamshid