How to warp an image using deformed mesh

Question

I'm trying to generate "crumpled" images using images obtained from a flatbed bed scanner.

By following the method described in the paper[Link] in the section 3.1. I've written the code generate the perturbed mesh but I don't know how to map this pixels from source image onto this mesh to form the perturbed image.

This is the code to generate the perturbed mesh.

import numpy as np
import matplotlib.pyplot as plt

mr = 88
mc = 68

xx = np.arange(mr-1, -1, -1)
yy = np.arange(0, mc, 1)
[Y, X] = np.meshgrid(xx, yy)
ms = np.transpose(np.asarray([X.flatten('F'), Y.flatten('F')]), (1,0))

perturbed_mesh = ms
nv = np.random.randint(20) - 1
for k in range(nv):
    #Choosing one vertex randomly
    vidx = np.random.randint(np.shape(ms)[0])
    vtex = ms[vidx, :]
    #Vector between all vertices and the selected one
    xv  = perturbed_mesh - vtex
    #Random movement 
    mv = (np.random.rand(1,2) - 0.5)*20
    hxv = np.zeros((np.shape(xv)[0], np.shape(xv)[1] +1) )
    hxv[:, :-1] = xv
    hmv = np.tile(np.append(mv, 0), (np.shape(xv)[0],1))
    d = np.cross(hxv, hmv)
    d = np.absolute(d[:, 2])
    d = d / (np.linalg.norm(mv, ord=2))
    wt = d
    
    curve_type = np.random.rand(1)
    if curve_type > 0.3:
        alpha = np.random.rand(1) * 50 + 50
        wt = alpha / (wt + alpha)
    else:
        alpha = np.random.rand(1) + 1
        wt = 1 - (wt / 100 )**alpha
    msmv = mv * np.expand_dims(wt, axis=1)
    perturbed_mesh = perturbed_mesh + msmv

plt.scatter(perturbed_mesh[:, 0], perturbed_mesh[:, 1], c=np.arange(0, mr*mc))
plt.show()

This is how the perturbed mesh looks like:

This is the screenshot from the paper illustrating the synthetic image generation:

Sample source image for testing: https://i.sstatic.net/26KN4.jpg

I'm stuck with mapping the source image pixels onto the mesh. I'll be grateful if someone can help.

Eighteenth

Answer 1

(1) Use cv2.copyMakeBorder to enlarge the image, to avoid the warpped points going out of range of the original image size.

cv2.copyMakeBorder(...)
    copyMakeBorder(src, top, bottom, left, right, borderType[, dst[, value]]) -> dst
    .   @brief Forms a border around an image.
    .
    .   The function copies the source image into the middle of the destination image. The areas to the
    .   left, to the right, above and below the copied source image will be filled with extrapolated
    .   pixels. This is not what filtering functions based on it do (they extrapolate pixels on-fly), but
    .   what other more complex functions, including your own, may do to simplify image boundary handling.

useage:

img = cv2.copyMakeBorder(img, dh, dh, dw, dw, borderType=cv2.BORDER_CONSTANT, value=(0,0,0))

Set dw=nw//2, dh=nh//2 maybe ok, adjust if necessary. The nh, nw is the height and width of the source image.

(2) Create perturbed mesh grid using the method from the paper

xs, ys = create_grid() # the result is like np.meshgrid

Notice make sure the type and the size.

# xs = xs.reshape(nh, nw).astype(np.float32)
# nh, nw is the height and width of the coppied image

(3) Use cv2.remap to remap:

cv2.remap(...)
    remap(src, map1, map2, interpolation[, dst[, borderMode[, borderValue]]]) -> dst
    .   @brief Applies a generic geometrical transformation to an image.
    .
    .   The function remap transforms the source image using the specified map:
    .   \f[\texttt{dst} (x,y) =  \texttt{src} (map_x(x,y),map_y(x,y))\f]

usage:

dst= cv2.remap(img, xs, ys, cv2.INTER_CUBIC)

This is a demo result:

(4) Crop the nonzero region and resize if necessary:

---

Related:

1. Converting opencv remap code from c++ to python

2. Split text lines in scanned document

Ogletree

how did you resize the grid ? I'm getting an assertion error when I use cv2.remap with full image size grid. cv2.error: OpenCV(3.4.2) /io/opencv/modules/imgproc/src/imgwarp.cpp:1728: error: (-215:Assertion failed) dst.cols < 32767 && dst.rows < 32767 && src.cols < 32767 && src.rows < 32767 in function 'remap' – Eighteenth

I think its better if cv2.copyMakeBorder is used after cv2.remap – Eighteenth

cv2.remap generate the result with the same size of the src. Use cv2.copyMakeBorder first to create a larger src with border, so we can the full wrapped src. Otherwise, it's of no use. Then to be honest, my xs, ys format is (nheight, nwidth), see the link of 1. – Lefthander

What is the version of your opencv ? – Eighteenth

I'm getting this cv2.error: OpenCV(3.4.2) /io/opencv/modules/imgproc/src/imgwarp.cpp:1728: error: (-215:Assertion failed) dst.cols < 32767 && dst.rows < 32767 && src.cols < 32767 && src.rows < 32767 in function 'remap'. When I googled I found its due to large number of elements in perturbed_mesh. I've been trying to fix this. Did you create meshgrid with original size of the image ? Did you use my code ? – Eighteenth

Yeah I've read your comments and I have used cv2.copyMakeBorder before cv2.remap and I have also changed my xs, ys format to (nheight, nwidth). It still throws an error. github.com/fengju514/Face-Pose-Net/issues/23 check this issue. – Eighteenth

I write the pertubed_mesh by myself. And before cv2.remap, I did this op: xs = xs.reshape(nh, nw).astype(np.float32). Notice, nh, nw is the height and width of the coppied image. – Lefthander

One last question how did you handle mesh points going out of range of image size. That is, after perturbing the mesh some points are going out of the range of (nh, nw). – Eighteenth

The first step cv2.copyMakeBorder do the job. I use dw=nw//2, dh=nh//2, almost ok. You can use bigger dw and dw if it's necessary. – Lefthander

@Kinght金 can you please post the code to generate the mesh (create_grid)? The code posted above by OP is wrong. – Ethelred

Hi could you please provide the code of create_grid? thanks! – Kinaesthesia

@Kinght金 Can you please share full code here? Mesh perturbation and warping using the source image you mentioned? – Georgeanngeorgeanna

Answer 2

I'm leaving a minimum working demo for other readers.

Currently, trying to figure out in which condition perturbed_mesh becomes None.

def deform(img, perturbed_mesh):
    h,w = img.shape[:2]

    perturbed_mesh_x = perturbed_mesh[:,0]
    perturbed_mesh_y = perturbed_mesh[:,1]
    
    perturbed_mesh_x =perturbed_mesh_x.reshape((h,w))
    perturbed_mesh_y =perturbed_mesh_y.reshape((h,w))

    remapped = cv2.remap(img, perturbed_mesh_x, perturbed_mesh_y, cv2.INTER_LINEAR) 

    return remapped

def get_perturbed_mesh(img):
    """
        Author: papabiceps
        https://mcmap.net/q/1217293/-how-to-warp-an-image-using-deformed-mesh
    """

    # mesh row & col
    mr,mc = img.shape[:2]

    xx = np.arange(0, mr, 1)
    yy = np.arange(mc-1, -1, -1)

    # yy = np.arange(0, mc, 1)
    [Y, X] = np.meshgrid(xx, yy)

    ms = np.transpose(np.asarray([X.flatten('F'), Y.flatten('F')]), (1,0))
 
    perturbed_mesh = ms
    nv = np.random.randint(20) - 1

    for k in range(nv):
        #Choosing one vertex randomly
        vidx = np.random.randint(np.shape(ms)[0])
        vtex = ms[vidx, :]
        #Vector between all vertices and the selected one
        xv  = perturbed_mesh - vtex
        #Random movement 
        mv = (np.random.rand(1,2) - 0.5)*20
        hxv = np.zeros((np.shape(xv)[0], np.shape(xv)[1] +1) )
        hxv[:, :-1] = xv
        hmv = np.tile(np.append(mv, 0), (np.shape(xv)[0],1))
        d = np.cross(hxv, hmv)
        d = np.absolute(d[:, 2])
        d = d / (np.linalg.norm(mv, ord=2))
        wt = d
        
        curve_type = np.random.rand(1)
        if curve_type > 0.3:
            alpha = np.random.rand(1) * 50 + 50
            wt = alpha / (wt + alpha)
        else:
            alpha = np.random.rand(1) + 1
            wt = 1 - (wt / 100 )**alpha
        msmv = mv * np.expand_dims(wt, axis=1)

        perturbed_mesh = perturbed_mesh + msmv

        perturbed_mesh = perturbed_mesh.astype(np.float32)

        if perturbed_mesh is not None:
            result = deform(img, perturbed_mesh)

            flipped = cv2.flip(result,1)

            cv2.imshow("before",result)
            cv2.imshow("after",flipped)
            cv2.waitKey(0)
 

def main():
    img = cv2.imread("./sample.png")

    get_perturbed_mesh(img)

Circulate

In mv = (np.random.rand(1,2) - 0.5)*20, 20 determines the intensity of deformation! – Circulate