How to remove the background from an image

Question

I want to remove the background, and draw the outline of the box shown in the image(there are multiple such images with a similar background) . I tried multiple methods in OpenCV, however I am unable to determine the combination of features which can help remove background for this image. Some of the approaches tried out were:

• Edge Detection - Since the background itself has edges of its own, using edge detection on its own (such as Canny and Sobel) didn't seem to give good results.
• Channel Filtering / Thresholding - Both the background and foreground have a similar white color, so I was unable to find a correct threshold to filter the foreground.
• Contour Detection - Since the background itself has a lot of contours, just using the largest contour area, as is often used for background removal, also didn't work.

I would be open to tools in Computer Vision or of Deep Learning (in Python) to solve this particular problem.

Aldenalder

deep learning. nothing less will do. -- I'll assume that the thing under the box is a conveyor belt and that it moves. don't be vague in your descriptions. you should have said what I just said, without prompting. -- show multiple examples or a video for better advice – Shermy

How about lowering the camera? – Impolitic

@ChristophRackwitz added another example. This is part of a public dataset - I dont have access to the camera, or the conveyor belt setup. Could you tell as to what specific tools/frameworks of Deep Learning can be used ? – Aldenalder

If it's part of a public dataset, please provide a link to it. If you would like folks to assist you, it's generally a good idea to make it easy for them. Thank you. – Impolitic

Okay the answer of Ann Zen is a really nice solution, I just want to mention ... there is also a special kind of algo class outside to solve your issue: background subtraction methods (opencv is including some of the basic algos) I made the best results with the quit awesome library: github.com/andrewssobral/bgslibrary – Tammany

Deep learning with salient object detection using U-2 Net should work. The output will give a mask, do some morphological operations to clean up the excess noise, find contours, draw the perfect outline, and/or bitwise_and to get the extracted result. Traditional image processing wont work here since the background is too noisy for any type of thresholding method (canny, Otsu's, adaptive, HSV color thresholding). For a robust approach, you have to use deep learning – Rebroadcast

Answer 1

The Concept

This is one of the cases where it is really useful to fine-tune the kernels of which you are using to dilate and erode the canny edges detected from the images. Here is an example, where the dilation kernel is np.ones((4, 2)) and the erosion kernel is np.ones((13, 7)):

The Code

import cv2
import numpy as np

def process(img):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray, (3, 3), 2)
    img_canny = cv2.Canny(img_blur, 50, 9)
    img_dilate = cv2.dilate(img_canny, np.ones((4, 2)), iterations=11)
    img_erode = cv2.erode(img_dilate, np.ones((13, 7)), iterations=4)
    return cv2.bitwise_not(img_erode)

def get_contours(img):
    contours, _ = cv2.findContours(process(img), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    cnt = max(contours, key=cv2.contourArea)
    cv2.drawContours(img, [cv2.convexHull(cnt)], -1, (0, 0, 255), 2)

img = cv2.imread("image2.png")
get_contours(img)
cv2.imshow("result", img)

cv2.waitKey(0)
cv2.destroyAllWindows()

The Output

Output for each of the two images provided:

Image 1:

Image 2:

Notes

Note that the processed image (which is binary) is inverted at cv2.bitwise_not(img_erode). Observe the processed version of both images (returned by the process() function defined above), with the inversion:

Processed Image 1:

Processed Image 2:

Tools

Finally, if you happen to have other images where the above program doesn't work properly on, you can use OpenCV Trackbars to adjust the values passed into the methods with the program below:

import cv2
import numpy as np

def process(img, b_k, b_s, c_t1, c_t2, k1, k2, k3, k4, iter1, iter2):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    b_k = b_k // 2 * 2 + 1
    img_blur = cv2.GaussianBlur(img_gray, (b_k, b_k), b_s)
    img_canny = cv2.Canny(img_blur, c_t1, c_t2)
    img_dilate = cv2.dilate(img_canny, np.ones((k1, k2)), iterations=iter1)
    img_erode = cv2.erode(img_dilate, np.ones((k3, k4)), iterations=iter2)
    return cv2.bitwise_not(img_erode)

d = {"Blur Kernel": (3, 50),
     "Blur Sigma": (2, 30),
     "Canny Threshold 1": (50, 500),
     "Canny Threshold 2": (9, 500),
     "Dilate Kernel1": (4, 50),
     "Dilate Kernel2": (2, 50),
     "Erode Kernel1": (13, 50),
     "Erode Kernel2": (7, 50),
     "Dilate Iterations": (11, 40),
     "Erode Iterations": (4, 40)}

cv2.namedWindow("Track Bars")
for i in d:
    cv2.createTrackbar(i, "Track Bars", *d[i], id)

img = cv2.imread("image1.png")

while True:
    img_copy = img.copy()
    processed = process(img, *(cv2.getTrackbarPos(i, "Track Bars") for i in d))
    contours, _ = cv2.findContours(processed, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    if contours:
        cnt = max(contours, key=cv2.contourArea)
        cv2.drawContours(img_copy, [cv2.convexHull(cnt)], -1, (0, 0, 255), 2)
    cv2.imshow("result", img_copy)
    
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cv2.waitKey(0)
cv2.destroyAllWindows()

Bort

How to resize this panel with sliders? I see only 7, 3 are hidden. – Felske

Answer 2

You Can use Rembg (tool to remove images background). This will work well even with pre-trained weights. I tried for the test image, here is my results using Rembg

You can Simply Download Rembg using pip

pip install rembg

Remove the background from a single file

rembg i path/to/input.png path/to/output.png

Remove the background from all images in a folder

rembg p path/to/input path/to/output

Crescentia

Yes, most of the case this package works perfectly. Some scenario its fails. – Crescentia

@ParthibanMarimuthu I think this should also work with Live Video Feed, Right ? – Expound

Yeah, I hope and check this one to github.com/nadermx/backgroundremover – Crescentia

Let us continue this discussion in chat. – Bort

Answer 3

You can try to use SIFT or SURF algorithms if you are given a perfect sample of what you are looking for. I took one from the 'intact' dataset as an example. The SIFT algorithm will try to match features in the sample with the actual image. From there, you can clean the matches and find an homography (RANSAC works best in this case) to find the contours.

Test image: Reference image (cut from a test one): Result (you can skip grayscale conversion):

The code I show here refers to execution in Colab with custom parameters you can tune some more.

import cv2 
import numpy as np
from google.colab.patches import cv2_imshow

img = cv2.imread("reference_1.png", cv2.IMREAD_GRAYSCALE) 
frame = cv2.imread("top.png", cv2.IMREAD_GRAYSCALE)

# if SIFT_create() gives problems, try downgrading opencv with
# pip uninstall opencv-python
# pip install opencv-contrib-python==3.4.2.17
sift = cv2.xfeatures2d.SIFT_create() 
kp_image, desc_image = sift.detectAndCompute(img, None) 
kp_frame, desc_frame = sift.detectAndCompute(frame, None) 

index_params = dict(algorithm=0, trees=5) 
search_params = dict() 
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(desc_image, desc_frame, k=2)

# clean the matches
good_points=[] 
for m, n in matches: 
    if(m.distance < 0.6 * n.distance): 
        good_points.append(m)

query_pts = np.float32([kp_image[m.queryIdx].pt for m in good_points]).reshape(-1, 1, 2) 
train_pts = np.float32([kp_frame[m.trainIdx].pt for m in good_points]).reshape(-1, 1, 2)

# find homography to find mask
matrix, mask = cv2.findHomography(query_pts, train_pts, cv2.RANSAC, 5.0) 
matches_mask = mask.ravel().tolist()
h,w = img.shape
pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts, matrix)
homography = cv2.polylines(frame, [np.int32(dst)], True, (255, 0, 0), 3) 

cv2_imshow(homography) 

Sussman