Google Lens APIs/SDK

Question

I was going through the Google Cloud Vision API to search for related products in the image. However, I found that I need to first create a product set (and upload the images) and then search on the set that I've created.

Is there any API via which I can provide an image (without creating any product set) and search google (or any specific website like amazon) for the related products (and their eCommerce links) in the image? Basically replicating what google lens does for its app. If not, is there any google lens mobile SDK for the same use-case, ie finding the related products (and their eCommerce URI) provided an image??

Update 1

I researched a bit more on this, and found Web Detection API. This does serve the purpose somewhat. However, I cant filter the results from a specific website (for eg:- Amazon). Is there a way to filter results from a specific domain?

Barred

Did my answer help you? – Griggs

Oh yes, it did. However, we opted out of this approach bcz of pricing issues. Nonetheless, thank you so much for the same. – Barred

Answer 1

With Vision API, you can detect from the web properties, faces, and text in the images.

An option is to use filtering to return more specific results when listing resources, evaluations, or operations. Using ?filter="[filter_name=]filter-value returns all JSON objects that match the filter-value. You can see more information in this link.

Another option is to filter before showing the results. You can see this example, which returns links from the Web and cloud storage that contain certain colors in the images.

You can see this example:

def detect_properties_uri(uri):
    """Detects image properties in the file located in Google Cloud Storage or
    on the Web."""
    from google.cloud import vision
    client = vision.ImageAnnotatorClient()
    image = vision.Image()
    image.source.image_uri = uri
 
    response = client.image_properties(image=image)
    props = response.image_properties_annotation
    print('Properties:')
 
    for color in props.dominant_colors.colors:
        print('frac: {}'.format(color.pixel_fraction))
        print('\tr: {}'.format(color.color.red))
        print('\tg: {}'.format(color.color.green))
        print('\tb: {}'.format(color.color.blue))
        print('\ta: {}'.format(color.color.alpha))
 
    if response.error.message:
        raise Exception(
            '{}\nFor more info on error messages, check: '
            'https://cloud.google.com/apis/design/errors'.format(
                response.error.message))
# [END vision_image_property_detection_gcs]

The next step is to iterate through the results like this example and filter the links that contain the domain. Looking at the domain inside the web links.

for entity in web_detection.web_entities:
    if (str(entity.description).find(domain) >-1)
        print(entity.description)

In this case, you will iterate each result and filter trying to find the domain inside the description and display it on the screen.

Another option would be to manipulate the result in JSON format. In that chase, you need to:

• Parse JSON
• Filter the JSON

You can see this example:

import json
 
input_json = """
[
    {
        "type": "1",
        "name": "name 1"
    },
    {
        "type": "2",
        "name": "name 2"
    },
    {
        "type": "1",
        "name": "name 3"
    }
]"""
 
# Transform json input to python objects
input_dict = json.loads(input_json)
 
# Filter python objects with list comprehensions
output_dict = [x for x in input_dict if x['type'] == '1']
 
# Transform python object back into json
output_json = json.dumps(output_dict)
 
# Show json
print output_json

Griggs

Answer 2

Make a GET request to https://lens.google.com/uploadbyurl?url=YOUR_IMAGE_URL

You will also need to play around with your headers. I think User-Agent is necessary along others.

Then parse the html response.

(Ik she ain't pretty but she runs, let me know how can I can improve it)

import json, bs4, request

def extract_data(soup):
    """This function is used to extract the data of a google lens page (response of bd_proxy_search  )

    Args:
        soup (bs4.BeautifulSoup): the html of the page (response of bd_proxy_search  )
    Returns:
        product_list (list): a list of dictionaries containing the data"""
    # Finds an element containig 'Visual Matches'  and returns a json object containing the data
    script_elements = soup.find_all('script')

    raw_data = [x for x in script_elements if 'Visual matches' in x.text]

    raw_data = raw_data[0].text

    start = raw_data.find('data:')+5
    end = raw_data.find('sideChannel') -2
    json_data = json.loads(raw_data[start:end])

    jason = []

    ###########################################
    # This is used beacuse sometimes the information is in json_data[1][0] and other times in json_data[1][1]
    try:
        jason = json_data[1][1][1][8][8][0][12] if len(json_data[1]) == 2 else json_data[1][0][1][8][8][0][12]
    except:
        print("The data is not in the expected format")
    ###########################################

    product_list = []
    for product in jason:
        information = {
            'google_image': product[0][0],
            'title': product[3],
            'redirect_url': product[5],
            'redirect_name': product[14],
            # 'price': product[0][7][1] if len(product[0]) > 6 else None

        }
        product_list.append(information)

    return product_list

This should return a list of dictionaries containing the information you need.

GL

Srinagar

This is an amazing hack. However, it does not display the related shopping products that the person is wearing in the image. IG, its only a functionality within the mobile app – Barred

Any further comments on this? – Barred

You should be getting back a dictionary containing info about related products. Could you describe your issue in further detail ? – Srinagar

@Srinagar can we add a title with it also? lens.google.com/uploadbyurl?url=YOUR_IMAGE_URL – Downing

@Srinagar what headers are neccesary? It might be nice to include a sample request along with the rest of the code. – Like

Answer 3

Here is the updated version from the answer by @Alvpvtos since recently there is an update for Google Lens website, since my answer is in PHP, I decide to add another answer in this question.

function extractData($html): array
{
    // Create a DOMDocument object
    $dom = new DOMDocument;

    // Suppress warnings due to malformed HTML
    libxml_use_internal_errors(true);

    // Load the HTML
    $dom->loadHTML($html);

    // Clear errors
    libxml_clear_errors();

    // Create an XPath object
    $xpath = new DOMXPath($dom);

    // Query all script tags
    $scriptElements = $xpath->query("//script");

    $raw_data = "";
    $offset = 2;

    foreach ($scriptElements as $element) {
        // The data is placed the third occurrence of the script tag content that contain "AF_initDataCallback"
        if (str_contains($element->nodeValue, "AF_initDataCallback")) {
            if ($offset == 0) {
                $raw_data = $element->nodeValue;
                break;
            } else {
                $offset--;
            }
        }
    }

    $start = strpos($raw_data, "data:") + 5;
    $end = strpos($raw_data, "sideChannel") - 2;
    $json_data = json_decode(substr($raw_data, $start, $end - $start), true);

    $jason = [];

    // This is used because sometimes the information is in $json_data[1][0] and other times in $json_data[1][1]
    try {
        $jason = count($json_data[1]) == 2 ? $json_data[1][1][1][8][8][0][12] : $json_data[1][0][1][8][8][0][12];
    } catch (Exception $e) {
        echo "The data is not in the expected format";
    }

    $product_list = [];
    foreach ($jason as $product) {
        $information = [
            'google_image' => $product[0][0],
            'title' => $product[3],
            'redirect_url' => $product[5],
            'redirect_name' => $product[14],
            'price' => $product[0][7][1] ?? null
        ];
        $product_list[] = $information;
    }

    return $product_list;
}

The html body no longer included the text "Visual matches", instead, I take the third script that containing the function AF_initDataCallback and continue the rest of the script from @Alvpvtos.

Mitigate

Note that this answer may returned result that contain "smartphone models" or not accurate sometimes. – Mitigate