How to get more than 100 results from Google Custom Search API

Question

I am trying to use Google Custom Search API for research purposes in Java. As a result I'm gonna need a big result set for each query. However it seems that I'm limited with first 100 results which is much less than what I need. I use the list method like this:

list.setStart(90L);

And when I set it to start from index 100 I get this error:

com.google.api.client.googleapis.json.GoogleJsonResponseException: 400 Bad Request

{   "code" : 400,   
    "errors" : [ {
    "domain" : "global",
    "message" : "Invalid Value",
    "reason" : "invalid"   
  } ],   
    "message" : "Invalid Value" 
}

Is there any way to remove this limitation?

Pious

Answer 1

There's a difference between two limitations: a) max 100 queries per day; and b) max 100 results per searched phrase (even split into 10 queries of 10 results per query). The limitation (a) can be solved by paying to Google, whereas limitation (b) cannot be solved at all, it seems to be a feature of the CSE product when searching the whole web (i.e. not just a site search, which can give more results), see e.g. https://productforums.google.com/forum/#!topic/customsearch/VM8_6trCxGU

Fact

Answer 2

You can only request 10 results per query with Google Custom Search, so you'd want to split up that search into multiple queries. (I don't know how your actual query code works so I don't know if you already know this)

Google Custom Search (Free) also has a limit of 100 queries per day, so if you want more, you'd have to pay.

You can find the pricing here: https://developers.google.com/custom-search/json-api/v1/overview

Deonnadeonne

Thanks. Yes I already know it. Problem is not the number of queries. As far as I understood, google just provides 10 results per query, so if I need more I have to repeat my query and change the starting index. BUT the limit for changing the starting index is 90. This means I cannot have more than 100 results per query. I wanna know is there any way to basically get limitless number of results. – Pious

Answer 3

If you guys are looking bypass the 100 queries per minute restriction, I haven't found a way to do it. So here's my solution with redis, still working on it so hit me up if you need an update I'll try to update it:

I've implemented a Queue with:

import express from 'express';
import Queue from 'bull';
import IORedis from 'ioredis';

The idea was to: await when we it the 100 query in a minutes threshold so we create space between fetch requiring the google api and avoid error.

const baseRedisConfig = {
  host: process.env.REDIS_HOST,
  port: parseInt(process.env.REDIS_PORT || '6379'),
  password: process.env.REDIS_PASSWORD,
};

// Configure separate Redis clients for Bull
const createClient = (type: 'client' | 'subscriber' | 'bclient') =>
  new IORedis({
    ...baseRedisConfig,
    // Custom configurations allowed for non-blocking clients
    enableReadyCheck: type === 'client',
    maxRetriesPerRequest: type === 'client' ? null : 0,
  });

const redisClient = createClient('client');
redisClient.on('connect', () => {
  redisClient.config('SET', 'maxmemory', '2gb');
  redisClient.config('SET', 'maxmemory-policy', 'noeviction');
});

const imageFetchQueue = new Queue('imageFetchQueue', {
  createClient: (type: 'client' | 'subscriber' | 'bclient') => createClient(type),
});

imageFetchQueue.process(async (job, done) => {
    const jobs = await imageFetchQueue.getJobs(['waiting', 'active', 'completed']);
    if (jobs.length > 100) {
        const timeDiff = new Date() - new Date(jobs[jobs.length - 101].timestamp);
        if (timeDiff < 60000) {
            setTimeout(() => {
                fetchHere(job.data.url);
                done();
            }, 60000 - timeDiff);
        } else {
            fetchHere(job.data.url);
            done();
        }
    } else {
        fetchImage(job.data.url);
        done();
    }
});

 

Strenuous

Answer 4

You can use lowRange and highRange to solve this problem. See: https://productforums.google.com/forum/#!topic/customsearch/2qilVDaCz0A

Mer

Please post the relevant code in your answer to add to the link you posted. meta.stackexchange.com/questions/8231/… – Nubilous

It's noted in that forum thread that this doesn't work. Just confirmed that myself. – Humiliating