Getting more than 10 results by Google Custom Search API V1 in Java

Question

I am using Google Custom Search API in Java to get results of Google in response to a query. I have written this code with the help of other posts, code is as follows:

    url = new URL("https://www.googleapis.com/customsearch/v1?key="+key+ "&cx="+ cx +"&q="+    searchText+"&alt=json"+"&start="+0+"&num="+30);
    HttpURLConnection conn2 = (HttpURLConnection) url.openConnection();
    System.out.println("Connection opened!");
    conn2.setRequestMethod("GET");
    conn2.setRequestProperty("Accept", "application/json");
    BufferedReader br = new BufferedReader(new InputStreamReader(
    (conn2.getInputStream())));

The problem is that whenever I am using the above code without num and start parameters it is executing properly, but giving only top 10 results. So I have used num and start parameters. But they are creating problems. Here I cannot understand where to put the num and start parameters in the url. It is always giving HTTP 400 i.e. Bad Request. I have read the Documentation page, there also no clear instruction is given about where to put these two parameters in Url.

So if anyone helps me to solve this problem I will be really grateful. Thank you.

Rm

Answer 1

You can't do it that way. num can only be a maximum of 10. See

https://developers.google.com/custom-search/json-api/v1/reference/cse/list#num

num - unsigned integer
Number of search results to return. Valid values are integers between 1 and 10, inclusive.

To show more results, Google suggests making multiple calls, incrementing the start parameter as needed:

https://developers.google.com/custom-search/json-api/v1/reference/cse/list#start

start - unsigned integer The index of the first result to return. Valid value are integers starting 1 (default) and the second result is 2 and so forth. For example &start=11 gives the second page of results with the default "num" value of 10 results per page. Note: No more than 100 results will ever be returned for any query with JSON API, even if more than 100 documents match the query, so setting (start + num) to more than 100 will produce an error. Note that the maximum value for num is 10.

Kadiyevka

Yes but I cannot understand where should I put the start parameter in the URL, because when I append start=0 or start=11 at the end of the URL like this: googleapis.com/customsearch/v1?key="+key+ "&cx="+ cx +"&q="+ searchText+"&alt=json"+"&start="+0, it is giving HTTP status 400 i.e. Bad Request. So could you please tell me exactly how to frame this url using start parameter. – Rm

In your code, print out the URL you are using. Something like System.out.println(url.toString()); Then copy and past that into your browser. It will show you the full error in JSON format, something like this (A sample - I just omitted the parameter "q" to get this). If you can't solve the problem, post the results you get here. { "error": { "errors": [ { "domain": "global", "reason": "required", "message": "Required parameter: q", "locationType": "parameter", "location": "q" } ], "code": 400, "message": "Required parameter: q" } } – Kadiyevka

Smartly designed by Google. Make us do more requests, instead of returning more results... -.- – Heyday

lol right @Heyday ? I can't believe it needs to be done this way. How the hell making more API requests is better for anyone? even for Google...it would be interesting to know the reason why they chose to set 10 as a maximum of results per search request – Nigercongo

Answer 2

First off, Google says: "The query parameters you can use with the JSON/Atom Custom Search API are summarized in this section. All parameter values need to be URL encoded." https://developers.google.com/custom-search/v1/using_rest#query-params Meaning that everything after the "?" should be encoded with an equivalent of the php url encoder which sets the standard for urlencoding. Thing is that Java's class URLEncoder isn't quite right, you have to do a couple of replaceAll's. You need to do this to your input:

String queryArguments = "key="+key+ "&cx="+ cx +"&q="+    searchText+"&alt=json"+"&start="+"0"+"&num="+"30";

Notice how there are quotes around the numbers. If you get these from variables use the following:

String thenum = Integer.toString(theinteger);

And then the proper encoding

String addition = URLEncoder.encode(queryArguments, "UTF-8")
.replaceAll("\\%28", "(") 
.replaceAll("\\%29", ")") 
.replaceAll("\\+", "%20") 
.replaceAll("\\%27", "'") 
.replaceAll("\\%21", "!") 
.replaceAll("\\%7E", "~");

Then you add that to the original unencoded url:

String url = "https://www.googleapis.com/customsearch/v1?"
String total = url + addition;

In conclusion your code will look like this:

String query = URLEncoder.encode("key="+key+ "&cx="+ cx +"&q="+    searchText+"&alt=json"+"&start="+"0"+"&num="+"30"), "UTF-8").replaceAll("\\%28", "(") 
.replaceAll("\\%29", ")") 
.replaceAll("\\+", "%20") 
.replaceAll("\\%27", "'") 
.replaceAll("\\%21", "!") 
.replaceAll("\\%7E", "~");
URL url = new URL("https://www.googleapis.com/customsearch/v1?" + query);
HttpURLConnection conn2 = (HttpURLConnection) url.openConnection();
System.out.println("Connection opened!");
conn2.setRequestMethod("GET");
conn2.setRequestProperty("Accept", "application/json");
BufferedReader br = new BufferedReader(new InputStreamReader(
(conn2.getInputStream())));

I hope this works for you. I did something very similar with the old deprecated image api, but the concept holds the same and I looked at the new docs. :)

EDIT: Make sure that your num parameter is between 0 and 10 inclusive.

Carilyn

Yes Sir, I have tried with your first suggested method i.e. I put quotes around the numbers in the URL. But still now it is giving the same error code 400 i.e. a Bad Request. Have you tried this yourself? – Rm

I've used the Google image search api before, which is exactly the same when it comes to use, besides the need for an api key and the base url. I forgot to mention that it is necessary for your num parameter to be ten or less, as the other answerer said. It should even work if you omit the num parameter, just as long as it is between 0 and 10 inclusive. I would like to clarify that the url encoding is necessary only because of queries which have spaces in them (they change to "%20") and a few other special cases. Also order of parameters does not matter in this case. Good luck! – Carilyn

Incorrect: num cannot be larger than 10 (inclusive); valid values are 1 ~ 10. start cannot be 0 either; valid values are integers greater than 0. – Delaryd

You're using UrlEncoder the wrong way. https://mcmap.net/q/24456/-java-url-encoding-of-query-string-parameters – Lunnete