Google Search API site limit

Question

According to the Google custom search API's docs: http://code.google.com/apis/customsearch/docs/start.html#sites there is a limit of up to 5000 sites that you can search. This is pretty lame. Is there any way around this so that I can search the entire web using Google's results?

Also if you include a bunch of url patterns that matches greater than 5000 websites, how would the API pick and choose which sites to include and which to exclude?

Oconnor

Answer 1

This is for a custom search, not a normal Google search. For example, if you owned abc.com and acme.com, you could set up a custom search on those two domains for your customers. That way, they could search your sites for information. The 5,000-site limit is actually huge. I'm not sure I can think of an application that would use that many specified sites.

I think what you are looking for is the Google Web Search API, which searched all of Google. Unfortunately, that is now depreciated. (reference: http://code.google.com/apis/websearch/) You can still use the old API, but it is a risk because Google reserves the rights to turn it off at any time. They will also limit the number of searches you perform per day (although I can't find a specific number for that limit). Here is a link to their terms: http://code.google.com/apis/websearch/terms.html

I would recommend looking at an API from another search engine if you really want to integrate it directly into your code. A different suggestion would be to put your search information behind an interface and code it to Google for now. Then if they turn it off or something better comes out, you could change just the search code to point to the newest and best API.

Ascension

yeah I know. I want to set up a search on my own domain that searches the entire web as a normal Google search – Oconnor

@Justin - unfortunately, that isn't what the Custom Search is for. I haven't found a good replacement for the Web Search API yet. – Ascension

I think Bings API lets you search the entire web, but I'd much rather use Google's results. Even though the Web Search API is deprecated, I believe it's still usable? What would be the potential downside of using the deprecated API? – Oconnor

@Justin - I understand about using Google. As for using the depreciated API, it will still work but it limits the number of searches you can do per day. They also reserve the right to pull the plug at any time without warning, which might not be a good thing for you. code.google.com/apis/websearch/terms.html – Ascension

What exactly do you mean by this? "A different suggestion would be to put your search information behind an interface and code it to Google for now" – Oconnor

@Justin - with whatever programming language you use, if you develop an interface with the standard commands and standard output, you can change the implementation of that interface as technologies change without changing all of your business logic code. Basically this would become part of the data access layer. – Ascension

Answer 2

Google Custom Search is actually capable of searching the entire web, although the setting is not obvious. See "Search the entire web".

The other problems you are likely to run into are:

1. You only get 100 results per search, and
2. You are limited to 100 queries per day.

Sadly, "upgrading" to Google Site Search eliminates problem #2 at the expense of being able to search the entire web.

Vaporing

Answer 3

you can break the 5000 limit if you host the linked specification CSE file on your own website https://www.google.com/cse/docs/cref.html

Tedious

Link is dead. I suspect google no longer allows this hack. – Disembowel