How to scalably implement something like Google alerts?

Question

The requirement is that we have a lot of saved searches, and when new documents come in, we want to be able to find which saved searches are matched by these new documents, and be able to notify the creators of these saved searches in real time.

Is something like this already supported in any of the popular free software search engines and libraries, either built-in or as a plugin? If not, what's the best way to do this in an efficient and scalable way?

I guess the general idea should be to index the saved searches as if they are documents, and use the incoming documents as queries to search them. But I haven't figured out exactly how to do this.

It's also best if we can place as little restrictions on the supported query operators as possible.

Thanks in advance.

Horsy

I know it's not what you asked for, but if you're not 100% wedded to Solr, you could give Elasticsearch a shot - the percolate feature there does what you're asking for - elasticsearch.org/guide/reference/java-api/percolate.html – Apothecary

Answer 1

How about doing the reverse -

• Maintain searches in some other database with user specific information instead of solr.
• Index the new document, so that it is available for all subsequent searches
• Re execute all the saved queries across the document after addition, to check if it matches the document.

Also, Solr supports wide range of query support so you can check on it .

Chantry