Cloudant Search Index Query Limit

Question

Why are results from search index queries limited to 200 rows, whereas standard view queries seem to have no limit?

Tortuga

Answer 1

Fundamentally because we hold a 200 item array in memory as we stream over all hits, preserving the top 200 scoring hits. A standard view just streams all rows between a start and end point. The intent of a search is to typically to find the needle in a haystack, so you don't generally fetch thousands of results (compare with Google, who clicks through to page 500?). If you don't find what you want, you refine your search and look again.

There are cases when retrieving all matches makes sense (and we can stream this in the order we find them, so there's no RAM issue). That's a feature we can (and should) add, but it's not currently available.

Zedekiah

I use search indexes (in conjunction with the keyword/perfield analyzer) everywhere to make up for CouchDB's very poor native querying capability, so that's why. – Tortuga

I reached this page because I'm also confused that my query with no index seems to have no limit, but as soon as I add an index it forces the limit to 200. My goal was actually to compare the query time with and without the index. – Vendee

Answer 2

It's also worth noting that the _view API (aka "mapreduce") is fundamentally different than search because of the ordering of results on disk. Materialized views are persisted in CouchDB b+ trees, so they are essentially sorted by key. That allows for efficient range queries (start/end key), and makes limit/paging trivial. However, it also means that you have to order the view rows on disk, which restricts the types of boolean queries that you can perform against the materialized views. That's where search helps, but Bob (aka "The Lucene Expert") notes the limitations.

Deduce