I've been working on a web app developed with Django which uses Postgres and ElasticSearch. It uses ElasticSearch since the database stores a lot of data which increases by the hour. To get things working quickly I used django-haystack 2.6.0 which allows me to access Elastic Search quite easily. So far so good.
The problem appeared when I made a query on ElasticSearch and the result was really big. The error I got was the following:
TransportError: TransportError(500, u'search_phase_execution_exception', u'Result window is too large, from + size must be less than or equal to: [10000] but was [12602]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter.')
After some research I tried increasing the result windows size but this won't work in the long term since the query-set size keeps increasing. Also django-haystack doesn't appear to implement scroll api so I can't follow the advice shown in the stack trace.
My question is: is there anyway I can fix this error and get a big query set with django-haystack maybe paginating the results in some way? Or is there a better approach without using haystack?
Thanks in advance!