How to filter haystack results with db query

Question

I need to text-search across my model and filter with db queries at the same time.

For example:

class MyModel(models.Model):
    text = models.TextField()
    users = models.ManyToMany(User)

class MyModelIndexIndex(indexes.SearchIndex, indexes.Indexable):
    text = indexes.CharField(document=True, model_attr='text')

    def get_model(self):
        return MyModel

So I want to filter all MyModel objects by user AND by some text via full-text search. Smth like these:

qs = MyModel.objects.filter(users=request.user)
sqs = MyModelIndex.objects.filter(text=request.GET['q'])
intersection = some_magic_function(qs, sqs)

or

intersection = some_other_magic_function(
    qs_kwargs={'users': request.user},
    sqs_kwargs={'text': request.GET['q']}
)

Of course desired db queries could be much more complicated.

I see some possible solutions, all with major flaws:

1. Make intersection in django: extract ids from qs and use them in sqs filter or vice versa. Problem: performance. We can workaround itby using pagination and do intersection only for given page and its predecessors. In this case we lose total count (

2. Index all m2m related fields. Problem: performance, duplicate functionality (I believe db will do such queries much better), db-features such as annotations etc.

3. Do not use haystack ( Go for mysql or posgresql built-in full-text search.

I believe I miss something obvious. Case seems to be quite common. Is there a conventional solution?

Cacia

What data do you need to be in your "intersection" variable? Is it supposed to contain MyModel as well as MyModelIndex objects? Or do you just need one or the other? If you could explain what you are trying to achieve that might help, there is no context as it is. – Cupid

@TitusP: In my intersection I want either queryset or searchresultset. In my example I want to filter all MyModel objects by user and by text via full-text search. – Cacia

Which haystack engine do you use? – Defibrillator

Answer 1

In the general case, it's (probably) not possible to solve your problem using just one query. For instance, if you are using ElasticSearch as a search backend engine and MySQL for django models, there is no way MySQL and ElasticSearch will communicate to produce a single, common query.

However, there should be a workaround if you are using a common SQL database for your Django models and your Haystack backend engine. You would have to create a custom haystack engine that would parse the query and filter the available models.

For example, to modify the behaviour of the SimpleSearchBackend, all you need to do is patch the search method:

class CustomSimpleSearchBackend(haystack.backends.SimpleSearchBackend):

    def search(self, query_string, **kwargs):
        ...
        if query_string:
            for model in models:
                ...
                if 'users' in kwargs:
                    qs = qs.filter(users=kwargs['users'])
                ...

class CustomSimpleEngine(haystack.backends.BaseEngine):
    backend = CustomSimpleSearchBackend
    query = haystack.backends.simple_backend.SimpleSearchQuery

And in settings.py:

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'myapp.backends.CustomSimpleEngine',
    },
}

Depending on which connection backend you use, the required patch will be different of course, but I suspect it should not be too hard to implement.

Lavernlaverna