Django Tastypie Advanced Filtering: How to do complex lookups with Q objects

Question

I have a basic Django model like:

class Business(models.Model):
    name = models.CharField(max_length=200, unique=True)
    email = models.EmailField()
    phone = models.CharField(max_length=40, blank=True, null=True)
    description = models.TextField(max_length=500)

I need to execute a complex query on the above model like:

qset = (
    Q(name__icontains=query) |
    Q(description__icontains=query) |
    Q(email__icontains=query)
    )
results = Business.objects.filter(qset).distinct()

I have tried the following using tastypie with no luck:

def build_filters(self, filters=None):
    if filters is None:
        filters = {}
    orm_filters = super(BusinessResource, self).build_filters(filters)

    if('query' in filters):
        query = filters['query']
        print query
        qset = (
                Q(name__icontains=query) |
                Q(description__icontains=query) |
                Q(email__icontains=query)
                )
        results = Business.objects.filter(qset).distinct()
        orm_filters = {'query__icontains': results}

    return orm_filters

and in class Meta for tastypie I have filtering set as:

filtering = {
        'name: ALL,
        'description': ALL,
        'email': ALL,
        'query': ['icontains',],
    }

Any ideas to how I can tackle this?

Thanks - Newton

Lanky

Answer 1

You are on the right track. However, build_filters is supposed to transition resource lookup to an ORM lookup.

The default implementation splits the query keyword based on __ into key_bits, value pairs and then tries to find a mapping between the resource looked up and its ORM equivalent.

Your code is not supposed to apply the filter there only build it. Here is an improved and fixed version:

def build_filters(self, filters=None):
    if filters is None:
        filters = {}
    orm_filters = super(BusinessResource, self).build_filters(filters)

    if('query' in filters):
        query = filters['query']
        qset = (
                Q(name__icontains=query) |
                Q(description__icontains=query) |
                Q(email__icontains=query)
                )
        orm_filters.update({'custom': qset})

    return orm_filters

def apply_filters(self, request, applicable_filters):
    if 'custom' in applicable_filters:
        custom = applicable_filters.pop('custom')
    else:
        custom = None

    semi_filtered = super(BusinessResource, self).apply_filters(request, applicable_filters)

    return semi_filtered.filter(custom) if custom else semi_filtered

Because you are using Q objects, the standard apply_filters method is not smart enough to apply your custom filter key (since there is none), however you can quickly override it and add a special filter called "custom". In doing so your build_filters can find an appropriate filter, construct what it means and pass it as custom to apply_filters which will simply apply it directly rather than trying to unpack its value from a dictionary as an item.

Epode

Dictionary has no method 'extend'. Should be: orm_filters.update({'custom': qset}) – Photomechanical

This solution causes calling the DB twice (for semi_filtered and then for custom filter). A slightly different code works for me: if 'custom' in applicable_filters: custom = applicable_filters.pop('custom') return Outreaches.objects.filter(custom) else: return super(OutreachResource, self).apply_filters(request, applicable_filters) – Dael

Answer 2

I solved this problem like so:

Class MyResource(ModelResource):

  def __init__(self, *args, **kwargs):
    super(MyResource, self).__init__(*args, **kwargs)
    self.q_filters = []

  def build_filters(self, filters=None):
    orm_filters = super(MyResource, self).build_filters(filters)

    q_filter_needed_1 = []
    if "what_im_sending_from_client" in filters:
      if filters["what_im_sending_from_client"] == "my-constraint":
        q_filter_needed_1.append("something to filter")

    if q_filter_needed_1:
      a_new_q_object = Q()
      for item in q_filter_needed:
        a_new_q_object = a_new_q_object & Q(filtering_DB_field__icontains=item)
      self.q_filters.append(a_new_q_object)

  def apply_filters(self, request, applicable_filters):
    filtered = super(MyResource, self).apply_filters(request, applicable_filters)

    if self.q_filters:
      for qf in self.q_filters:
        filtered = filtered.filter(qf)
      self.q_filters = []

    return filtered

This method feels like a cleaner separation of concerns than the others that I've seen.

Lowrie

It's a really bad idea to put request-specific information on a resource instance. So self.q_filters.append(a_new_q_object). This is because in a deployed environment with multiple threads, you might end up with one request's state influencing another's. So for example, all the filters built up in one request could actually be applied to a completely different one, depending on the timing. See the docs here: django-tastypie.readthedocs.io/en/latest/… This is the problem that passing a bundle object around everywhere solves. – Willpower

Answer 3

Taking the idea in astevanovic's answer and cleaning it up a bit, the following should work and is more succinct.

The main difference is that apply_filters is made more robust by using None as the key instead of custom (which could conflict with a column name).

def build_filters(self, filters=None):
    if filters is None:
        filters = {}
    orm_filters = super(BusinessResource, self).build_filters(filters)

    if 'query' in filters:
        query = filters['query']
        qset = (
                Q(name__icontains=query) |
                Q(description__icontains=query) |
                Q(email__icontains=query)
                )
        orm_filters.update({None: qset}) # None is used as the key to specify that these are non-keyword filters

    return orm_filters

def apply_filters(self, request, applicable_filters):
    return self.get_object_list(request).filter(*applicable_filters.pop(None, []), **applicable_filters)
    # Taking the non-keyword filters out of applicable_filters (if any) and applying them as positional arguments to filter()

Willpower