Django Rest Framework: Dynamically return subset of fields

Question

Problem

As recommended in the blogpost Best Practices for Designing a Pragmatic RESTful API, I would like to add a fields query parameter to a Django Rest Framework based API which enables the user to select only a subset of fields per resource.

Example

Serializer:

class IdentitySerializer(serializers.HyperlinkedModelSerializer):
    class Meta:
        model = models.Identity
        fields = ('id', 'url', 'type', 'data')

A regular query would return all fields.

GET /identities/

[
  {
    "id": 1,
    "url": "http://localhost:8000/api/identities/1/",
    "type": 5,
    "data": "John Doe"
  },
  ...
]

A query with the fields parameter should only return a subset of the fields:

GET /identities/?fields=id,data

[
  {
    "id": 1,
    "data": "John Doe"
  },
  ...
]

A query with invalid fields should either ignore the invalid fields or throw a client error.

Goal

Is this possible out of the box somehow? If not, what's the simplest way to implement this? Is there a 3rd party package around that does this already?

Capitalist

Answer 1

You can override the serializer __init__ method and set the fields attribute dynamically, based on the query params. You can access the request object throughout the context, passed to the serializer.

Here is a copy&paste from Django Rest Framework documentation example on the matter:

from rest_framework import serializers

class DynamicFieldsModelSerializer(serializers.ModelSerializer):
    """
    A ModelSerializer that takes an additional `fields` argument that
    controls which fields should be displayed.
    """

    def __init__(self, *args, **kwargs):
        # Instantiate the superclass normally
        super(DynamicFieldsModelSerializer, self).__init__(*args, **kwargs)

        fields = self.context['request'].query_params.get('fields')
        if fields:
            fields = fields.split(',')
            # Drop any fields that are not specified in the `fields` argument.
            allowed = set(fields)
            existing = set(self.fields.keys())
            for field_name in existing - allowed:
                self.fields.pop(field_name)


class UserSerializer(DynamicFieldsModelSerializer, serializers.HyperlinkedModelSerializer):

    class Meta:
        model = User
        fields = ('url', 'username', 'email')

Petasus

I finally came around to implement this, and it works perfectly! Thanks. I ended up writing a mixin for this, composition is a bit more flexible than subclassing :) gist.github.com/dbrgn/4e6fc1fe5922598592d6 – Capitalist

You'll need to change QUERY_PARAMS to query_params in recent versions of Django, but other than that this works like a charm. – Engineer

You probably should check that requests exists as a member of context. While it does in production it doesn't when running unit tests that create the objects manually. – Dejection

Doesn't work for nested resources, and check the comments for the gist above for a few fixes. – Kare

i have made a patch to handle cases where none of values given in fields param doesn't belong to model class for applied serialiser gist.github.com/manjitkumar/e6580296d634b0a48487 – Naman

I just published this as a package on PyPI! github.com/dbrgn/drf-dynamic-fields – Capitalist

FYI: This example is a verbatim copy of DRF documentation found here: django-rest-framework.org/api-guide/serializers/#example It's a bad form to not provide link to original authors – Barton

Can someone elaborate on how this trick could improve performance? Thanks. – Apheliotropic

The DRF documentation, from which this answer was copied, has been improved since this answer was posted. – Tailrace

It was returning blank JSON data, then I tried to get the data in to_representation(), there also it is showing blank OrderedDict() to me. – Mcilroy

Answer 2

This functionality is available from a 3rd-party package.

pip install djangorestframework-queryfields

Declare your serializer like this:

from rest_framework.serializers import ModelSerializer
from drf_queryfields import QueryFieldsMixin

class MyModelSerializer(QueryFieldsMixin, ModelSerializer):
    ...

Then the fields can now be specified (client-side) by using query arguments:

GET /identities/?fields=id,data

Exclusion filtering is also possible, e.g. to return every field except id:

GET /identities/?fields!=id

disclaimer: I'm the author/maintainer.

Blunger

Hi. What is the difference between this and github.com/dbrgn/drf-dynamic-fields (as linked in the comments of the chosen answer)? – Capitalist

Thanks, I had a look at that implementation, and it looks like it's the same basic idea. But the dbrgn implementation has some differences: 1. doesn't support exclude with fields!=key1,key2. 2. also modifies serializers outside of GET request context, which can and will break some PUT/POST requests. 3. does not accumulate fields with e.g. fields=key1&fields=key2, which is a nice-to-have for ajax apps. It also has zero test coverage, which is somewhat unusual in OSS. – Blunger

@Blunger Which versions of DRF and Django does your library support? I didn't find anything in the docs. – Autobahn

Django 1.7-1.11+, basically any configuration that DRF supports. This comment might go out of date, so check the test matrix for the CI, here. – Blunger

Works great for me: Django==2.2.7, djangorestframework==3.10.3, djangorestframework-queryfields==1.0.0 – Infarction

this work very well when the api returns list but when it return single object like /identities/1/?fields=id,data it don't work? any ideas? – Whitford

@alial-karaawi It certainly does work on the detail response and there are tests for that. Perhaps you should post a question with a reproducible example. – Blunger

Answer 3

serializers.py

class DynamicFieldsSerializerMixin(object):

    def __init__(self, *args, **kwargs):
        # Don't pass the 'fields' arg up to the superclass
        fields = kwargs.pop('fields', None)

        # Instantiate the superclass normally
        super(DynamicFieldsSerializerMixin, self).__init__(*args, **kwargs)

        if fields is not None:
            # Drop any fields that are not specified in the `fields` argument.
            allowed = set(fields)
            existing = set(self.fields.keys())
            for field_name in existing - allowed:
                self.fields.pop(field_name)


class UserSerializer(DynamicFieldsSerializerMixin, serializers.HyperlinkedModelSerializer):

    password = serializers.CharField(
        style={'input_type': 'password'}, write_only=True
    )

    class Meta:
        model = User
        fields = ('id', 'username', 'password', 'email', 'first_name', 'last_name')


    def create(self, validated_data):
        user = User.objects.create(
            username=validated_data['username'],
            email=validated_data['email'],
            first_name=validated_data['first_name'],
            last_name=validated_data['last_name']
        )

        user.set_password(validated_data['password'])
        user.save()

        return user

views.py

class DynamicFieldsViewMixin(object):

 def get_serializer(self, *args, **kwargs):

    serializer_class = self.get_serializer_class()

    fields = None
    if self.request.method == 'GET':
        query_fields = self.request.QUERY_PARAMS.get("fields", None)

        if query_fields:
            fields = tuple(query_fields.split(','))


    kwargs['context'] = self.get_serializer_context()
    kwargs['fields'] = fields

    return serializer_class(*args, **kwargs)



class UserList(DynamicFieldsViewMixin, ListCreateAPIView):
    queryset = User.objects.all()
    serializer_class = UserSerializer

Biophysics

Answer 4

Configure a new pagination serializer class

from rest_framework import pagination, serializers

class DynamicFieldsPaginationSerializer(pagination.BasePaginationSerializer):
    """
    A dynamic fields implementation of a pagination serializer.
    """
    count = serializers.Field(source='paginator.count')
    next = pagination.NextPageField(source='*')
    previous = pagination.PreviousPageField(source='*')

    def __init__(self, *args, **kwargs):
        """
        Override init to add in the object serializer field on-the-fly.
        """
        fields = kwargs.pop('fields', None)
        super(pagination.BasePaginationSerializer, self).__init__(*args, **kwargs)
        results_field = self.results_field
        object_serializer = self.opts.object_serializer_class

        if 'context' in kwargs:
            context_kwarg = {'context': kwargs['context']}
        else:
            context_kwarg = {}

        if fields:
            context_kwarg.update({'fields': fields})

        self.fields[results_field] = object_serializer(source='object_list',
                                                       many=True,
                                                       **context_kwarg)


# Set the pagination serializer setting
REST_FRAMEWORK = {
    # [...]
    'DEFAULT_PAGINATION_SERIALIZER_CLASS': 'DynamicFieldsPaginationSerializer',
}

Make dynamic serializer

from rest_framework import serializers

class DynamicFieldsModelSerializer(serializers.ModelSerializer):
    """
    A ModelSerializer that takes an additional `fields` argument that
    controls which fields should be displayed.

    See:
        http://tomchristie.github.io/rest-framework-2-docs/api-guide/serializers
    """

    def __init__(self, *args, **kwargs):
        # Don't pass the 'fields' arg up to the superclass
        fields = kwargs.pop('fields', None)

        # Instantiate the superclass normally
        super(DynamicFieldsModelSerializer, self).__init__(*args, **kwargs)

        if fields:
            # Drop any fields that are not specified in the `fields` argument.
            allowed = set(fields)
            existing = set(self.fields.keys())
            for field_name in existing - allowed:
                self.fields.pop(field_name)
# Use it
class MyPonySerializer(DynamicFieldsModelSerializer):
    # [...]

Last, use a homemage mixin for your APIViews

class DynamicFields(object):
    """A mixins that allows the query builder to display certain fields"""

    def get_fields_to_display(self):
        fields = self.request.GET.get('fields', None)
        return fields.split(',') if fields else None

    def get_serializer(self, instance=None, data=None, files=None, many=False,
                       partial=False, allow_add_remove=False):
        """
        Return the serializer instance that should be used for validating and
        deserializing input, and for serializing output.
        """
        serializer_class = self.get_serializer_class()
        context = self.get_serializer_context()
        fields = self.get_fields_to_display()
        return serializer_class(instance, data=data, files=files,
                                many=many, partial=partial,
                                allow_add_remove=allow_add_remove,
                                context=context, fields=fields)

    def get_pagination_serializer(self, page):
        """
        Return a serializer instance to use with paginated data.
        """
        class SerializerClass(self.pagination_serializer_class):
            class Meta:
                object_serializer_class = self.get_serializer_class()

        pagination_serializer_class = SerializerClass
        context = self.get_serializer_context()
        fields = self.get_fields_to_display()
        return pagination_serializer_class(instance=page, context=context, fields=fields)

class MyPonyList(DynamicFields, generics.ListAPIView):
    # [...]

Request

Now, when you request a resource, you can add a parameter fields to show only specified fields in url. /?fields=field1,field2

You can find a reminder here : https://gist.github.com/Kmaschta/e28cf21fb3f0b90c597a

Gossip

Answer 5

You could try Dynamic REST, which has support for dynamic fields (inclusion, exclusion), embedded / sideloaded objects, filtering, ordering, pagination, and more.

Bellringer

Answer 6

If you want something flexible like GraphQL, you can use django-restql. It supports nested data (both flat and iterable).

Example

from rest_framework import serializers
from django.contrib.auth.models import User
from django_restql.mixins import DynamicFieldsMixin

class UserSerializer(DynamicFieldsMixin, serializers.ModelSerializer):
    class Meta:
        model = User
        fields = ('id', 'username', 'email', 'groups')

A regular request returns all fields.

GET /users

    [
      {
        "id": 1,
        "username": "yezyilomo",
        "email": "[email protected]",
        "groups": [1,2]
      },
      ...
    ]

A request with the query parameter on the other hand returns only a subset of the fields:

GET /users/?query={id, username}

    [
      {
        "id": 1,
        "username": "yezyilomo"
      },
      ...
    ]

With django-restql you can access nested fields of any level. E.g

GET /users/?query={id, username, date_joined{year}}

    [
      {
        "id": 1,
        "username": "yezyilomo",
        "date_joined": {
            "year": 2018
        }
      },
      ...
    ]

For iterable nested fields, E.g groups on users.

GET /users/?query={id, username, groups{id, name}}

    [
      {
        "id": 1,
        "username": "yezyilomo",
        "groups": [
            {
                "id": 2,
                "name": "Auth_User"
            }
        ]
      },
      ...
    ]

Fairground

Answer 7

Such functionality we've provided in drf_tweaks / control-over-serialized-fields.

If you use our serializers, all you need is to pass ?fields=x,y,z parameter in the query.

Juggins

Answer 8

For nested data, I am using Django Rest Framework with the package recommended in the docs, drf-flexfields

This allows you to restrict the fields returned on both the parent and child objects. The instructions in the readme are good, just a few things to watch out for:

The URL seems to need the / like this '/person/?expand=country&fields=id,name,country' instead of as written in the readme '/person?expand=country&fields=id,name,country'

The naming of the nested object and its related name need to be completely consistent, which isn't required otherwise.

If you have 'many' e.g. a country can have many states, you'll need to set 'many': True in the Serializer as described in the docs.

Fecundity

Answer 9

The solution suggested at the [DRF-Documentation][1] worked for me, however when I called the serializer from the View with:

class SomeView(ListAPIView):
    def get(self, request, *args, **kwargs):
        qry=table.objects.filter(column_value=self.kwargs['urlparameter'])
        fields=['DBcol1','DBcol2','DBcol3']    
        serializer=SomeSerializer(qry,many=True,fields=fields)

I had to add many=True, otherwise it was not working.

  [1]: https://www.django-rest-framework.org/api-guide/serializers/#example

Oberland

Answer 10

Another alternative is to make use of GraphWrap: https://github.com/PaulGilmartin/graph_wrap

By adding /graphql to your urlpatterns, you add layer your REST API with a fully compliant GraphQL queryable API.

Stairs