Django rest api - searching a method field with the search filter

Question

im trying to filter search a rest api page and want to use a method field as one of the search fields, however when I do this I get an error stating the field is not valid and it then lists the field in my model as the only valid source

serialiser:

class SubnetDetailsSerializer(QueryFieldsMixin, serializers.HyperlinkedModelSerializer):
    subnet = serializers.SerializerMethodField()
    device = serializers.ReadOnlyField(
        source='device.hostname',
    )
    circuit_name = serializers.ReadOnlyField(
        source='circuit.name',
    )
    subnet_name = serializers.ReadOnlyField(
        source='subnet.description',
    )
    safe_subnet = serializers.SerializerMethodField()

    def get_safe_subnet(self, obj):
        return '{}{}'.format(obj.subnet.subnet, obj.subnet.mask.replace('/','_')) 

    def get_subnet(self, obj):
        return '{}{}'.format(obj.subnet.subnet, obj.subnet.mask) 

    class Meta:
        model = DeviceCircuitSubnets   
        fields = ('id','device_id','subnet_id','circuit_id','subnet','safe_subnet','subnet_name','device','circuit_name') 

views:

class SubnetDetailsSet(viewsets.ReadOnlyModelViewSet):
    queryset = DeviceCircuitSubnets.objects.all().select_related('circuit','subnet','device')
    serializer_class = SubnetDetailsSerializer
    permission_classes = (IsAdminUser,)
    filter_class = DeviceCircuitSubnets
    filter_backends = (filters.SearchFilter,)
    search_fields = (
        'device__hostname',
        'circuit__name',
        'subnet__subnet',
        'safe_subnet'
    )

how can include the safe_subnet in the search fields?

Thanks

EDIT This is the code now

views.py

class SubnetDetailsSet(viewsets.ReadOnlyModelViewSet):
    queryset = DeviceCircuitSubnets.objects.all()
    serializer_class = SubnetDetailsSerializer
    permission_classes = (IsAdminUser,)
    filter_class = DeviceCircuitSubnets
    filter_backends = (filters.SearchFilter,)
    search_fields = (
        'device__hostname',
        'circuit__name',
        'subnet__subnet',
        'safe_subnet'
    )

    def get_queryset(self):
        return (
            super().get_queryset()
            .select_related('circuit','subnet','device')
            .annotate(
                safe_subnet=Concat(
                    F('subnet__subnet'),
                    Replace(F('subnet__mask'), V('/'), V('_')),
                    output_field=CharField()
                )
            )
        )

serializer.py

class SubnetDetailsSerializer(QueryFieldsMixin, serializers.HyperlinkedModelSerializer):
    subnet = serializers.SerializerMethodField()
    device = serializers.ReadOnlyField(
        source='device.hostname',
    )
    circuit_name = serializers.ReadOnlyField(
        source='circuit.name',
    )
    subnet_name = serializers.ReadOnlyField(
        source='subnet.description',
    )
    def get_safe_subnet(self, obj):
        return getattr(obj, 'safe_subnet', None)

    def get_subnet(self, obj):
        return '{}{}'.format(obj.subnet.subnet, obj.subnet.mask) 

    class Meta:
        model = DeviceCircuitSubnets   
        fields = ('id','device_id','subnet_id','circuit_id','subnet','safe_subnet','subnet_name','device','circuit_name')  

Model:

class DeviceCircuitSubnets(models.Model):
    device = models.ForeignKey(Device, on_delete=models.CASCADE)
    circuit = models.ForeignKey(Circuit, on_delete=models.CASCADE, blank=True, null=True)
    subnet = models.ForeignKey(Subnet, on_delete=models.CASCADE)
    active_link = models.BooleanField(default=False, verbose_name="Active Link?")
    active_link_timestamp = models.DateTimeField(auto_now=True, blank=True, null=True)

Error:

Exception Type: ImproperlyConfigured at /api/subnets/
Exception Value: Field name `safe_subnet` is not valid for model `DeviceCircuitSubnets`.

Stickney

searches are performed using queries, you can't search on a method field. – Mayst

Can you post the model and error as well? – Ought

any ideas on this? thank – Stickney

Do you need to use a ?search= for all of these, or can you use a custom filter that supports ?subnet=192.168/24&device=django&circut=31 style queries? The latter is preferable, in this kind of situation. – Devotional

Answer 1

Going in a completely different direction from the other (excellent) answers. Since you want to be able to filter frequently on the safe_subnet field, why not just let it be an actual database field in your model? You could calculate and populate/update the value during one of your save methods and then just let django-filters do it's thing. This also has the advantage of allowing the filtering to be done directly through SQL which would theoretically provide better performance.

Tetherball

I did end up going with this, overriding the form valid method and creating it on the fly in the DB – Stickney

Answer 2

You need to annotate your queryset with the safe_subnet attribute so it becomes searchable.

from django.db.models import F, Value as V
from django.db.models.functions import Concat, Replace

class SubnetDetailsSet(viewsets.ReadOnlyModelViewSet):
    queryset = DeviceCircuitSubnets.objects.all()
    serializer_class = SubnetDetailsSerializer
    permission_classes = (IsAdminUser,)
    filter_class = DeviceCircuitSubnets
    filter_backends = (filters.SearchFilter,)
    search_fields = (
        'device__hostname',
        'circuit__name',
        'subnet__subnet',
        'safe_subnet'
    )

    def get_queryset(self):
        return (
            super().get_queryset()
            .select_related('circuit','subnet','device')
            .annotate(
                safe_subnet=Concat(
                    F('subnet__subnet'),
                    Replace(F('subnet__mask'), V('/'), V('_')),
                    output_field=CharField()
                )
            )
        )

Then in your serializer you can use the following.

def get_safe_subnet(self, obj):
    return obj.safe_subnet

Manrope

Im seeing the error Field name safe_subnet is not valid for model DeviceCircuitSubnets. stil – Stickney

If the error is caused by the filter then it's possibly a result of this DRF issue and you should be able to solve by upgrading. Otherwise, if the serializer is used outside SubnetDetailsSet and the queryset isn't annotated then you would also get this error. You could change to return getattr(obj, 'safe_subnet', None) if you don't want to annotate for some reason. – Manrope

I've tried upgrading and adding what was suggested but I still get the same error – Stickney

@Stickney does the annotation work when you use it in the shell? You need to look at the traceback and see what's causing the error. Likely SearchFilter doesn't call the view's .get_queryset() method before filtering so you need to figure out how to do that. – Manrope

the annotation works through shell yes, I have no idea how to edit the search filter to include that, would you happen to know how to do this? Thanks – Stickney

Answer 3

Previous answer with annotate is a really good start:

from .rest_filters import DeviceCircuitSubnetsFilter  

class SubnetDetailsSet(viewsets.ReadOnlyModelViewSet):
    queryset = DeviceCircuitSubnets.objects.all()
    serializer_class = SubnetDetailsSerializer
    permission_classes = (IsAdminUser,)
    # That's where hint lays
    filter_class = DeviceCircuitSubnetsFilter
    #filter_backends = (filters.SearchFilter,)
    search_fields = (
        'device__hostname',
        'circuit__name',
        'subnet__subnet',
        'safe_subnet'
    )

    #No need to override your queryset

Now in rest_filters.py

from django_filters import rest_framework as filters
from django.db.models import F, Value as V
from django.db.models.functions import Concat, Replace
#.... import models

class DeviceCircuitSubnets(filters.FilterSet):

    safe_subnet = filters.CharFilter(
        name='safe_subnet',
        method='safe_subnet_filter')

    def safe_subnet_filter(self, queryset, name, value):
        """
        Those line will make ?safe_subnet=your_pk available
        """
        return  queryset.annotate(
                    safe_subnet=Concat(
                        F('subnet__subnet'),
                        Replace(F('subnet__mask'), V('/'), V('_')),
                        output_field=CharField()
                )
            ).filter(safe_subnet=value)
        )
    class Meta:
        model = DeviceCircuitSubnets
        # See https://django-filter.readthedocs.io/en/master/guide/usage.html#generating-filters-with-meta-fields
        # This pattern is definitely a killer!
        fields = {
            'device': ['exact', 'in'],
            'circuit': ['exact', 'in'],
            'subnet': ['exact', 'in'],
            'active_link': ['exact'],
            'active_link_timestamp': ['lte', 'gte']
        }

Please note: I'm annotating safe_subnet within the filer, depending on how much you use this, you might want to set this up in your model's manager!

Rianna

I'm getting Error: safe_subnet = django_filters.IntegerField( AttributeError: module 'django_filters' has no attribute 'IntegerField' – Stickney

I still see the same AttributeError: module 'django_filters.rest_framework' has no attribute 'IntegerField' – Stickney

Sorry... so used to Serializers/Models... Correct name was NumberFilter... response is updated with CharFilter: If filter does not match your data type, you can find full list here: django-filter.readthedocs.io/en/master/ref/filters.html#filters My guess is you'll need CharFilter as safe_subnet is CharField – Rianna

ok its gone back to the original error now for some reason "Cannot resolve keyword 'safe_subnet' into field" – Stickney

Did you keep safe_subnet from your APIView search_fields? If yes can you try without this ? – Rianna

I now get __init__() got an unexpected keyword argument 'name' which I think is erring under kwargs {'label': '[invalid name]', 'name': 'safe_subnet', 'required': False} – Stickney