How to create list field in django

Question

How do I create a ListField in Django (Python) like the ListProperty property in Google App Engine (Python)? My data is a list like this : 3,4,5,6,7,8.

What property do I have to define and how would I fetch values from it?

Sofar

Answer 1

Revisiting this with a ListField type you can use. But it makes a few of assumptions, such as the fact that you're not storing complex types in your list. For this reason I used ast.literal_eval() to enforce that only simple, built-in types can be stored as members in a ListField:

from django.db import models
import ast

class ListField(models.TextField):
    __metaclass__ = models.SubfieldBase
    description = "Stores a python list"

    def __init__(self, *args, **kwargs):
        super(ListField, self).__init__(*args, **kwargs)

    def to_python(self, value):
        if not value:
            value = []

        if isinstance(value, list):
            return value

        return ast.literal_eval(value)

    def get_prep_value(self, value):
        if value is None:
            return value

        return unicode(value)

    def value_to_string(self, obj):
        value = self._get_val_from_obj(obj)
        return self.get_db_prep_value(value)

class Dummy(models.Model):
    mylist = ListField()

Taking it for a spin:

>>> from foo.models import Dummy, ListField
>>> d = Dummy()
>>> d.mylist
[]
>>> d.mylist = [3,4,5,6,7,8]
>>> d.mylist
[3, 4, 5, 6, 7, 8]
>>> f = ListField()
>>> f.get_prep_value(d.numbers)
u'[3, 4, 5, 6, 7, 8]'

There you have it that a list is stored in the database as a unicode string, and when pulled back out it is run through ast.literal_eval().

Previously I suggested this solution from this blog post about Custom Fields in Django:

An alternative to the CommaSeparatedIntegerField, it allows you to store any separated values. You can also optionally specify a token parameter.

from django.db import models

class SeparatedValuesField(models.TextField):
    __metaclass__ = models.SubfieldBase

    def __init__(self, *args, **kwargs):
        self.token = kwargs.pop('token', ',')
        super(SeparatedValuesField, self).__init__(*args, **kwargs)

    def to_python(self, value):
        if not value: return
        if isinstance(value, list):
            return value
        return value.split(self.token)

    def get_db_prep_value(self, value):
        if not value: return
        assert(isinstance(value, list) or isinstance(value, tuple))
        return self.token.join([unicode(s) for s in value])

    def value_to_string(self, obj):
        value = self._get_val_from_obj(obj)
        return self.get_db_prep_value(value)

Habitat

App Engine supports lists natively; storing them comma separated seems like a terrible idea. – Milli

App Engine makes you think it stores lists natively but it really doesn't. How do you think it's actually stored in the database? Probably as a serialized version of a list (aka text). I added my own whack at a ListField. – Habitat

I'm on the App Engine team, and I know how it stores lists: as a series of fields. Eg, natively. – Milli

Ha, wow! That's awesome! :) I'd be interested to see how you would solve it with Django. – Habitat

Unfortunately, I'm not familiar with Django's database layer, so I couldn't say. I'm just pointing out that serializing your data to a single field is not a great solution when the underlying datastore supports lists natively. – Milli

I don't disagree with you fundamentally. Django relies on marshaling native objects to some sort of serialized format such as Pickle or JSON for storage in the database (get_prep_value()), and then reversing that upon retrieval of the data (to_python()). – Habitat

Nick, I'd love to see your alternative, until then jathanism's SeparatedValuesField approach worked great for my GAE Django-nonrel project. Thanks. – Longfellow

@NickJohnson What do you mean by "natively?" Surely you don't mean that it's normalized in a relational database. I expected GAE to use something document-driven. – Huarache

Django (1.5.2) Field object is asking for a "connection" arg on get_db_prep_value() method. But not using it at all? – Hypercorrect

@NickJohnson keeps saying "App Engine supports lists natively", but I can't find any information about that. WTF are you talking about!? – Irreligion

@Irreligion I'm not sure what the confusion is. An App Engine datastore model can contain a ListField, which stores the values as individual items, unlike relational databases which require hacks like the SeparatedValuesField above. – Milli

I have ended up using the custom ListField that's in alukach's answer. @NickJohnson could you post an answer that shows what you are talking about? – Irreligion

@Irreligion As I said in my original answer, I'm not familiar enough with Django DB internals to write a list field that uses App Engine's support for storing multi-valued fields - I was just pointing out that the support exists, and that there's no need to do string concatenation and parsing here. – Milli

I now understand what you're saying. Your reading of the question is different to mine. I came to this question looking for the answer to "how do I do a ListField in Django?" (A list field being a similar thing to in App Engine). – Irreligion

To answer about the "connection" issue. As of django 1.5 the method's sugnatire has changed and you should define: def get_db_prep_value(self, value,connection,prepared=False) – Drumhead

What about using get_prep_value instead of get_db_prep_value, to avoid the connection issue? – Socialite

Answer 2

Try using a CommaSeparatedIntegerField which is documented here: http://docs.djangoproject.com/en/1.2/ref/models/fields/#commaseparatedintegerfield

Manwell

Can you show us the code which fetches a single value for a commaseperatedintegerfield ?? – Analogous

If your data is in a list, say [3,4,5] then you would assign to the CommaSeparatedIntegerField like this: my_object.my_field = ",".join([str(x) for x in my_list]) – Manwell

App Engine supports lists natively; storing them comma separated seems like a terrible idea. – Milli

Note: CommaSeparatedIntegerField was deprecated in 2016. github.com/django/django/pull/6101 – Noleta

Answer 3

Consider django-jsonfield, the advantages are:

• Save and load native list objects, no conversion needed
• Well tested and mature solution
• Could have other advantages within your project
• Supports filtering on the database and regex (if needed)

also:

• Cross database support
• Supports Python 2.7 to Python 3.4 and Django 1.4 to 1.8
• Really easy :)

Supersaturate

This is a great idea and I like this simple and clean approach. Thanks! – Miru

Or now, the Django native JSONField: docs.djangoproject.com/en/4.1/ref/models/fields/…. – Noleta

yes, this answer is very old now. The Django native field would be the logical way to do this in a more recent version. – Supersaturate

Answer 4

While jathanism's answer is great, I was getting the following error when trying to use the dumpdata command:

Error: Unable to serialize database: get_db_prep_value() takes at least 3 arguments (2 given)

The issue is that self.get_db_prep_value call in the value_to_string method requires a connection value to be provided (at least in Django 1.4.10, which is what I am using). In the end, I didn't really see what was being gained by calling the value_to_string method in the first place and removed it, along with the unnecessary __init__ method. This is what I ended up with:

class ListField(models.TextField):
    __metaclass__ = models.SubfieldBase
    description = "Stores a python list"

    def to_python(self, value):
        if not value:
            value = []

        if isinstance(value, list):
            return value

        converted = ast.literal_eval(value)
        if not isinstance(converted, list):
            raise ValueError('Value "%s" not a list' % converted)

        return converted

Awn

what you had to do was change the method as follows: def get_db_prep_value(self, value,connection,prepared=False) as the method's signature was changed – Drumhead

@OritK: how? What's the right way to implement the ListField avoiding this get_db_prep_value problem? Your comment is ambiguous: the implementation by @Habitat is not defining get_db_prep_value - just using it, in a place where there is no connection parameter available. – Socialite

Must admit that i'm not 100% sure i remember it.. but according to my comment: you should change jathanism's implementation of ListField. instead of "def get_prep_value(self, value)", put "def get_db_prep_value(self, value,connection,prepared=False)" and it should work. – Drumhead

I agree with @OritK, that would be the minimal required change to get @jathanism's implementation working. I thought there was some unnecessary changes in that implementation though, so I chopped out all but what I thought were actually required to create a ListField – Awn

Answer 5

If you are using postgresql, django supports postgres with arrayfield.

Messinger

Answer 6

I do this:

def get_comma_field(self, field):
    data = getattr(self, field)
    if data:
        return data.split(',')
    return []


def set_comma_field(self, val, field):
    if isinstance(val, types.StringTypes):
        setattr(self, field, val)
    else:
        setattr(self, field, ','.join(val))


def make_comma_field(field):
    def getter(self):
        return get_comma_field(self, field)

    def setter(self, val):
        return set_comma_field(self, val, field)

    return property(getter, setter)

class MyModel(models.Model):
    _myfield = models.CharField(max_length=31)
    myfield = make_comma_field('_myfield')

But I suppose now it might be overkill. I needed quite a few of them, which is why I wrote the make_comma_field function.

Fetal

It's nearly the same as your SeparatedValuesField class, just isn't a Field class, and doesn't have a configurable delimiter. – Fetal

Well functionally, yes, but I meant as far as following conventions set within the Django framework. – Habitat

Answer 7

Simply, you can store the list as a string and whenever use it, use ast.literal_eval() to convert into list first from string. eg:

import ast

class MyModel(models.Model):
    field_1 = models.any kind of field()
    list_field = models.CharField(max_length=255)

    def get_list(self):
        list = ast.literal_eval(self.list_field)
        return list

same way in views etc. When saving, make oprations on list and lastly convert it into a string by:

model.list_field = str(list)
model.save()

Godred