Django 1.8 ArrayField append & extend

Question

Django 1.8 will come with new advanced field types including ArrayField these rely on PostgreSQL and are implemented at a DB level.

PostgreSQL's array field implements an append method.

However I can't find any documentation on appending an item to an ArrayField. This would clearly be very useful as it would allow the field to be updated without transferring its entire contents from the db then back to it.

Is this possible? If not will it be possible in future? Any pointers to docs I've missed would be much appreciated.

To clarify what I'm asking about, this would be great:

Note: this is fantasy code and I assume won't work (I haven't tried)

# on model:
class Post(models.Model):
    tags = ArrayField(models.CharField(max_length=200))

# somewhere else:
p = Post.objects.create(tags=[str(i) for i in range(10000)])
p.tags.append('hello')

Is there currently any way of doing this without resorting to raw sql?

Beers

Answer 1

I think the features you are looking for are currently not implemented (and may not be planned). Many of the Postgres contrib features originated based on this kickstarter project.

I find the most useful documentation for the new features come from the source code itself. Which includes a link to the original pull request for many of these features.

An important note in regards to the Array Functions mentioned, they are Functions and arguably outside the scope of a typical ORM.

I hope this information is useful and you find a nice solution to this issue.

Donovan

Thanks, that's what I had expected/feared. I'll leave the question open on the off chance someone supplies an elegant solution in code. – Beers

These futures are implemented. Please check out my answer. – Epidaurus

@Daniil everything you said is completely accurate. I might be misunderstanding the question but I think SColvin is looking to use a Postgres function instead of doing an UPDATE and sending the entire array back to Postgres from Python. – Donovan

@erik-e, my bad, sorry! – Epidaurus

Answer 2

Note: OP code will absolutely work. We just need to save the model (because these is just a model field, not relation). Let's see:

>>> p = Post.objects.create(tags=[str(i) for i in range(10000)])
>>> p.tags.append("working!")
>>> p.save()
>>> working_post = Post.objects.get(tags__contains=["working!"])
<Post: Post object>
>>> working_post.tags[-2:]
[u'9999', u'working!']

Going deeper

Django gets ArrayField as python list

Code reference

Everything you could do with list, you can do with ArrayField. Even sorting

Django saves ArrayField as python list

Code reference

These means that it saves structure and elements of python list.

Epidaurus

It doesn't actually append on the DB level. It will retrieve all of the elements in the array, does a list append, and then send the whole list back the the DB. – Bruin

Answer 3

This works:

from django.db.models import F
from django.db.models.expressions import CombinedExpression, Value

post = Post.objects.get(id=1000)
post.tags = CombinedExpression(F('tags'), '||', Value(['hello']))
post.save()

or in an update clause:

Post.objects.filter(created_on__lt=now() - timespan(days=30))\
    .update(tags=CombinedExpression(F('tags'), '||', Value(['old'])))

Edibles

This isn't working for me. I get a type error on postgres' side: It's treating the value to append ('old') in your exemple) as type text(], and the value already in the DB as type varying[], and then complains that it can't operate on those types. Any idea on how to solve this ? I've tried specifying an output_field of ArrayField(CharField(..)) for Value, same behaviour. – Eupatrid

Answer 4

Another solution is using a custom expression. I tested the following code with Django 1.11 and Python 3.6 (f-strings).

from django.db.models.expressions import Func

class ArrayAppend(Func):

    function = 'array_append'
    template = "%(function)s(%(expressions)s, %(element)s)"
    arity = 1

    def __init__(self, expression: str, element, **extra):
        if not isinstance(element, (str, int)):
            raise TypeError(
                f'Type of "{element}" must be int or str, '
                f'not "{type(element).__name__}".'
            )

        super().__init__(
            expression,
            element=isinstance(element, int) and element or f"'{element}'",
            **extra,
        )

The expression can be used in update():

Post.objects \
    .filter(pk=1) \
    .update(tags=ArrayAppend('tags', 'new tag'))

Ashok

Answer 5

You could use django_postgres_extensions. It supports a lot of functions like append, prepend, remove, concatenate.

But if you are using Django 1.8 like me, you should use only the required classes from this package. That way, you won't have to change database backend too. I've pasted the required classes here. Use them as described in first link.

Mulciber

Answer 6

This 100% works

from django.db.models import F
from django.db.models.expressions import CombinedExpression, Value

Post.objects.filter(created_on__lt=now() - timespan(days=30)).update(tags=CombinedExpression(F('tags'), '||', Value('{hello}')))

Note the difference between my variant and Yotam Ofek's:

Value(["hello"]) should be changed to Value("{hello}")

Also if you need to exclude entries with some specific tag use this:

.exclude(tags__contains="{hello}")

One more moment, field "tags" should have default=list

tags = ArrayField(models.CharField(max_length=200), default=list)

Eady