GROUP_CONCAT equivalent in Django

Question

Say I have the following table called fruits:

id | type   | name
-----------------
 0 | apple  | fuji
 1 | apple  | mac
 2 | orange | navel

My goal is to ultimately come up with a count of the different types and a comma-delimited list of the names:

apple, 2, "fuji,mac"
orange, 1, "navel"

This can be easily done with GROUP_CONCAT in MySQL but I'm having trouble with the Django equivalent. This is what I have so far but I am missing the GROUP_CONCAT stuff:

query_set = Fruits.objects.values('type').annotate(count=Count('type')).order_by('-count')

I would like to avoid using raw SQL queries if possible.

Any help would be greatly appreciated!

Thanks! =)

Antidromic

Answer 1

The Django ORM does not support this; if you don't want to use raw SQL then you'll need to group and join.

Trouveur

A colleague of mine maintains an open source project that exposes mysql specific functions like GROUP_CONCAT in django. Take a look github.com/adamchainz/django-mysql – Thymic

Just note that for Postgres we have django.contrib.postgres.aggregates.StringAgg. – Valerle

Answer 2

You can create your own Aggregate Function (doc)

from django.db.models import Aggregate

class Concat(Aggregate):
    function = 'GROUP_CONCAT'
    template = '%(function)s(%(distinct)s%(expressions)s)'

    def __init__(self, expression, distinct=False, **extra):
        super(Concat, self).__init__(
            expression,
            distinct='DISTINCT ' if distinct else '',
            output_field=CharField(),
            **extra)

and use it simply as:

query_set = Fruits.objects.values('type').annotate(count=Count('type'),
                       name = Concat('name')).order_by('-count')

I am using django 1.8 and mysql 4.0.3

Chokebore

NOTICE that Django (>=1.8) provides Database functions – Cambodia

For the sake of completeness, there is also: django.contrib.postgres.aggregates.StringAgg in case you want the same in Postgres – Soak

This function is also present in django-mysql: django-mysql.readthedocs.io/en/latest/… – Altimetry

For django 2.2 I needed to add allow_distinct = True to Concat class – Corky

@LorenzoPeña That is the only thing which worked amongst the jungle of Group Concat articles and answers. You should add that as an answer StringAgg(fieldname, delimiter) – Iconology

A word of caution: MySQL will truncate GROUP_CONCAT result to 1024 characters, by default - I've spent some time figuring out why I was getting non-existent ids. You can set group_concat_max_len, example is here: #36475640 – Smalltime

Note: if someone faces not defined issue of CharField in line output_field=CharField(), then you may just replace from django.db.models import Aggregate to from django.db.models import Aggregate, CharField – Millham

Answer 3

NOTICE that Django (>=1.8) provides Database functions support. https://docs.djangoproject.com/en/dev/ref/models/database-functions/#concat

Here is an enhanced version of Shashank Singla

from django.db.models import Aggregate, CharField

class GroupConcat(Aggregate):
    function = 'GROUP_CONCAT'
    template = '%(function)s(%(distinct)s%(expressions)s%(ordering)s%(separator)s)'

    def __init__(self, expression, distinct=False, ordering=None, separator=',', **extra):
        super(GroupConcat, self).__init__(
            expression,
            distinct='DISTINCT ' if distinct else '',
            ordering=' ORDER BY %s' % ordering if ordering is not None else '',
            separator=' SEPARATOR "%s"' % separator,
            output_field=CharField(),
            **extra
        )

Usage:

LogModel.objects.values('level', 'info').annotate(
    count=Count(1), time=GroupConcat('time', ordering='time DESC', separator=' | ')
).order_by('-time', '-count')

Cambodia

It doesn;t work for me. Another fine example: gist.github.com/ludoo/ca6ed07e5c8017272701 – Geognosy

@Iliaw495Nikitin this works well in my project using Django 1.10.x – Cambodia

Works well in Django 1.11.x. Thanks! – Greig

Answer 4

Use GroupConcat from the Django-MySQL package ( https://django-mysql.readthedocs.org/en/latest/aggregates.html#django_mysql.models.GroupConcat ) which I maintain. With it you can do it simply like:

>>> from django_mysql.models import GroupConcat
>>> Fruits.objects.annotate(
...     count=Count('type'),
...     name_list=GroupConcat('name'),
... ).order_by('-count').values('type', 'count', 'name_list')
[{'type': 'apple', 'count': 2, 'name_list': 'fuji,mac'},
 {'type': 'orange', 'count': 1, 'name_list': 'navel'}]

Guadalajara

I know this is an old answer but I am somehow confused. May be am missing something. So how does the GroupConcat know that it will concatenate values of field 'name' ? As I don't see you indicating the field 'name' anywhere in the query, and yet we are concatenating values in field 'name' – Herc

Great! Your answer was helpful. I will give you a vote. :) @adam-chainz – Herc

Answer 5

If you are using PostgreSQL, you can use ArrayAgg to aggregate all of the values into an array.

https://www.postgresql.org/docs/9.5/static/functions-aggregate.html

Meerschaum

Answer 6

If you don't mind doing this in your template the Django template tag regroup accomplishes this

Kanal

Answer 7

As of Django 1.8 you can use Func() expressions.

query_set = Fruits.objects.values('type').annotate(
    count=Count('type'),
    name=Func(F('name'), 'GROUP_BY')
).order_by('-count')

Mytilene

Answer 8

Similar Aggregate Function for PostgreSQL in case someone needs:

from django.db.models import Aggregate, CharField

class GroupConcat(Aggregate):
    function = "STRING_AGG"
    template = "%(function)s(%(expressions)s::text, %(separator)s%(ordering)s)"

    def __init__(self, expression, ordering=None, separator=',', **extra):
        super().__init__(
            expression,
            ordering=" ORDER BY %s" % ordering if ordering is not None else "",
            separator="'%s'" % separator,
            output_field=CharField(),
            **extra
        )

queryset = Fruits.objects.values('type').annotate(
    count=Count('type'),
    name=GroupConcat('name')
).order_by('-count')

Incline

Answer 9

Not supported by Django ORM, but you can build your own aggregator.

It's actually pretty straightforward, here is a link to a how-to that does just that, with GROUP_CONCAT for SQLite: http://harkablog.com/inside-the-django-orm-aggregates.html

Note however, that it might be necessary to handle different SQL dialects separately. For example, the SQLite docs say about group_concat:

The order of the concatenated elements is arbitrary

While MySQL allows you to specify the order.

I guess that may be a reason why GROUP_CONCAT it's not implemented in Django at the moment.

Barbarism

Answer 10

To complete the answer of @WeizhongTu, Notice that you can not use the keyword SEPARATOR with SQLITE. In cases where you are using MySQL and SQLite for your tests, you can write :

class GroupConcat(Aggregate):
    function = 'GROUP_CONCAT'
    separator = ','

    def __init__(self, expression, distinct=False, ordering=None, **extra):
        super(GroupConcat, self).__init__(expression,
                                          distinct='DISTINCT ' if distinct else '',
                                          ordering=' ORDER BY %s' % ordering if ordering is not None else '',
                                          output_field=CharField(),
                                          **extra)

    def as_mysql(self, compiler, connection, separator=separator):
        return super().as_sql(compiler,
                              connection,
                              template='%(function)s(%(distinct)s%(expressions)s%(ordering)s%(separator)s)',
                              separator=' SEPARATOR \'%s\'' % separator)

    def as_sql(self, compiler, connection, **extra):
        return super().as_sql(compiler,
                              connection,
                              template='%(function)s(%(distinct)s%(expressions)s%(ordering)s)',
                              **extra)

Entomology

Answer 11

I just wanted to say a word of caution if you go with any of the proposed solutions for MySQL: by default, MySQL will truncate GROUP_CONCAT result to 1024 characters. I've spent some time figuring out why I was getting non-existent ids (they were truncated existent ids).

You can avoid the limitation by setting group_concat_max_len in Django settings. An example is here: Include multiple statements in Django's raw queries

Smalltime

Answer 12

this is the best way of working in Django ORM

f1 = Fruits.objects.values('type').annotate(count = Count('type'),namelist= GroupConcat('namelist')).distinct()

Streamline