Issue with ManyToMany Relationships not updating immediately after save

Question

I'm having issues with ManytoMany Relationships that are not updating in a model when I save it (via the admin) and try to use the new value within a function attached to the post_save signal or within the save_model of the associated AdminModel. I've tried to reload the object within those functions by using the get function with the id.. but it still has the old values.

Is this a transaction issue? Is there a signal thrown when the transaction ends?

Thanks,

Callihan

So, are you changing the pk of objects? – Inconvertible

I have one object, and there is a relationship manytomany with other, but i can get a updated relationship – Callihan

See my answer here: https://mcmap.net/q/501729/-manytomany-field-not-saved-when-using-django-admin – Photoactinic

Answer 1

When you save a model via admin forms it's not an atomic transaction. The main object gets saved first (to make sure it has a PK), then the M2M is cleared and the new values set to whatever came out of the form. So if you are in the save() of the main object you are in a window of opportunity where the M2M hasn't been updated yet. In fact, if you try to do something to the M2M, the change will get wiped out by the clear(). I ran into this about a year ago.

The code has changed somewhat from the pre-ORM refactor days, but it boils down to code in django.db.models.fields.ManyRelatedObjectsDescriptor and ReverseManyRelatedObjectsDescriptor. Look at their __set__() methods and you'll see manager.clear(); manager.add(*value) That clear() complete cleans out any M2M references for the current main object in that table. The add() then sets the new values.

So to answer your question: yes, this is a transaction issue.

Is there a signal thrown when the transaction ends? Nothing official, but read on:

There was a related thread a few months ago and MonkeyPatching was one method proposed. Grégoire posted a MonkeyPatch for this. I haven't tried it, but it looks like it should work.

Farmelo

I ran into this problem as well, I used a solution found here #6200733 – Leptospirosis

Correct me if I am wrong but this answer by @peterrowell is still valid as of current version of Django (1.10). The M2M field is cleared first and then is filled with the form's data. – Gladine

@Gladine : It's been quite a while since I last looked at this. The fundamental problem is that this is inherently not an atomic transaction because the primary model must be created/modified before the m2m is created/updated (because the m2m references the PKIDs of the parents). There may be a work-around mentioned here and here. Good luck! – Farmelo

Yeap! Django's m2m_changed signal is the way to do it. Although, in that case you lose the ability to do it in the django admin's save_model where you have access to the request obj and want to add a message to the admin. Anyway, can live without it! – Gladine

Answer 2

When you are trying to access the ManyToMany fields in the post_save signal of the model, the related objects have already been removed and will not be added again until after the signal is finished.

To access this data you have to tie into the save_related method in your ModelAdmin. Unfortunately you'll also have to include the code in the post_save signal for non-admin requests that require customization.

see: https://docs.djangoproject.com/en/1.7/ref/contrib/admin/#django.contrib.admin.ModelAdmin.save_related

Example:

# admin.py
Class GroupAdmin(admin.ModelAdmin):
    ...
    def save_related(self, request, form, formsets, change):
        super(GroupAdmin, self).save_related(request, form, formsets, change)
        # do something with the manytomany data from the admin
        form.instance.users.add(some_user)

Then in your signals you can make the same changes that you want to execute on a save:

# signals.py
@receiver(post_save, sender=Group)
def group_post_save(sender, instance, created, **kwargs):
    # do something with the manytomany data from non-admin
    instance.users.add(some_user)
    # note that instance.users.all() will be empty from the admin: []

Loring

Still my solution chosen for Django 3.2. – Aeromechanics

Answer 3

I have a general solution to this that seems a bit cleaner than monkey-patching the core or even using celery (although I'm sure someone could find areas where it fails). Basically I add a clean() method in the admin for the form that has the m2m relationships, and set the instance relations to the cleaned_data version. This make the correct data available to the instance's save method, even though it's not "on the books" yet. Try it and see how it goes:

def clean(self, *args, **kwargs):
    # ... actual cleaning here
    # then find the m2m fields and copy from cleaned_data to the instance
    for f in self.instance._meta.get_all_field_names():
        if f in self.cleaned_data:
            field = self.instance._meta.get_field_by_name(f)[0]
            if isinstance(field, ManyToManyField):
                setattr(self.instance,f,self.cleaned_data[f])

Bitt

Answer 4

See http://gterzian.github.io/Django-Cookbook/signals/2013/09/07/manipulating-m2m-with-signals.html

problem: When you manipulate the m2m of a model within a post or pre_save signal receiver, your changes get wiped out in the subsequent 'clearing' of the m2m by Django.

solution: In you post or pre_save signal handler, register another handler to the m2m_changed signal on the m2m intermediary model of the model whose m2m you want to update.

Please note that this second handler will receive several m2m_changed signals, and it is key to test for the value of the 'action' arguments passed along with them.

Within this second handler, check for the 'post_clear' action. When you receive a signal with the post_clear action, the m2m has been cleared by Django and you have a chance to successfully manipulate it.

an example:

def save_handler(sender, instance, *args, **kwargs):
    m2m_changed.connect(m2m_handler, sender=sender.m2mfield.through, weak=False)


def m2m_handler(sender, instance, action, *args, **kwargs):
    if action =='post_clear':
        succesfully_manipulate_m2m(instance)


pre_save.connect(save_handler, sender=YouModel, weak=False)

see https://docs.djangoproject.com/en/1.5/ref/signals/#m2m-changed

Destructive

The links no longer work. – Vinic

Answer 5

It would help if you use on_commit, so ManyToMany Relationships will be updated.

class Publication(models.Model):
    title = models.CharField(max_length=30)

class Article(models.Model):
    publications = models.ManyToManyField(Publication)

@receiver(post_save, sender=Article)
def my_signal(sender, **kwargs):
    from django.db import transaction
    transaction.on_commit(
        lambda: do_something_after_commit()
    )

    def do_something_after_commit():
        # Here you have access to updated m2m fields
        pass

Turbellarian

This answer is cleaner and simpler than the others and worked well for me, thanks! – Supersedure

Answer 6

You can find more informations in this thread : Django manytomany signals?

Barroom

I tried it in django 1.4, it's useless, using m2m_changed signal, assigning m2m values would cause resursing error, if you disconnect then assign, then instance.save(), the update didn't quite work. – Zhdanov

Answer 7

One of the solutions to update m2m, along with updating one of your models.

Django 1.11 and higher

First of all, all requests via admin panel are atomic. You can look at ModelAdmin:

@csrf_protect_m
def changeform_view(self, request, object_id=None, form_url='', extra_context=None):
    with transaction.atomic(using=router.db_for_write(self.model)):
        return self._changeform_view(request, object_id, form_url, extra_context)

@csrf_protect_m
def delete_view(self, request, object_id, extra_context=None):
    with transaction.atomic(using=router.db_for_write(self.model)):
        return self._delete_view(request, object_id, extra_context)

The behavior which you can observe during updating, when changes which you made with m2m records were not saved, even after you made them in a save method one of your models or in a signal, happens only because m2m form rewrites all records after the main object is updated.

This is why, step by step:

1. The main object is updated.

2. Your code(in a save method or in a signal) made changes (you can look at them, just put a breakpoint in ModelAdmin):

 def save_related(self, request, form, formsets, change):
     breakpoint()
     form.save_m2m()
     for formset in formsets:
         self.save_formset(request, form, formset, change=change)

3. form.save_m2m() takes all m2m values which were placed on a page(roughly speaking) and replace all m2m records via a related manager. That's why you can't see your changes at the end of a transaction.

There is a solution: make your changes with m2m via transaction.on_commit. transaction.on_commit will make your changes after form.save_m2m() when the transaction is committed.

Unfortunately, the downside of this solution - your changes with m2m will be executed in a separate transaction.

Porkpie