Django import export - Unable to import model with BinaryField

Question

I'm trying to import data from a model which has nullable BinaryField. The data doesn't contain the field and I want it to be imported with a null value in the field. If the field already exists in the database for a given id, it should keep the value as it is.

I removed the field from the fields whitelist in the corresponding Resource object and added it to the exclude blacklist. However, I'm getting this error while importing - can't pickle memoryview objects.

Traceback:

Traceback (most recent call last):
File "/lib/python3.5/site-packages/import_export/resources.py", line 451, in import_row
original = deepcopy(instance)
File "/lib/python3.5/copy.py", line 182, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/lib/python3.5/copy.py", line 297, in _reconstruct
state = deepcopy(state, memo)
File "/lib/python3.5/copy.py", line 155, in deepcopy
y = copier(x, memo)
File "/lib/python3.5/copy.py", line 243, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/lib/python3.5/copy.py", line 174, in deepcopy
rv = reductor(4)
TypeError: can't pickle memoryview objects

Package versions - django==1.11, django-import-export==0.6

EDIT:

class ABC(models.Model):
    name = models.CharField('Name', max_length=128, blank=False, null=False)
    binary_field = models.BinaryField('Some name', null=True, blank=True)

class ABCResource(resources.ModelResource):
    class Meta:
        model = ABC
        fields = (
            'id',
            'name',
        )
        import_id_fields = ('id',)
        exclude = ('binary_field',)

class ABCAdmin(ImportExportModelAdmin):
    form = ABCModelForm
    list_display = (
        'id',
        'name',
    )
    exclude = ('binary_field',)
    resource_class = ABCResource

class ABCModelForm(forms.ModelForm):
    class Meta:
        model = ABC
        exclude = ['binary_field']

Leslileslie

Proprietary code, so added a reduced version of it... – Leslileslie

what version of django-import-export are you using? – Mores

django-import-export is 0.6 – Merrymaker

brain-fart, sorry. @Leslileslie does this occur with the latest 0.7 release? – Mores

I tried this using for django 1.11 and django-import-export==0.5.1 but I haven't faced any issues. – Ginsburg

tried with 0.7. Same issue – Merrymaker

please provide forms.py and admin.py also – Deary

Added ModelForm – Leslileslie

For what it's worth, I was not able to recreate your issue using for same version of django and django-import-export. django-import-export ignores the BinaryField when it's missing from the fields list or included in the exclude list, as expected, and never attempt to serialize the memoryview, because it never creates an instance of a model with one populated. – Wilkes

It's really a huge waste of resources, time and effort to use django import export or to even attempt to dump a database at application level at all. Use the tools that are custom made for this - that is your databases dump tool. – Margalo

Answer 1

Sadly this seems to be a problem with django-import-export module.

In Python 2.7+ and 3.5.2+, pickling memoryview objects, is permitted. Try the following code just to confirm that:

import pickle

test = memoryview(b'abc')
test.__reduce__()

The above will raise the following error:

Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
    raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle memoryview objects

Apparently, in line 451 of django-import-export (version 0.6), the BinaryField is passed as a memoryview object to

original = deepcopy(instance)

which causes the error (instance contains the BinaryField as a memoryview object).
In the current version (1.0.0) this happens in line 446.

The instance retrieved/generated by the instance_loader of the module doesn't take into consideration the BinaryField.

You should probably open a ticket at django-import-export about this.

Croteau

Answer 2

have you added binary_field in ABCModelForm ? If yes, then it is creating issue because It is not possible to include a BinaryField in a ModelForm. For reference : binaryfield

Chisolm

No, I have exclude = ['binary_field'] in the ModelForm's Meta class – Leslileslie

Answer 3

Maybe passing the actual array instead of the memory view can solve your problem. If you want to execute a function in parallel, all of it parameters have to be picklable if i recall correctly. At least that is the case with python multiprocessing. So you could pass the array to the function and create the memoryview inside your function.

reference: moreinfo

Deary

Um, I'm not passing a memoryview anywhere. It's probably done implicitly by django-import-export – Leslileslie

The memoryview is created in BinaryFields.to_python method: <github.com/django/django/blob/…> The link is from the current head, but the behavior has been consistent for awhile. – Wilkes