Login/Register
Plone: collective.transmogrifier.sections.constructor doesn't write objects while importing
Asked Answered
plonedata-migrationplone-4.xtransmogrifier
R

1

8

I'm almost successful in my efforts to transfer an excerpt of my ZODB (a small subset of folders, including resources taken from other folders) to another Plone instance; the showstopper is: during import, my objects are not written. Here is my export script:

[transmogrifier]
pipeline =
    sitewalker
    xmlwalker
    uidextractor
    encapsulator
    manifestexporter
    fileexporter
    marshaller
    propertiesexporter
    commentsexporter
    datacorrector
    portletsexporter
    writer
    EXPORTING

[sitewalker]
blueprint = quintagroup.transmogrifier.sitewalker
path =
    folder1
    folder2

[xmlwalker]
blueprint = collective.transmogrifier.sections.xmlwalker

[uidextractor]
# my own section which parses the html text of the given fields,
# checks the href and src attributes for UIDs, and injects entries
# for the refered objects into the pipeline
blueprint = my.transmogrifier.uidextractor
inspect_fields =
    text
    description
    notes

[encapsulator]
blueprint = plone.app.transmogrifier.mimeencapsulator
mimetype = item/_mimetype
field = string:datafield

[manifestexporter]
blueprint = quintagroup.transmogrifier.manifestexporter

[fileexporter]
blueprint = quintagroup.transmogrifier.fileexporter

[marshaller]
blueprint = quintagroup.transmogrifier.marshaller

[propertiesexporter]
blueprint = quintagroup.transmogrifier.propertiesexporter

[commentsexporter]
blueprint = quintagroup.transmogrifier.commentsexporter

[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
sources =
    marshall

[portletsexporter]
blueprint = quintagroup.transmogrifier.portletsexporter

[writer]
blueprint = quintagroup.transmogrifier.writer
context = directory
path = var/export/
prefix = structure

[EXPORTING]
blueprint = quintagroup.transmogrifier.logger
keys =
    _type
    _path

Based on the default export and import scripts from quintagroup.transmogrifier, here is my import script:

[transmogrifier]
pipeline =
    reader
    pathfixer
    constructor
    schemaupdater
    datacorrector
    demarshaller
    uidupdater
    referencesimporter
    propertiesimporter
    commentsimporter
    portletsimporter
    printcounters
    IMPORTING

[reader]
blueprint = quintagroup.transmogrifier.reader
prefix = 
path = /path/to/var/export
context = directory
# unchanged entries I don't really understand:
.objects.xml = manifest
.marshall.xml = marshall
.properties.xml = propertymanager
.comments.xml = comments
.file-fields.xml = file-fields
.interfaces.xml = interfaces
.portlets.xml = portlets

[pathfixer]
blueprint = plone.app.transmogrifier.pathfixer
stripstring = Plone/
prependstring = 

[constructor]
blueprint = collective.transmogrifier.sections.constructor

[schemaupdater]
blueprint = plone.app.transmogrifier.atschemaupdater

[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
type = import
sources =
    marshall

[fileimporter]
blueprint = quintagroup.transmogrifier.fileimporter

[demarshaller]
blueprint = quintagroup.transmogrifier.demarshaller

[uidupdater]
blueprint = plone.app.transmogrifier.uidupdater

[referencesimporter]
blueprint = quintagroup.transmogrifier.referencesimporter

[propertiesimporter]
blueprint = quintagroup.transmogrifier.propertiesimporter

[commentsimporter]
blueprint = quintagroup.transmogrifier.commentsimporter

[portletsimporter]
blueprint = quintagroup.transmogrifier.portletsimporter

[printcounters]
blueprint = collective.transmogrifier.sections.summary
count = true

[IMPORTING]
blueprint = quintagroup.transmogrifier.logger
keys = 
    _type
    _path

As far as I understand, the reader comes up to the writer; the marshaller comes up for the demarshaller, etc. However, while importing, the objects are not actually written to the database. Tweaking the blueprints with a counting facility (in the add-info branch of my forks of the packages), I got the following overview:

Items summary
~~~~~~~~~~~~~
[reader]:
  created:   714
[pathfixer]:
  got:         714
  forwarded:   714
  stripped:    390
[constructor]:
  got:            714
  missing-type:   714
  missing-info:   714
  forwarded:      714
[datacorrector]:
  got:         714
  forwarded:   714
[fileimporter]:
  got:          31
  forwarded:    31
[demarshaller]:
  got:          31
  forwarded:    31
[uidupdater]:
  got:          31
  forwarded:    31
[referencesimporter]:
  passed-through:    31
[printcounters]:
  passed-through:    31
[IMPORTING]:
  got:          31
  forwarded:    31

When the constructor doesn't do anything about the item, I count the reasons; obviously the constructor section doesn't write anything because it doesn't know the types of the objects to create. But shouldn't this information have been created somewhere?!

(There is another problem: apparently the fileimporter receives only 31 objects of the 714 passed on by the datacorrector. But first I'd like to see anything imported.)

What am I doing wrong?

Oh, and online documentation for the collective.transmogrifier.sections would be nice;they even lack docstrings ...

Edit: My counting-enabled forks are here (branches add-info):

2nd Edit:

When I move the IMPORTING section before constructer, the counters output is:

Items summary
~~~~~~~~~~~~~
[reader]:
  created:   714
[pathfixer]:
  got:         714
  forwarded:   714
  stripped:    390
[IMPORTING]:
  got:         714
  forwarded:   714
[constructor]:
  got:            714
  missing-type:   714
  missing-info:   714
  forwarded:      714
[datacorrector]:
  got:         714
  forwarded:   714
[fileimporter]:
  got:          31
  forwarded:    31
[demarshaller]:
  got:          31
  forwarded:    31
[uidupdater]:
  got:          31
  forwarded:    31
[referencesimporter]:
  passed-through:    31
[printcounters]:
  passed-through:    31

Thus, the collector still can't find _type information.

3rd Edit:

I added a little facility which prints a short info about the items found (the 1st item per section by default; _path and _type with values, if present, and a list of the other keys). The result is:

[reader], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[pathfixer], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[pathfixer], item #2:
    _path=''
    other keys: _import_context (DirectoryImportContext)
2015-08-18 18:39:56 INFO IMPORTING _path=
[constructor], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[reader], item #2:
    _path='some/archetypes/object/containing/a/video'
    other keys: _files (dict), _import_context (DirectoryImportContext)

Indeed there are no _type keys anywhere, so I need a section which provides them.

4th Edit:

After re-inserting manifestimporter before constructor, I got:

[reader], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[pathfixer], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[pathfixer], item #2:
    _path=''
    other keys: _import_context (DirectoryImportContext)
2015-08-19 10:15:43 INFO IMPORTING _path=
[constructor], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[manifestimporter], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[reader], item #2:
    _path='    [reader], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[pathfixer], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[pathfixer], item #2:
    _path=''
    other keys: _import_context (DirectoryImportContext)
2015-08-19 10:15:43 INFO IMPORTING _path=
[constructor], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[manifestimporter], item #1:
    _path=''
    other keys: _import_context (DirectoryImportContext)
[reader], item #2:
    _path='some/archetypes/object/containing/a/video'
    other keys: _files (dict), _import_context (DirectoryImportContext)
...
2015-08-19 10:15:44 INFO IMPORTING
Pipeline processing time: 00:00:00
         715 items were generated in source sections
           2 went through full pipeline
         713 were discarded in some section

The manifestimporter sections doesn't forward any previous items, so all items from reader are thrown away.

5th Edit: Tried the recommended way to use the "Site Configuration Export Step;" I edited the default export script and tried the export, but I got the following traceback:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.GenericSetup.tool, line 598, in manage_exportSelectedSteps
  Module Products.GenericSetup.tool, line 1053, in _doRunExportSteps
  Module quintagroup.transmogrifier.exportimport, line 74, in exportSiteStructure
  Module collective.transmogrifier.utils, line 121, in constructPipeline
  Module quintagroup.transmogrifier.sitewalker, line 29, in __init__
TypeError: ('Could not adapt', {'manifestexporter': {'blueprint': 'quintagroup.transmogrifier.manifestexporter'}, 'transmogrifier': {'pipeline': '\nsitewalker\nuidextractor\npathfixer\nmanifestexporter\nfileexporter\nmarshaller\npropertiesexporter\ncommentsexporter\ndatacorrector\nportletsexporter\nwriter\nEXPORTING'}, 'uidextractor': {'blueprint': 'my.transmogrifier.uidextractor', 'trace-first': 'true', 'inspect_fields': '\ntext\ndescription\nnotes'}, 'sitewalker': {'blueprint': 'quintagroup.transmogrifier.sitewalker', 'start-path': '\nakademie/vortraege/d-02-verlegung-lektion-02-leitungsgraben', 'exclude-contained': 'false'}, 'xmlwalker': {'blueprint': 'collective.transmogrifier.sections.xmlwalker'}, 'encapsulator': {'blueprint': 'plone.app.transmogrifier.mimeencapsulator', 'mimetype': 'item/_mimetype', 'field': 'string:datafield'}, 'writer': {'blueprint': 'quintagroup.transmogrifier.writer', 'path': 'var/export/', 'prefix': '', 'context': 'tarball'}, 'commentsexporter': {'blueprint': 'quintagroup.transmogrifier.commentsexporter'}, 'pathfixer': {'blueprint': 'plone.app.transmogrifier.pathfixer', 'stripstring': 'unitracc/'}, 'echo': {'blueprint': 'visaplan.transmogrifier.echo'}, 'marshaller': {'blueprint': 'quintagroup.transmogrifier.marshaller'}, 'propertiesexporter': {'blueprint': 'quintagroup.transmogrifier.propertiesexporter'}, 'datacorrector': {'blueprint': 'quintagroup.transmogrifier.datacorrector', 'sources': '\nmarshall'}, 'breakpoint': {'blueprint': 'collective.transmogrifier.sections.breakpoint'}, 'EXPORTING': {'blueprint': 'quintagroup.transmogrifier.logger', 'keys': '\n_type\n_path'}, 'portletsexporter': {'blueprint': 'quintagroup.transmogrifier.portletsexporter'}, 'fileexporter': {'blueprint': 'quintagroup.transmogrifier.fileexporter'}}, <InterfaceClass zope.annotation.interfaces.IAnnotations>)

It didn't matter whether or not my uidexporter section was included.

6th Edit:

Here is the current export profile which I used for the Site Configuration Export Step:

[transmogrifier]
pipeline =
    sitewalker
    manifestexporter
    fileexporter
    marshaller
    propertiesexporter
    commentsexporter
    datacorrector
    portletsexporter
    writer
    EXPORTING

[sitewalker]
blueprint = quintagroup.transmogrifier.sitewalker
exclude-contained = false
start-path =
    Plone/some/existing/structure

[manifestexporter]
blueprint = quintagroup.transmogrifier.manifestexporter

[fileexporter]
blueprint = quintagroup.transmogrifier.fileexporter

[marshaller]
blueprint = quintagroup.transmogrifier.marshaller

[propertiesexporter]
blueprint = quintagroup.transmogrifier.propertiesexporter

[commentsexporter]
blueprint = quintagroup.transmogrifier.commentsexporter

[datacorrector]
blueprint = quintagroup.transmogrifier.datacorrector
sources =
    marshall

[portletsexporter]
blueprint = quintagroup.transmogrifier.portletsexporter

[writer]
blueprint = quintagroup.transmogrifier.writer
context = tarball
path = var/export/
prefix =

[EXPORTING]
blueprint = quintagroup.transmogrifier.logger
keys =
    _type
    _path

7th Edit: A short information which kind of data is required by the reader to enable the constructor to create objects, and which section of the export pipeline (with an idea about the necessary options) might earn easy 150 reputation points ;-)

Reactance answered 13/8, 2015 at 16:0 Comment(11)
The constructor blueprint needs the '_type' key or similar to work out which type to create & the _path key to work out where to put it. Move the IMPORTING section before the constructor section in the pipeline & tell us what gets printed. – Athapaskan
@Danimal: You are right, there is a rst page for most of the blueprints. I edited my question and added some information. – Reactance
can you pastie your export pipeline please? – Athapaskan
What is the visaplan.transmogrifier.echo in your traceback? It's not in your pipeline :-/ Maybe you need to restart (or rerun buildout)? – Athapaskan
Sorry, I overlooked that. It doesn't do anything special; it just prints the items and yields them, similar to other blueprints I found later. – Reactance
Try using pdb In sitewalker? – Athapaskan
I think you would do better just trying to prove q.tm is broken in a vanilla Plone install. If so - raise a bug against in github, if not then build from there. – Athapaskan
If the same goal could be achieved e.g. with plone.app.transmogrifier blueprints, this would be fine with me. Sadly I couldn't find useful hints about (pretty standard, I'd say) working export and corresponding import pipelines. Oh, and I don't insist in using the "Site configuration export" method. – Reactance
Yep, I vaguely remember similar difficulties finding out how to get starting the Transmogrifier. I mentioned the common use stories as I saw it in my blog - but you've already read that, but basically I think most people do use quintagroup.transmogrifier to export / import, so it should work. In conclusion: you are on a mainstream track with q.tm - so try to prove it broken or otherwise – Athapaskan
@Danimal: I issued an error for default export script which is IMHO definitely broken. – Reactance
Ok, hopefully that gets a response. Add a foolproof way to prove the error would improve your chances. – Athapaskan
A
0

From a comparison to quintagroup.transmogrifier's import.cfg You are missing a manifestimporter section in your pipeline. Their pipeline reads:

pipeline =
    reader
    manifestimporter
    constructor
    datacorrector
    fileimporter
    demarshaller
    referencesimporter
    propertiesimporter
    commentsimporter
    portletsimporter
    IMPORTING
Athapaskan answered 18/8, 2015 at 18:6 Comment(4)
True. I had commented out this one because the manifestimporter doesn't forward any of the previous items, and because I could not find any documentation about it. See my update. – Reactance
Ok, but you seem to be far off the beaten track here. Can you not try following one of the How to's at the bottom of this page and report back? "Content migration from Plone 3 to Plone 4" will probably suit you even if you are doing 4 to 4... – Athapaskan
Yes, sorry; urgent things to do meanwhile. Tried the Site configuration Export, but no success; traceback following. – Reactance
updated the question, including a traceback. I'm not sure the site configuration export step works currently, but this might depend on the versions involved ... – Reactance

© 2022 - 2024 — McMap. All rights reserved.