How to make a Luigi task generate an in-memory list as target

Question

I'm trying to write an etl pipeline using luigi. As far as I understand from the documentation a task in luigi can generate a target that can be either some type of file storage or a database. To decrese the processing time I would like to have as an output an in-memory list. Is this possible? Do I have to create a custom target?

Vacillating

Related to question #42515963, which some useful looking answers – Mas

Answer 1

I found out I can use a MockFile as a target. A good example:

http://gouthamanbalaraman.com/blog/building-luigi-task-pipeline.html

Vacillating

the author noted that: "I use the MockFile here just as a way to see the results on the console. This is usually a good testing tool, though I am not sure if it is production ready approach." So have you tried MockFile for (semi-)production pipelines? Is it right way for in-memory pipelining? – Kilan