I am trying to configure Luigi's retry mechanism so that failed tasks will be retried a few times. However, while the task is retried successfully, Luigi exits unsuccessfully: ===== Luigi Executio...

As far as I know, a luigi.Target can either exist, or not. Therefore, if a luigi.Target exists, it wouldn't be recomputed. I'm looking for a way to force recomputation of the task, if one of its d...

Say I have a task with the following dependency structure class ParentTask(luigi.Task): def requires(self): return [ChildTask(classLevel=x) for x in self.class_level_list] def run(self): yaday...

I'm starting to port a nightly data pipeline from a visual ETL tool to Luigi, and I really enjoy that there is a visualiser to see the status of jobs. However, I've noticed that a few minutes after...

We are trying to run a simple pipeline distributed on a docker swarm cluster. The luigi workers are deployed as replicated docker services. They start successfully and after a few seconds of asking...

We have a Luigi Task that request a piece of information from a 3rd party service. We are limited on the number of call requests we can perform per minute to that API call. Is there a way to speci...

I'm trying to grasp how luigi works, and I get the idea, but actual implementation is a bit harder ;) This is what i have: class MyTask(luigi.Task): x = luigi.IntParameter() def requires(self)...

I have a luigi python task which includes some pyspark libs. Now I would like to submit this task on mesos with spark-submit. What should I do to run it? Below is my code skeleton: from pyspark.sq...

I'm trying to write an etl pipeline using luigi. As far as I understand from the documentation a task in luigi can generate a target that can be either some type of file storage or a database. To d...

I defined three tasks T1, T2, and T3, and then a task T4 as follows: class T4(luigi.Task) def requires(self): return [T1(), T2(), T3()] Is there a natural way to tell Luigi that I want these t...

I am using django as a web framework. I need a workflow engine that can do synchronous as well as asynchronous(batch tasks) chain of tasks. I found celery and luigi as batch processing workflow. My...

I am using Luigi to launch some pipeline. Let's take a simple exemple task = myTask() w = Worker(scheduler=CentralPlannerScheduler(), worker_processes=1) w.add(task) w.run() Now let's say that ...

My initial files are in AWS S3. Could someone point me how I need to setup this in a Luigi Task? I reviewed the documentation and found luigi.S3 but is not clear for me what to do with that, then ...

I am trying to mock something that supplies a default value for a luigi parameter. A dumb example showing what I'm trying to accomplish: Task under test: import luigi from bar import Bar bar = ...