luigi Questions
1
I have a luigi task that performs some non-stable computations. Think of an optimization process that sometimes does not converge.
import luigi
MyOptimizer(luigi.Task):
input_param: luigi.Param...
Afton asked 4/5, 2020 at 16:4
4
Solved
I am trying to learn in a very simple way how luigi works. Just as a newbie I came up with this code
import luigi
class class1(luigi.Task):
def requires(self):
return class2()
def output(self...
3
When looping over files with Luigi I do not what to be forced to save empty files just to show that the task was complete, and let the next task check if there are any rows in the txt, etc.
How ca...
6
Solved
Currently, I have a bunch of luigi tasks queued together, with a simple dependency chain( a -> b -> c -> d). d gets executed first, and a at the end. a is the task that gets triggered.
A...
4
Solved
I have a Luigi task that requires a subtask. The subtask depends on parameters passed through by the parent task (i.e. the one that is doing the requireing). I know you can specify a parameter that...
1
I have built a pipeline of Tasks in Luigi. Because this pipeline is going to be used in different contexts, it was possible that it would require to include more tasks at the beginning of or the en...
Shaker asked 2/3, 2017 at 18:30
2
Solved
I have a luigi workflow that downloads a bunch of large files via ftp and deposits them on s3.
I have one task that reads a list of files to download then creates a bunch of tasks that actually do...
Spirituality asked 8/1, 2018 at 12:34
4
Solved
I am having problems running a Luigi task through the Windows cmd. Here are the facts:
Running Anaconda installed in C:\ProgramData\Anaconda2 (Python 2.7)
Anaconda has added its paths to the PA...
Windywindzer asked 16/3, 2017 at 19:48
1
Solved
I have a Luigi task with a boolean parameter that is set to True by default:
class MyLuigiTask(luigi.Task):
my_bool_param = luigi.BoolParameter(default=True)
When I run this task from terminal...
1
I am building a wrapper for Luigi Tasks and I ran into a snag with the Register class that's actually an ABC metaclass and not being pickable when I create a dynamic type.
The following code, mor...
3
I am copying a pdf file to local, using the following piece of code:
with self.input_target().open('r') as r:
with self.output_target().open('w') as w:
for line in r:
w.write(line)
Which is...
Jumpoff asked 7/11, 2015 at 5:18
3
I've been trying to integrate Luigi as our workflow handler. Currently we are using concourse, however many of the things we're trying to do is a hassle to get around in concourse so we made the sw...
Instantaneous asked 28/2, 2017 at 19:49
2
Solved
I was coding my first project in Python 3.6 using Spotify's Luigi to arrange some Natural Language Processing Tasks in a pipeline.
I noticed that the output() function of a Task class always retur...
Lascar asked 28/2, 2017 at 17:45
2
I am using Luigi for my workflow. My workflow is divided into three general parts - import, analysis, export. Within each part, there are multiple Luigi tasks.
I could have everything in a single ...
Evans asked 20/10, 2016 at 18:8
2
Solved
I have Sidekiq running with a Rails app. I need to be able to run a job from a Python script (as I'm using Luigi to run tasks in general). I'm searching for a Python library to work with the Sideki...
2
Solved
I am trying to build a strategy to log from Luigi in such a way that there is a configurable list of outputs including stdout and a custom list of files. I would like to be able to set the logging ...
Entropy asked 16/6, 2016 at 18:48
2
Solved
I am having troubles to write a binary LocalTarget in a Luigi pipeline in my project. I isolated the problem here:
class LuigiTest(luigi.Task):
def output(self):
return luigi.LocalTarget('test.n...
Instead asked 27/8, 2016 at 9:33
2
Solved
I am running on Windows 7, Python 2.7 via Anaconda 4.3.17, Luigi 2.4.0, Pandas 0.18, sklearn version 0.18. Per below, I am trying to have a luigi.LocalTarget output be a pickle to store a few diffe...
Lucie asked 7/6, 2017 at 15:46
1
Solved
Consider the following tasks:
import luigi
class YieldFailTaskInBatches(luigi.Task):
def run(self):
for i in range(5):
yield [
FailTask(i, j)
for j in range(2)
]
class YieldAllFailTasksA...
2
Solved
I have installed luigi by pip command and I would like to change the port for the web UI. I tried to find the config file but I couldn't. Do I need to create one?
1
Solved
I am new to luigi, came across it while designing a pipeline for our ML efforts. Though it wasn't fitted to my particular use case it had so many extra features I decided to make it fit.
Basically...
Woollen asked 26/6, 2018 at 15:51
1
I am using luigi to execute a chain of tasks, like so:
class Task1(luigi.Task):
stuff = luigi.Parameter()
def output(self):
return luigi.LocalTarget('test.json')
def run(self):
with self.ou...
1
Solved
I am looking at several open source workflow schedulers for a DAG of jobs with heterogeneous RAM usage. The scheduler should not only schedule less than a maximum number of threads, but should also...
Notions asked 2/9, 2018 at 20:52
3
Solved
How do I pass in parameters to Luigi? if I have a python file called FileFinder.py with a class named getFIles:
class getFiles(luigi.Task):
and I want to pass in a directory to this class such as...
1
Solved
In the Luigi docs, the use of a luigi.Config class is recommended for global configuration.
However, I am running into issues when using such a config class in order to pass a commandline argument...
1 Next >
© 2022 - 2024 — McMap. All rights reserved.