airflow Questions
2
Solved
I have a use case in which, I am downloading some json files and parsing them. Depending on the files that are downloaded, the program needs to populate data in different tables. Once the data is l...
2
Solved
We have a requirement to schedule spark jobs, since we are familiar with apache-airflow we want to go ahead with it to create different workflows. I searched web but did not find a step by step gui...
Kirkman asked 16/11, 2018 at 19:35
3
Solved
I am starting to use Apache Airflow and I am wondering how to effectively make it use secrets and passwords stored in Vault. Unfortunately, search does not return meaningful answers beyond a yet-to...
Confirm asked 28/10, 2019 at 14:57
2
I am facing a issue wherein it is not able to import the airflow modules properly giving the below error. Could someone please help me understand it
10:39:36 ERROR: FileIngestion.test_download_file...
Centistere asked 22/12, 2023 at 12:22
2
Solved
Is there a way to run airflow DAG in a loop?
When trying to create a cycle (connecting the last component to the upstream of the last one) I got "Cycle detected in DAG. Faulty task: ..."
Generally...
Dumond asked 24/12, 2017 at 8:12
3
Solved
I'm trying to write a Python operator in an airflow DAG and pass certain parameters to the Python callable.
My code looks like below.
def my_sleeping_function(threshold):
print(threshold)
fmfd...
14
Airflow seems to be skipping the dags I added to /usr/local/airflow/dags.
When I run
airflow list_dags
The output shows
[2017-08-06 17:03:47,220] {models.py:168} INFO - Filling up the DagBag f...
Mikvah asked 6/8, 2017 at 17:10
3
I'm using Debian/Ubuntu, and I am trying to install an Airflow provider in my Python virtual environment:
$ pip install apache-airflow-providers-mysql
error: subprocess-exited-with-error
× Getti...
Apogee asked 10/8, 2023 at 11:52
4
Recently updated my Airflow and saw BigQueryExecuteQueryOperator has been deprecated in favor of BigQueryInsertJobOperator. The documentation seemed rather vague, linking to REST Resource: jobs (an...
Sitin asked 31/3, 2021 at 21:6
3
I need to get last two successful execution dates of Airflow job to use in my current run.
Example :
Execution date Job status
2020-05-03 success
2020-05-04 fail
2020-05-05 success
Question :
Wh...
Cahoot asked 8/5, 2020 at 3:40
4
Solved
I'm testing the use of Airflow, and after triggering a (seemingly) large number of DAGs at the same time, it seems to just fail to schedule anything and starts killing processes. These are the logs...
Postboy asked 29/8, 2019 at 15:26
4
Whenever I try to run a DAG, it will be in the running state but the tasks will not run. I have set my start date to datetime.today() and my schedule interval to "* * * * *". Manually triggering a ...
Almoner asked 13/9, 2017 at 20:23
4
I am trying to execute a task after 5 minutes from the parent task inside a DAG.
DAG : Task 1 ----> Wait for 5 minutes ----> Task 2
How can I achieve this in Apache Airflow? Thanks in advance.
4
Solved
I installed Python, Docker on my machine and am trying to import the
from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator but when I connect the docker, I get the mes...
Pontias asked 24/5, 2019 at 16:57
4
Solved
I am looking to write docker compose file to locally execute airflow in production similar environent.
For older airflow v1.10.14, docker compose is working fine. But same docker compose is not wor...
Phrenology asked 25/1, 2021 at 17:57
21
Solved
I have started the Airflow webserver and scheduled some dags. I can see the dags on web GUI.
How can I delete a particular DAG from being run and shown in web GUI? Is there an Airflow CLI command ...
Koenraad asked 17/11, 2016 at 9:59
4
Solved
I want to run a simple Dag "test_update_bq", but when I go to localhost I see this: DAG "test_update_bq" seems to be missing.
There are no errors when I run "airflow initdb", also when I run test a...
Trantrance asked 10/10, 2019 at 9:31
2
Solved
Does anyone use MWAA in production?
We currently have around 500 DAGs running and we see an unexpected behavior with tasks staying in a "queued" state for unknown reasons.
Task is in the...
Bellini asked 6/1, 2022 at 12:29
4
Solved
We have 100 of dags which has a prefix with "dag_EDW_HC_*" . we have below command to pause the dag
Command: airflow pause dag_id
Is there any way we can pause all the 100 dags "dag...
Jovitta asked 6/8, 2020 at 14:8
3
Solved
We use Amazon MWAA Airflow, rarely some task as marked as "FAILED" but there is no logs at all. As if the container had been shut down without noticing us.
I have found this link:
https:/...
Caterinacatering asked 30/4, 2021 at 10:12
5
Solved
Our airflow installation is using CeleryExecutor.
The concurrency configs were
# The amount of parallelism as a setting to the executor. This defines
# the max number of task instances that shoul...
3
I am trying to write unittests for some of the tasks built with Airflow TaskFlow API. I tried multiple approaches for example, by creating a dagrun or only running the task function but nothing is ...
Abate asked 13/7, 2022 at 22:52
4
Solved
I am trying to pass a list of strings from one task to another one via XCom but I do not seem to manage to get the pushed list interpreted back as a list.
For example, when I do this in some funct...
4
Solved
I am totally new to Airflow. I would like to run a simple DAG at a specified date. I'm struggling to make difference between the start date, the execution date, and backfilling. And what is the com...
2
Solved
I'm trying to setup an Airflow DAG that provides default values available from dag_run.conf. This works great when running the DAG from the webUI, using the "Run w/ Config" option. Howeve...
Industrialist asked 1/10, 2021 at 21:45
© 2022 - 2024 — McMap. All rights reserved.