airflow Questions
4
How to deploy apache airflow (formally known as airbnb's airflow) scheduler in high availability?
I am not asking about the backend DB or RabbitMQ that should obviously be deployed in high availab...
Webworm asked 19/9, 2016 at 11:27
4
Solved
I have the following DAG with 3 tasks:
start --> special_task --> end
The task in the middle can succeed or fail, but end must always be executed (imagine this is a task for cleanly closing ...
Curren asked 7/8, 2018 at 13:49
3
Solved
I have a dag that has been triggered by another dag. I have passed through to this dag some configuration variables via the DagRunOrder().payload dictionary in the same way the official example has...
4
Solved
Is there a way to ssh to different server and run BashOperator using Airbnb's Airflow?
I am trying to run a hive sql command with Airflow but I need to SSH to a different box in order to run the hi...
7
Solved
I've just installed Apache Airflow, and I'm launching the webserver for the first time, and it asks me for username and password, I haven't set any username or password.
Can you let me know what is...
Breastbone asked 11/2, 2021 at 18:36
2
I am running Airflow on Google's Cloud Composer. I am using the KubernetesPodOperator and would like to mount a google storage bucket to a directory in pod via gcsfuse. It seems like to do this I n...
Janettajanette asked 22/11, 2019 at 20:56
2
Solved
I am using AWS's MWAA service (2.2.2) to run a variety of DAGs, most of which are implemented with standard PythonOperator types. I bundle the DAGs into an S3 bucket alongside any shared requiremen...
Daemon asked 13/4, 2022 at 17:44
4
I'd like to get some clarification on whether Cloud Dataflow or Cloud Composer is the right tool for the job, and I wasn't clear from the Google Documentation.
Currently, I'm using Cloud Dat...
Gunslinger asked 11/1, 2019 at 22:20
3
Solved
It is not clear to me if the aws mwaa documentation says the following "With Amazon Managed Workflows for Apache Airflow (MWAA) you pay only for what you use. There are no minimum fees or upfr...
Ganef asked 11/12, 2021 at 20:46
2
Solved
I have a spark job that runs via a Kubernetes pod . Till now I was using an Yaml file to run my jobs manually.
Now , I want to schedule my spark jobs via airflow.
This is the first time I am using ...
Meador asked 24/10, 2019 at 9:28
3
Solved
I'm learning Airflow and have a simple question. Below is my DAG called dog_retriever:
import airflow
from airflow import DAG
from airflow.operators.http_operator import SimpleHttpOperator
from air...
Spiculate asked 10/10, 2017 at 21:39
3
Solved
Is it possible to parse JSON string inside an airflow template?
I have a HttpSensor which monitors a job via a REST API, but the job id is in the response of the upstream task which has xcom_push ...
Doorway asked 27/11, 2017 at 8:0
2
Solved
I am trying to understand catchup and backfill in airflow. I understood what catchup is but I have not completely understood what backfill exactly is and how it is used.
I have read the documentat...
Risibility asked 30/7, 2019 at 9:31
2
What's the best way to handle having a different schedule interval for backfilling and ongoing running?
For backfilling I want to use a daily interval, but for ongoing running I want to use an hou...
1
While running docker image, I am receiving the following error:
/entrypoint.sh: line 7: USER: unbound variable
Code of entrypoint.sh.
#!/bin/bash
set -euo pipefail
export SPARK_DIST_CLASSPAT...
Respondent asked 8/6, 2017 at 7:38
2
Solved
I am currently using Airflow Taskflow API 2.0. I am having an issue of combining the use of TaskGroup and BranchPythonOperator.
Below is my code:
import airflow
from airflow.models import DAG
from ...
Revenge asked 27/5, 2021 at 10:38
6
I have installed apache airflow on Ubuntu 18.4 using this link https://airflow.apache.org/docs/apache-airflow/stable/start/local.html
now when i run airflow with
airflow webserver --port 8080
and...
Rosalvarosalyn asked 11/4, 2021 at 14:0
2
Solved
I see from the log the following info:
[2019-02-28 16:33:14,766] {python_operator.py:95} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=email_operator_with_log_attachment_example
AIR...
Alage asked 28/2, 2019 at 21:42
4
Solved
Is it possible to pass parameters to Airflow's jobs through UI?
AFAIK, 'params' argument in DAG is defined in python code, therefore it can't be changed at runtime.
Counterglow asked 20/11, 2017 at 16:44
5
I recently have built a Python program that runs on Google Colaboratory, I need to run the program every day at a specific time, So Is there any way to schedule it to run on Google Colab?
Smetana asked 12/5, 2020 at 21:13
2
Solved
I am new to airflow, tried to run a dag by starting airflow webserver and scheduler. After I closed the scheduler and airflow webserver, the airflow processes are still running.
ps aux | grep airfl...
Episcopate asked 10/1, 2021 at 17:26
2
Solved
I am reading Data Pipelines with Apache Airflow book In the book the picture shows a tree view however my Airflow installation doesn't contain tree view, is it removed in late versions?
My current ...
Ringworm asked 20/7, 2022 at 11:3
17
Problem
Is there any way in Airflow to create a workflow such that the number of tasks B.* is unknown until completion of Task A? I have looked at subdags but it looks like it can only work with a...
4
Solved
I need to implement the waiting task in Airflow.
Waiting time is to be around a couple of hours.
First,
TimeDeltaSensor is just not working.
SLEEP_MINUTES_1ST = 11
sleep_task_1 = TimeDeltaSenso...
15
Solved
I am new to Airflow. I am following a tutorial and written following code.
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime, timed...
Amenity asked 17/8, 2016 at 9:38
© 2022 - 2024 — McMap. All rights reserved.