Airflow Standalone Cannot use relative path:

Question

I just installed Airflow 2.3.0 using the command

pip install "apache-airflow==2.3.0" --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.3.0/constraints-3.8.txt"

in a virtual environment. Then tried running airflow standalone as specified in the official documentation here: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html

But I keep getting this error:

standalone | Starting Airflow Standalone
standalone | Checking database is initialized
INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, Add ``is_encrypted`` column in ``connection`` table
INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, Maintain history for compatibility with earlier migrations
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, Add ``operator`` and ``queued_dttm`` to ``task_instance`` table
INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, Add indices in ``job`` table
INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding ``extra`` column to ``Log`` table
INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, Add ``dag_run`` table
INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, Change ``task_instance.task_duration`` type to ``FLOAT``
INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, Add ``conf`` column in ``dag_run`` table
INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, Add ``password`` column to ``user`` table
INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, Add ``start_date`` and ``end_date`` in ``dag_run`` table
INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add ``notification_sent`` column to ``sla_miss`` table
INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, Add ``is_encrypted`` column to variable table
INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, Rename user table
INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, Add TI state index
INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, Add ``task_fail`` table
INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, Add ``dag_stats`` table
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to MySQL tables
INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, Add indices on ``xcom`` table
INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, Add ``pid`` field to ``TaskInstance``
INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add ``dag_id``/``state`` index on ``dag_run`` table
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, Add ``max_tries`` column to ``task_instance``
INFO  [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
INFO  [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, Create index on ``job_id`` column in ``task_instance`` table
INFO  [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, Add Kubernetes resource check-pointing
INFO  [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, Add ``executor_config`` column to ``task_instance`` table
INFO  [alembic.runtime.migration] Running upgrade 27c6a30d7c24 -> 86770d1215c0, Add kubernetes scheduler uniqueness
INFO  [alembic.runtime.migration] Running upgrade 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566, Merge migrations Heads
INFO  [alembic.runtime.migration] Running upgrade 05f30312d566 -> f23433877c24, Fix MySQL not null constraint
INFO  [alembic.runtime.migration] Running upgrade f23433877c24 -> 856955da8476, Fix Sqlite foreign key
INFO  [alembic.runtime.migration] Running upgrade 856955da8476 -> 9635ae0956e7, Create index on ``task_fail`` table
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> dd25f486b8ea, Add index on ``log`` table
INFO  [alembic.runtime.migration] Running upgrade dd25f486b8ea -> bf00311e1990, Add index to ``task_instance`` table
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, Add ``task_reschedule`` table
INFO  [alembic.runtime.migration] Running upgrade 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815, Merge migrations Heads
INFO  [alembic.runtime.migration] Running upgrade 03bc53e68815 -> 41f5f12752f8, Add superuser field
INFO  [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, Add ``description`` and ``default_view`` column to ``dag`` table
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag
INFO  [alembic.runtime.migration] Running upgrade dd4ecb8fbee3 -> 939bb1e647c8, task reschedule foreign key on cascade delete
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make ``TaskInstance.pool`` not nullable
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> d38e04c12aa2, Add ``serialized_dag`` table
INFO  [alembic.runtime.migration] Running upgrade d38e04c12aa2 -> b3b105409875, Add ``root_dag_id`` to ``DAG``
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, Increase queue name size limit
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove ``dag_stat`` table
INFO  [alembic.runtime.migration] Running upgrade a56c9515abdc, 004c1210f153, 74effc47d867, b3b105409875 -> 08364691d074, Straighten out the migrations
INFO  [alembic.runtime.migration] Running upgrade 08364691d074 -> fe461863935f, Increase length for connection password
INFO  [alembic.runtime.migration] Running upgrade fe461863935f -> 7939bcff74ba, Add ``DagTags`` table
INFO  [alembic.runtime.migration] Running upgrade 7939bcff74ba -> a4c2fd67d16b, Add ``pool_slots`` field to ``task_instance``
INFO  [alembic.runtime.migration] Running upgrade a4c2fd67d16b -> 852ae6c715af, Add ``RenderedTaskInstanceFields`` table
INFO  [alembic.runtime.migration] Running upgrade 852ae6c715af -> 952da73b5eff, Add ``dag_code`` table
INFO  [alembic.runtime.migration] Running upgrade 952da73b5eff -> a66efa278eea, Add Precision to ``execution_date`` in ``RenderedTaskInstanceFields`` table
INFO  [alembic.runtime.migration] Running upgrade a66efa278eea -> da3f683c3a5a, Add ``dag_hash`` Column to ``serialized_dag`` table
INFO  [alembic.runtime.migration] Running upgrade da3f683c3a5a -> 92c57b58940d, Create FAB Tables
INFO  [alembic.runtime.migration] Running upgrade 92c57b58940d -> 03afc6b6f902, Increase length of ``Flask-AppBuilder`` ``ab_view_menu.name`` column
INFO  [alembic.runtime.migration] Running upgrade 03afc6b6f902 -> cf5dc11e79ad, Drop ``user`` and ``chart`` table
INFO  [alembic.runtime.migration] Running upgrade cf5dc11e79ad -> bbf4a7ad0465, Remove id column from xcom
INFO  [alembic.runtime.migration] Running upgrade bbf4a7ad0465 -> b25a55525161, Increase length of pool name
INFO  [alembic.runtime.migration] Running upgrade b25a55525161 -> 3c20cacc0044, Add ``run_type`` column in ``dag_run`` table
INFO  [alembic.runtime.migration] Running upgrade 3c20cacc0044 -> 8f966b9c467a, Set ``conn_type`` as non-nullable
INFO  [alembic.runtime.migration] Running upgrade 8f966b9c467a -> 8d48763f6d53, Add unique constraint to ``conn_id``
INFO  [alembic.runtime.migration] Running upgrade 8d48763f6d53 -> e38be357a868, Add ``sensor_instance`` table
INFO  [alembic.runtime.migration] Running upgrade e38be357a868 -> b247b1e3d1ed, Add queued by Job ID to TI
INFO  [alembic.runtime.migration] Running upgrade b247b1e3d1ed -> e1a11ece99cc, Add external executor ID to TI
INFO  [alembic.runtime.migration] Running upgrade e1a11ece99cc -> bef4f3d11e8b, Drop ``KubeResourceVersion`` and ``KubeWorkerId``
INFO  [alembic.runtime.migration] Running upgrade bef4f3d11e8b -> 98271e7606e2, Add ``scheduling_decision`` to ``DagRun`` and ``DAG``
INFO  [alembic.runtime.migration] Running upgrade 98271e7606e2 -> 52d53670a240, fix_mssql_exec_date_rendered_task_instance_fields_for_MSSQL
INFO  [alembic.runtime.migration] Running upgrade 52d53670a240 -> 364159666cbd, Add ``creating_job_id`` to ``DagRun`` table
INFO  [alembic.runtime.migration] Running upgrade 364159666cbd -> 45ba3f1493b9, add-k8s-yaml-to-rendered-templates
INFO  [alembic.runtime.migration] Running upgrade 45ba3f1493b9 -> 849da589634d, Prefix DAG permissions.
INFO  [alembic.runtime.migration] Running upgrade 849da589634d -> 2c6edca13270, Resource based permissions.
Traceback (most recent call last):
File "c:\program files\python38\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
File "c:\program files\python38\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
File "D:\Projects\Airflow\env\Scripts\airflow.exe\__main__.py", line 7, in <module>
File "D:\Projects\Airflow\env\lib\site-packages\airflow\__main__.py", line 38, in main
    args.func(args)
File "D:\Projects\Airflow\env\lib\site-packages\airflow\cli\cli_parser.py", line 51, in command
    return func(*args, **kwargs)
File "D:\Projects\Airflow\env\lib\site-packages\airflow\cli\commands\standalone_command.py", line 48, in entrypoint
    StandaloneCommand().run()
File "D:\Projects\Airflow\env\lib\site-packages\airflow\cli\commands\standalone_command.py", line 64, in run
    self.initialize_database()
File "D:\Projects\Airflow\env\lib\site-packages\airflow\cli\commands\standalone_command.py", line 172, in initialize_database
    db.initdb()
File "D:\Projects\Airflow\env\lib\site-packages\airflow\utils\session.py", line 71, in wrapper
    return func(*args, session=session, **kwargs)
File "D:\Projects\Airflow\env\lib\site-packages\airflow\utils\db.py", line 645, in initdb
    upgradedb(session=session)
File "D:\Projects\Airflow\env\lib\site-packages\airflow\utils\session.py", line 68, in wrapper
    return func(*args, **kwargs)
File "D:\Projects\Airflow\env\lib\site-packages\airflow\utils\db.py", line 1411, in upgradedb
    command.upgrade(config, revision=to_revision or 'heads')
File "D:\Projects\Airflow\env\lib\site-packages\alembic\command.py", line 320, in upgrade
    script.run_env()
File "D:\Projects\Airflow\env\lib\site-packages\alembic\script\base.py", line 563, in run_env
    util.load_python_file(self.dir, "env.py")
File "D:\Projects\Airflow\env\lib\site-packages\alembic\util\pyfiles.py", line 92, in load_python_file
    module = load_module_py(module_id, path)
File "D:\Projects\Airflow\env\lib\site-packages\alembic\util\pyfiles.py", line 108, in load_module_py
    spec.loader.exec_module(module)  # type: ignore
File "<frozen importlib._bootstrap_external>", line 848, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "D:\Projects\Airflow\env\lib\site-packages\airflow\migrations\env.py", line 107, in <module>
    run_migrations_online()
File "D:\Projects\Airflow\env\lib\site-packages\airflow\migrations\env.py", line 101, in run_migrations_online
    context.run_migrations()
File "<string>", line 8, in run_migrations
File "D:\Projects\Airflow\env\lib\site-packages\alembic\runtime\environment.py", line 851, in run_migrations
    self.get_context().run_migrations(**kw)
File "D:\Projects\Airflow\env\lib\site-packages\alembic\runtime\migration.py", line 620, in run_migrations
    step.migration_fn(**kw)
File "D:\Projects\Airflow\env\lib\site-packages\airflow\migrations\versions\0074_2c6edca13270_resource_based_permissions.py", line 342, in upgrade
    remap_permissions()
File "D:\Projects\Airflow\env\lib\site-packages\airflow\migrations\versions\0074_2c6edca13270_resource_based_permissions.py", line 290, in remap_permissions
    appbuilder = create_app(config={'FAB_UPDATE_PERMS': False}).appbuilder
File "D:\Projects\Airflow\env\lib\site-packages\airflow\www\app.py", line 84, in create_app
    raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: Cannot use relative path: `sqlite:///D:\Projects\Airflow/airflow.db` to connect to sqlite. Please use absolute path such as `sqlite:////tmp/airflow.db`.

I have made sure the AIRFLOW_HOME variable does not have spaces in it.

Operating system: Windows 11 Python: 3.8 Airflow: 2.3.0

Wilk

Answer 1

I looked around and this is what I found:

Please set AIRFLOW_HOME. Then you can prevent this issue.

export AIRFLOW_HOME=~/airflow

The same issue is addressed here: sqlite3 raised an error after running Airflow command line

How to set AIRFLOW_HOME: https://youtu.be/M521KLHGaZc?t=768. The video sets airflow home under the C: path. Works for older versions but not 2.3.0, at least in my experience.

Cruzcruzado

I have this set in my machine, though I am using windows. I started to think it's not supported on windows without WSL and other people failed at using it directly as well – Wilk

@Wilk set it to just "/airflow" and check – Barstow

Answer 2

In Mac, If you are using a local directory, add sudo command before each airflow command.

Explanation:

This error mainly happens for two reasons, your system does not allow airflow to access path. or you did not defines the path of airflow.

export AIRFLOW_HOME=~/airflow

, which is the default location of airflow should solve the issue in case you have not defines it. If you did, and still facing issue, there is a chance your system is not allowing airflow to access it. So before every airflow command, use sudo (unix) or equivalent to other machines.

Honorable

could you include the full command? – Dorfman

Answer 3

I had faced the same issue on my Mac. The issue was that a special chararacter was not supported. So I deleted the generated files from airflow db init and updated the AIRFLOW_HOME with:

export AIRFLOW_HOME=`pwd`

This sets home as your current directory.

Elysian