ERROR: jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3 occurs in Airflow/papermill running as a docker container

Question

I am trying to run an example papermill notebook that you can find here: at the Official Apache Airflow url

airflow-github

with a corresponding dag file, found within the same directory as specified above.

I am running this locally using a docker-compose assembly that you can find here at the Airflow website:

airflow-docker-page

I had to custom build the image with Papermill because it looks like it is not baked into the image:

enter image description here

Here is an error I am getting:

raise NoSuchKernel(kernel_name)

jupyter_client.kernelspec.NoSuchKernel: No such kernel named python3

How do I add the kernel?

UPDATE: I managed to resolve this issue but now have this:

[2021-10-28, 22:47:57 UTC] {execute.py:84} INFO - Input Notebook: /opt//dags/example_notebook.ipynb [2021-10-28, 22:47:57 UTC] {execute.py:85} INFO - Output Notebook: out-2021-10-28T22:47:55.826195+00:00.ipynb [2021-10-28, 22:47:57 UTC] {driver.py:192} INFO - Generating grammar tables from /home//.local/lib/python3.6/site-packages/blib2to3/Grammar.txt [2021-10-28, 22:47:57 UTC] {driver.py:195} INFO - Writing grammar tables to /home//.cache/black/21.9b0/Grammar3.6.15.final.0.pickle [2021-10-28, 22:47:57 UTC] {driver.py:199} INFO - Writing failed: [Errno 2] No such file or directory: '/home/airflow/.cache/black/21.9b0/tmp27kcm8y_' [2021-10-28, 22:47:57 UTC] {driver.py:192} INFO - Generating grammar tables from /home//.local/lib/python3.6/site-packages/blib2to3/PatternGrammar.txt [2021-10-28, 22:47:57 UTC] {driver.py:195} INFO - Writing grammar tables to /home//.cache/black/21.9b0/PatternGrammar3.6.15.final.0.pickle [2021-10-28, 22:47:57 UTC] {driver.py:199} INFO - Writing failed: [Errno 2] No such file or directory: '/home/airflow/.cache/black/21.9b0/tmpnexc_sob' [2021-10-28, 22:47:57 UTC] {warnings.py:99} WARNING - /home//.local/lib/python3.6/site-packages/IPython/paths.py:67: UserWarning: IPython parent '/home/***' is not a writable location, using a temp directory. " using a temp directory.".format(parent))

Flyback

Answer 1

I got same issue with you,

I resovled it by add there rows in Docker file

RUN pip install --upgrade pip ipython ipykernel
RUN ipython kernel install --name "python3" --user

And then

docker-compose build
docker-compose up

Here is my full Dockerfile

FROM apache/airflow:2.2.1   
COPY requirement.txt .   
RUN pip install -r requirement.txt
RUN pip install --upgrade pip ipython ipykernel   
RUN ipython kernel install --name "python3" --user
CMD python -m ipykernel_launcher -f $DOCKERNEL_CONNECTION_FILE

Fetor

really appreciated – Fruitful

pip install --upgrade pip ipython ipykerne was enough for me to solve the error. thank you. – Bronez