Docker is pushing all layers instead of the last one

Question

Yesterday i pushed the base image layer for my app that contained the environment needed to run my_app.

That push was massive but it is done and up in my repo.

This is currently the image situation in my local machine:

➜  docker images                          
REPOSITORY               TAG                 IMAGE ID       CREATED          SIZE
dr_prof_patrick/my_app   my_app_v0           7e4cb75b4735   22 minutes ago   5.36GB
dr_prof_patrick/my_app   my_app_base_image   b1cccd87e4f7   37 hours ago     5.35GB
python                   3.8                 67ec76d9f73b   8 days ago       909MB
python                   3                   f48ea80eae5a   8 days ago       917MB

I introduced some minor changes to the image as you can see here:

➜  docker history dr_prof_patrick/my_app:my_app_v0
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
7e4cb75b4735   22 minutes ago   /bin/sh -c #(nop)  CMD ["python3" "main.py"]    0B        
55fe27affa9a   22 minutes ago   /bin/sh -c pip install -r requirements.txt      16.1kB    
3eba19411e42   22 minutes ago   /bin/sh -c #(nop) WORKDIR /my_app            0B        
54249235bcba   22 minutes ago   /bin/sh -c #(nop) COPY dir:7f9da91b4e3f9ed60…   6.03MB    
b1cccd87e4f7   37 hours ago     /bin/sh -c pip install -r requirements.txt      4.04GB    
<missing>      37 hours ago     /bin/sh -c #(nop) COPY file:8e879a6a889ff22f…   305B      
<missing>      37 hours ago     /bin/sh -c apt-get install ffmpeg libsm6 lib…   385MB     
<missing>      37 hours ago     /bin/sh -c apt-get update                       17.7MB    
<missing>      8 days ago       /bin/sh -c #(nop)  CMD ["python3"]              0B        
<missing>      8 days ago       /bin/sh -c set -ex;   wget -O get-pip.py "$P…   8.31MB    
<missing>      8 days ago       /bin/sh -c #(nop)  ENV PYTHON_GET_PIP_SHA256…   0B        

It tries to push the base layer again which takes like 2 hours.

➜  docker push dr_prof_patrick/my_app:my_app_v0    
The push refers to repository [docker.io/dr_prof_patrick/my_app]
548f7bc62c43: Layer already exists 
fa5433c54740: Layer already exists 
f09b381b1d57: Pushing  1.102MB/4.041GB
a0f39ee33d3d: Layer already exists 
a1e799af3370: Pushing  8.842MB/384.5MB
c93f6a714096: Pushing   1.77MB/17.7MB
e9df9d3bdd45: Layer already exists 
1271cc224a6b: Layer already exists 
740ef99eafe1: Pushing   5.42MB/48.56MB
b7b662b31e70: Pushing  6.628MB/18.47MB
6f5234c0aacd: Waiting 
8a5844586fdb: Waiting 
a4aba4e59b40: Waiting 
5499f2905579: Waiting 
a36ba9e322f7: Waiting 

I dont know how my workflow prevented docker from uploading only the latest changes - any help would be appreciated.

EDIT

Thought the docker file that was used to create the my_app_v0 could help:

FROM dr_prof_patrick/my_app:my_app_base_image

COPY . /my_app

WORKDIR /my_app

RUN pip install -r requirements.txt

CMD ["python3", "main.py"]

Ribal

Answer 1

Docker invalidates COPY layers once the context changes -- regardless of what the next steps actually depend on. Copy files at the last possible moment -- in your case, copy requirements.txt first, and the rest later. Like this:

FROM dr_prof_patrick/my_app:my_app_base_image

WORKDIR /my_app

COPY requirements.txt .

RUN pip install -r requirements.txt

COPY . .

CMD ["python3", "main.py"]

Also take a look at your .dockerignore and don't copy useless files. The best strategy I see used is to use .dockerignore as a whitelist, not a blacklist, by ignoring everything first and then un-ignoring the files you need:

*
!requirements.txt

Americano

This would improve a lot of the performance in future image updates since Docker could often use cache if the "requirements.txt" file doesn't change. – Deracinate

Thanks for the response, though i tried both replacing Dockerfile with you suggested one as well as adding the .dockerignore and it still tries to upload everything – Ribal

It will try to upload everything because the image changed. The image on the server is built using the old Dockerfile -- wait for this one to upload, then the next changes should behave better. – Culottes

you are right. Now everything works. Thanks! – Ribal

Answer 2

docker push pushes all layers (5 at the time by default) of the image that are not equal to the image in the repository (aka the layers that did not change), not a single layer, in the end resulting in a new image in your repository.

You can see it as if Docker made a diff between the local and the remote image and pushed only the differences between those two, which will end up being a new image - equal to the one you have in your machine but with "less work" to reach the desired result since it doesn't need to push literally all the layers.

In your case it's taking a lot of time since the 4 Gb layer changed (since the content of what you are copying is different now), making Docker push a big part of the size of your image.

Link for the docker push documentation, if needed: https://docs.docker.com/engine/reference/commandline/push/

Deracinate

#39830033 This guy says otherwise – Ribal

Ok, I should have been more specific, my bad, I will update the answer. My "all images" was not very accurate. Thank you for pointing it out. – Deracinate