Can't keep postgres data persistent using Github CodeSpaces with Docker-Compose

Question

I set up a Github codespaces environment using devcontainer.json and docker-compose.yaml. Everything works fine, but the postgres database defined in docker-compose.yml loses its data every time the container needs to be re-built.

Here's the bottom part of the docker-compose.yml

      db:
        image: postgres:latest
        restart: unless-stopped
        volumes:
          - postgres-data:/var/lib/postgresql/data
        environment:
          POSTGRES_USER: test_user
          POSTGRES_DB: test_db
          POSTGRES_PASSWORD: test_pass
       
   volumes:
     postgres-data:

as you can see, I am trying to map the postgres data volume into a postgres-data volume, but this doesn't work for some reason.

What am I doing wrong that's preventing postgres data from persisting between container builds?

Isar

Maybe I have misunderstood but can't you solve by creating the volume outside the docker-compose and set external: true? – Runabout

This is Github codespaces -- I don't have access to the "external". It's their own VM and my starting point is the docker-compose. I can't get to the machine running the docker-compose.... – Isar

Right, sorry, I have never used Codespaces, but I did some researches and maybe this can help you. – Runabout

What's the purpose of the data? If you need a predictable data set on each rebuild I'd recommend to create an own managed postgres image with redefined data. – Retractile

From what I know Codespace does not give you any warranty on persistence. It is not meant for that. – Einberger

@Isar facing the exact same issue when uising VS code with github codespace. How dod you end up solving this isssue to be able to work with your local dev database? – Thomson

Answer 1

according to https://docs.github.com/en/codespaces/customizing-your-codespace/configuring-codespaces-for-your-project#dockerfile , only docker images can be pulled from source and set-up, nowhere they mention that volume persistence is guaranteed.

and after going through this https://code.visualstudio.com/docs/remote/devcontainerjson-reference looks like mounts and few other features related to volumes are not supported for codespaces.

workspaceMount : Not yet supported in Codespaces or when using Clone Repository in Container Volume.

workaround :

in .devcontainer folder where your dockerfile is present add a line like this

RUN curl https://<your_public_cloud>/your_volume.vol -O

here <your_public_cloud> can be google drive, aws or any endpoint where you have access to download the volume. its also the volume you needed to be persist.

• and once its downloaded you can mount the volume to postgres service or make a hotswap.

• and when you want to save, just upload the volume to your cloud storage provider.

• repeat the process every time you build, and save and upload before "unbuild" or dismissing your codespace whatever you like to call.

hope that eases your issue, happy coding!

Kahle

Answer 2

Another option would be to look into using Spawn. (Disclaimer - I'm one of the devs working on it).

We've written some documentation about exactly how to use Spawn-hosted databases with GitHub codespaces here: https://docs.spawn.cc/blog/2021/08/01/spawn-and-codespaces

This will allow you to provision a database thats independent from the GitHub codespace and preserve data between restarts.

You get some extra features with Spawn like arbitrary save points, resets and loading back to saved revisions with Spawn - but the key functionality of spinning up a database for a GitHub codespace and preserving data is one of the things it works extremely well for.

Malines

Answer 3

As long as you don't remove the volume with docker-compose down --volumes as an example, the data should persist.

I had the same issue; and it turned out that I had put a crontab running docker system prune -af every 15 minutes!

Directed

This is Github codespaces -- It's their own VM and my starting point is the docker-compose. I can't get to the machine running the docker-compose.... and I cannot run docker-compose myself in that environment. – Isar

@Isar I think this might be a docker or compose issue, I noticed that docker-compose is creating new volumes instead of using the named ones for a new project but it's not happening for an older project. I have to investigate this further! – Directed

Answer 4

As you don't have access to VM, maybe the directory containing your docker-compose.yml changes.

In that case, volume name may change too.

Indeed, by default, your volume name would be the following :

<directory_name>_postgres-data

Could you try a named volume (starting with compose 3.4):

  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: test_user
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: test_pass
       
volumes:
  postgres-data:
    external: false
    name: postgres-data

documentation of docker-compose can be found here :

https://docs.docker.com/compose/compose-file/compose-file-v3/#name

EDIT 1

If your VM is created at each build, docker dependencies too.

volumes, networks, etc...

A persistent volume is needed somewhere (surviving VM builds).

You may have to create a directory in your local workspace, like:

/local/workspace/postgres-data/

which become in codespaces according to my understanding :

./postgres-data

Check permissions, your user may not exist in the container.

As a result your compose file become:

  db:
    image: postgres:latest
    restart: unless-stopped
    volumes:
      - ./postgres-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: test_user
      POSTGRES_DB: test_db
      POSTGRES_PASSWORD: test_pass

Emulsoid

tried that. did not work. once the Github CodSpaces VM built from scratch and I re-enter the machine, all psql data is erased. – Isar

Could you try by following this article tips blog.3-4.fr/2020/09/02/docker-volumes-in-codespace you volume is local to the docker host which is created at each build. A directory mount from your development environment into the docker container should be reliable. – Emulsoid

Answer 5

You could just mount a host directory, instead of using a docker volume:

volumes:
  - /home/me/postgres_data:/var/lib/postgresql/data

This guarantees that no volume cleanup (accidental or deliberate) nukes your database.

Indeed the postgres docs do this in their examples. See the PGDATA environment variable.

Flowers

when Github codespaces VMs are re-built -- everything gets deleted, including the home directories. – Isar