Create custom Neo4j Docker image with intial data from cypher file

Question

I tried create an docker image of neo4j that already provide some data, when you start an container. For my approach I inherited from the neo4j docker image, added some data via the neo4j cypher shell. But when i build the image and run a container from it the data did not appear in the database but the custom password is set. This is my current dockerfile:

From neo4j:3.4
ENV NEO4J_AUTH=neo4j/password
COPY data.cypher /var/lib/neo4j/import/
USER neo4j
RUN bin/neo4j-admin set-initial-password password || true && \
    bin/neo4j start && sleep 5 && \
    cat /var/lib/neo4j/import/data.cypher | NEO4J_USERNAME=neo4j NEO4J_PASSWORD=password /var/lib/neo4j/bin/cypher-shell --fail-fast
CMD [neo4j]

I added also an match query to the data.cypher file to make sure that the shell added the data to neo4j. Maybe it has something to do that /data is defined as volume in the neo4j image?

Fenestration

Answer 1

I was using Bierbarbar's approach. I got it working after getting over the following two pitfalls:

Firstly, $NEO4J_HOME/data was symlinked to /data, which seem to have permission issues. Changing the default data folder: by adding dbms.directories.data=mydata line to $NEO4J_HOME/conf/neo4j.conf fixed this.

Secondly, make sure data.cypher file contains correct format for cypher-shell: 1) Semicolon is needed at the end of each cypher statemnt; 2) there are :begin and :commit commands in some versions (or all versions?) of cypher-shell

Benedictus

Answer 2

In your Dockerfile, these commands happen at Docker container build time. At that time, there is no neo4j docker container, and so calls to cypher-shell and data import have no chance to succeed. So this overall approach just won't work.

What I would recommend is you build your database locally with something like Neo4j Desktop, then load the data into that. Then take a copy of your graph.db folder that resulted from the loading. And attach that to a docker container, no new Dockerfile needed.

Additionally, there are configuration options to set the default password from the outside, so you also don't need the neo4j-admin part.

Rostand

Sorry, but your answer is not correct. At first when you build an image an intermediate container is created. In this intermediate container neo4j is started and this is the reason why the import succeed in this case. The Problem is that i didn't found the right why to persist the data file. At Second: I know that there are options to set the default password from outside, but i added this to make clear that parts of the code that is running on build time is written into an data file and commited to the image. So why is the other data not kept? – Fenestration

I really don't recommend doing this in the way that you're doing, but probably the reason that the file isn't copying as you expect is that as you say, you're creating a layered image. You should do something like FROM neo4:3.4 AS build your commands, and then COPY --from=build /some/data /some/other/path – Rostand

This gives me an error. Maybe the operation is not possible? invalid from flag value build: pull access denied for build, repository does not exist or may require 'docker login' – Fenestration

Apologies but you haven't posted an updated dockerfile and this is becoming a separate question. In my answer above, you have my best recommendation on how to proceed and accomplish what you're trying to do. – Rostand

With this method I fear that when image is rebuilt from Dockerfile (let's say FROM neo4j:latest), there might be a version mismatch between graph.db file and neo4j. This problem wouldn't exist, if I maintain the cypher query instead of the graph.db file. Wouldn't you agree? – Benedictus

@SidaZhou if you keep your neo4j within a minor version (i.e. 3.4.* or 3.5.*) then this issue should not arise. If you move your neo4j across minor versions, then you need to set allow_store_upgrade=true and still it will work. If you have the option to maintain the whole thing as cypher that is nice, but not feasible if it's quite large – Rostand

Answer 3

Another approach is to copy the inital base data into a new to build neo4j docker image.

1. Start your neo4j container

2. Load this with your base data, using cypher, csv, a seeder or whatever

3. Stop your database (not the container) e.g. cypher: stop DATABASE neo4j

4. Copy the data from the docker container to your local storage: e.g. docker cp <containerid>:/var/lib/neo4j/data /path/to/local/storage. This should contain the directories /databases and /transactions

5. Create a new neo4j docker image with a dockerfile:

   FROM neo4j:latest

   COPY /path/to/local/storage /var/lib/neo4j/data

6. Build the image with docker build -t neo4j-loaded .

7. Run your new image with initial data loaded, docker run -d --name my-neo4j-container -p 7474:7474 -p 7687:7687 neo4j-loaded the actual database within the container may be offline. Use cypher in the browser view to start the database: start DATABASE neo4j or use show DATABASES to see troubleshooting info.

Swanskin