Setting up MySQL and importing dump within Dockerfile
Asked Answered
S

9

151

I'm trying to setup a Dockerfile for my LAMP project, but i'm having a few problems when starting MySQL. I have the folowing lines on my Dockerfile:

VOLUME ["/etc/mysql", "/var/lib/mysql"]
ADD dump.sql /tmp/dump.sql
RUN /usr/bin/mysqld_safe & sleep 5s
RUN mysql -u root -e "CREATE DATABASE mydb"
RUN mysql -u root mydb < /tmp/dump.sql

But I keep getting this error:

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111)

Any ideas on how to setup database creation and dump import during a Dockerfile build?

Sickly answered 18/9, 2014 at 18:37 Comment(6)
This is due to the fact that each RUN command is executed in a different container. It is well explained here: #17892169Bonney
This just explains that RUN commands have different contexts. But i'm depending on a daemon, not context.Sickly
Yes, but that explains why you cannot connect to MySQL. It is because, it is only running in your first RUN line.Bonney
To execute your SQL statements, your need to start MySQL, and use MySQL client in the same container: one RUN with several steps. You can find an example with a multi-step installation of a software here: https://mcmap.net/q/120938/-how-to-install-nvm-in-docker/…Bonney
You can also have a look on service with docker-compose: docs.docker.com/compose/wordpress/#build-the-project with that, mysql can be attached to your appEc
I know this is not what you asked for but you can also use docker cp to copy files over to a container which is mighty convenient: https://mcmap.net/q/40504/-how-to-copy-files-from-host-to-docker-containerPrismoid
B
137

Each RUN instruction in a Dockerfile is executed in a different layer (as explained in the documentation of RUN).

In your Dockerfile, you have three RUN instructions. The problem is that MySQL server is only started in the first. In the others, no MySQL are running, that is why you get your connection error with mysql client.

To solve this problem you have 2 solutions.

Solution 1: use a one-line RUN

RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \
  sleep 5 && \
  mysql -u root -e "CREATE DATABASE mydb" && \
  mysql -u root mydb < /tmp/dump.sql

Solution 2: use a script

Create an executable script init_db.sh:

#!/bin/bash
/usr/bin/mysqld_safe --skip-grant-tables &
sleep 5
mysql -u root -e "CREATE DATABASE mydb"
mysql -u root mydb < /tmp/dump.sql

Add these lines to your Dockerfile:

ADD init_db.sh /tmp/init_db.sh
RUN /tmp/init_db.sh
Bonney answered 18/9, 2014 at 19:28 Comment(14)
Do you have any idea how i can persist that information? The databases and tables i'm creating on the Dockerfile build are not available when i run stuff on containers. :(Sickly
That is weird that you cannot see your databases and tables in your containers. Try to build with docker build -t mysql ., then docker run -t -i mysql /bin/bash. Here, you should be able to see everything with mysql client.Bonney
That's exactly the problem. Not even the databases exist anymore.Sickly
I get it. That is due to VOLUME ["/etc/mysql", "/var/lib/mysql"]. These folders are marked as Docker volumes. So they are mounted when you docker run your final container. Either you remove this line, or your add -v option to your docker run command. You can find further documentation here: docs.docker.com/userguide/dockervolumes/#volume-defBonney
Can i persist data without a volume?Sickly
It depends of what you mean by persist. If you want to have the data persisted across several docker run executions, you need to mount volumes. If you just want to have a container with your dump, but you don't want to persist further modifications, you can get rid of the VOLUME instruction in your Dockerfile.Bonney
Should be nice to include the solution given by @Megathere that indicates the volume where to store sql or sh files that are launched at startup (to import data, for example). That avoid the need to build our own image. Note that this init dir is also provided with Postgres docker image, and a pull-request is in progress to make the same with Mongo docker image.Karajan
I get ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying Solution 1. Following the logs mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql and mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid endedLandlordism
I find it's much better to have a script check on startup to see if mysql has created a database. If it has, then leave it alone, otherwise run mysql_init_db and load in your database structure. This allows the flexibility of having the database reset automatically when testing, but if you want it to persist or test different data sets, you just mount a volume at /var/lib/mysql using docker run -v ....Cysticercoid
@Vituel Re: ERROR 2002 -- don't use localhost when connecting to your mysql instance, it's going to try to connect through the socket file. If you have another host pointed to your machine that should be fine, or use the ip as your host. Perhaps: -h 127.0.0.1Triangle
@Triangle - That doesn't work either. Now it gives error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)Quaternity
This does not work for me. It got /bin/sh: /bin/bash: not foundLeger
The sleep 5 part looks a bit hackish to me as it bases correct execution on an assumption. I consider the other approaches around here with /docker-entrypoint-initdb.d/ to be much more reliable as that's clearly the intention of the msyql image providers. If you also want to use bash scripts besides sql scripts please see my answer: https://mcmap.net/q/119098/-setting-up-mysql-and-importing-dump-within-dockerfileRunoff
i trued same was as RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \ sleep 5 && \ mysql -u root -e "CREATE DATABASE mydb" && \ mysql -u root mydb < /tmp/dump.sql but when i push this image to AWS ECR and pull it from docker-compose of another project, the db created is not present.Alberta
M
188

The latest version of the official mysql docker image allows you to import data on startup. Here is my docker-compose.yml

data:
  build: docker/data/.
mysql:
  image: mysql
  ports:
    - "3307:3306"
  environment:
    MYSQL_ROOT_PASSWORD: 1234
  volumes:
    - ./docker/data:/docker-entrypoint-initdb.d
  volumes_from:
    - data

Here, I have my data-dump.sql under docker/data which is relative to the folder the docker-compose is running from. I am mounting that sql file into this directory /docker-entrypoint-initdb.d on the container.

If you are interested to see how this works, have a look at their docker-entrypoint.sh in GitHub. They have added this block to allow importing data

    echo
    for f in /docker-entrypoint-initdb.d/*; do
        case "$f" in
            *.sh)  echo "$0: running $f"; . "$f" ;;
            *.sql) echo "$0: running $f"; "${mysql[@]}" < "$f" && echo ;;
            *)     echo "$0: ignoring $f" ;;
        esac
        echo
    done

An additional note, if you want the data to be persisted even after the mysql container is stopped and removed, you need to have a separate data container as you see in the docker-compose.yml. The contents of the data container Dockerfile are very simple.

FROM n3ziniuka5/ubuntu-oracle-jdk:14.04-JDK8

VOLUME /var/lib/mysql

CMD ["true"]

The data container doesn't even have to be in start state for persistence.

Megathere answered 28/10, 2015 at 17:3 Comment(8)
This is the better answer IMHO. It allows to NOT create an image ourself. Thanks for this tip.Karajan
One minor change for volumes: - ./docker/data:/docker-entrypoint-initdb.d —removed the . in front of container directory.Alexis
Adding note that compressed archives, ie. sql.gz appears to be supported now too: *.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${mysql[@]}"; echo ;;Fermanagh
While this is the correct procedure it doesn't address the use case very well. One of the advantages of Docker is that you can spin up an environment very quickly. If you have to wait for 3-4 minutes while Docker imports a MySQL DB during start up, you lose this advantage. The objective is to have a container that already contains the data in the database, so that you can use it as quickly as possible.Cherimoya
is this answer still working with the latest versions? does not seem to work anymoreLinlithgow
Why ubuntu-oracle-jdk for the data volume? Why not oraclelinux:7 as for the mysql image?Gymnastics
Note that the docker-compose example here is v2 not v3. "volumes_from:" is not supported in v3.Shep
This seems to be supported by the MariaDB image(s) as well: hub.docker.com/_/mariadbCrossfade
B
137

Each RUN instruction in a Dockerfile is executed in a different layer (as explained in the documentation of RUN).

In your Dockerfile, you have three RUN instructions. The problem is that MySQL server is only started in the first. In the others, no MySQL are running, that is why you get your connection error with mysql client.

To solve this problem you have 2 solutions.

Solution 1: use a one-line RUN

RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \
  sleep 5 && \
  mysql -u root -e "CREATE DATABASE mydb" && \
  mysql -u root mydb < /tmp/dump.sql

Solution 2: use a script

Create an executable script init_db.sh:

#!/bin/bash
/usr/bin/mysqld_safe --skip-grant-tables &
sleep 5
mysql -u root -e "CREATE DATABASE mydb"
mysql -u root mydb < /tmp/dump.sql

Add these lines to your Dockerfile:

ADD init_db.sh /tmp/init_db.sh
RUN /tmp/init_db.sh
Bonney answered 18/9, 2014 at 19:28 Comment(14)
Do you have any idea how i can persist that information? The databases and tables i'm creating on the Dockerfile build are not available when i run stuff on containers. :(Sickly
That is weird that you cannot see your databases and tables in your containers. Try to build with docker build -t mysql ., then docker run -t -i mysql /bin/bash. Here, you should be able to see everything with mysql client.Bonney
That's exactly the problem. Not even the databases exist anymore.Sickly
I get it. That is due to VOLUME ["/etc/mysql", "/var/lib/mysql"]. These folders are marked as Docker volumes. So they are mounted when you docker run your final container. Either you remove this line, or your add -v option to your docker run command. You can find further documentation here: docs.docker.com/userguide/dockervolumes/#volume-defBonney
Can i persist data without a volume?Sickly
It depends of what you mean by persist. If you want to have the data persisted across several docker run executions, you need to mount volumes. If you just want to have a container with your dump, but you don't want to persist further modifications, you can get rid of the VOLUME instruction in your Dockerfile.Bonney
Should be nice to include the solution given by @Megathere that indicates the volume where to store sql or sh files that are launched at startup (to import data, for example). That avoid the need to build our own image. Note that this init dir is also provided with Postgres docker image, and a pull-request is in progress to make the same with Mongo docker image.Karajan
I get ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) when trying Solution 1. Following the logs mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql and mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid endedLandlordism
I find it's much better to have a script check on startup to see if mysql has created a database. If it has, then leave it alone, otherwise run mysql_init_db and load in your database structure. This allows the flexibility of having the database reset automatically when testing, but if you want it to persist or test different data sets, you just mount a volume at /var/lib/mysql using docker run -v ....Cysticercoid
@Vituel Re: ERROR 2002 -- don't use localhost when connecting to your mysql instance, it's going to try to connect through the socket file. If you have another host pointed to your machine that should be fine, or use the ip as your host. Perhaps: -h 127.0.0.1Triangle
@Triangle - That doesn't work either. Now it gives error ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (111)Quaternity
This does not work for me. It got /bin/sh: /bin/bash: not foundLeger
The sleep 5 part looks a bit hackish to me as it bases correct execution on an assumption. I consider the other approaches around here with /docker-entrypoint-initdb.d/ to be much more reliable as that's clearly the intention of the msyql image providers. If you also want to use bash scripts besides sql scripts please see my answer: https://mcmap.net/q/119098/-setting-up-mysql-and-importing-dump-within-dockerfileRunoff
i trued same was as RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \ sleep 5 && \ mysql -u root -e "CREATE DATABASE mydb" && \ mysql -u root mydb < /tmp/dump.sql but when i push this image to AWS ECR and pull it from docker-compose of another project, the db created is not present.Alberta
P
42

What I did was download my sql dump in a "db-dump" folder, and mounted it:

mysql:
 image: mysql:5.6
 environment:
   MYSQL_ROOT_PASSWORD: pass
 ports:
   - 3306:3306
 volumes:
   - ./db-dump:/docker-entrypoint-initdb.d

When I run docker-compose up for the first time, the dump is restored in the db.

Primeval answered 17/11, 2015 at 13:25 Comment(3)
+1, with one addition: the link to the script being run for better understanding what is actually happening: github.com/docker-library/mariadb/blob/…Pipeline
latest mysql seems doesn't load the dumped sql file, and even using 5.6 I have problem with InnoDb storage engines.Bennet
same in here, it has such instruction and i even seen in logging that's loading init files, but db is empty!Qktp
G
16

Here is a working version using v3 of docker-compose.yml. The key is the volumes directive:

mysql:
  image: mysql:5.6
  ports:
    - "3306:3306"
  environment:
    MYSQL_ROOT_PASSWORD: root
    MYSQL_USER: theusername
    MYSQL_PASSWORD: thepw
    MYSQL_DATABASE: mydb
  volumes:
    - ./data:/docker-entrypoint-initdb.d

In the directory that I have my docker-compose.yml I have a data dir that contains .sql dump files. This is nice because you can have a .sql dump file per table.

I simply run docker-compose up and I'm good to go. Data automatically persists between stops. If you want remove the data and "suck in" new .sql files run docker-compose down then docker-compose up.

If anyone knows how to get the mysql docker to re-process files in /docker-entrypoint-initdb.d without removing the volume, please leave a comment and I will update this answer.

Gauntry answered 29/8, 2018 at 19:30 Comment(1)
this answer helped me. the note on docker-compose down was very important as I was seeing no changes despite restarting docker-compose. MYSQL_DATABASE is a required variable in this caseCentimeter
Z
13

I used docker-entrypoint-initdb.d approach (Thanks to @Kuhess) But in my case I want to create my DB based on some parameters I defined in .env file so I did these

1) First I define .env file something like this in my docker root project directory

MYSQL_DATABASE=my_db_name
MYSQL_USER=user_test
MYSQL_PASSWORD=test
MYSQL_ROOT_PASSWORD=test
MYSQL_PORT=3306

2) Then I define my docker-compose.yml file. So I used the args directive to define my environment variables and I set them from .env file

version: '2'
services:
### MySQL Container
    mysql:
        build:
            context: ./mysql
            args:
                - MYSQL_DATABASE=${MYSQL_DATABASE}
                - MYSQL_USER=${MYSQL_USER}
                - MYSQL_PASSWORD=${MYSQL_PASSWORD}
                - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
        ports:
            - "${MYSQL_PORT}:3306"

3) Then I define a mysql folder that includes a Dockerfile. So the Dockerfile is this

FROM mysql:5.7
RUN chown -R mysql:root /var/lib/mysql/

ARG MYSQL_DATABASE
ARG MYSQL_USER
ARG MYSQL_PASSWORD
ARG MYSQL_ROOT_PASSWORD

ENV MYSQL_DATABASE=$MYSQL_DATABASE
ENV MYSQL_USER=$MYSQL_USER
ENV MYSQL_PASSWORD=$MYSQL_PASSWORD
ENV MYSQL_ROOT_PASSWORD=$MYSQL_ROOT_PASSWORD

ADD data.sql /etc/mysql/data.sql
RUN sed -i 's/MYSQL_DATABASE/'$MYSQL_DATABASE'/g' /etc/mysql/data.sql
RUN cp /etc/mysql/data.sql /docker-entrypoint-initdb.d

EXPOSE 3306

4) Now I use mysqldump to dump my db and put the data.sql inside mysql folder

mysqldump -h <server name> -u<user> -p <db name> > data.sql

The file is just a normal sql dump file but I add 2 lines at the beginning so the file would look like this

--
-- Create a database using `MYSQL_DATABASE` placeholder
--
CREATE DATABASE IF NOT EXISTS `MYSQL_DATABASE`;
USE `MYSQL_DATABASE`;

-- Rest of queries
DROP TABLE IF EXISTS `x`;
CREATE TABLE `x` (..)
LOCK TABLES `x` WRITE;
INSERT INTO `x` VALUES ...;
...
...
...

So what happening is that I used "RUN sed -i 's/MYSQL_DATABASE/'$MYSQL_DATABASE'/g' /etc/mysql/data.sql" command to replace the MYSQL_DATABASE placeholder with the name of my DB that I have set it in .env file.

|- docker-compose.yml
|- .env
|- mysql
     |- Dockerfile
     |- data.sql

Now you are ready to build and run your container

Zara answered 4/4, 2017 at 0:33 Comment(2)
I like the approach of using a .env file. However according to this, the .env file feature only works when you use the docker-compose up command and does not work with docker stack deploy. Thus, in case you want to use the .env file in production, you might want to check out docker secrets that introduced with docker 17.06. Then you can use your .env file in conjuction with secrets in both development docker compose up and production docker stack deploy phasesCaravette
Nice approach but I would suggest to use RUN sed -i '1s/^/CREATE DATABASE IF NOT EXISTS $MYSQL_DATABASE;\nUSE $MYSQL_DATABASE;\n/' data.sql instead of the sed command you suggested. This way you can use any dump file you have - it's useful if you're working with big amount of data :)Stripy
R
1

edit: I had misunderstand the question here. My following answer explains how to run sql commands at container creation time, but not at image creation time as desired by OP.

I'm not quite fond of Kuhess's accepted answer as the sleep 5 seems a bit hackish to me as it assumes that the mysql db daemon has correctly loaded within this time frame. That's an assumption, no guarantee. Also if you use a provided mysql docker image, the image itself already takes care about starting up the server; I would not interfer with this with a custom /usr/bin/mysqld_safe.

I followed the other answers around here and copied bash and sql scripts into the folder /docker-entrypoint-initdb.d/ within the docker container as this is clearly the intended way by the mysql image provider. Everything in this folder is executed once the db daemon is ready, hence you should be able rely on it.

As an addition to the others - since no other answer explicitely mentions this: besides sql scripts you can also copy bash scripts into that folder which might give you more control.

This is what I had needed for example as I also needed to import a dump, but the dump alone was not sufficient as it did not provide which database it should import into. So in my case I have a script named db_custom_init.sh with this content:

mysql -u root -p$MYSQL_ROOT_PASSWORD -e 'create database my_database_to_import_into'
mysql -u root -p$MYSQL_ROOT_PASSWORD my_database_to_import_into < /home/db_dump.sql

and this Dockerfile copying that script:

FROM mysql/mysql-server:5.5.62
ENV MYSQL_ROOT_PASSWORD=XXXXX
COPY ./db_dump.sql /home/db_dump.sql
COPY ./db_custom_init.sh /docker-entrypoint-initdb.d/

Runoff answered 7/8, 2020 at 14:28 Comment(0)
D
1

Based on Kuhess response, but without hard sleep:

RUN /bin/bash -c "/usr/bin/mysqld_safe --skip-grant-tables &" && \
  while ! mysqladmin ping --silent; do sleep 1; echo "wait 1 second"; done && \
  mysql -u root -e "CREATE DATABASE mydb" && \
  mysql -u root mydb < /tmp/dump.sql
Deciliter answered 9/11, 2021 at 13:50 Comment(0)
D
0

any file or script added to /docker-entrypoint-initdb.d will executed at the starting of the container

make sure that you do not add or run any sql or sh file that can use mysql servies from the Dockerfile .they will fail and stop the image build becuase mysql servies did not start yet when this files or scripts called .the best way to add .sh file is to ADD them on /docker-entrypoint-initdb.d directory from your Dockerfile

working exmple

FROM mysql
ADD mysqlcode.sh /docker-entrypoint-initdb.d/mysqlcode.sh
ADD db.sql /home/db.sql
RUN chmod -R 775 /docker-entrypoint-initdb.d
ENV MYSQL_ROOT_PASSWORD mypassword

and the mysqlcode.sh will do some command when mysql service is active

mysqlcode.sh

#!/bin/bash
mysql -u root -pmypassword --execute "CREATE DATABASE IF NOT EXISTS mydatabase;"
mysql -u root -pmypassword mydatabase < /home/db.sql    
Discernment answered 26/6, 2022 at 5:15 Comment(0)
S
-1

I have experienced the same problem, but managed to get it working by separating the MySQL start-up commands:

sudo docker build -t MyDB_img -f Dockerfile.dev
sudo docker run --name SomeDB -e MYSQL_ROOT_PASSWORD="WhatEver" -p 3306:3306 -v $(pwd):/app -d MyDB_img

Then sleep for 20 seconds before running the MySQL scripts, it works.

sudo docker exec -it SomeDB sh -c yourscript.sh

I can only presume that the MySQL server takes a few seconds to start up before it can accept incoming connections and scripts.

Sickroom answered 9/11, 2021 at 9:25 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.