Getting Started: Dolt with Docker
For a long time, Dolt resisted Docker. Here at DoltHub, we take a lot of pride in the fact that Dolt is a single program with no external dependencies. To use Dolt, you simply download the current build for your operating system, put the directory you stored the program in your PATH, and type dolt. For something that simple, Docker seemed like overkill.
But people just kept asking for a Docker image. So, eventually we caved and put two Dolt images on DockerHub: one for the Dolt CLI and the more popular SQL Server image, modeled after the MySQL image.
We still get questions about how to use Dolt with Docker even though our documentation is pretty good. This blog will start from the basics and work up to answering some of the common questions we get about using Dolt with Docker.
Install Docker
To use Docker, you must first install Docker. On my Mac, I went to the Docker Getting Started page and clicked on "Download for Mac - Apple Chip". The correct OS was selected by default. That started a download for Docker.dmg. Once the download was complete, I was prompted to move Docker into my Applications directory.
I then ran the application and was greeted with this application home screen.
That's it. You are good to go. Docker will start in the background whenever you restart your computer.
Install MySQL
Dolt is a standalone database. You connect to it with any MySQL client. For this example, let's grab a copy of MySQL so we can connect with that client. Head over to the MySQL Getting Started documentation and install MySQL on your machine. I used Homebrew to install the MariaDB flavored MySQL on my Mac.
MySQL comes with a MySQL server called mysqld and a MySQL client called mysql. You're only interested in the client. After following the instructions from MySQL's documentation, make sure you have a copy of the mysql client on your path:
$ mysql --version
mysql from 11.1.2-MariaDB, client 15.2 for osx10.18 (arm64) using EditLine wrapperGet the Dolt Docker Image
You have two modes of using Docker: through the above graphical user interface or through the command line. I am going to switch to the command line, mostly because I don't want a bunch of screen shots in this blog. So, open a terminal.
DoltHub publishes two repositories on dockerhub, updated on every Dolt release. The two repositories are a simple Dolt CLI repository with no bells and whistles and a Dolt SQL Server repository modeled after the MySQL Docker image. This article will focus on the more popular Dolt SQL Server repository.
If you're a Docker purist, use the Dolt CLI repository but it's overkill. We recommend you just install the Dolt binary. It's a single program.
To get a repository from dockerhub, you run docker pull.
$ docker pull dolthub/dolt-sql-server
Using default tag: latest
latest: Pulling from dolthub/dolt-sql-server
Digest: sha256:953f11230b841d6d38fdf8071ceb54599ba7d1565f905dc3a24e1c139092aaf8
Status: Image is up to date for dolthub/dolt-sql-server:latest
docker.io/dolthub/dolt-sql-server:latest
What's Next?
View a summary of image vulnerabilities and recommendations → docker scout quickview dolthub/dolt-sql-server
$Start a Server
My goal is to start a Dolt SQL Server in a docker container and connect to it with a MySQL client outside of Docker on my local machine.
Let's first start by starting the image with the default arguments:
$ docker run dolthub/dolt-sql-server
2023-10-10 20:51:27+00:00 [Note] [Entrypoint]: Entrypoint script for Dolt Server 1.18.1 starting.
2023-10-10 20:51:27+00:00 [Note] [Entrypoint]: Running init scripts
2023-10-10 20:51:27+00:00 [Warn] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
Starting server with Config HP="0.0.0.0:3306"|T="28800000"|R="false"|L="info"
2023-10-10T20:51:28Z INFO [no conn] Server ready. Accepting connections. {}
2023-10-10T20:51:28Z WARN [no conn] secure_file_priv is set to "", which is insecure. {}
2023-10-10T20:51:28Z WARN [no conn] Any user with GRANT FILE privileges will be able to read any file which the sql-server process can read. {}
2023-10-10T20:51:28Z WARN [no conn] Please consider restarting the server with secure_file_priv set to a safe (or non-existent) directory. {}Looks like it's working. If I head over to another terminal and try to connect on port 3306, I cannot connect:
$ mysql -h 0.0.0.0 -P 3306 -u root
ERROR 2002 (HY000): Can't connect to server on '0.0.0.0' (36)This is the most common Dolt with Docker mistake. The Dolt SQL Server is running in a container with port 3306 exposed but you must map 3306 to another port on your local machine in order to connect to the container.
You have two options:
- Go into the container to use Dolt
- Stop that container and start a new one with port 3306 mapped to a local port.
I'll show you both.
Use Dolt from a Shell in the Container
I need to connect to a running Docker container with a shell to use Dolt. To do this I use docker exec to start bash. First, I need the container id of my running container. To do that I use docker ps and note the `container id``.
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6f86aad96980 dolthub/dolt-sql-server "tini -- docker-entr…" 13 seconds ago Up 12 seconds 3306/tcp, 33060/tcp inspiring_kapitsaThen, I use docker exec with that container id to start bash. I need to pass the -i for "interactive" and -t for "tty" or terminal so docker exec will give me a running terminal.
$ docker exec -it 6f86aad96980 /bin/bashAnd, I'm in! Now, I have a bash terminal in my container. I make sure Dolt is there and I can run the dolt command line.
root@6f86aad96980:/var/lib/dolt# dolt version
dolt version 1.18.1
root@6f86aad96980:/var/lib/dolt# dolt sql -q "show databases"
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
+--------------------+Use a local MySQL client to connect to Dolt
Now, what if you want to connect from your local machine to your Docker container with a normal MySQL client? Currently, we aren't running your container correctly if we want to do this. We have no external port for MySQL to connect to. Stop the first container and run the following command to start a new container with local port 3307 mapped to port 3306 of the container. Remember, local port first, then docker port.
$ docker run -p 3307:3306 dolthub/dolt-sql-server
2023-10-10 20:56:17+00:00 [Note] [Entrypoint]: Entrypoint script for Dolt Server 1.18.1 starting.
2023-10-10 20:56:17+00:00 [Note] [Entrypoint]: Running init scripts
2023-10-10 20:56:17+00:00 [Warn] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
Starting server with Config HP="0.0.0.0:3306"|T="28800000"|R="false"|L="info"
2023-10-10T20:56:18Z INFO [no conn] Server ready. Accepting connections. {}
2023-10-10T20:56:18Z WARN [no conn] secure_file_priv is set to "", which is insecure. {}
2023-10-10T20:56:18Z WARN [no conn] Any user with GRANT FILE privileges will be able to read any file which the sql-server process can read. {}
2023-10-10T20:56:18Z WARN [no conn] Please consider restarting the server with secure_file_priv set to a safe (or non-existent) directory. {}Now, in your other shell you should be able to connect via port 3307.
$ mysql -h 0.0.0.0 -P 3307 -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.9-Vitess Dolt
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]>You are connected to your Dolt SQL Server running in Docker through a local MySQL client!
Make Some Tables
From here, I'm going to create a database and make a few tables following the example laid out in Getting Started: Version Controlled Database.
MySQL [(none)]> create database getting_started;
Query OK, 1 row affected (0.087 sec)
MySQL [(none)]> use getting_started;
Database changed
MySQL [getting_started]> create table employees (
id int,
last_name varchar(255),
first_name varchar(255),
primary key(id));
Query OK, 0 rows affected (0.069 sec)
MySQL [getting_started]> create table teams (
id int,
team_name varchar(255),
primary key(id));
Query OK, 0 rows affected (0.034 sec)
MySQL [getting_started]> create table employees_teams(
team_id int,
employee_id int,
primary key(team_id, employee_id),
foreign key (team_id) references teams(id),
foreign key (employee_id) references employees(id));
Query OK, 0 rows affected (0.063 sec)
MySQL [getting_started]> show tables;
+---------------------------+
| Tables_in_getting_started |
+---------------------------+
| employees |
| employees_teams |
| teams |
+---------------------------+
3 rows in set (0.013 sec)Use Version Control
Now it's time to use our first Dolt version control features. We're going to make a Dolt commit and examine it in the Dolt log. Dolt version control write operations (ie. add, commit, branch, merge) are exposed as procedures and Dolt version control read operations (ie. log, diff, status) are exposed as system tables.
MySQL [getting_started]> call dolt_add('employees', 'employees_teams', 'teams');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (0.046 sec)
MySQL [getting_started]> call dolt_commit('-m', 'Created tables');
+----------------------------------+
| hash |
+----------------------------------+
| a33td2539t02opn5jr07669s45mj9nhd |
+----------------------------------+
1 row in set (0.044 sec)
MySQL [getting_started]> select * from dolt_log;
+----------------------------------+---------------------+----------------------+---------------------+----------------------------+
| commit_hash | committer | email | date | message |
+----------------------------------+---------------------+----------------------+---------------------+----------------------------+
| a33td2539t02opn5jr07669s45mj9nhd | root | root@% | 2023-10-10 21:52:23 | Created tables |
| g176lbjg300nc7qaa5pirn9kn6qnua37 | Dolt System Account | doltuser@dolthub.com | 2023-10-10 21:43:31 | Initialize data repository |
+----------------------------------+---------------------+----------------------+---------------------+----------------------------+Dolt is a full featured SQL database combined with the power of Git-style version control. Feel free to continue on with Getting Started: Version Controlled Database to get the full tour of Dolt's SQL database and version control functionality. But this article is about Docker, so let's stick to containers.
Get a Database out of Docker
All the Docker experts at this point are shouting, "You aren't doing it right! Docker storage is ephemeral". They're right. When you start a Docker container the life of the disk is limited to the life of the container. So, when I started my Dolt SQL server container without mounting to a local disk, I'm basically saying "I'm fine with losing data if this container goes away". So, how do I get my data out?
The Dolt SQL Server Docker image stores database data in /var/lib/dolt/. We would like to get that data out of the Docker container onto our local machine into ~/docker_dolts. To do this we use the docker cp command.
First, we need to find the id of the container. To do this, we run docker container ls.
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
da12c699b644 dolthub/dolt-sql-server "tini -- docker-entr…" About an hour ago Up About an hour 33060/tcp, 0.0.0.0:3307->3306/tcp flamboyant_kilbyWe note the container id for the copy command and copy the data out of the container onto our local machine using docker cp.
$ docker cp da12c699b644:/var/lib/dolt/getting_started ~/docker_dolts
Successfully copied 1.06MB to /Users/timsehn/docker_dolts
$ ls -al ~/docker_dolts
total 0
drwxr-xr-x 3 timsehn staff 96 Oct 10 15:14 .
drwxr-x---+ 53 timsehn staff 1696 Oct 10 15:14 ..
drwxr-xr-x@ 3 timsehn staff 96 Oct 10 14:43 getting_started
$ cd ~/docker_dolts/getting_started
$ dolt ls
Tables in working set:
employees
employees_teams
teams
$ dolt log
commit a33td2539t02opn5jr07669s45mj9nhd (HEAD -> main)
Author: root <root@%>
Date: Tue Oct 10 14:52:23 -0700 2023
Created tables
commit g176lbjg300nc7qaa5pirn9kn6qnua37
Author: Dolt System Account <doltuser@dolthub.com>
Date: Tue Oct 10 14:43:31 -0700 2023
Initialize data repositoryAs you can see, the Dolt database is cleanly copied out of Docker and can be accessed by the local Dolt CLI.
Get a local Database into Docker
docker cp is not the best way to make sure your data is persisted after the container goes away. It is better to mount a local directory when you start your container.
To show this off I'll make another Dolt database called already_started to live alongside the getting_started database I copied out of Docker in the last section. I use the Dolt CLI to create it and add a simple test table.
$ pwd
/Users/timsehn/docker_dolts
$ mkdir already_started
$ cd already_started
$ dolt init --fun
Successfully initialized dolt data repository.
$ dolt sql -q "create table t (id int primary key, words varchar(100))"
$ dolt add .
$ dolt commit -am "Added test table"
commit n33i79rtir76oknc390b7rlq6h9nubnj (HEAD -> main)
Author: timsehn <tim@dolthub.com>
Date: Tue Oct 10 15:54:42 -0700 2023
Added test table
$Now, I'm going to start a container with the ~/docker_dolts directory mounted to the /var/lib/dolt directory where the Dolt SQL server image is configured to look for Dolt databases. To do this I use the -v option for "volume list". Just like port local directory first then Docker directory.
$ docker run -p 3307:3306 -v ~/docker_dolts:/var/lib/dolt dolthub/dolt-sql-server
2023-10-10 23:04:25+00:00 [Note] [Entrypoint]: Entrypoint script for Dolt Server 1.18.1 starting.
2023-10-10 23:04:25+00:00 [Note] [Entrypoint]: Running init scripts
2023-10-10 23:04:25+00:00 [Warn] [Entrypoint]: /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
Starting server with Config HP="0.0.0.0:3306"|T="28800000"|R="false"|L="info"
2023-10-10T23:04:25Z INFO [no conn] Server ready. Accepting connections. {}
2023-10-10T23:04:25Z WARN [no conn] secure_file_priv is set to "", which is insecure. {}
2023-10-10T23:04:25Z WARN [no conn] Any user with GRANT FILE privileges will be able to read any file which the sql-server process can read. {}
2023-10-10T23:04:25Z WARN [no conn] Please consider restarting the server with secure_file_priv set to a safe (or non-existent) directory. {}
2023-10-10T23:04:28Z INFO [conn 1] NewConnection {DisableClientMultiStatements=false}Now, in another shell I can see my mounted databases.
$ mysql -h 0.0.0.0 -P 3307 -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.7.9-Vitess Dolt
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| already_started |
| getting_started |
| information_schema |
| mysql |
+--------------------+
4 rows in set (0.018 sec)
MySQL [(none)]> use already_started;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MySQL [already_started]> show tables;
+---------------------------+
| Tables_in_already_started |
+---------------------------+
| t |
+---------------------------+
1 row in set (0.011 sec)This is the recommended way to use the Docker SQL Server container to preserve your databases.
Define and Persist Configuration
The Dolt SQL Server comes with two other directories you should be aware of. These are used to store two types of configuration:
- Server configuration like port and log level.
- Database configuration like privileges and branch permissions.
Server configuration is defined using YAML and read from /etc/dolt/servercfg.d in Docker. This directory can contain only one .yaml file, usually called config.yaml, for the server to use.
Server configuration is defined using SQL statements and is stored in /etc/dolt/doltcfg.d. There are different .db files for different purposes. For instance, .doltcfg/privileges.db stores users and grants while .doltcfg/branch_control.db stores branch permissions. Generally, you shouldn't worry about this directory other than making sure it is mounted somewhere locally so your database doesn't lose its permissions if the container is restarted.
For a simple example, let's change the port of our Dolt database to 5432 for you Postgres lovers. First let's make a docker_dolts_config directory and populate it with our desired config.yaml.
Note, the host must be set to 0.0.0.0 which the default Dolt Docker configuration knows. But Dolt itself defaults to the host being localhost so if you don't define the host, you get a ERROR 2013 (HY000): Lost connection to server at 'handshake: reading initial communication packet', system error: 35 when you try to connect because Docker doesn't expose a localhost.
$ mkdir ~/docker_dolts_config/
$ cat <<EOF > ~/docker_dolts_config/config.yaml
listener:
host: 0.0.0.0
port: 5432
EOF
$ cat ~/docker_dolts_config/config.yaml
listener:
host: 0.0.0.0
port: 5432Now, just like above, we tell Docker to mount that directory to /etc/dolt/servercfg.d.
$ docker run -p 3307:5432 -v ~/docker_dolts:/var/lib/dolt -v ~/docker_dolts_config:/etc/dolt/servercfg.d dolthub/dolt-sql-server
2023-10-12 20:52:30+00:00 [Note] [Entrypoint]: Entrypoint script for Dolt Server 1.18.1 starting.
2023-10-12 20:52:30+00:00 [Note] [Entrypoint]: Checking for config provided in /etc/dolt/servercfg.d
2023-10-12 20:52:30+00:00 [Note] [Entrypoint]: /etc/dolt/servercfg.d/config.yaml file is found
Starting server with Config HP="0.0.0.0:5432"|T="28800000"|R="false"|L="info"
2023-10-12T20:52:31Z INFO [no conn] Server ready. Accepting connections. {}
2023-10-12T20:52:31Z WARN [no conn] secure_file_priv is set to "", which is insecure. {}
2023-10-12T20:52:31Z WARN [no conn] Any user with GRANT FILE privileges will be able to read any file which the sql-server process can read. {}
2023-10-12T20:52:31Z WARN [no conn] Please consider restarting the server with secure_file_priv set to a safe (or non-existent) directory. {}
2023-10-12T20:52:40Z INFO [conn 1] NewConnection {DisableClientMultiStatements=false}
2023-10-12T20:52:56Z INFO [conn 1] ConnectionClosed {}As you can see, the server started on port 5432 and I should still be able to connect on port 3307.
$ mysql -h 0.0.0.0 -P 3307 -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.9-Vitess Dolt
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| already_started |
| getting_started |
| information_schema |
| mysql |
+--------------------+
4 rows in set (0.022 sec)
MySQL [(none)]>We are now running on port 5432.
Clone a Database from DoltHub
Another model to handle ephemeral Docker storage would be to use DoltHub or DoltLab as permanent storage instead of a mounted local volume. For public databases, cloning a database locally is as simple as calling the dolt_clone() procedure from a running Dolt SQL Server.
MySQL [(none)]> call dolt_clone('timsehn/almost_finished');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (1.376 sec)
MySQL [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| almost_finished |
| already_started |
| getting_started |
| information_schema |
| mysql |
+--------------------+
5 rows in set (0.017 sec)I have the almost_finished database from DoltHub.
Push a Database to DoltHub
This gets a little hairy when you need to write the database back to DoltHub or DoltLab because that requires the credentials created by dolt login. So, run dolt login locally on your machine if you have not yet. Here is the guide.
After doing this you should see your Dolt credentials in ~/.dolt/creds. This directory needs to be mounted in order for the Docker container to use them. The directory you want to mount to on the Docker container is /root/.dolt. This is because /root is $HOME on ubuntu for the root user and Dolt looks in $HOME/.dolt for credentials. I restart the container with the new mount.
$ docker run -p 3307:5432 -v ~/docker_dolts:/var/lib/dolt -v ~/docker_dolts_config:/etc/dolt/servercfg.d -v ~/.dolt:/root/.dolt dolthub/dolt-sql-server
2023-10-12 22:15:05+00:00 [Note] [Entrypoint]: Entrypoint script for Dolt Server 1.18.1 starting.
2023-10-12 22:15:05+00:00 [Note] [Entrypoint]: Checking for config provided in /etc/dolt/servercfg.d
2023-10-12 22:15:05+00:00 [Note] [Entrypoint]: /etc/dolt/servercfg.d/config.yaml file is found
Starting server with Config HP="0.0.0.0:5432"|T="28800000"|R="false"|L="info"
2023-10-12T22:15:06Z INFO [no conn] Server ready. Accepting connections. {}
2023-10-12T22:15:06Z WARN [no conn] secure_file_priv is set to "", which is insecure. {}
2023-10-12T22:15:06Z WARN [no conn] Any user with GRANT FILE privileges will be able to read any file which the sql-server process can read. {}
2023-10-12T22:15:06Z WARN [no conn] Please consider restarting the server with secure_file_priv set to a safe (or non-existent) directory. {}Now, I make a change and push it to DoltHub.
$ mysql -h 0.0.0.0 -P 3307 -u root
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.9-Vitess Dolt
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MySQL [(none)]> use almost_finished;
Database changed
MySQL [almost_finished]> insert into t values (1, "I have permissions");
Query OK, 1 row affected (0.027 sec)
MySQL [almost_finished]> call dolt_add('t');
+--------+
| status |
+--------+
| 0 |
+--------+
1 row in set (0.030 sec)
MySQL [almost_finished]> call dolt_commit('-m', 'I committed this in Docker and now I\'m going to push');
+----------------------------------+
| hash |
+----------------------------------+
| 5nm4c1jm0hta7ff3og10ttfjbjbtp5a1 |
+----------------------------------+
1 row in set (0.012 sec)
MySQL [almost_finished]> call dolt_push();
+--------+---------+
| status | message |
+--------+---------+
| 0 | |
+--------+---------+
1 row in set (2.959 sec)And I can confirm that the database was pushed to DoltHub by looking at the Commit log.
Conclusion
There you have it, a guide to using Dolt with Docker. This blog should help you work through any issues you have. Still stuck, come talk to us on our Discord. We're here to help.