Database configuration isn't the most exciting topic. Developers dislike prolonged onboarding, preferring to dive straight into application building. Because Dolt marries the concepts of Git And MySQL, we maintain two sets of configurations. Unfortunately, double database configuration is twice as difficult for users.

As we make bolder strides supporting transactional applications, we think it is important to acknowledge a key piece of our infrastructure that has failed to keep pace with the rest of our development work. We will discuss how the Git and MySQL configuration stories led to Dolt's present configuration, and a recent feature that highlighted holes in our sql-server configuration.

Git and MySQL Configs

Git reads configuration settings hierarchically. This means that a local .git/config is is read before defaulting to a global ~/.gitconfig. More info can be found here.

MySQL separates the documentation of server settings from option files to Even though you can configure a MySQL server with an "option file," it is equally valid to run SET queries to initialize new servers.

Dolt has Git-like local/global configs and MySQL-like server option files. that behave similarly to their Git/MySQL equivalents. When Dolt was primarily a CLI tool, the local/global configs stored metadata like user.name and user.email. When we added server support, the server option file was used to configure a server's host, port, name, password, ...etc.

Bridging Configs

We recently added a server feature that writes SQL variables to Git-style configs.

The SET PERSIST syntax is responsible for persisting SQL variables. To understand variable persistence, first note that regular variable setting is transient between sessions:

$ dolt sql -q "SELECT @@GLOBAL.max_connections"
+--------------------------+
| @@GLOBAL.max_connections |
+--------------------------+
| 151                      |
+--------------------------+
$ dolt sql -q "SET @@GLOBAL.max_connections = 47"
$ dolt sql -q "SELECT @@GLOBAL.max_connections"
+--------------------------+
| @@GLOBAL.max_connections |
+--------------------------+
| 151                      |
+--------------------------+

If we spin up a server, we can set a global variable that outlives client sessions. But restarting the server again resets system variables. Only the PERSIST prefix saves variables durably:

$ dolt sql -q "SET @@PERSIST .max_connections = 47;"
$ dolt sql -q "SELECT @@GLOBAL.max_connections"
+--------------------------+
| @@GLOBAL.max_connections |
+--------------------------+
| 47                       |
+--------------------------+

This is cool because servers can save and restore configuration between restarts. If you are interested in learning more about system variables, there is more info here.

The Gaps

Persistent configs are cool, but there are a lot of reasons why we can do better.

Server option files are currently too weak to support persistence, so the implementation bridges CLI and SQL configs:

dolt config --local --add sqlserver.global.max_connections 47
Config successfully updated.
$ dolt sql -q "SELECT @@GLOBAL.max_connections"
+--------------------------+
| @@GLOBAL.max_connections |
+--------------------------+
| 47                       |
+--------------------------+

The result is that settings might be split between three files: a global config, local config, and server option file. In an umbrella sql-server with three databases, we might even have five configs: a server option file, global config, and three local configs (one for each database).

The server option file should be able to do more, at the very least persist variables. Ideally, Dolt sql-server users would have all of their needs addressed by one single server config. In the future, we will ship an improved sql-server option file to improve this experience.

Summary

We discussed how Dolt evolved two configuration patterns in the image of Git and MySQL. The server config has failed to keep pace with the rate SQL engine improvements. We introduced a change for SET PERSIST syntax recently that highlighted the server option file's shortcomings.

We shipped the desired MySQL support, but we can do better! Developers want to write applications, not get stuck fighting configuration files.

If you are interested in learning more about Dolt reach out to us on Discord!