Advanced config.yaml
Last week, I wrote an article going over the basic configuration options in Dolt's main SQL Server configuration file, config.yaml. That article was already really long, clocking in at a 23 minute read according to Gatsby, our blog publishing platform. There is a whole section of config.yaml I labeled "Advanced Configuration" that I didn't cover. This article covers those configuration values.
Overview
Here is a complete config.yaml file populated with all the default values for every key. We'll dive into each key after the "Advanced Configuration" comment individually. If your looking for documentation on the other basic configuration settings, this article covers those.
log_level: info
behavior:
read_only: false
autocommit: true
disable_client_multi_statements: false
dolt_transaction_commit: false
event_scheduler: "ON"
user:
name: ""
password: ""
listener:
host: localhost
port: 3306
max_connections: 100
read_timeout_millis: 28800000
write_timeout_millis: 28800000
tls_key: null
tls_cert: null
require_secure_transport: null
allow_cleartext_passwords: null
max_logged_query_len: 0
data_dir: .
cfg_dir: .doltcfg
privilege_file: .doltcfg/privileges.db
branch_control_file: .doltcfg/branch_control.db
# Advanced Configuration
metrics:
host: null
port: -1
labels: {}
remotesapi:
port: null
read_only: null
system_variables: {}
user_session_vars: []
jwks: []
# Cluster configuration has required defaults.
# cluster: {}metrics
This set of configuration values configures a Dolt metrics HTTP endpoint. Dolt emits metrics in Prometheus format.
host
The host defines the host Dolt will use to serve the metrics endpoint.
Default: null
Values: localhost or an IPv4 or IPv6 address
port
The port defines the port Dolt will use to expose the metrics endpoint.
Default: -1
Values: Any integer between 1024 to 49151
Example:
host and port must be defined together to enable a metrics endpoint. In this example, I define host as localhost and port as 11111 in config.yaml and start a server.
$ grep -3 metrics config.yaml
cfg_dir: .doltcfg
metrics:
labels: {}
host: localhost
port: 11111
$ dolt sql-server --config config.yaml
Starting server with Config HP="127.0.0.1:3310"|T="28800000"|R="false"|L="debug"
INFO[0000] Server ready. Accepting connections. Then, I can access the metrics by making an HTTP request to http://localhost:11111/metrics
$ curl http://localhost:11111/metrics | HEAD -n 10
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6041 0 6041 0 0 4776k 0 --:--:-- --:--:-- --:--:-- 5899k
# HELP dss_concurrent_connections Number of clients concurrently connected to this instance of dolt sql server
# TYPE dss_concurrent_connections gauge
dss_concurrent_connections 0
# HELP dss_concurrent_queries Number of queries concurrently being run on this instance of dolt sql server
# TYPE dss_concurrent_queries gauge
dss_concurrent_queries 0
# HELP dss_connects Count of server connects
# TYPE dss_connects counter
dss_connects 0
# HELP dss_disconnects Count of server disconnectsFor more information on how to scrape the metrics from this endpoint consult our metrics documentation.
labels
Labels can be added to any Dolt metrics emitted using this optional configuration setting. This is often used to differentiate metrics coming from multiple sources to a single Prometheus collector. The label map will be applied to every metric Dolt emits.
Default: {}
Values: A map of the form {"label": "value"}
Example:
I add the {"process": "dolt-sql-server"} label value in config.yaml and start a Dolt SQL Server.
$ grep -3 metrics config.yaml
cfg_dir: .doltcfg
metrics:
host: localhost
port: 11111
labels: {"process": "dolt-sql-server"}
$ dolt sql-server --config config.yaml
Starting server with Config HP="127.0.0.1:3310"|T="28800000"|R="false"|L="debug"
INFO[0000] Server ready. Accepting connections. Now all the metrics emitted are labeled with process="dolt-sql-server".
$ curl http://localhost:11111/metrics | HEAD -n 10
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6425 0 6425 0 0 3115k 0 --:--:-- --:--:-- --:--:-- 3137k
# HELP dss_concurrent_connections Number of clients concurrently connected to this instance of dolt sql server
# TYPE dss_concurrent_connections gauge
dss_concurrent_connections{process="dolt-sql-server"} 0
# HELP dss_concurrent_queries Number of queries concurrently being run on this instance of dolt sql server
# TYPE dss_concurrent_queries gauge
dss_concurrent_queries{process="dolt-sql-server"} 0
# HELP dss_connects Count of server connects
# TYPE dss_connects counter
dss_connects{process="dolt-sql-server"} 0
# HELP dss_disconnects Count of server disconnectsremotesapi
A running Dolt SQL server can serve as a Dolt remote by enabling these configuration values. With a remote endpoint enabled, you can clone, push, pull, and fetch from a running Dolt SQL Server by connecting with a user with the appropriate permissions. Additional documentation on how to push can be found in this blog article where we announced push support.
port
Default: null
Values: Any integer between 1024 to 49151
Example:
$ grep -3 remotesapi config.yaml
port: 11111
labels: {"process": "dolt-sql-server"}
remotesapi:
port: 22222
read_only: null
$ dolt sql-server --config config.yaml
Starting server with Config HP="127.0.0.1:3310"|T="28800000"|R="false"|L="debug"
INFO[0000] Server ready. Accepting connections.
INFO[0000] Starting http server on :22222 In another shell, I can now clone the database by specifying the DOLT_REMOTE_PASSWORD environment variable and a --user. Note, cloning from DoltHub or DoltLab uses a different authentication method and thus, does not require a user or password.
$ DOLT_REMOTE_PASSWORD= dolt clone --user root http://localhost:22222/config_blog
cloning http://localhost:22222/config_blog
$ cd config_blog
$ dolt log
commit im7qq2ja3nfqnc75khtuli8krla3s3fm (HEAD -> main, remotes/origin/main)
Author: root <root@%>
Date: Thu Dec 05 12:19:04 -0800 2024
Commit created using an event
commit vmikac4f7s4395v0v43dtfcbhrmtmo41
Author: configblog <tim@dolthub.com>
Date: Wed Dec 04 16:51:32 -0800 2024
Transaction commit
commit rgifn94i58hqov4mdv0efsjju0qpg964
Author: root <root@%>
Date: Wed Dec 04 16:48:50 -0800 2024
Manual commit
commit do1tvb8g442jvggv4e3nfqp3fmqt0u5a
Author: timsehn <tim@dolthub.com>
Date: Tue Dec 03 11:16:49 -0800 2024
Inіtialіze datа rеposіtory
I now have a cloned copy of the database in the location I cloned to.
read_only
If a Dolt remote endpoint is enabled by setting a valid port, the endpoint can be made read only by setting read_only to true. The endpoint will accept clone, pull, and fetch requests but not push requests.
Default: null
Values: null, true, or false
Example:
I now set the read_only configuration value to true and start the Dolt SQL server.
$ grep -3 remotesapi config.yaml
port: 11111
labels: {"process": "dolt-sql-server"}
remotesapi:
port: 22222
read_only: true
$ dolt sql-server --config config.yaml
Starting server with Config HP="127.0.0.1:3310"|T="28800000"|R="false"|L="debug"
INFO[0000] Starting http server on :22222 If I make a change and attempt to push it will fail.
$ dolt sql -q "insert into t values (6, 'Can I push this');
dquote> "
Query OK, 1 row affected (0.00 sec)
$ dolt sql -q "select * from t;"
+----+-----------------+
| id | words |
+----+-----------------+
| 0 | first modified |
| 1 | second |
| 2 | third |
| 3 | fourth |
| 4 | dolt commit |
| 6 | Can I push this |
+----+-----------------+
$ dolt commit -am "Added row to push"
commit 0vkmfbrt3d1uljrh0ie0mdikoc9tcsss (HEAD -> main)
Author: timsehn <tim@dolthub.com>
Date: Wed Dec 11 14:07:35 -0800 2024
Added row to push
$ DOLT_REMOTE_PASSWORD= dolt push --user root origin main
- Uploading...unknown push error; rpc error: code = PermissionDenied desc = this server only provides read-only accesssystem_variables
Dolt features a number of custom system variables and supports many of MySQL's system variables. These variables can be set for a running server using a map of system variable to value in this section of the configuration.
Default: {}
Values: A map of system variable to value.
Example:
I will enable the dolt_show_system_tables system variable which changes the behavior of show tables to include Dolt system tables.
$ grep system_variables config.yaml
system_variables: {"dolt_show_system_tables": 1}Connecting a client to the server now has that variable set and exhibits the proper behavior:
MySQL [config_blog]> select @@dolt_show_system_tables;
+---------------------------+
| @@dolt_show_system_tables |
+---------------------------+
| 1 |
+---------------------------+
1 row in set (0.000 sec)
MySQL [config_blog]> show tables;
+------------------------------+
| Tables_in_config_blog |
+------------------------------+
| dolt_branches |
| dolt_commit_ancestors |
| dolt_commit_diff_t |
| dolt_commits |
| dolt_conflicts |
| dolt_conflicts_t |
| dolt_constraint_violations |
| dolt_constraint_violations_t |
| dolt_diff_t |
| dolt_history_t |
| dolt_log |
| dolt_remote_branches |
| dolt_remotes |
| dolt_status |
| dolt_workspace_t |
| t |
+------------------------------+
16 rows in set (0.000 sec)user_session_vars
If instead of setting system variables globally, you would rather set them for individual users, Dolt supports a user_session_vars list of maps in config.yaml.
Default: []
Values: A list of user to variable map
Example:
Let's again set the dolt_show_system_tables variable but this time only for user root. I modify my config.yaml as such.
$ grep -3 user_session config.yaml
system_variables: {}
user_session_vars:
- name: "root"
vars:
"dolt_show_system_tables": 1And then in a connected client with user root, the server now has that variable set and exhibits the proper behavior:
MySQL [config_blog]> select user();
+--------+
| user() |
+--------+
| root@% |
+--------+
1 row in set (0.000 sec)
MySQL [config_blog]> select @@dolt_show_system_tables;
+---------------------------+
| @@dolt_show_system_tables |
+---------------------------+
| 1 |
+---------------------------+
1 row in set (0.000 sec)
MySQL [config_blog]> show tables;
+------------------------------+
| Tables_in_config_blog |
+------------------------------+
| dolt_branches |
| dolt_commit_ancestors |
| dolt_commit_diff_t |
| dolt_commits |
| dolt_conflicts |
| dolt_conflicts_t |
| dolt_constraint_violations |
| dolt_constraint_violations_t |
| dolt_diff_t |
| dolt_history_t |
| dolt_log |
| dolt_remote_branches |
| dolt_remotes |
| dolt_status |
| dolt_workspace_t |
| t |
+------------------------------+
16 rows in set (0.000 sec)jwks
The jwks section of config.yaml is used to configure JSON web token (JWT) authentication. This configuration section is used to authenticate users of the Hosted Workbench to running Hosted Dolt servers. If your interested in this authentication method for your own Dolt use case, please come to our Discord and let us know.
cluster
This section of config.yaml is used to configure "Direct to Standby" or cluster replication. Refer to the documentation for replication for this section of config.yaml. This configuration requires multiple Dolt instances configured so it is out of scope for this article.
Conclusion
That covers all the advanced configuration values for config.yaml. More configuration questions? Come by our Discord and just ask. We're happy to help.