Dolt for Game Development
Dolt is a version-controlled SQL database which gives you the power of relational databases, with the safety and ease of use of a Git-like versioning system. When we began working on Dolt five years ago we believed that a versioned database would be a game changer for developers. It is a tool that is useful across a wide variety of domains and products. As such we cast a wide net in our marketing and outreach, but the one domain that we receive the most interest from is game development.
There are numerous use cases for Dolt in game development, however we have several game developers actively using Dolt to manage their game's configuration, and that will be the focus of this article. We'll demonstrate how to use Dolt to manage game configuration with our fork of the open-source game Endless Sky.
What is Game Configuration And How is it Stored?
Game configuration is the data that defines the behavior of a game. It can include things like the stats of a weapon, AI behavior, the layout of a level, the color of text, or the cost of an item. This data is often stored in plain text files because they can be edited with any text editor, and diffed and merged by the same version control system that manages the source code. Line based file formats such as yaml, or custom text file formats seem to be a common choice. The issue with this is, as games grow, the number of configuration files increases, and the complexity of the configuration can become unwieldy. We have spoken with dev teams that store gigabytes of configuration across hundreds of files, and the process of finding specific entities that need to be changed can become very difficult, grepping through configuration files looking for the right value.
Some games choose to use binary formats for configuration, but in doing so they largely lose the ability to collaborate with others, or require custom tooling to merge changes to configuration.
With Dolt, you don't have to manage gigabytes of text files, or write custom tooling. You can write queries to find and update the entities you are looking for quickly and easily, and you get all the regular version control features that you're used to. Manage your changes and workflow with branches. See what's changed with diffs, and collaborate with merges. You can even get pull requests using DoltHub, DoltLab, Hosted Dolt, or the Dolt SQL Workbench.
The Demo
I have created a simple demo to show how you can use Dolt to manage game configuration using the open source game, Endless Sky which is a 2D space trading and combat game. It is a very active project still being developed against and expanded. I forked the source and made this demo available here.
The Original Configuration
The configuration for endless sky lives in the data directory. Within it and its subdirectories are numerous text
files which use a custom text file format. Which looks like this:
color "hover" 1. 1. 1. 0.
color "active" .75 .75 .75 0.
color "inactive" .25 .25 .25 0.
outfit "Ka'het Shield Restorer"
category "Systems"
cost 138000
thumbnail "outfit/ka'het shield restorer"
"mass" 14
"outfit space" -17
"shield generation" 1.02
"energy consumption" 1.53
description "The Builders never relied much on shield matrices, preferring instead to strengthen their hull. The reality of war, however, required that the Builders develop strong shield technology, leading to some Het vessels making extensive use of this module."
outfit "Ka'het Nullifier"
category "Guns"
cost 3062000
thumbnail "outfit/ka'het nullifier"
"mass" 68
"outfit space" -97
"weapon capacity" -97
"gun ports" -1
weapon
sprite "projectile/ionball"
"frame rate" 15
"rewind"
sound "ionball"
"fire effect" "finisher sparks" 20
"hit effect" "ionball ring" 15
"hit effect" "ionball impact"
"hit effect" "ionball fragment" 24
"live effect" "ionball spark" 140
"live effect" "ionball trail" 140
"live effect" "ionball twinkle" 560
"inaccuracy" 0.2
"velocity" 9
"lifetime" 200
"reload" 350
"cluster"
"firing energy" 710
"firing force" 90
"firing heat" 160
"turn" 73
"shield damage" 120
"hull damage" 40
"energy damage" 3200
"ion damage" 220
"scrambling damage" 10
"hit force" 50
"blast radius" 20
description "The Nullifier was one of the largest weapons the Builders ever developed, created specifically to be carried in battle by the Vareti. Although it does almost no damage to shields and hull, one shot is capable of disabling most kinds of reactors in a matter of seconds."
effect "ionball spark"
sprite "effect/ion impact"
"no repeat"
"frame rate" 30
"lifetime" 11
"random angle" 360
"random spin" 10
"random velocity" 3
"velocity scale" 1.1
effect "ionball trail"
sprite "effect/spark"
"no repeat"
"frame rate" 30
"lifetime" 9
"random angle" 360
"random frame rate" 6
"random velocity" 5
"random spin" 10
"velocity scale" 0.3The format declares an entity type, and its name on the first line. Subsequent lines contain entity attributes and sub-entities. Sub-entities are declared in a nested format which is defined using tabs. It's a very straightforward format that's easy to edit, and easy to read.
The Dolt Schema
Dolt is a SQL database, which means that any data stored goes in tables. To get Endless Sky configuration data into Dolt,
we have to convert it to tabular format. For the demo, I chose a handful of entity types, analyzed them, and created a schema for each.
Many of them, such as colors, are very simple as the data they store is extremely regular. The most difficult was the
outfits entity type. An "outfit" is a piece of equipment that can be equipped to a ship. It has a number of attributes,
and can be used to modify the behavior of the ship. Some outfits are weapons, and weapons have a number of attributes
that are unique to them. As such I created a schema for the weapons table and a schema for the outfits table. The
outfits table has a foreign key relationship to the weapons table. Additionally sprite entities are used to
define the appearance of the outfits in game and are also referenced via a foreign key. outfits and weapons both support
a tremendous number of unique attributes. Putting each of these fields in the schema would have made the schema
large, and sparsely filled. Because of this, only attributes that were in a majority of entity instances would be added
to the schema. This is the schema for our Dolt database:
CREATE TABLE color (
name varchar(32) primary key,
red float,
green float,
blue float,
alpha float
);
CREATE TABLE galaxy (
name varchar(32) primary key,
posx int,
posy int,
sprite varchar(32)
);
CREATE TABLE star (
name varchar(32) primary key,
power float,
wind float
);
create table sprites (
id varchar(36) primary key,
name varchar(128),
frame_time float,
delay int,
scale float,
frame_rate float,
random_start_frame bool,
rewind bool,
no_repeat bool
);
create table weapons (
id varchar(36) primary key,
lifetime int,
velocity float,
reload float,
firing_energy float,
firing_heat float,
inaccuracy float,
shield_damage float,
hull_damage float,
attributes json,
sprite_id_fk varchar(36),
FOREIGN KEY (sprite_id_fk) REFERENCES sprites(id) on delete set null
);
create table outfits (
name varchar(128) primary key,
category varchar(32),
description text,
thumbnail varchar(128),
cost int,
mass float,
attributes json,
weapon_id_fk varchar(36) references weapons(id),
FOREIGN KEY (weapon_id_fk) REFERENCES weapons(id) on delete set null
);
create table outfitters (
name varchar(128) primary key
);
create table outfitter_outfits (
outfitter_name_fk varchar(128),
outfit_name_fk varchar(128),
PRIMARY KEY (outfitter_name_fk, outfit_name_fk),
FOREIGN KEY (outfitter_name_fk) REFERENCES outfitters(name),
FOREIGN KEY (outfit_name_fk) REFERENCES outfits(name)
);I created my tables in my dolt database via the Dolt shell. I then wrote a script to parse out all my entities from the game config files, and insert them into the database.
Taking the outfits table as an example, category receives its own column, as it is a common attribute in nearly all
(604 of 627 instances) of the outfits. On the other side, shield generation did not as it was only present in 29 of 627
outfits. The attributes column is a JSON column, and is used to store all the attributes that did not make it into the schema.
Using a JSON column allows us to store entity attributes without having to modify the schema every time we add a new attribute.
This is a common pattern we see being used by our customers for game configuration, and a great feature of Dolt
is the ability to merge JSON columns. This means that if two people add a new attribute to an entity, or modify two different
attributes, Dolt will merge the changes automatically.
Now we can query to see what the "Ka'het Shield Restorer" shown in the original config above looks like within Dolt:
-- select from outfits omitting the description for brevity
datadb/main> SELECT name, category, thumbnail, cost, mass, attributes, weapon_id_fk
-> FROM outfits
-> WHERE name = "Ka'het Shield Restorer";
+------------------------+----------+-------------------------------+--------+------+------------------------------------------------------------------------------+--------------+
| name | category | thumbnail | cost | mass | attributes | weapon_id_fk |
+------------------------+----------+-------------------------------+--------+------+------------------------------------------------------------------------------+--------------+
| Ka'het Shield Restorer | Systems | outfit/ka'het shield restorer | 138000 | 14 | {"outfit space": -17, "shield generation": 1.02, "energy consumption": 1.53} | NULL |
+------------------------+----------+-------------------------------+--------+------+------------------------------------------------------------------------------+--------------+
1 row in set (0.00 sec)The Endless-Sky Configuration Database is available on DoltHub
and you can clone it and check it out by running dolt clone dolthub/endless-sky databdb and then dolt sql to start the SQL shell.
Note: The Endless-Sky Demo expects the database to be named datadb which is why
the clone command clones the DoltHub database to a directory named datadb.
Loading the Configuration
In the original source code, the configuration was loaded by going through all the files in the data directory structure,
and parsing them. When loading the configuration from Dolt we use SQL to query the database for
the entities we are interested in. Because Endless Sky is written in C++ we need to include
some kind of SQL library to interact with Dolt. I chose to use the mysql-connector-c++.
I wrote some convenience functions to make the database code a bit easier to read for this demo, but I won't focus on
the implementation specifics too much as they will be different for different platforms. Dolt is
mysql compatible, and you can use various connectors supported in many environments and languages.
Here is a function that loads the colors from the database:
void UniverseObjects::LoadColors(DB *db, const std::set<std::string> &changed, bool debugMode)
{
if (!setContains(changed, "color")) {
return;
}
const char * loadColorsQuery = "SELECT name, red, green, blue, alpha "
"FROM color "
"ORDER BY name;";
Rows *rows = db->SelectQuery(loadColorsQuery);
while (rows->Next()) {
string name;
if (rows->String("name", &name)) {
double r = 1.0;
double g = 1.0;
double b = 1.0;
double a = 1.0;
rows->Double("red", &r);
rows->Double("green", &g);
rows->Double("blue", &b);
rows->Double("alpha", &a);
colors.Get(name)->Load(r, g, b, a);
}
}
}This function is straightforward, and the query being run is very clear. It selects the name, red, green, blue, and alpha
from the color table, and then uses the existing colors object to load the colors into the game.
Loading outfits is a bit more complicated as we need to reify the outfit object. Here I join the outfits table with weapons,
and sprites to recreate our original outfit object.
void UniverseObjects::LoadOutfits(DB *db, const std::set<std::string> &changed, bool debugMode)
{
if (!setContains(changed, "outfits") && !setContains(changed, "weapons") && !setContains(changed, "sprites")) {
return;
}
const char * loadOutfitsQuery = "SELECT outfits.name as name,"
"outfits.category as category,"
"outfits.description as description,"
"outfits.thumbnail as thumbnail,"
"outfits.cost as cost,"
"outfits.mass as mass,"
"outfits.attributes as attributes,"
"outfits.weapon_id_fk as weapon_id,"
"weapons.lifetime as lifetime,"
"weapons.velocity as velocity,"
"weapons.reload as reload,"
"weapons.firing_energy as firing_energy,"
"weapons.firing_heat as firing_heat,"
"weapons.inaccuracy as inaccuracy,"
"weapons.shield_damage as shield_damage,"
"weapons.hull_damage as hull_damage,"
"weapons.attributes as weapon_attributes,"
"sprites.name as sprite_name,"
"sprites.frame_time as frame_time,"
"sprites.delay as delay,"
"sprites.scale as scale,"
"sprites.frame_rate as frame_rate,"
"sprites.random_start_frame as random_start_frame,"
"sprites.rewind as rewind,"
"sprites.no_repeat as no_repeat "
"FROM outfits "
"LEFT JOIN weapons ON outfits.weapon_id_fk = weapons.id "
"LEFT JOIN sprites ON sprites.id = weapons.sprite_id_fk "
"ORDER BY name;";
Rows *rows = db->SelectQuery(loadOutfitsQuery);
while (rows->Next()) {
// Load each outfit row by rowThe actual loading of the row data into game objects can be seen here. Most of it is taking the load code from the original game, and modifying it to use the data from the database instead of the text files.
To run the code you will need run dolt sql-server -H127.0.0.1 -udolt from the datadb directory that you cloned the
database into earlier. Then you can run the demo.
Live Loading of Configuration Changes
One feature that I think is particularly beneficial for game development is the ability to live load configuration changes for development. Sometimes it may take a long time to reproduce an issue, or test a configuration change. It may require some amount of gameplay to reach the point where your configuration change is visible. As you balance things, or make tweaks you may need to test many different values, and restarting the game for each change is time-consuming.
Live loading is a feature that allows you to change the configuration of your game while it is running. For text file based configuration this can mean watching all the configuration files for updates to their timestamps. Custom binary formats will require custom tooling to support live loading. Dolt's storage layer uses a Merkle DAG to store data, so it can be diffed, and merged efficiently. Every item stored in the DAG is content addressable which makes it easy to tell if anything has changed since you last checked, and if you want to find out the exact rows that have changed, you can do that by querying the dolt system tables to get the diff.
The demo includes live loading of the Dolt based configuration.
Playing with the Demo Config
To test everything was working right away I made a command line mysql connection to the same database server that Endless-Sky is connected to. I then ran:
UPDATE color
SET red=1,green=0,blue=0
WHERE name = "active";and I saw the change in game:
Reverting that change is as simple as:
call dolt_reset('--hard');Next I created a branch that gave access to all the outfits in the game, and set their cost to one. I then committed those changes to my branch. I did this using the command line mysql client;
~/dev/es>mysql -h127.0.0.1 -udolt -Ddatadb
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 4146
Server version: 8.0.33 Dolt
Copyright (c) 2000, 2024, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> call dolt_checkout('-b','debug-outfits');
+--------+------------------------------------+
| status | message |
+--------+------------------------------------+
| 0 | Switched to branch 'debug-outfits' |
+--------+------------------------------------+
1 row in set (0.04 sec)
mysql> INSERT IGNORE INTO outfitter_outfits SELECT 'Common Outfits', outfits.name FROM outfits;
Query OK, 586 rows affected, 41 warnings (0.07 sec)
mysql> UPDATE outfits SET cost = 1;
Query OK, 627 rows affected, 41 warnings (0.05 sec)
Rows matched: 627 Changed: 627 Warnings: 0
mysql> call dolt_commit('-Am','Made all outfits available to the "Common Outfits", and set the cost of all outfits to 1');
+----------------------------------+
| hash |
+----------------------------------+
| scok1cl9qfgsp10sgu8gjjer9h7qnd15 |
+----------------------------------+
1 row in set (0.01 sec)At the moment these changes do not appear in game as they were made on the branch "debug-outfits". If I go to an outfitter that is part of the "Common Outfit" group you will see the following types of ammunition available:
To see the changes in game I need to get them onto main. I may not want them on main permanently. I may be using these config changes that are sitting on this branch to debug an issue that requires buying a ton of outfits. Whatever the reason I'll merge the changes without committing them.
mysql> call dolt_checkout('main');
+--------+---------------------------+
| status | message |
+--------+---------------------------+
| 0 | Switched to branch 'main' |
+--------+---------------------------+
1 row in set (0.00 sec)
mysql> call dolt_merge('--squash','debug-outfits');
+----------------------------------+--------------+-----------+------------------+
| hash | fast_forward | conflicts | message |
+----------------------------------+--------------+-----------+------------------+
| scok1cl9qfgsp10sgu8gjjer9h7qnd15 | 1 | 0 | merge successful |
+----------------------------------+--------------+-----------+------------------+
1 row in set (0.01 sec)Now these changes are staged on main, and if I exit the outfitter and come back in I will see the changes.
When I'm done debugging and I want to get rid of these changes I can simply reset the branch.
call dolt_reset('--hard');Other things you can do with your Game Config in Dolt
With your data in a relational database you can do a lot of things that are difficult to do with text files. You can write queries to find specific entities and update them. You can write queries to analyze your data, and find patterns. You can use this data to help you balance your game. Let's try some of these out.
First, I'm curious what kinds of outfits are in the game, and how many of each there are.
SELECT category, COUNT(*) as cnt
FROM outfits
GROUP BY category
ORDER BY cnt DESC;
+-------------------+-----+
| category | cnt |
+-------------------+-----+
| Engines | 144 |
| Systems | 98 |
| Power | 79 |
| Turrets | 72 |
| Guns | 64 |
| Ammunition | 40 |
| Secondary Weapons | 37 |
| NULL | 23 |
| Hand to Hand | 22 |
| Licenses | 22 |
| Minerals | 14 |
| Unique | 8 |
| Special | 4 |
+-------------------+-----+
13 rows in set (0.01 sec)Now let's look at outfits in the "Engines" category that modify the "turn" attribute. Let's see which outfits make you the most maneuverable. We'll also look at which of these take up the most space on our ship.
SELECT name, cost, mass, attributes->"$.turn" as turn, attributes->"$.outfit space" as space
FROM outfits
WHERE category = 'Engines'
HAVING turn IS NOT NULL
order by turn desc;
+----------------------------+----------+------+--------+-------+
| name | cost | mass | turn | space |
+----------------------------+----------+------+--------+-------+
| Steering (Stellar Class) | 2435000 | 89 | 4005 | -89 |
| Subarashii Rift Steering | 1577000 | 90 | 3900 | -90 |
| A865 Atomic Steering | 1920000 | 92 | 3597 | -92 |
| Tyrant Plasma Steering | 324700 | 101 | 3549 | -101 |
| "Bufaer" Atomic Steering | 2100000 | 76 | 2967 | -76 |
| Nelmeb GP Hybrid Steering | 1439000 | 67 | 2800 | -67 |
| X5200 Ion Steering | 225000 | 89 | 2609 | -89 |
| Fusion Drive | 80000000 | 480 | 2496 | -480 |
| Vareti Engine Block | 1815000 | 137 | 2333.9 | -168 |
| Orca Plasma Steering | 174000 | 74 | 2283 | -74 |
| Smelter-Class Steering | 880000 | 55 | 2123 | -55 |
| A525 Atomic Steering | 960000 | 60 | 2101 | -60 |
| Ookii Rift Steering | 940000 | 53 | 2099 | -53 |
| Steering (Planetary Class) | 1077000 | 52 | 2069.6 | -52 |
| Pug Lohmar Steering | 1580000 | 64 | 1955 | -64 |
| "Bondir" Atomic Steering | 950000 | 49 | 1714 | -49 |
| Korath Jak'parat Steering | 2740000 | 67 | 1680 | -67 |
| Medium Graviton Steering | 16000000 | 50 | 1600 | -50 |
| Gaktem GP Hybrid Steering | 833000 | 42 | 1590 | -42 |
| Type 4 Radiant Steering | 1100000 | 47 | 1466 | -47 |
| X4200 Ion Steering | 102000 | 59 | 1426 | -59 |
| Telis Engine Nacelles | 1372000 | 81 | 1379.2 | -101 |
| Pug Cormet Steering | 880000 | 46 | 1300 | -46 |
| A375 Atomic Steering | 480000 | 38 | 1222 | -38 |
| Nami Rift Steering | 442000 | 31 | 1137 | -31 |
| Impala Plasma Steering | 79000 | 43 | 1133 | -43 |
| Steering (Lunar Class) | 473000 | 30 | 1056 | -30 |
| "Biroo" Atomic Steering | 530000 | 32 | 1028 | -32 |
| RG3 Torch Steering | 70000 | 64 | 1004 | -64 |
| Forge-Class Steering | 393000 | 28 | 995 | -28 |
| Maeri Engine Nacelles | 928000 | 59 | 931.4 | -76 |
| Pug Akfar Steering | 472000 | 33 | 863 | -33 |
| Type 3 Radiant Steering | 590000 | 30 | 825 | -30 |
| Farves GP Hybrid Steering | 413000 | 24 | 800 | -24 |
| Large Steering Module | 269000 | 25 | 747 | -25 |
| X3200 Ion Steering | 46000 | 35 | 708 | -35 |
| A255 Atomic Steering | 240000 | 25 | 704 | -25 |
| SC-14 Plasma Steering | 41500 | 34 | 638 | -34 |
| Greyhound Plasma Steering | 36000 | 26 | 590 | -26 |
| Steering (Comet Class) | 221000 | 18 | 568.8 | -18 |
| "Benga" Atomic Steering | 250000 | 20 | 563 | -20 |
| Chiisana Rift Steering | 201000 | 17 | 539 | -17 |
| RG18 Torch Steering | 32000 | 39 | 523 | -39 |
| Type 2 Radiant Steering | 325000 | 20 | 476 | -20 |
| Crucible-Class Steering | 172000 | 14 | 456 | -14 |
| Ka'het Sustainer Nacelles | 357000 | 27 | 432.1 | -34 |
| A125 Atomic Steering | 120000 | 16 | 402 | -16 |
| Anvil-Class Engine | 290000 | 26 | 369.6 | -26 |
| X2200 Ion Steering | 21000 | 20 | 368 | -20 |
| Arkrof GP Hybrid Steering | 131000 | 12 | 345 | -12 |
| Fission Drive | 2650000 | 71 | 319.8 | -71 |
| Chipmunk Plasma Steering | 16000 | 15 | 307 | -15 |
| "Basrem" Atomic Steering | 120000 | 12 | 301 | -12 |
| SC-12 Plasma Steering | 17000 | 19 | 297 | -19 |
| RG15 Torch Steering | 15000 | 25 | 287 | -25 |
| Steering (Asteroid Class) | 95000 | 10 | 280 | -10 |
| "Baellie" Atomic Engines | 230000 | 22 | 244 | -22 |
| Bow Drive (Meteor Class) | 187000 | 18 | 240 | -18 |
| Engine (Meteor Class) | 187000 | 18 | 240 | -18 |
| Korath Ark'parat Steering | 36000 | 12 | 240 | -12 |
| Ka'het Compact Engine | 172000 | 21 | 234.4 | -26 |
| X1200 Ion Steering | 10000 | 12 | 192 | -12 |
| Type 1 Radiant Steering | 115000 | 9 | 181 | -9 |
| Small Steering Module | 60000 | 7 | 178.8 | -7 |
| SC-1 Plasma Engines | 21200 | 21 | 160 | -21 |
| X1050 Ion Engines | 20000 | 20 | 132 | -20 |
+----------------------------+----------+------+--------+-------+Releases
At some point during in the game development cycle you will need to create release builds. It is unlikely that you would want to kick off a running database server when your users start your game, and as such you will need a solution for packaging your database with your game.
If you are using Golang for your game, you can use embedded Dolt to read the Dolt database directly from your game. However, Golang isn't a popular language for game development, and as such this likely will not be an option for your title.
For everything else you will likely need to take another approach.
One option would be to export the tables from dolt and to create a sqlite database which can be loaded in game. This approach provides a solution which can largely use the same code as your debug builds. A second option is to have a build stage that runs the queries that you use to load your configuration, and then packages the resulting data into a format that is easy to read from your game. You can write a shell script to run the queries and write the data to an intermediate format. Looking at the color load code we talked about above this table could be written to a file like this:
dolt sql -r csv -q 'SELECT name, red, green, blue, alpha FROM color ORDER BY name' >releasedata/colors.csv
The resulting csv data can easily be turned into row data that can be loaded in place of our active database connection. Alternatively it can be parsed into a binary format that is faster to load for your game.
Setting up Dolt for your Game
Depending on the scale and requirements of your game, there are several different ways to work with Dolt. For small non-collaborative projects you can simply run Dolt on your local machine without any need of a remote to push to and pull from.
For small to medium collaborative projects I'd recommend using DoltHub to host your database. If the game is open source then creating a public database on DoltHub is free and easy. It works just like Github and you can clone, push, and pull from it just like you would with a git repository. For closed source projects you can create a private database by creating a DoltHub Pro account. As long as your data is less than 100MB you can these databases are free. If you exceed 100MB it is just $50 a month and allows your private repos to be up to 1GB in size and $1 for each additional gigabyte.
For projects developed in environments where you need to have all data stored on premises, or in your private cloud, you have two options. The first is to set up a file based remote on a network share, or a cloud storage based remote using a service such as S3 or GCS as your remote. The second is to run DoltLab within your local network or your private cloud. With DoltLab you get the features of DoltHub run on your infrastructure.
Finally, you could set up Hosted Dolt which is a fully managed service that runs on AWS or GCP. This will give you collaboration, backups, and pull requests in a fully managed environment where your data and server are fully isolated from other customers.
If you have any issues or questions about using Dolt in your game's development, reach out to us on Discord.