Dolt is a version controlled database. How would one use such a thing?

• Do you build an application to collect configuration data?
• Is data quality an issue?
• Do you detect defects when data changes or do defects manifest downstream as production errors?
• How long does it take to debug and correct an error in new data? Can you examine differences?
• Can you instantly roll back?
• How do you implement human review?

With Dolt, you get all the convenience of Git-style source code control in your database, for both data and schema. We think a primary use case is to add Git-style version control to your configuration management application; adding branches, differences, rollback, and human review to these applications with minimal source code or schema changes.

Why Version Control Configuration?

Much of the logic in our systems has moved from code to configuration. This is a good thing. We can build wildly complex applications with less and less code over time.

But that means configuration is big and complex, more so over time. Take a look at how much YAML it takes to deploy DoltHub (and other DoltHub infrastructure), almost 54,000 lines!

ld $ find . | grep yaml | xargs cat | wc -l
   53680

Small changes to configuration have big consequences just like code. Breaking configuration is just like breaking code. Enter GitOps, a movement to version control configuration. GitOps promises and delivers faster development with fewer errors.

GitOps is great but it only works works up to a point. Eventually you have so many files, GitOps breaks. There is too much complexity and duplication that finding the configuration you need to change and changing it is harder than it needs to be.

This is where you start to think about a custom application to manage configuration. Configuration becomes data and you need a database. You need a better tool than files to manage it all. Databases have types, schema, and are queryable. These features make ensuring configuration quality easier. You can also build applications on top of databases much easier than files.

But until Dolt, you would lose version control unless you went through the cumbersome process of building it into your database schema. We think Dolt is the next evolution in configuration management. DoltOps anyone?

We've had a number of customers ask us about using Dolt to manage all manner of configuration that has crossed the boundary into data: SMS rates, translations, school information. This blog examines network configuration.

How it Started

Network To Code are thought leaders in the Network Automation space. They provide solutions to clients to automate their network deployment and maintenance. One of the core tools they were deploying for customers was an open source tool called NetBox. NetBox is a web application used to create, store, visualize, and modify network configuration. The goal is for NetBox to be the "Source of Truth" (SoT) for network configuration.

After deploying NetBox a number of times, the Network To Code team noticed a pattern. Despite network engineer's best effort, data in NetBox was often wrong. An engineer mistyped the name of a router. Someone deployed configuration in production and did not update NetBox. Some of these same errors were not happening with network configuration stored as files in Git. But with files in Git, network engineers had a hard time finding and modifying configuration. What if NetBox had Git-style versioning?

This observation manifested before Dolt existed. Damien Garros went looking for a version controlled database for NetBox. He found Dolt's forebear, Noms, tracked it through the Attic Labs acquisition, and Noms eventual shuttering. He went as far as trying to create a demonstration but Noms was not far enough off along.

Damien's dreams were answered in 2019.

OMG, @Liquidata1 put together one of my secret dream.
A Relational Database with the property of a Git repo. https://t.co/aP2GkhV5FJ
The possibilities in infrastructure management and SOT are so big. can't wait to give it a try.

— Damien Garros (@damgarros) November 21, 2019

Damien's colleague, Ken Celenza, reached out to us in the Spring of 2020 to build a NetBox with version control demo for a network automation conference in the Fall of 2020 with the goal of co-presenting what we built. We built SQL Sync and Webhooks for the demo.

We got a working prototype of NetBox + Dolt working. Ken walks through the demo in this presentation.

Let's Build it for Real

The demo generated enough buzz that Ken and team came to us. Let's build it for real.

Network To Code let us know that they were building a fork of NetBox called Nautobot which would diverge from NetBox in a number of ways. For our use case, the two key differences were:

1. The database would be swappable from Postgres to MySQL.
2. Nautobot would support plugins, making a version control extension feasible.

This meant that we wouldn't need a SQL Sync process and DoltHub involved like in the demo. We could back Nautobot with Dolt and use the plug in architecture to implement branches, pull requests and diffs natively in the application. Nautobot could have native version control powered by Dolt.

The first step was to abstract away Postgres. NetBox is a Django application. Django ships with a pretty sophisticated Object Relational Model (ORM). This covered most of the database swapping issues. We just had to change a few types only supported in Postgres and fix the associated business logic. After this was done, Nautobot could run on top of Postgres or MySQL by changing some configuration. Nautobot could also in theory run on top of Dolt but in practice Dolt was missing a key feature.

Dolt didn't support transactions. If you wanted write isolation you had to do it on a branch and write custom logic to merge. That wasn't going to cut it for a drop in replacement to MySQL so off we went building transactions.

Once we had transactions, we needed to write all the plug in code to use Dolt to give Nautobot version control when the plug in was enabled. From start to finish this was about 3 months for one developer.

The Tour

Success. Take a look for yourself. The credentials are in the banner.

Here's a few features you get with the version control plug in enabled in Nautobot. This is a bit of a recap of Andy's launch blog so check that out for more detail.

Branches

If you want to make a change, you can do it on a branch, just like the best practice for source code changes.

Diffs

Once you've made a bunch of changes, you can see what you changed in diff form. This helps you make sure you made only the changes you intended to make before making them permanent.

Pull Requests

Once you're satisfied with what you've done, you can submit the change as a "Pull Request" to your colleagues for additional human review. This multi-step change process is shown to decrease bugs in code by 36%. Now, you can reduce errors in your network configuration changes using the same method.

Rollback

If a bad change makes it through the battery of additional defenses, you also have the option to instantly rollback to a known good state.

Conclusion

So, if you're in the network configuration business, you should definitely try Dolt powered Nautobot. The Git-style version control features will improve your network configuration source of truth, making your network easier to change and reducing downtime.

If you're a software developer building a configuration management application, that application will be better with native Git style versioning powered by Dolt. The configuration will have fewer errors and operators will be able to recover faster from errors that do sneak through. You can build all this easily using the MySQL and Git patterns you already know.

Interested? Come chat with us on our Discord. Let's get you started building your next application on top of a Dolt database.