One of the cool things about building Dolt for almost eight years now is when we get to innovate on top of the Git model. For those of you unfamiliar with Dolt, it is the world’s first version-controlled database. Dolt offers you all the functionality of Git on database tables instead of files.

There are couple key differences between Git and Dolt usage:

1. Databases are generally much bigger than files.
2. Developers tend to build applications on top of databases.

Both of these differences make history management a bigger concern in Dolt than Git. In fact, I just wrote an article called “My Dolt Got Too Big” about history management. It’s clearly top of mind.

Today, we’re announcing a new tool in the history management tool box: dolt_squash_history(). This article explains why it’s useful and how to use it.

Why Squash History?#

Two reasons: save disk space and make version control operations faster.

To see why squashing helps with both, it helps to know how history is stored. Dolt stores your database as a commit graph, just like Git. Every commit points at a complete tree of your database where table data is stored as Prolly Trees. Prolly Trees are built from content-addressed chunks so that two commits that are 99% the same share 99% of their storage. Structural sharing means a commit is cheap. But cheap is not free. Every commit pins the chunks it references, so a long history of genuinely different data is a long list of chunks that garbage collection is not allowed to touch. Any operation that walks history — dolt log, dolt blame, merge base calculation — does work proportional to the number of commits it walks.

Squashing collapses a string of commits into one. The intermediate versions stop being referenced, garbage collection can finally throw their chunks away, and history walks get shorter.

This matters more in Dolt than it does in Git because of how Dolt gets used. Git commits are usually hand-crafted by humans (and now agents), dozens per day at most. Dolt commits are often made by machines. A common pattern is commit-every-write: an application or agent calls dolt_commit() after every insert or update so that every change is attributable and revertable. That’s a great pattern, right up until you have a million commits and you realize you only care about the last hundred. Nobody hand-writes a million commits in Git. In Dolt, an application can do it in a day.

How to Squash History in Git#

Git gives you two ways to do this, and they make an interesting contrast.

The first is interactive rebase. You run git rebase -i, mark all but one commit as squash, and Git replays your history, writing new commits as it goes. Rebase is a powerful tool: you can squash every second commit, keep one commit per day, reword messages, drop commits entirely. But that power comes from replaying each commit one at a time, so it’s slow on long histories, and it wants an editor session in the middle.

Here’s what that looks like on a repository with 100 commits of churn on top of an initial commit:

$ git log --oneline | wc -l
     101
$ git rebase -i $(git rev-list --max-parents=0 HEAD)

Git drops you into an editor with a 100-line plan. You change every pick after the first to squash:

pick ff4fade # churn 1
squash e672b70 # churn 2
squash 53ce088 # churn 3
squash 9d567ce # churn 4
...96 more lines of this...

Save, quit, get prompted again for the combined commit message, and Git replays all 100 commits:

$ git log --oneline
1214ea2 churn commits, squashed
7265870 create table

It works, but you just did a lot of typing, and Git did a lot of replaying.

The second is the pointer trick: make a backup branch, git reset --soft back to your first commit, and git commit --amend. No replay, so it’s fast. It’s also finnicky. You’re holding your entire history’s worth of changes in the staging area mid-flight, the incantation has to be exactly right, and if you mistype, you get to learn about git reflog.

$ git branch backup
$ git reset --soft $(git rev-list --max-parents=0 HEAD)
$ git commit --amend -m "One hundred churn commits, squashed"
$ git log --oneline
7145c62 One hundred churn commits, squashed

Fast, no replay, done in a blink. But notice what happened: --amend rewrote the initial commit itself, so the whole history is now one commit, initial commit included. If you wanted to keep the initial commit and stack the squash on top, that was a plain git commit, not --amend. This is what I mean by finnicky: two nearly identical incantations, two different histories, and the only reason I get to shrug about it is the backup branch from step one.

So in Git you choose: slow and powerful, or fast and complicated.

Dolt has had both of the above options for a while. dolt_rebase() works just like Git’s, and it’s the tool I used in the “My Dolt Got Too Big” article. It is still the right tool for surgical history editing. But when what you want is “collapse all of this into one commit”, making a rebase plan, updating the plan table, and replaying N commits is a lot of work for a pointer move.

dolt_squash_history()#

dolt_squash_history() is the fast path, without the finnicky. One call, no plan table, no editor, no replay:

CALL dolt_squash_history('-m', 'One hundred churn commits, squashed');

That collapses your current branch’s entire history into a single commit holding exactly the data at HEAD, sitting right on top of the initial commit. Under the hood it’s the pointer trick done properly: Dolt writes one new commit that carries HEAD’s tree, reparents it onto the start of your history, and moves the branch. No commits are replayed, so it runs in roughly constant time no matter how deep your history is.

If you only want to squash part of your history, --first marks the oldest commit to include. Everything from --first through HEAD collapses into one commit, and history older than that is preserved. It takes a commit hash, a HEAD~N expression, or a branch name.

-- Collapse the last ten commits into one; keep everything older.
CALL dolt_squash_history('-m', 'last ten commits squashed', '--first', 'HEAD~9');

There are a few gotchas. You must provide a message with -m. Your working set must be clean:

error: cannot squash history with uncommitted changes

You can’t run it mid-merge or mid-rebase. And the initial commit always survives, because it’s the permanent base your squashed commit gets reparented onto. The procedure returns the hash of the new commit.

One warning applies here just like it does to rebase: this rewrites history. The intermediate versions are gone once you garbage collect, and any clone of your database now has divergent history. If you want to keep the full history around, push to a remote first and treat the remote as your archive of record. I covered that pattern in the disk space article.

Demo#

Let’s see it work. I built a database with a commit-every-write style history: 100 commits, each inserting 50 rows and updating a slice of the table. That leaves 5,000 rows, 102 commits, and 8.5M on disk.

$ dolt log --oneline | wc -l
     102
$ du -sh .dolt
8.5M	.dolt

First, the old way, squashing everything with dolt_rebase():

$ time dolt sql -q "call dolt_rebase('-i', '$ROOT');
                    update dolt_rebase set action='squash' where rebase_order > 1;
                    call dolt_rebase('--continue');"
real	0m1.60s

Now the same squash on an identical copy with dolt_squash_history():

$ time dolt sql -q "call dolt_squash_history('-m', 'One hundred churn commits, squashed');"
real	0m0.15s

Ten times faster on a hundred commits, and this is a toy database. Rebase replays every commit, so its cost grows with the length and size of your history. The squash writes one commit and moves a pointer, so it doesn’t. On a million commit database, the time will be the same, some milliseconds. This is the difference between an instant and some downtime.

Either way, you end up here:

$ dolt log --oneline
d531qouc4mnj7t7vtgucm9n4mktl4606 One hundred churn commits, squashed
m8oobvkhvoooe6agil5jkl40kku0shcb Initialize data repository

The history is two commits. The data is untouched, all 5,000 rows exactly as they were at HEAD. Now that nothing references the intermediate versions, garbage collection can do its job:

$ dolt gc --full
$ du -sh .dolt
172K	.dolt

8.5M down to 172K, a 50x reduction, and the version control operations on this branch now walk a history of two.

Conclusion#

History management matters more in Dolt than Git because databases are bigger than files and machines make commits faster than people do. dolt_rebase() remains the surgical tool when you want to reshape history. When you just want it gone, dolt_squash_history() collapses any suffix of your history into a single commit with one call, in constant time, no editor required. Follow it with dolt gc --full to get your disk back, and push to a remote first if you want the full history archived.

Curious about history management, or want to tell us what history tools you’re missing? Come chat with us on our Discord.