Doltgres, the world’s first version-controlled Postgres-compatible database, just hit 1.0, meaning that it’s ready for production use. But before 1.0, Doltgres was in beta for over 18 months. Dozens of brave customers took a chance on building with Doltgres during this Beta period, helping us find problems and incompatibilities before the 1.0 release, and we are forever grateful for the trust those early adopters placed in us.

Last month, one of them found a critical Beta error in the worst possible way: by experiencing data loss. This is, to our knowledge, the only incident of unrecoverable data loss in our company’s eight-year history. It’s a bad feeling. In the words of our CEO Tim, while talking to the affected customer shortly after the incident,

This is really bad and I literally have an upset stomach reporting this to you. Data loss is never something you want to report to a customer as a database company.

In the spirit of transparency, and as an advisory to any similarly situated customers, we are sharing the details of the incident here.

Incident summary#

In August, a Doltgres customer performed an upgrade from version 0.57.3 to 1.2.0 (the latest at the time). Before running the upgrade, they took a backup of the database using the hosted web interface. Shortly after the upgrade completed, they began experiencing errors with SELECT queries on some tables. A manual downgrade to the previous version of Doltgres was performed but the errors persisted. A new database instance restored from the most recent backup had the same problem. DoltHub engineers were granted access to the instance and determined that data loss had occurred in both the running server and in the backup. Engineers used admin tools to repair the database structure so that queries could execute again, but data was still irretrievably missing. Irretrievable values were replaced by NULL. The customer lost a small amount of their overall data, but some smaller tables had 100% of their rows impacted in at least one column.

Root cause analysis#

Dolt and Doltgres store rows as tuples on disk. Large values that cannot fit inline are stored out of band, with a reference to the out-of-band storage location in place of the actual value in tuple storage. This occurs for data types like TEXT, JSON, and others. Additionally, beginning in April 2025, some data types use an adaptive encoding where values that are small enough to fit inline are stored there, with out-of-band storage being used only for larger values. In either case, Dolt storage records the set of all out-of-band data addresses at the end of each row tuple.

During data movement operations including clone, pull, and backup, Dolt walks the tree of address references looking for data chunks to move. The tuple store participates in this process, with each row providing the set of addresses to walk. These addresses represent the out-of-band values in that row. A similar process occurs during garbage collection, where the set of data to keep is determined by the same reference walk.

A bug in the logic that records the set of out-of-band storage addresses in a row caused Doltgres rows containing adaptively encoded values (TEXT, JSON, etc.) to omit such addresses from their bookkeeping data. This meant that the chunks for these out-of-band values would not be moved during clone, push, backup, etc. It also meant that garbage collection would inappropriately delete such chunks after deciding they were unreferenced.

This bug was not discovered and fixed until April 30, 2026 (PR), during testing for the inclusion of adaptively encoded values for Dolt 2.0. Every release of Doltgres between the release of adaptive encoding (April 2025) and 0.56.3 (May 2026) wrote tuples missing this out-of-band bookkeeping data, and all such values would fail to be moved during a backup or clone, or retained during a garbage collection for releases prior to 1.3.0. The bug wasn’t discovered during Doltgres development because data movement and garbage collection operations were unsupported at the time of its introduction, and testing gaps after they were developed failed to trigger the bug. Automatic garbage collection was enabled very late in the development cycle leading up to the Doltgres 1.0 release, well after the underlying bug was fixed, so that failure path also went unnoticed.

When the customer created their final backup prior to upgrading to 1.2.0, all rows written by versions earlier than 0.56.3 failed to include out-of-band values in the backup. After upgrading, automatic garbage collection immediately ran on their data store, permanently erasing all such values. The customer downgraded to the previous version quickly, but the data had already been lost. Restoring from backup was also ineffective, because the backup was missing the same values that were lost during automatic garbage collection.

Mitigation and follow-up#

Every customer running a Doltgres server with values written by a release prior to 0.56.3 was subject to this data loss bug prior to release 1.3.0, which fixed it.

Additionally, during mitigation work for this release, we discovered that this bug also impacted all key columns for adaptively encoded types stored out of band, making them subject to potential data loss on garbage collection just like non-key columns. This bug is also fixed in 1.3.0.

In the aftermath of this incident, we proposed the following five mitigations. Four of them are now fully implemented, with the fifth in progress.

• ✅ Publish this post-mortem as an advisory warning for existing customers who have not yet upgraded to 1.3.0 or later.
• ✅ Provide an administrative tool to rewrite all rows in affected data stores.
• ✅ Perform a data integrity check on startup for Doltgres releases starting with 1.3.0. This check refuses to start a server on affected databases.
• ✅ Retain N recently deployed versions in the downgrade selector for Hosted Doltgres, which previously contained only the 5 most recently released versions. These versions may not have included the most recently deployed version for a customer, which made undoing a bad upgrade involve a support call to DoltHub engineering.
• 🟠 Use fail-safe backup methods for Hosted instances, such as EBS volume snapshots, to act as a backstop against future defects in dolt_backup(). This work is in progress and we expect completion soon.

What current Doltgres customers need to do#

Current Doltgres customers running on a version prior to 1.3.0 are advised to upgrade to at least that version as soon as possible, as well as to make a backup of their data directories prior to doing so. Not every customer is impacted — they must have used certain data types, written certain larger values with them, and done so with releases prior to May. Doltgres 1.3.0 and later will automatically scan data on first startup and refuse to start the server if any data is impacted by the bug.

If you cannot upgrade to 1.3.0 or later immediately, then the following advisories apply:

• DO NOT run the dolt_gc() procedure
• DO NOT restore from a backup
• DO use a backup method other than dolt_backup()
• DO make a durable copy of your data directory before any upgrades

We have already notified our Hosted Doltgres customers of this problem and taken steps to correct it. But since Doltgres is free and open-source, most of our customers are using self-hosted deployments, which means we have no way of contacting them to tell them about this issue. We are publishing this notification here, in our release notes, and in our Discord server to warn as many potentially impacted customers as we can.

Conclusion#

Losing data is the worst thing that a database product can do, and we are ashamed that this happened to one of our customers. But it’s important that we own the error and communicate transparently about it to minimize any future impact.

If you are concerned you may be affected by this bug, please reach out. As always, you can visit us on the DoltHub Discord, where our engineering team hangs out all day, to discuss this issue or anything else on your mind.