We recently launched DoltgreSQL version 1.0, which was a landmark release for us after working on it for almost 3 years. DoltgreSQL is a version of Dolt built to be a drop-in replacement for PostgreSQL. Dolt is a drop-in replacement for MySQL that is built from the ground up with Git-influenced version control features, and Doltgres is built directly on top of Dolt’s internal systems. This means that you can use branch, merge, diff, and more with your data and schema.
Last year, we announced that we were adding support for extensions by loading their native libraries, however we’ve decided to change our extension approach entirely. In this blog post, we’ll discuss why we decided to change directions and emulate extensions instead, and why this will be better overall.
Native Extensions?#
One of Postgres’ most exciting features is its ability to be extended. Although it’s open source, the most common way that users add functionality to the product has been through extensions. Over the years, Postgres has built up a large library of extensions that many would consider core to the Postgres experience. With Doltgres, since we’re building it on top of Dolt’s battle-tested engine, this means that we’re working from our completely custom implementation that adheres to Postgres’ documentation. We’re even implemented in different languages (Go for Doltgres, C for Postgres).
This posed a problem to adding extensions, and we had a few potential ways of how we may resolve it. Eventually, we settled on emulating the API that Postgres exposes, using cgo as the glue between our API and the native extension libraries. As a proof of concept, we were able to get uuid-ossp working, however it posed three core challenges. The first was that larger extensions would require a fairly substantial API surface to be implemented before they would work. The second was that these extensions were not designed to work with our versioning features. The third was that these extensions had to reside on the target machine, meaning a user would be unable to use a database on a machine that didn’t replicate the exact same extension versions that the original machine used (weakening the usefulness of push, pull, and clone).
We tried to tackle these problems, but it became clearer over time that this approach just would not work with the selling point of Doltgres: being a fully versioned database. Therefore, we decided to take a new direction.
Emulated Extensions#
We initially discarded the idea of emulating extensions as that would mean that we cannot handle any arbitrary extension that a user may want to use. After looking through over 100 Postgres Dumps, it became clear that the majority of extension usage was focused on a few core ones, and an extremely small minority use their own in-house extensions. Therefore, if we can implement the most commonly used Postgres extensions, then we can support the majority of cases that actually matter for users.
This is actually faster to develop than we initially thought, as we’re not working across languages, but keeping everything in our native Go environment where we’ve built up our experience over the past 8 years. This also allows us to completely bypass the versioning issues, as we’re building the extensions directly into the product. We treat the extension surface as a contract (types, functions, views, etc.), so that users are able to use them exactly like native extensions on a Postgres instance. The underlying logic, however, is fully tailored to Doltgres’ internals, meaning we can add extensions with perfect compatibility.
So far we’ve implemented uuid-ossp and pgvector, proving that the approach can work for more complex extensions. Our previous estimate for implementing pgvector via native extension loading was in the ballpark of several months, but through emulation we were able to get it done in a week.
What’s Next?#
As of the writing of this blog post, we’re working on emulating PostGIS. Afterwards, we’ll continue to add new extensions based on what seems to be the most commonly used ones. pgcrypto, citext, and hstore are just a few that we have planned for the near future, with more to come after that. If you have a particular extension that you need for Doltgres adoption, then let us know via our Discord!
You can also get in touch with us on Twitter/X, where you can stay up-to-date on Doltgres and Dolt! These past three years have been exciting as an engineer working on Doltgres, and I’m excited for the many more years to come. If you’ve not yet tried Doltgres or Dolt then I highly recommend it, as there are no other databases on the market like it. The power of a version-controlled database is something that must be experienced to be believed! Thank you for reading my blog post!