Introduction

We're writing Dolt, the world's first version controlled SQL database. Dolt is based on the MySQL dialect and wire protocol, and built to be a drop-in replacement for any MySQL application. But many potential customers expressed their preference for a PostgreSQL variant, and we listened. Last month we announced our initial release of DoltgreSQL, which is the PostgreSQL version of Dolt, with the goal of eventually being a drop-in replacement for any PostgreSQL application.

Today we're excited to announce another milestone: running our suite of sysbench performance tests against DoltgreSQL.

What is sysbench?

sysbench is a database benchmarking tool that has become an industry standard. It comes with a bunch of different reference benchmarks, and we added several of our own to highlight performance problems we discovered and addressed in Dolt. We publish updated latency metrics against these benchmarks on every release of Dolt. For each benchmark, we run it against both Dolt and MySQL, and publish how much worse Dolt's performance is, as a multiplier of MySQL. For the latest release, Dolt is 2.1 times slower than MySQL for reads and 1.3 times slower for writes.

Since releasing our alpha of DoltgreSQL, we've been naturally curious about a few key questions related to its performance:

1. How much slower is DoltgreSQL than original Dolt?
2. How much slower is DoltgreSQL than Postgres?
3. How much slower is MySQL than Postgres?

Today development on DoltgreSQL has progressed far enough that it's possible to run the benchmarks and answer these questions. The last question, comparing MySQL and Postgres to each other, is especially interesting to us considering that we compare ourselves to MySQL for performance purposes. It's commonly accepted that Postgres is a faster database than MySQL, but we've never measured this difference for ourselves until today. How do Dolt and Doltgres stack up against their more mature equivalents?

Results

These numbers were obtained with MySQL 8.0.34 and Postgres 16.1, both running as Windows services on my laptop. Each measurement records the median latency for the benchmark operation in milliseconds. Most of these benchmarks are bundled with sysbench, and the rest can be found in our benchmark repository.

For each benchmark, we calculate the latency multiplier between MySQL and Postgres, Dolt and MySQL, DoltgreSQL and Postgres, and finally between DoltgreSQL and Dolt. These multipliers give us a good intuitive sense to the question of "how much slower" that raw latency numbers (under 5ms on most benchmarks) don't.

First, the results for read operations.

test name	postgres	mysql	dolt	doltgres	mysql v. postgres	doltgres v. postgres	dolt v. mysql	doltgres v. dolt
covering_index_scan	0.95	1.1	1.12	1.21	1.16	1.27	1.02	1.08
index_join	1.55	0.78	2.76	7.04	0.50	4.54	3.54	2.55
index_join_scan	0.39	0.74	1.01	5.99	1.90	15.36	1.36	5.93
oltp_point_select	0.09	0.12	0.22	0.26	1.33	2.89	1.83	1.18
oltp_read_only	2	2.26	3.68	7.3	1.13	3.65	1.63	1.98
select_random_points	0.75	1.52	2.07	10.09	2.03	13.45	1.36	4.87
select_random_ranges	0.64	1.25	0.94	0.99	1.95	1.55	0.75	1.05
index_scan	10.09	18.95	28.67	127.81	1.88	12.67	1.51	4.46
groupby_scan	2.71	7.43	7.43	20.74	2.74	7.65	1.00	2.79
table_scan	9.73	19.65	29.19	130.13	2.02	13.37	1.49	4.46
types_table_scan	27.66	44.17	80.03	missing	1.60	missing	1.81	missing
read multiplier average					1.66	7.64	1.57	3.04

As you can see, Postgres is faster than MySQL across the board, with MySQL being about 1.6 times slower than Postgres on most benchmarks. And as we expected, DoltgreSQL is slower than Postgres, with most of that difference coming from DoltgreSQL's much worse performance relative to Dolt: about 3 times slower on reads on average. We've invested a lot of time iterating on Dolt's wire path to make it fast, and we need to do the same for DoltgreSQL as well to close that gap. We're confident that we can get DoltgreSQL to parity with Dolt, which implies it will be about 2.5x slower than Postgres for reads.

Next up, here are the results for write operations:

test name	postgres	mysql	dolt	doltgres	mysql v. postgres	doltgres v. postgres	dolt v. mysql	doltgres v. dolt
oltp_delete_insert	0.37	1.89	2.52	2.43	5.11	6.57	1.33	0.96
oltp_insert	0.17	0.65	1.18	1.23	3.82	7.24	1.82	1.04
oltp_read_write	2.76	3.82	7.43	10.65	1.38	3.86	1.95	1.43
oltp_update_index	0.2	1.14	1.3	1.27	5.70	6.35	1.14	0.98
oltp_update_non_index	0.2	2.61	1.25	1.37	13.05	6.85	0.48	1.10
oltp_write_only	0.57	3.43	3.49	3.36	6.02	5.89	1.02	0.96
types_delete_insert	0.5	5.57	2.61	2.66	11.14	5.32	0.47	1.02
write multiplier average					6.60	6.01	1.17	1.07

What's especially interesting here is that MySQL is over 6 times slower than Postgres on writes. This was surprising enough that I went digging through my local Postgres server configuration file to see if different durability settings were enabled (for example, not requiring an fsync on writes, which makes write operations much faster at the expense of being susceptible to non-recovery after some crashes). But as far as I can tell, Postgres's durability settings out of the box are the same as MySQL, and it really is just that much faster.

The other interesting thing here is that, unlike for reads, DoltgreSQL and Dolt have very similar write performance. This is more evidence in favor of the idea that something on the wire or serialization path for DoltgreSQL is badly in need of optimization, and that we can close that gap with some attention there.

And also notice that our method of averaging the multipliers gives a funny result where DoltgreSQL appears less-slower than MySQL compared to Postgres. This is an artifact of MySQL doing especially badly on a couple of the benchmarks, despite being faster than Dolt on most benchmarks.

Conclusion

DoltgreSQL is slower than Postgres, which in turn is faster than MySQL. Being 7x slower than Postgres sounds like a lot, until you realize that we're talking about the difference between 2ms and 7ms for a typical benchmark operation. It's a difference in latency most applications won't notice, swamped by things like network latency to the web browser. We'll be addressing closing this performance gap as we continue developing the product. Look for updated versions of these numbers as we release new versions of DoltgreSQL.

We're early on this journey, and we still have a long way to go to get DoltgreSQL to production-ready quality. But it's already good enough for you to start playing with it and experimenting with how you could use it in your applications. All the Dolt version-control procedures like CALL DOLT_COMMIT(...) and the version-control system_tables like SELECT * FROM dolt_diff work just fine. If you're curious, it's not too early to try it out. We'd love to get more feedback from early adopters as we prioritize how we build out missing features.

Have questions about Dolt or DoltgreSQL? Join us on Discord to talk to our engineering team and meet other Dolt users.