Two weeks ago we announced DumboDB, a new database that combines the best of MongoDB and Git. It uses the storage engine of Dolt, which is our product where MySQL and Git had a baby. There are also Postgres and SQLite babies we nurture here at DoltHub. It’s a wild time to be alive!
DumboDB is very early in development, and as a “vibe coded” project, we depend heavily on parity testing to compare how DumboDB performs side by side with MongoDB. This applies to behavior, such as query results, but also to performance. Tim wrote about agents needing tests recently, but a high level of testing predated the agent era here at DoltHub. I’ve been working in the software industry for 23 years, and I’ve never seen a product with such a strong culture of automated testing as we have with Dolt. For every PR, we run tens of thousands of tests. It’s a critical piece in enabling our 24-hour promise!
For DumboDB, we currently have three categories of parity tests:
- Behavioral Parity: We run the same queries against both DumboDB and MongoDB and compare the results. This ensures that DumboDB is returning the same data as MongoDB for the same queries. This test suite of more than 1,400 tests was the main driver for our initial v0.1.0 release.
- Query Performance Parity: We run the same queries against both databases and compare the execution times. This helps us identify any performance regressions in DumboDB compared to MongoDB. This also enables us to see where the query planner in DumboDB is making different decisions than MongoDB, which can lead to performance differences.
- Dolt Storage Parity: Since DumboDB uses the Dolt storage engine, we also run tests to ensure that the underlying storage layer is performing as expected. Comparable data in Dolt and DumboDB should have similar performance characteristics. This includes verifying that merging branches is comparatively fast and that the storage engine is not a bottleneck for DumboDB. This piece will be useless until we support garbage collection in DumboDB, which is currently a work in progress.
In this blog, we’ll cover what we’ve seen so far with query performance.
Running the Tests#
Currently we don’t run the query performance tests in continuous integration. We’ll get there, but for the time being we haven’t really optimized any aspect of the product yet. Furthermore, there is little point in gating on performance when we are still building the basic features of the product. Today, we run them manually when we see the need to.
Using the dumbodb-parity-testing repository, we can run the query performance tests with the following command:
go run ./benchmarks/cmd/compare -benchtime=2s -csv /tmp/results.csvThe compare executable will first start MongoDB and DumboDB Docker containers. Then it will perform a variety of data loading, data updates, and querying operations against both databases. It will measure the execution time for each operation and output the results in CSV format to /tmp/results.csv. This uses the standard Go benchmarking framework, which is a great way to get consistent and reliable performance measurements. The -benchtime=2s flag tells the benchmark to run each test for at least 2 seconds, which helps get more accurate measurements by reducing the impact of outliers. See the Go docs for more options.
Results#
Given that we haven’t really focused deeply on performance yet, the results are… fine. DumboDB is anywhere from 1.5x to 17x slower than MongoDB.
There are 61 tests currently. We won’t cover all of them. We consider anything less than 2x slower to be “Green”, between 2x and 5x to be “Yellow”, and anything above 5x to be “Red”. Green results are pleasant surprises, mainly because we haven’t done any deep performance work yet. Yellow results are expected at this stage, but there is room to improve. Red results are areas where there is likely a significant issue that we need to address.
Single Document Reads#
Of all the categories of tests, single document reads are possibly the most important to optimize for speed. The key-value access pattern is the bread and butter of document databases. In this category, we are about 50% slower than MongoDB, which is a pleasant surprise. We tested across a variety of different types of _id fields, including integers, strings, ObjectIds, and even documents. The _id didn’t seem to make much of a difference, which is great.
The other axis we tested was the size of the document returned. As the document gets larger, the performance gap widens. Getting documents up to 10KB is still in the green, but once we get to 100KB and above, we are in the yellow. Not terrible, and maybe not even necessary to fix. It’s not clear to us if there are that many users who want documents that are 100KB or larger. If this matters to you, please hop on our Discord and let us know!
| Benchmark | DumboDB µs/op | MongoDB µs/op | Multiplier |
|---|---|---|---|
| Find One (int _id) | 472 | 331 | 1.43 |
| Find One (Document _id) | 478 | 325 | 1.47 |
| Find One (ObjectId _id) | 472 | 329 | 1.44 |
| Find One (String _id) | 473 | 322 | 1.47 |
| Find One (1KB result) | 516 | 353 | 1.46 |
| Find One (10KB result) | 633 | 371 | 1.71 |
| Find One (100KB result) | 1,911 | 902 | 2.12 |
| Find One (1MB result) | 13,500 | 6,890 | 1.96 |
Single Document Inserts#
Inserting documents is also a critical operation for a document database. We tested inserting very small documents, basically a single field in addition to _id, up to large documents of 1MB. We are about 3x-5x slower than MongoDB on single document inserts, and the size of the document doesn’t seem to directly correlate with the performance variance. Why it’s slow is currently unknown, but it stands to reason that Dolt’s storage mechanism is more complex due to the prolly indexing of JSON documents. That gives us something to look into!
| Benchmark | DumboDB µs/op | MongoDB µs/op | Multiplier |
|---|---|---|---|
| Insert One | 1,372 | 320 | 4.28 |
| Insert One (1KB doc) | 1,387 | 354 | 3.91 |
| Insert One (10KB doc) | 2,014 | 565 | 3.56 |
| Insert One (100KB doc) | 8,238 | 3,149 | 2.62 |
| Insert One (1MB doc) | 71,684 | 15,822 | 4.53 |
Single Document Updates#
Updating documents is where we see our largest performance gap. For these tests, we updated a single field in documents of varying sizes. The worst performance we see in any of our 61 tests is in the UpdateOne_1MB test, where we are 17x slower than MongoDB. This is definitely an area we need to focus on. If I had to guess, DumboDB is replacing the entire document on update, when it could probably update a single prolly leaf node. This is just a guess, but we’ll be looking into it to get the real reason before too long.
| Benchmark | DumboDB µs/op | MongoDB µs/op | Multiplier |
|---|---|---|---|
| Update One | 1,271.52 | 329.43 | 3.86 |
| Update One (1KB doc) | 1,255.15 | 330.70 | 3.8 |
| Update One (10KB doc) | 1,347.67 | 343.44 | 3.92 |
| Update One (100KB doc) | 2,331.02 | 358.41 | 6.5 |
| Update One (1MB doc) | 11,406.02 | 654.50 | 17.43 |
Filtering Queries#
We have two categories of filtering queries: equality filters and range filters. These types of filters benefit very much from indexes, and we used these tests primarily as a way to validate that DumboDB’s indexes were working at all. Surprisingly, the equality filters perform significantly worse than the range filters. This is a very good indication that the indexes aren’t working as well as they should be for equality! Range filters, on the other hand, are all holding solidly in the green at about 1.2x to 1.7x MongoDB’s runtime.
Again, the performance work we’ve done thus far has been pretty minimal. There is a good chance we need to totally redo indexes based on some other investigations in progress. Nevertheless, it’s good to have a baseline to work from.
| Benchmark | DumboDB µs/op | MongoDB µs/op | Multiplier | DumboDB (indexed) µs/op | MongoDB (indexed) µs/op | Multiplier (indexed) |
|---|---|---|---|---|---|---|
| Filter Eq (1K docs) | 2,617 | 984 | 2.66 | 1,730 | 1,426 | 1.21 |
| Filter Eq (10K docs) | 28,520 | 5,051 | 5.65 | 16,217 | 3,388 | 4.79 |
| Filter Eq (50K docs) | 156,986 | 21,868 | 7.18 | 77,945 | 14,910 | 5.23 |
| Filter Range (1K docs) | 2,209 | 968 | 2.28 | 1,700 | 1,331 | 1.28 |
| Filter Range (10K docs) | 10,410 | 3,254 | 3.2 | 1,756 | 1,043 | 1.68 |
| Filter Range (50K docs) | 61,123 | 14,196 | 4.31 | 8,132 | 4,814 | 1.69 |
Conclusion#
There are additional tests as well, but we thought this was enough to cover for today. Overall, we aren’t embarrassed by this as a starting point. Dolt was 10x slower than MySQL many years ago, and now the performance is nearly identical.
Maybe more than anything, we want you to know that we are building the testing and performance harnesses to ensure that DumboDB is stable. This new era of vibe coding requires a new level of testing that many products don’t have. Given that the code is kind of a black box now, we need to have a multitude of tests to ensure that the code is correct and performant. We’ll keep adding to the tests as we go, and we’ll keep sharing the results with you.
Hop on our Discord to ask questions and nerd out about version-controlled databases!