Two weeks ago we published A Spooky Performance Regression in AWS EBS Volumes that highlighted a performance regression in writes against AWS EBS volumes we saw while benchmarking Dolt and MySQL.

Initially we suspected that a recent self-imposed change to our benchmarking infrastructure resulted in the write performance slow-down, but after numerous tests and sanity checks, we concluded that something outside of our control had changed with the AWS EBS volumes we were using.

Well, today we have an update—a Christmas update! (Since Thanksgiving has come and gone).

Soon after our previous blog went out, a user reached out on Discord with this comment:

For benchmarking Dolt's performance, we've always used a 30GB gp2 volume. gp2 volumes are general purpose solid state drives (SSDs), which guarantee a minimum of 100 input/output operations per second (IOPS). For these volume types, the IOPS scales linearly with volume size, so if we increased the size of the volume, we'd also increase the provisioned IOPS.

In 2020, AWS released gp3 volumes, a new SSD type that decouples IOPS from volume size allowing users to provision the required performance without provisioning additional block storage capacity. Contrary to gp2 volumes, gp3 provisions a minimum of 3000 IOPS, regardless of volume size. More, of course, can be provisioned, but the volume size does not need to increase in tandem.

So, we went ahead and decided to give the gp3 volumes a shot. The result was the most wonderful, early Christmas gift us Dolt's could have asked for—our Sysbench write performance regression was gone!

On the gp2 backed benchmarking hosts we saw MySQL perform worse than Dolt on multiple benchmarks (though both servers saw increased write latency, MySQL's write latency increased significantly more):

write_tests	server_name	server_version	latency_median	server_name	server_version	latency_median	multiplier
oltp_delete_insert	mysql	8.0.35	7.98	dolt	c7b85e8c	6.79	0.9
oltp_insert	mysql	8.0.35	3.75	dolt	c7b85e8c	3.36	0.9
oltp_read_write	mysql	8.0.35	8.43	dolt	c7b85e8c	14.46	2.1
oltp_update_index	mysql	8.0.35	3.82	dolt	c7b85e8c	3.36	0.9
oltp_update_non_index	mysql	8.0.35	3.82	dolt	c7b85e8c	3.3	0.9
oltp_write_only	mysql	8.0.35	5.37	dolt	c7b85e8c	7.56	1.4
types_delete_insert	mysql	8.0.35	7.7	dolt	c7b85e8c	7.3	1.0

Each multiplier of 0.9 means that Dolt was faster on gp2 than MySQL was, but this didn't match our expectation. At best, we expect Dolt to be equally as fast as MySQL.

On gp3 backed hosts, though, we now see write latency down across both servers to pre-Halloween levels, which is what we expect to see from our benchmarking tests:

write_tests	server_name	server_version	latency_median	server_name	server_version	latency_median	multiplier
oltp_delete_insert	mysql	8.0.35	5.57	dolt	1.29.2	5.99	1.1
oltp_insert	mysql	8.0.35	2.81	dolt	1.29.2	2.91	1.0
oltp_read_write	mysql	8.0.35	7.3	dolt	1.29.2	14.73	2.0
oltp_update_index	mysql	8.0.35	2.86	dolt	1.29.2	3.02	1.1
oltp_update_non_index	mysql	8.0.35	2.97	dolt	1.29.2	3.02	1.1
oltp_write_only	mysql	8.0.35	4.1	dolt	1.29.2	7.17	1.7
types_delete_insert	mysql	8.0.35	5.47	dolt	1.29.2	6.55	1.2

As a result, we've updated our benchmarking infrastructure to now provision gp3 EBS volumes for benchmarking Dolt, and we'll keep an eye out for any other changes to Dolt's performance, especially if it's unexpected.

Conclusion

Big shout out to fernomac for the early gp3 Christmas gift.

Fixing our benchmarking infrastructure only required a one line change to our Terraform code, so that was nice too.

We love having users and blog readers reach out to us on Discord so come by and chat with us about how you're using Dolt.

We still aren't sure what happened with gp2 EBS volumes on Halloween, but if you're experiencing any slowness we'd highly recommend migrating to gp3 as well.

Happy Holidays and don't forget to check out each of our different product offerings below, to find which ones are right for you:

• Dolt—it's Git for data.
• DoltHub—it's GitHub for Dolt.
• DoltLab—it's GitLab for data.
• Hosted Dolt—it's RDS for Dolt databases.