How We Built DoltHub: Introduction
Towards the end of last month, we launched a totally reworked and redesigned version of DoltHub, our web application for hosting and collaborating on Dolt repositories. Now that we've had a little while to iron the kinks out, it seems like a good time to talk about some of the underlying technology.
If you're not yet familiar with Dolt and DoltHub, you might want to check out some of these resources before reading on:
- Dolt and DoltHub: Getting Started, an introductory blog post by our company's founder and CEO
- The DoltHub homepage
- The Dolt project page on GitHub
- The DoltHub documentation website
For the rest of this article, I'm going to categorize and summarize some of the major technical choices and considerations behind DoltHub. Each major category will be expanded upon in a future article in the series, which will be linked to from here once posted:
Without further ado, let's get geeky!
Stack and system architecture
The heart of DoltHub's backend is an API of GRPC services written in Go. The frontend is a Next.js application using TypeScript and React. Both the frontend and backend are deployed to AWS with Kubernetes.
The initial version of DoltHub did not use GraphQL but it had been on our radar from the start. The redesign presented the perfect opportunity to add a GraphQL layer to our stack, which involved deploying a NestJS server using the fantastic TypeGraphQL library under the hood.
Finally, we recently added a documentation website and ported this blog from the our old site to DoltHub. Both of those are separate Gatsby apps, hosted at "subdirectories" of dolthub.com. This arrangement presented a few interesting challenges.
Sound interesting? Want to know more about how we glue together all these technologies I just name-dropped? Go read the full article!
Front-end architecture
The front-end architecture of DoltHub consists of a monorepo of NPM packages managed with a Yarn workspace. We used to use Lerna as well but ditched it after discovering it wasn't buying us anything. Additionally, we now use TypeScript project references for fast incremental builds of our internal dependencies.
We also extracted a package of components to be shared between the main DoltHub app, the documentation website, and this blog. We encountered challenges and limitations here as well, but came up with something that works...with a few warts.
Other recent architectural changes were consequences of the decisions to migrate to GraphQL for data fetching and Tailwind CSS with CSS Modules for styling. Both of these are elaborated on in their own sections below.
If you're setting up your own TypeScript monorepo, or you're just plain curious, you'll definitely want to read the full article.
Switching the front-end to GraphQL
In earlier iterations of DoltHub we used generated GRPC clients directly from the frontend. This had a number of issues, the foremost being that the mutable object-oriented style of the clients meant that we gave up a lot of the benefits of static typing. In an effort to combat this, we ended up writing lots of code to convert back and forth between class instances and plain old objects.
Adding a GraphQL layer to our infrastructure has given us a single, manageable place for such messy conversions, and allows us to provide the front-end with data that is extremely convenient to work with. Furthermore, GraphQL is statically-typed and supports excellent code generation of TypeScript types, so our typing has never been more comprehensive, helpful, or effortless.
Finally, the developer tooling available makes it harder than ever to make mistakes. We don't have statistics on exasperated sighs, tears, four-letter words, or fits of rage from our developers, but we're confident that all of them would show a significant downwards trend.
Interested in our journey to the data graph? Then you won't want to skip the full scoop when it lands.
Using Tailwind CSS and CSS Modules for sane styling
In the past, our styling needs were primarily handled by styled-components. It's a nice library but we began to find that it encouraged some less-than-best practices, and didn't always play well with TypeScript. Furthermore, you give up much of the innovation happening in the CSS world: the tooling support is spotty, it doesn't work with PostCSS, etc.
Since our redesign, we've switched to Tailwind CSS and CSS Modules (both of which are kind of having a moment right now) and we're rather happy with them.
Want to learn more about how we make stuff pretty? You'll definitely want to read the full article when it's published.
Testing
We encountered a number of challenges in trying to get a comprehensive testing setup working. In fact, it's something we're still heavily iterating on. We'd like to have a suite of integration(-ish) tests against our components using React Testing Library, but getting a sane mocking setup working proved more difficult than anticipated.
We do, however, have a suite of Cypress tests running against our production deployment to alert us to problems users may see, whether they be from errors in application code, network failures, infrastructure misconfigurations, or whatever else.
Once we have this part of our story figured out a bit better, we'll eagerly share our results. Don't miss it.
Conclusion
Hopefully the above has given you some interesting insights into how a fairly complex system comes together to support what you see when you point your browser at DoltHub. We hope you're as excited with the results as we are.
Did we leave something out of the list that you want to know about? Give us a shout here; we're constantly looking for topics to blog.
And please do let us know your thoughts on DoltHub. We welcome your feedback, and look forward to building the future of data with you.