Dolt is Git for data and DoltHub is our web application that houses Dolt repositories. DoltHub consists of three separate React applications: our main Next.js app, as well as two Gatsby apps for our blog and documentation.

Our dependency problem

We use Yarn workspaces to manage our front-end packages. We currently have 14 packages, each including its own package.json, but sharing a node_modules and yarn.lock file. If you're curious about our front-end architecture you can learn more about it here.

As of today, we have 2,481 top-level dependencies, our yarn.lock is 27,439 lines, and our root node_modules is 1.4G. We redesigned and rebuilt DoltHub only a year ago, and these numbers will continue to grow every day. Vinai shared this very accurate and not at all exaggerated comparison in our internal chat the other day:

We use dependabot to automate our dependency updates. It checks each package's package.json and opens a pull request for each dependency bump according to the designated schedule. In the early fall, I realized our robot friend had been turned off since we had rebuilt DoltHub months earlier. I had never dealt with dependencies before and was bombarded with some serious dependency update pull requests. Naive falltime Taylor started going through dependabot's requests and merged the ones that passed continuous integration (CI). We had minimal test coverage in CI at the time, but what was the worst that could happen? The website seemed to be working fine.

Shortly after, one of my coworkers informed me that he could not run our GraphQL server. I tried as well, hoping for user error on his part, but it was indeed broken. NestJS had introduced a breaking change with the update to their newest version. It took me about a day to track it down. I tried running our server and found a new error. Turns out all 9 of our NestJS dependencies needed to be updated to the same version.

After two days, our GraphQL server was running again. I felt like I had wasted two days of my life, but it seemed not completely horrible for the months we spent blissfully ignorant of our dependencies and their needs.

I went to run our Next.js DoltHub server to continue with one of my other projects I had been forced to neglect and ran into another error. Days went by, one error leading to another, leading to another. One dependency update led to another dependency breaking, and bumping that dependency would cause a breaking change for our website. I was filled with regret, sadness, and frustration, and later learned of a name for these dependency-induced feelings: "dependency hell", a common issue among developers like me.

The dependency bump process

I emerged from my dependency hell after over a week. I never wanted to go through that again, and because I also wouldn't wish it on my worst enemies, I came up with a plan.

I configured dependabot to update our dependencies once a month. While the PTSD creeps in when I wake up to an inbox that looks like this on the first of every month, I also know the volume of pull requests and number of breaking changes will be more manageable if I deal with this on a more consistent basis.

My longer term plan includes significantly increasing our React test coverage to reduce parts of this manual process, but these are the steps I go through now on Dependency Day:

1. Properly caffeinate, take a deep breath, and pray to the dependency gods for good fortune.
2. Start with the easy ones. Some of our dependencies are only used in one or two components in one application. The likelihood these majorly mess things up are low, and I feel productive and successful getting those in first.
3. Pull down the first updated branch from dependabot.
4. Make sure all our services run without issues (GraphQL server, DoltHub Next.js server, and our two Gatsby app servers). Fix any issues or bugs.
5. Poke around the affected application and components. Fix any issues or bugs.
6. Push changes, if any. Make sure the pull request is still passing CI and merge (dependabot will automatically rebase any open PRs that are affected by the change).
7. Repeat.

Some things to note

It now rarely takes me more than a day to sort out our dependency updates. Here are a few things that I ran into that are worth noting in case they're helpful for others dealing with dependencies in React apps.

Different versions of React

We have four packages that use React. Have you ever seen this error before?

You may have more than one version of React. And the React dependency listed in your package.json may not be the cause of the error. It can sometimes be a dependency that uses React as a dependency at a different version than the one you're using in your app, which is extra fun!

After some googling, I added "resolutions" to our root package.json that looks like this:

{
  "resolutions": {
    "**/react": "^17.0.1",
    "**/react-dom": "^17.0.1"
  }
}

Now whenever I run into that dreaded React error, I can run yarn install in our root directory (web), and the error (usually) goes away.

Breaking changes in eslint

We use eslint to catch problems in our code. I personally like it and think it helps enforce consistency in our code. However, we currently have 14 eslint dependencies listed in our root package.json:

{
  "@typescript-eslint/eslint-plugin": "^4.9.0",
  "@typescript-eslint/parser": "^4.9.0",
  "eslint": "^7.14.0",
  "eslint-config-airbnb-typescript": "^12.0.0",
  "eslint-config-prettier": "^7.1.0",
  "eslint-plugin-css-modules": "^2.11.0",
  "eslint-plugin-import": "^2.19.1",
  "eslint-plugin-jest": "^24.1.0",
  "eslint-plugin-jest-dom": "^3.2.4",
  "eslint-plugin-jsx-a11y": "^6.4.1",
  "eslint-plugin-prettier": "^3.0.1",
  "eslint-plugin-react": "^7.22.0",
  "eslint-plugin-react-hooks": "^2.3.0",
  "eslint-plugin-testing-library": "^3.10.0"
}

And they introduce breaking changes ALL the time. While these don't affect running our servers or break our website, they will cause errors in CI and we don't want that in our master branch.

Every time a new rule is introduced, or an old one is changed, we need to make the decision about whether it's worth investing the time to update our code. Sometimes we do, but sometimes we decide the investment isn't worth it. When that's the case, we update our root .eslintrc.js (which is used in all our packages that use eslint) to warn instead of error, like so:

{
    "@typescript-eslint/prefer-nullish-coalescing": [
        "warn",
        { forceSuggestionFixer: true },
    ],
}

Test coverage

Since we rebuilt DoltHub, we've struggled to come to an agreement for how to best test DoltHub. We currently have a suite of Cypress tests that run against DoltHub in production. While this works for catching end-to-end production errors, breaking changes from PRs are not caught in CI before a pull request is merged. This makes it difficult to feel confident merging in dependency updates without pulling the branch and manually poking around, even for the most insignificant of dependencies.

We decided recently to take some time to write more unit tests for our React components. We also may one day aim to improve the stability of our Cypress tests so we can reliably run them against PRs and our development environment. We have a long road ahead, but are hoping longer-term this will help with some of our dependency issues.

Conclusion

Managing dependencies for multiple React applications can be a painful, yet necessary part of life as a front-end developer. Have you run into similar dependency woes or have better processes for managing your dependencies? I'd love to hear from you! Come chat with me in our Discord in our #dolthub channel.