Dolt is Git for data and DoltHub is our web application that houses Dolt repositories. At the beginning of Dolt, we adopted Bash Automated Testing System (Bats) for end-to-end testing of the Dolt command-line (check out our blog about Bats here). These tests define our desired behavior for Dolt, and we're always working toward zero skipped Bats tests.

When we redesigned DoltHub earlier this year, we wanted a similar testing suite that could reflect the behavior we have and want for DoltHub. We looked at a few options, and were excited about the potential of Cypress. So we gave it a try.

What is Cypress?

Cypress is an end-to-end testing framework for anything that runs in the browser. Cypress runs in the web browser (we use Chrome), making the debugging process shorter and less painful . It also takes snapshots as your tests run that you can refer back to later in time. You can read more about their unique features in their docs.

Why Cypress works for DoltHub

In what we call DoltHub V1 (before the redesign), we used react-testing-library for our all of our front-end testing. This worked for us at the time, but mocking data and adding tests for someone who doesn't touch the front end as often was pretty difficult. We wanted a testing solution that made it easy for anyone to come in and write a test.

Mocking data is a still a painpoint for testing DoltHub. While mocking data for a singular component is more manageable, mocking the data for a whole page, especially our repository pages which can have over 20 calls to our API, is unrealistic. With Cypress, we can directly test our product without any data mocking. The interface is also very intuitive and easy to use, and makes debugging a more visually-enjoyable experience.

Setting up Cypress

We use Cypress to test against DoltHub in production. That means if we're testing our homepage, all our Cypress tests run against the elements and data you see on that page (as opposed to fake/mocked/seeded data).

We use yarn as a package manager, so installing Cypress was as easy as yarn add cypress.

In our cypress.json file, we set the baseUrl to https://www.dolthub.com to run our tests against our production deployment. In our package.json, we set up some scripts to easily run the Cypress commands we need:

{
  "scripts": {
    "cy-open": "cypress open", // runs tests using the full UI in Chrome against prod
    "cy-run": "cypress run", // runs tests against prod (default browser is Electron)
    "cy-chrome": "cypress run --browser chrome --headless" // runs tests headless against prod (using Chrome)
  }
}

We use data-cy tags to identify elements within our React components, so sometimes we need to run our Cypress tests against our locally running DoltHub server (i.e. localhost:3000). We can set a Cypress env variable when running our tests so do so. This is what our local DoltHub scripts look like:

{
  "scripts": {
    "cylo-open-dolthub": "CYPRESS_BASE_URL=http://localhost:3000 cypress open", // runs tests in Chrome against local server
    "cylo-run-dolthub": "CYPRESS_BASE_URL=http://localhost:3000 cypress run" // runs tests against local server
  }
}

This gets even more complicated when testing our blog or docs, which are separate applications (learn more about our front-end stack and architecture here). Cypress can only run against a single host, so running our blog tests against our local DoltHub server won't work (localhost:3000/blog does not exist, but dolthub.com/blog does). We added the following scripts to run our tests against our blog or docs local web servers:

{
  "scripts": {
    "cylo-open-blog": "CYPRESS_BASE_URL=http://localhost:8000 cypress open",
    "cylo-open-docs": "CYPRESS_BASE_URL=http://localhost:7000 cypress open",
    "cylo-run-blog": "CYPRESS_BASE_URL=http://localhost:8000 cypress run",
    "cylo-run-docs": "CYPRESS_BASE_URL=http://localhost:7000 cypress run"
  }
}

Once in production, our Cypress tests run every 10 minutes using Jenkins.

Writing Cypress tests

Dustin put in a lot of initial work to set up Cypress and put together some utility functions that make even easier to come in and start writing tests for DoltHub. We're working to make our tests open-source, but in the meantime the following examples will not use these utility functions.

Our homepage is one of our most important pages, so we want to write some tests to make sure all the information we want is visible and clicking does what's expected. Here are two simple homepage tests:

// defaultTimeout is the time in ms cypress will wait attempting
// to .get() an element before failing
const defaultTimeout = 5000;
const currentPage = "/";

describe("Homepage renders expected components on different devices", () => {
  describe("Homepage renders expected components on macbook-15", () => {
    // Runs once before all tests in the block
    before(() => {
      const ga = cy.stub().as("ga"); // Create the stub here
      // [Some code to prevent google analytics from loading]
      cy.visit(currentPage);
    });

    // Runs before each test in the block
    beforeEach(() => {
      cy.viewport(device);
      cy.wait(200); // Number of milliseconds to wait before moving on to the next command
    });

    it("should have navbar", () => {
      cy.get("[data-cy=signed-out-navbar]", {
        timeout: defaultTimeout,
      }).should(["be.visible"]);
    });

    it("should have hero with 'Welcome to DoltHub'", () => {
      cy.get("[data-cy=home-page-hero-desktop]", {
        timeout: defaultTimeout,
      }).should(["be.visible.and.contain", "Welcome to DoltHub"]);
    });
  });
});

What's really awesome about Cypress is being able to watch your tests run in Chrome in real time. This is what the above tests would look like running in Chrome:

Testing clicks and page flows are especially important for ensuring the behavior of buttons and links. On our homepage, we have a See pricing button that opens a modal with our pricing information. We want to test to make sure the modal opens, shows the correct information, and closes. To do so, we can add this assertion, which uses clicks, to the above homepage tests:

it("should open and close the pricing modal", () => {
  // Click `See pricing` to open modal
  cy.get("[data-cy=see-pricing-button-desktop]", {
    timeout: defaultTimeout,
  }).click();

  // Assert modal and correct headers are present
  cy.get("[data-cy=pricing-info]", {
    timeout: defaultTimeout,
  }).should(
    "be.visible.and.contain",
    "Pricing",
    "DoltHub Basic",
    "DoltHub Pro"
  );

  // Click to close modal
  cy.get("[data-cy=close-modal]", {
    timeout: defaultTimeout,
  }).click();
});

You can see the modal open and close as the tests are running:

Cypress makes cross-device testing simple as well. Our homepage needs to render the correct components on both desktop and mobile. We can test this my passing in different devices to cy.viewport(). Here's an example:

const devices = ["macbook-15", "macbook-11", "ipad-2", "iphone-x"];

describe("Homepage renders expected components on different devices", () => {
  devices.forEach((device) => {
    describe(`Homepage renders expected components on ${device}`, () => {
      before(() => {
        // Same code as above
      });

      beforeEach(() => {
        cy.viewport(device);
        cy.wait(200);
      });

      // Assertions
    });
  });
});

This is what the tests we wrote look like running against the four different devices above:

You'll notice that some of our iPad and iPhone tests fail :(. Because our testing are running in Chrome, we have access to the Chrome dev tools and can check out what went wrong where.

The failed test tells us it timed out trying to find an element with a tag [data-cy=see-pricing-button-desktop]. If we inspect the pricing button element, we can see that the See pricing button has the CSS property display: none. The element showing is now a link and has a different data-cy tag on mobile ([data-cy=see-pricing-link-mobile]). Oops! On the bright side, it took me less than a minute to find this mistake, so very little time was lost. We can change the assertions on mobile so that the correct See pricing element is clicked.

Modals aren't very mobile-friendly, so you can see that our See pricing button on mobile takes us to the /pricing page instead. We're able to test for this behavior on mobile devices.

Some downsides

As we've been setting up and writing Cypress tests, we've been somewhat iffy about whether this is the best solution for testing DoltHub. While it's great for alerting us to problems users may see that tests using mocked data may not catch (network failures, infrastructure misconfigurations, etc.), there are some downsides to Cypress that we have not yet found solutions to:

• Our tests run against production, so the tests can't run in CI before changes are merged in and deployed to production.
• We've found some difficulties testing against authed pages, and are still working toward getting more test coverage for these pages.
• Having to ensure state is consistent in a live application isn't always ideal. We had to create automated_testing organization with some repositories in different states we know as a team not to change. This makes it difficult to test updating or changing information.

Conclusion

Cypress has an intuitive UI for anyone to step in and start writing tests for our front end against DoltHub in production. While it has it's drawbacks, we feel it's worth it in combination with integration/unit testing using react-testing-library (more updates on our progress here coming soon). We relaunched the redesign of DoltHub early this year, and are still working to increase our test coverage. Our goal is to make our Cypress tests open-source as an example for others to use for their own front-end test coverage. We'd love for you to check out DoltHub and please contact us with any issues so we can write a Cypress test to ensure it never happens again!