We adopted Bash Automated Testing System (Bats) to test the Dolt command-line. As of March 10, 2020 we are up to 473 tests, though 55 are skipped because they currently fail. The tests define desired behavior so we're constantly working to get skipped tests to zero. The tests are open source so feel free to use Dolt source code as example to write your own Bats tests. We couldn't find many examples when we were starting our Bats journey.

We run the tests on Windows and Linux against GitHub Pull Requests as part of our continuous integration testing. This approach was extremely successful for us to prevent regressions in Dolt. Given how useful the tool was for us, we were surprised to find out there was not much written about Bats. This blog attempts to rectify that.

What is Bats?

Bats is a Bash-based testing framework. It's designed to test software on the command-line. You write tests in Bash script. Bats provides useful test fixtures like setup(), teardown(), @test, skip, and run. Once you've defined a test file called foo.bats, you run the tests in that file by running bats foo.bats.

A simple test file for a command like touch would look like:

#!/usr/bin/env bats

setup() {
    mkdir foo
    cd foo
}

teardown() {
    cd ..
    rm -rf foo
}

@test "touch creates files" {
    run touch bar.txt # run sets $status and $output
    [ $status -eq 0 ] # use bash conditional operators to test
    [ "$output" = "" ]
    [ -f bar.txt ]
}

After you install Bats, running that file looks like this:

timsehn$ bats foo.bats
 ✓ touch creates files

1 test, 0 failures

If any command exits with a non-zero exit code the test fails. run is special in that it allows a tested command to return non-zero and fills some variables like $status and $output so you can do further assertions on the results of the command.

That's basically all you need to know to get started. Bats is simple.

Why does Bats work for Dolt?

The Dolt command-line tries to look exactly like the Git command-line, except the unit of versioning is tables instead of files. Bats can construct the state of the repository starting at dolt init, run a state modification command, like dolt checkout -b, and then inspect the state of the repository using another Dolt command, like dolt branch. Unless you want to test remotes (ie. clone, push, pull, fetch), you can construct and inspect the state of a repository from the command-line with no external dependencies. This meant we could get really great test coverage using just Dolt and Bats.

Bats tips and tricks

We learned a lot in writing our 9,607 lines of Bats tests. If you adopt Bats, here are some helpful tips and tricks we gleaned from the experience.

${#lines[@]} to test the size of the output

It's often useful to make an assertion about the number of lines in $output, not the contents. Using [ ${#lines[@]} -eq 3 ] asserts that the number of lines in the output from the previously-run command is 3.

Use bash -c if you want to use pipes and run

If you want to use pipes like in the command run find -type f | grep foo, you must use run bash -c "find -type f | grep foo". This is because run, which has no output, is being piped to grep, meaning this grep will always return nothing.

Regexes require || false to work on MacOS

Regexes in Bash take the following form [[ "foo bar" =~ "foo"]]. In earlier versions of Bash there is a non-deterministic problem where regexes that evaluated false would return true. So, in order to use regexes in Bats and have them work on pre-4.0 versions of Bash, you need to append || false. So, the above would need to be [[ "foo bar" =~ "foo"]] || false. The version of Bash that ships on MacOS is very old. On mine it is 3.2.57(1)-release. Beware.

Complex regexes must be defined in variables

If you are composing a regex with special characters like ., *, or +, the only way we could get those to work was by defining the regex in a variable and then using the variable in the regex. [[ "foo bar" =~ "fo.+"]] || false does not work but the following does work:

regex = "fo.*"
[[ "foo bar" =~ $regex]] || false

Using pid to enable multiple parallel Bats jobs

When creating directories to use during your tests, it is helpful to append the pid of the process to the end of the directory like so: mkdir directory-$$. This allows multiple Bats processes to run on the same machine at the same time without stomping on each other.

Starting secondary processes by shutting off &3

Bats uses &3 to communicate with itself. Bats will hang if you use it. If you run something that uses &3, you need to turn it off using 3>&-. We had to do this to start our remote mock server so we could test dolt clone, push, pull, and fetch. If you are starting longer-running processes and then testing input and output from those processes, you may run into this issue.

Making Bats work on Windows

The standard Windows command prompt will not work for Bats because Bats uses Bash commands. We tried a number of Linux environment emulators (CygWin, MinTTY, etc.) and found that WSL (Windows Subsystem for Linux) provided the best compatibility. We recommend that you install Bats through WSL.

Due to how WSL communicates with Windows, the temporary folder used by Bats is not available to any Windows processes. You need to set a temporary folder that exists in the standard Windows filesystem. We chose not to use the standard Windows temporary folder as some users ran into issues with permissions. For Dolt, we assume that the user installed Windows on the C drive, and then create a folder that we use as a base temp folder for all of our tests.

In addition to the above, paths in WSL are a bit different than what a Windows program expects. For example, in WSL a valid path may be /mnt/c/folder, whereas a Windows program would expect C:\folder. Thankfully, WSL provides wslpath, and our tests wrap a function that calls wslpath when running in WSL, just passing the path through for other environments.

Lastly, to share environment variables between WSL and Windows, we use WSLENV. With a little bit of extra work, getting tests to run in POSIX and Windows environments can be as simple as adding an extra function call.

Conclusion

If you want to see Bats in action on Dolt, clone the Dolt repo from GitHub, cd to the bats directory and run bats .. If you find a problem with Dolt, please send a pull request with a skipped Bats test. Or better yet, a working Bats test with the Dolt code patched. Happy testing.