We ship a lot of code at DoltHub, and coding agents, like Claude and Cursor, help us produce code even faster. However, we still need a human to review all of that code and sign off on it before it goes into our products. About six weeks ago, we started using Ito on the repository for our Postgres-compatible version-controlled database, doltgresql. Ito is an AI QA agent a runtime analysis code review tool that works off of GitHub PRs and takes a different approach than other AI tools that analyze PRs: instead of just reading your PR diff and commenting on it, Ito actually builds your application from the PR branch, runs it in a real environment, and exercises it like a user would. On Doltgres that means standing up a real doltgres server and running SQL queries against it. Ito can handle other types of applications, too, for example, running your web application and capturing a screen recording of the UI testing. This post is our review of Ito. We’ll cover what Ito is, how to enable it, dig into a couple of concrete examples of bugs it’s helped us find, and close out with the numbers on how effective it’s actually been over the six weeks we’ve been running it, including our estimated signal-to-noise ratio on the findings it’s raised. Ito doesn’t replace a human reviewer, but it aguments our review process and has been successful at helping us find bugs and broken edge cases.

What is Ito?#

AI tools that analyze GitHub PRs tend to work off the diff of your changes: they feed the changed lines (and maybe some surrounding context) to an LLM and ask it to spot problems. That’s useful for catching sloppy code and some obvious bugs, but it can’t tell you how your product actually runs with these changes, and it can’t test if your product is behaving as expected. A good code reviewer understands the codebase and the product experience and reviews the changes in that larger context.

Ito is different. Ito builds a containerized version of your application from the PR branch and actually runs it. It reads the diff and the PR description to figure out what area of your app is affected, generates a set of test scenarios targeting that area, and then executes them against the running application. For a web app, that means an agent driving a real browser session against your frontend and backend together. For Doltgres, it means opening a Postgres connection to a running doltgres server and issuing SQL statements. Either way, the approach is the same: it’s checking how your code actually behaves, not just what the diff looks like.

Every PR gets a comment from Ito summarizing what it ran, what passed, what failed, and a severity rating for anything it flagged, plus a top-line verdict on whether Ito thinks the PR is safe to merge. Failures come with reproduction instructions that make it easy to repro the failure yourself. For web-based applications, you even get a screen recording you can view that shows you exactly what Ito tested and how your application responded. The Ito website provides a more detailed view of the testing Ito performed. Here’s an example of a detailed report from the Ito website for changes it tested against the open-source chatwoot project.

Below, you can see an example of what an Ito summary comment looks like on one of our PRs. It shows us that it ran 15 tests and two of them failed. It gives a summary of the findings and a recommendation that this PR is not safe to merge yet.

In that same comment, you can drill into the “Tests run by Ito” section to see more detail. We’ll revisit these same findings below, when we talk about what kinds of bugs Ito has found for us.

Turning Ito On#

Setting up Ito is very easy. You create an account at app.ito.ai, install the Ito GitHub App on your organization (or a personal account), and select which repositories you want it watching. There’s no YAML to write and no config to tune before your first run. Ito maps out your codebase from what’s already there, and from that point on, it automatically picks up new and updated pull requests on the repos you selected. As PRs are reviewed, a bot account, itoqa, comments with results as they come in.

You can add custom variables, seed data, or secrets later if you want to point it at specific fixtures, but we haven’t needed to yet. It’s been running against our repo with the defaults.

Ito provides a free trial for startups and small teams, which makes it easy to test out Ito with your app and see how it can help your team.

Ito in Action#

Let’s take a closer look at some real examples of how Ito has helped us catch some pretty tricky bugs. One of the things we’ve noticed is that Ito does a realy good job of testing edge cases in SQL behavior. SQL is a well-documented spec, so it makes sense that a generative AI tool like Ito could use that information to effectively test edge cases and find bugs. This is where we’ve seen Ito be most helpful so far in our experience testing our product.

Two particularly good examples come out of PR #2913, which added native window function support to DoltgreSQL so that we could match PostgreSQL behavior exactly. Previously, Doltgres was relying on go-mysql-server’s implementation, which is implemented to match MySQL behavior, and doesn’t always match PostgreSQL’s behavior, particularly for return types. When I thought I was done with the work, I looked at the PR and noticed that Ito had added comments for two interesting issues it had found.

Bug: Named window reference OVER w produces a full-partition SUM instead of a running SUM within each partition#

SQL window function syntax allows you to define a “window” of result rows over which aggregate functions will operate. These window definitions can be inline (e.g. SELECT sum(y) OVER (ORDER BY z)) or they be defined as a named window (e.g. SELECT sum(y) over (w1) FROM a WINDOW w1 AS (order by z);). In either syntax form, if the ORDER BY clause is not specified for a window, then the window function operates over the full partition. In other words, there is a single partition containing all rows that the window function operates on. However, there was a bug hiding deep in our database engine where the correct window framing was applied only if the ORDER BY clause was defined inline. If the ORDER BY clause was defined in a named window, our code saw that there wasn’t an ORDER BY clause specified inline, and incorrectly defaulted to framing the window over the full result set. To execute this query correctly, our engine needed to recognize that a named window was being used and examine it to see if it declared an ORDER BY clause. Only when no inline window definition and no named window definition contained an ORDER BY clause should the engine default the window framing to a single window over the entire result set.

To make this more concrete, check out these statements that repro the issue:

CREATE TABLE a (x INT PRIMARY KEY, y INT, z INT);                                                                                                          
INSERT INTO a VALUES (0,0,0),(1,1,0),(2,2,0),(3,0,0),(4,1,0),(5,3,0);

-- the inline version worked correctly
SELECT sum(y) over (ORDER BY z) FROM a ORDER BY x;
-- correct:         0, 1, 3, 3, 4, 7   (running/cumulative sum)

-- referencing a named window did NOT work correctly
SELECT sum(y) over (w1) FROM a WINDOW w1 AS (order by z) ORDER BY x;
-- buggy (pre-fix): 7, 7, 7, 7, 7, 7   (full-partition sum on every row)                                                                                   
-- correct:         0, 1, 3, 3, 4, 7   (running/cumulative sum)

Note that when the window definition was defined inline (i.e. SELECT sum(y) over (order by z) FROM a ORDER BY x;), this query produced the correct results. This bug was specific to referencing a named window definition.

Here’s the comment Ito added for this issue:

Inside that comment, there is a lot of detail, including a summary of the finding, evidence for repro’ing the problem, and even a sample prompt you can pass to a coding agent to start debugging the issue. I was particularly interested in the repro instructions. I find that’s usually the fastest way for me to understand an issue. When testing a UI application, Ito will actually provide evidence as a screen recording of the testing with your UI, so you can see exactly how it was triggered and how your app responded. Since DoltgreSQL is a server process and not a UI, the evidence Ito provided for us is a repro script and some analysis. You can see the full evidence Ito provided below.

DROP TABLE IF EXISTS t_named;
CREATE TABLE t_named(id int, grp int, amt int);
INSERT INTO t_named VALUES (1,1,10),(2,1,20),(3,2,5);

SELECT id, SUM(amt) OVER w AS s
FROM t_named
WINDOW w AS (PARTITION BY grp ORDER BY id)
ORDER BY id;

SELECT id, SUM(amt) OVER (PARTITION BY grp ORDER BY id) AS s
FROM t_named
ORDER BY id;

This issue was interesting for several reasons. First off, it’s a legitimate bug deep in our query processor, so it affected multiple products, including Dolt and Doltgres. It’s also syntax that we did have test coverage for in our query processor. Unfortunately, the test coverage we had was asserting incorrect results! This is a great example of a latent bug where it looked like we had good coverage, and we were indeed executing this syntax in tests, but we were asserting the wrong results. Ito provided great value here to help us catch this before a customer had to report it to us.

This also illustrates a powerful aspect of Ito. Nothing in the code diff in the PR that Ito reviewed had any direct sign of this bug. If Ito had only been looking at the diff lines from the PR, then it wouldn’t have caught this. Instead, Ito analyzed what was changing, used knowledge of window function syntax in SQL, and tested edge cases to see if it could find any problems, and sure enough, it did.

Bug: Inheriting or overriding a named window’s ORDER BY drops it#

The next issue that Ito found is a similar, but separate bug. Just like the previous issue, this one is a bug in how a named window definition is merged into a statement, specifically when an ORDER BY clause is provided in the statement to override the ordering in the named window definition. In this case, the overridden ORDER BY clause wasn’t getting applied correctly, and resulted in the aggregate function being incorrectly applied over a window covering all result rows, instead of using the correct window framing required by the ORDER BY clause.

Here’s a concrete example of this bug and how it affects the returned results:

CREATE TABLE t(id int, grp int, amt int);
INSERT INTO t VALUES (1,1,10),(2,1,20),(3,1,30),(4,2,5),(5,2,15);

SELECT id, SUM(amt) OVER (w1 ORDER BY id) AS s FROM t
    WINDOW w1 AS (PARTITION BY grp) ORDER BY id;
-- buggy:   60, 60, 60, 20, 20   (full-partition sum, w1's frame leaking through)                                                                          
-- correct: 10, 30, 60,  5, 20   (running sum, from the inline baseline SUM(amt) OVER (PARTITION BY grp ORDER BY id))

Here’s the comment Ito added for this issue:

As with the previous bug, Ito provided clear steps to reproduce this bug, which made it very easy to get started debugging it.

DROP TABLE IF EXISTS t_inherit;
CREATE TABLE t_inherit(id int, grp int, amt int);
INSERT INTO t_inherit VALUES (1,1,10),(2,1,20),(3,1,30),(4,2,5),(5,2,15);

-- Inheritance chain
SELECT id, SUM(amt) OVER w2 AS s FROM t_inherit
  WINDOW w1 AS (PARTITION BY grp), w2 AS (w1 ORDER BY id) ORDER BY id;

-- Explicit named-window override
SELECT id, SUM(amt) OVER (w1 ORDER BY id) AS s FROM t_inherit
  WINDOW w1 AS (PARTITION BY grp) ORDER BY id;

-- Inline baseline (control)
SELECT id, SUM(amt) OVER (PARTITION BY grp ORDER BY id) AS s FROM t_inherit ORDER BY id;

Like the previous bug, this bug was deep in our query processor, in the shared go-mysql-server module, so it affected all of our database products built on our query processor. Unlike the previous bug, this one was a gap in our test coverage. Thanks to Ito, we found this gap and added new tests for this case, ensuring that same tests will run for each of our database products and prevent a regression.

Not Just for SQL#

It’s worth noting that DoltgreSQL is not the typical product Ito can handle testing. Ito’s initial target was web applications: an agent drives a real browser against your app, clicks around like a user, and hands back a video replay of anything that broke, alongside the usual severity rating and repro steps. DoltgreSQL has no UI to click through. It’s a database engine that other programs talk to over the Postgres wire protocol. On our repo, Ito’s “user” is a SQL client instead of a browser, and its test scenarios are queries instead of clicks.

Ito Improvements#

We’ve only been using Ito for a little over a month, but even in that short amount of time, we’ve seen improvements in how well Ito is able to test our product. Like we mentioned earlier, the majority of Ito users are using it to test web-based applications. Because of that, Ito defaulted to taking screen recordings of the testing and showing that as the evidence for each bug report. As you can imagine… these screen recordings weren’t super interesting or helpful for a database server. The Ito team was receptive to this feedback and quickly rolled out changes so that non-UI products, like Doltgres, don’t include screen recordings, and instead, get a text file containing the evidence Ito used to determine each bug.

We had another issue where Ito was reporting false alarms about SQL syntax that didn’t work. This was initially confusing because the PR where we saw this was successfully running tests that showed the new syntax working. When we reported this to the Ito team, they were responsive and quickly identified that Ito hit a problem building the Doltgres binary on our PR branch and instead used an older binary, which didn’t have support for the new syntax added in the PR. The Ito team quickly rolled out a change to prevent Ito from falling back to an older binary. Now if the binary couldn’t be successfully built from the PR branch, Ito would fail with a clear error message. We were happy with the quick response from the Ito team and haven’t seen this issue again.

For the Doltgres database server, we’ve noticed that Ito does really well finding issues with some PRs, but on other PRs, there are sometimes comments or reported issues that aren’t as helpful and can add noise to your PRs. How effectively Ito can find bugs in your PRs seems to depend on what type of application you’re building and what type of changes you have in your PR. For example, in our experience, we’ve noticed that Ito does really well at finding bugs and broken edge cases when we’re implementing something well-documented, like features from the SQL spec. In the examples above, we were implementing well-defined syntax for SQL window functions, and Ito was able to identify a couple of broken edge cases. We’ve seen similarly helpful comments from Ito in other PRs where we implemented SQL functions, like COALESCE(). In PRs where we were making internal performance optimizations that didn’t directly affect SQL features, Ito wasn’t as helpful or sometimes even added comments that didn’t warrant any action.

By the Numbers#

The examples above are illustrative of the kinds of issues Ito has found for us. In addition to that, I also want to share some stats on the overall effectivness of Ito with our product so far. In the six weeks we’ve been using Ito, it has commented on 92 pull requests in the doltgresql repo. A little under half of those are dependabot-style PRs (e.g. automated dependency bumps, post-release metadata updates) with no real logic for Ito to exercise, so we’ll exclude those from our analysis. That leaves 43 substantive PRs where Ito actually had something to test.

Of the 38 of those 43 PRs that have since merged or closed: 20 came back clean on every run, and on the 18 remaining PRs, Ito reported at least one issue. So, Ito is reporting issues for about half of our PRs. 8 of those 18 show a fix-and-re-verify cycle directly in Ito’s comment history before merge. That shows that for about half of the PRs where Ito is reporting issues, the PR author is seeing those issues and addressing them before merging the PR. This is still undercounting by a bit, since some Ito reported issues get fixed in a follow-up PR or moved to be tracked in our GitHub backlog. Based on that data, and rounding up a bit for issues that are addressed in follow-up PRs or added to the backlog, roughly 33% of our PRs are benefiting from the findings Ito is reporting.

It’s also worth noting how often Ito finds bugs that have nothing to do with the PR it’s reviewing. In 26 of those 43 PRs (~60%), Ito surfaced at least one issue explicitly flagged as pre-existing and unrelated to the PR specific changes. Ito found those by understanding the functionality changing and testing edge cases around it, not because the PR touched that code path. That’s a different kind of value than catching a regression in the PR: it’s deeply testing parts of the product and finding existing issues that we didn’t know about yet.

As one final angle, we went back through every review thread looking for cases where someone on the team weighed in directly on a specific Ito finding, calling it either legitimate or a false alarm. Across those judgment calls, we confirmed more than twice as many findings as we dismissed. A more than 2:1 ratio of confirmed-to-dismissed, on a tool that’s finding real bugs, is a worthwhile trade.

Summary#

We’ve been using the Ito automated review tool on PRs in the Doltgres repository for about six weeks now. After a few initial bumps getting our product working with Ito, which were all quickly resolved by the Ito team, we’ve been getting real value from the comments Ito leaves on our PRs. The numbers above bear that out, including a more than 2:1 signal-to-noise ratio on the findings we’ve actually sat down and judged. It’s been particularly helpful at catching broken edge cases in PRs related to SQL features. In this blog, we showed examples of Ito finding two bugs with SQL window framing, that both existed deep in our query processor dependency.

We’re still experimenting with Ito and seeing where it is most helpful. When we’ve hit an occassional issue using the tool, the Ito team has been responsive and quick to roll out a solution to improve our experience, including making their product work well for non-web-UI products, like Doltgres.

If you’re curious about Ito, you should try it out! It’s easy to create an account with Ito and hook up the ItoQA GitHub action to your GitHub repository. If you are hesitant to change your repository settings, you can also try out Ito by analyzing a single PR. This gives you a really lightweight way to see the Ito experience on your own PR. You can find more details on Ito’s pricing online, which includes options for a free trial, as well as free use for approved open-source projects.

Last, but not least, if you want to talk about version-controlled databases, or AI tooling, please come by the DoltHub Discord. We’re always around Discord and happy to talk about the benefits of version-controlled databases and how AI tooling is changing the software development experience.