DoltLite is proudly agent engineered. Don’t you dare call it vibe code!
This week I had an agent generate documentation for DoltLite. I reviewed the documentation and it was thorough. I even had an agent test the calls in CI and this testing uncovered a garbage collection bug. So, this blog is a feature announcement? Nope.
Somehow, I felt dirty about the documentation. If a DoltLite user can have an agent answer any question about the product from the code, what use is the documentation? Isn’t documentation likely to drift just like comments? I’m on the record saying agent-engineered projects should delete their comments. Should you also delete your documentation? I’m torn. This article is an attempt to collect my thoughts.
The Documentation#
Before we get into the “do you or don’t you?”, let’s look at what the agent actually produced.
The reason I decided to make a documentation change in the first place is the DoltLite README was getting very long. It was becoming a catch-all for all documentation about the DoltLite project. There were things like a detailed concurrency contract in the README. Upstream SQLite has a doc directory. I could just add a doltlite directory in there and offload some of the README weight into it.
That’s what happened. Sections of the README were shortened and new, more detailed docs were added to doc/doltlite. The result is a shorter README, with detailed documentation available in a more discreet location. Each DoltLite SQL function has its own dedicated documentation.
Since the documentation is in the source and agents need tests, I decided to have a CI job that validates the documentation is correct. The validation caught a garbage collection bug because it exercised some strange options.
This is all to say, DoltLite documentation exists and I’m not deleting it. I’m happy I did this documentation exercise.
The Case for Deletion#
These docs took one agent session to generate. I did review them and suggest some tweaks. But still, there’s not much value add over a “Explain how dolt_merge works” or “How do I get the root hash of a database?” prompt with DoltLite source available. Are humans really going to navigate the code to find some markdown files to read or are they going to ask an agent? I think every day, the answer is more likely the latter. Ask an agent.
Moreover, after the documentation was added, I went about my day turning the open source flywheel. As I was reviewing agent-generated pull requests, I started to notice that every change included a change to the documentation, even the most mundane changes. What the heck was going on?
As with comments, it seems that agents are tuned to love documentation updates. They love writing docs. I asked my Codex agent what was up and it said it misinterpreted some lines in AGENTS.md to mean every change requires a documentation change. I told it to fix the offending lines and this is the diff:
No human would ever interpret those instructions as “Every change requires a docs change”. Agents love to write! I had to additionally instruct agents to only change the docs when told explicitly to do so.
Documentation#
Update user-facing documentation only when the user explicitly requests
documentation work. A bug fix, feature, refactor, or observable behavior change
does not by itself authorize documentation edits. Explain the change in the PR
and add regression coverage; do not turn each fix into another paragraph in a
guide or a new published contract claim.Keep existing contract evidence accurate when tests or implementation change.
This does not require expanding documentation or adding contract rows for every
regression test. When documentation work is requested, keep the affected docs,
contract claims, and evidence consistent.
This seems to have fixed the problem. But if left unchecked, I’d have a crazy mess of documentation as confounders in my code repository, much like the problem I observed with code comments. The mere existence of a directory full of documentation was excuse enough for agents to “slop em up”.
Docs or Nah?#
So, let’s review the pros and cons of in-repository documentation for agent-engineered projects like DoltLite.
Pros#
- Shorter README
- Additional dimension for testing
- Humans could read them
Cons#
- Not much value add over a prompt
- Drift can cause confounders
- Documentation is a slop magnet
Verdict#
I’m still torn, leaning documentation in repository. With the proper instructions in AGENTS.md, I’ve been able to successfully avoid the unnecessary documentation changes. Time will tell if documentation drift causes confounders.
Conclusion#
Do you have documentation in your agent-engineered repository? Any hot takes? Come by our Discord. Meet me in the #doltlite🪶 channel and let’s share notes.