An agent that answers, and a tool with no SQL parameter
I built an AI agent that answers questions about our own software, in Slack, for people who don’t write code.
The retrieval was the easy half. The work was deciding how it answers, and building limits it can’t talk its way around.
Questions that just die
Somebody on support gets asked why a building’s heat didn’t come on last Tuesday at six. The answer is in the software and nowhere else.
They either interrupt an engineer, guess, or let it go. Honestly, mostly they let it go.
Nobody reports a dead question. So it never showed up as a problem anybody owned. It showed up as a customer getting a vague answer.
My role, and what isn’t mine
I built Alakazam. David, a staff engineer here, built the platform it runs on and reviewed my work.
Mine: how it answers, the rules it answers under, what it can reach, the data subagent, and the performance work.
Not mine: we have a second agent that opens pull requests on our marketing site. David built that one.
Read-only, and still not simple
Alakazam doesn’t change anything. It reads three repositories and answers. That’s the easier half of the agent problem and I’d rather say so.
But an agent that only answers still gets to be wrong. Being wrong confidently in front of a customer-facing team is its own kind of expensive.
Two constraints set that up. The audience isn’t technical, so no code, ever. And it calls things by their on-screen names, not the names in the codebase.
That second one sounds small.
It’s the most common way these things fail, because the model has read the internal names ten thousand times and the person asking has never seen one.
Five rules for how it talks
- Lead with what the person experiences, in business language. Implementation after, and only as much as the question needs.
- Give the mental model, not the isolated fact. A correct answer that can’t predict the next case hasn’t landed.
- Dig deep, answer short. Thoroughness belongs in the searching.
- Say when you’re assuming, in a few words, instead of presenting it as settled.
- Answer as of today’s code. When code and docs disagree, say so.
No restating the question, no “great question,” no sign-off.
Introducing Athena, and a tool with no SQL parameter
People kept asking why a specific building was in a specific state, which needs the database, not the code.
Alakazam was pointing them at another bot for numbers. That kind of handoff kills a tool.
So I added a data subagent. The interesting part isn’t that it queries. It’s what I didn’t give it.
A planner returns a handle to a named, reviewed recipe. The thing that runs against the database accepts only handles. There is no parameter to pass raw SQL into.
Not a rule telling it to prefer recipes. No such parameter exists.
The subagent also runs with its other tools switched off, so it can’t reach the shell or the filesystem.
Same argument I make about install gates. If what stops this going wrong is a sentence in a prompt, nothing stops it going wrong.
The round trip that cost 86 seconds
It used to spend a whole turn asking permission. Ask for eight things, two don’t exist, and it would stop to check whether you wanted the other six.
I measured one of those. 86 seconds and a regenerated 36-row table to learn something you could have assumed.
Now it delivers what exists alongside the caveat about what doesn’t. Having fewer columns than somebody asked for isn’t a fork worth stopping at.
I also moved the repository cloning to build time. It used to happen on every session and hung for 29.8 minutes the first time I traced it.
The prompt is a budget, so I made it a test
The always-on instructions get paid on every turn. It’s the file everybody wants to add a paragraph to, and nobody feels the cost of their own paragraph.
So there’s a test that fails the build if that file passes 215 lines. It was 187 when I wrote the test.
New procedures go in a skill that loads on demand. The test is what makes you do that instead of arguing about it.