QA for AI agents and workflow automations
An AI automation or agent can keep running while it gets things wrong. The same evals cover them: built in the Implementation Sprint, kept true by the Evals Retainer.
- Implementation Sprint
- From USD 24,000, 3 to 4 weeks
- Evals Retainer
- From USD 6,000/month, capped at 16 hours a month
- Building the automation or agent
- Scoped on a call, quoted separately
What you get
- A test set built from real cases, signed off by your domain expert
- Checks that fail loudly, so you hear from an alert, not a customer
- A CI gate for agent and prompt changes, with traces behind every result
- A runbook and a team handover
Not "automate everything" with no owner, and not a demo with no plan for production.
The same evals as everywhere else: a test set from real cases, checks that fail loudly, kept true every month.
Questions
What are the three offers and what do they cost?
Eval Audit: from USD 8,000, 1 week. Implementation Sprint: from USD 24,000, 3 to 4 weeks. Evals Retainer: from USD 6,000/month, capped at 16 hours a month. Each has a fixed scope, written down before we start.
Do you cover workflow automations and AI agents?
Yes, as part of the same offers. An AI automation can keep running while it gets things wrong: a misread document, a ticket sent to the wrong queue, a report with bad numbers. The Implementation Sprint puts a test set and checks that fail loudly around an automation or agent, and the Retainer keeps them true. If you also need the automation or an agent setup built, we scope that on a call and quote it separately.
Does this work with our stack?
Yes. If you already use Braintrust, Langfuse, LangSmith or a similar tool, I build on top of it instead of replacing it. If you have nothing yet, I set up a light harness in your repo. It works with any model provider, any agent framework, Python or TypeScript, and the CI you already run.
How long does it take?
The Eval Audit takes 1 week from the day I have access. The Implementation Sprint takes 3 to 4 weeks. The Evals Retainer is monthly, with a 3-month minimum and then month to month.
More questions (1)
What do you need from us?
Read access to a sample of real traces or workflow data (redacted is fine), one named domain expert who can give a few hours across the week to label cases and sign off on results, and repo and CI access where needed for the Sprint.
Full scope and detail
Automations and agents you can trust to keep working
- A test set built from real cases of the workflow or agent task, signed off by your domain expert
- Checks that fail loudly and alerts when quality drops, so you find out from an alert, not a customer
- A CI gate for agent and prompt changes, with traces behind every result
- A runbook and a team handover, so your team owns it
- Implementation Sprint
- From USD 24,000, 3 to 4 weeks. Builds the test set and checks around the automation or agent.
- Evals Retainer
- From USD 6,000/month. Capped at 16 hours a month. Keeps them true.
- Building the automation or agent setup
- Scoped on a call and quoted separately. It ships with evals from day one.
Workflow automations
Where automations quietly go wrong
A misread document, a ticket sent to the wrong queue, a report with bad numbers. The automation keeps running, and nobody notices until a customer or a reviewer does. Intake and triage, document extraction and classification, reporting and ops handoffs all benefit from a test set of their own.
AI agents
Agents in your product, and coding agents in your team
For an agent that is the core of your product, the offers are the same: audit the failure modes, build the gate, keep it true. For engineering teams whose coding agents are hit and miss, the checks show whether the agents are actually helping. An agent harness, repo-level config, routing and guardrails are scoped separately.
Field notes
Where the checks go
Short outlines on support triage, patient intake and compliance reporting automations are in the resources section.
What it is not
- Not "automate everything" with no clear process and no owner
- Not a demo or prototype with no plan to run it in production
- Not a separate price list: the automation or agent build is scoped and quoted on a call
Know where your AI fails
Book a call and we will scope it on your real traces or workflow.