Implementation Sprint: evals in CI that stop bad changes before they merge

Evals in CI, built in your repo and handed over: a gold set, scorers and a gate that blocks a bad prompt, model or code change before it merges.

Book an eval audit

From USD 24,000

Price: From USD 24,000, fixed scope
Timebox: 3 to 4 weeks
Who it is for: AI-native teams that know their top failure modes and ship prompt, model or code changes every week.

An eval harness and gold set in your repo
Scorers for your top 3 to 5 failure modes
A CI gate that fails loudly, with tracing hooked up
Before and after scores, and a model-swap cost comparison
A runbook and a 90-minute team handover

Not a hosted platform, not a replacement for your domain expert, not fine-tuning.

Sample CI run · illustrative data

ci / eval-gate

$ eval suite run --gate
refund-policy-bot    42/45 passed
document-extraction  38/40 passed
support-triage       31/34 passed
 
3 regressions in support-triage: case-114, case-119, case-127
eval gate: 3 regressions, blocking merge
exit 1

How it works

Agree what good looks like, build the checks, gate every change in CI. Your team runs it without me.

Questions

What does the model-swap cost comparison show?

On the gold set built from your own cases, it compares the models you are considering on quality and on cost per case, so a cheaper model is chosen only where the scores show quality holds. It is part of the Eval Audit and the Implementation Sprint, and the Retainer includes one run a month.

Does this work with our stack?

Yes. If you already use Braintrust, Langfuse, LangSmith or a similar tool, I build on top of it instead of replacing it. If you have nothing yet, I set up a light harness in your repo. It works with any model provider, any agent framework, Python or TypeScript, and the CI you already run.

How long does it take?

The Eval Audit takes 1 week from the day I have access. The Implementation Sprint takes 3 to 4 weeks. The Evals Retainer is monthly, with a 3-month minimum and then month to month.

What do you need from us?

Read access to a sample of real traces or workflow data (redacted is fine), one named domain expert who can give a few hours across the week to label cases and sign off on results, and repo and CI access where needed for the Sprint.

A full eval setup in your repo, owned by your team

An eval harness in your repo, with a gold set of real cases (roughly 150 to 300) built from your traces
Scorers for your top 3 to 5 failure modes: code checks where a rule is enough, LLM judges calibrated against human labels where it is not
A CI gate that fails loudly, and tracing hooked up to the tool you use (Braintrust, Langfuse, LangSmith or similar)
Before and after scores on the same gold set, and a model-swap cost comparison
A runbook, a 90-minute team handover, and a short "how to keep this true" guide

Price: From USD 24,000, fixed scope
Timebox: 3 to 4 weeks
Who it is for: AI-native teams that already know their top failure modes, often from an Eval Audit, and ship prompt, model or code changes every week.

How it works

Agree what good looks like, then build the checks

I turn real cases into a gold set, including the edge cases. Your domain expert signs off on what a correct result looks like, so the checks test what matters to you.

I build the eval harness with scorers that run against the gold set: code checks where a rule is enough, LLM judges where it isn't, each judge checked against human labels.

Checks run in CI or on a schedule, and they fail loudly. Every result links to its trace. You get a runbook and a team handover, and your team can run it all without me.

Your stack

Built on what you already use

If you already use Braintrust, Langfuse, LangSmith or a similar tool, I build on top of it instead of replacing it. If you have nothing yet, I set up a light harness in your repo. It works with any model provider, any agent framework, Python or TypeScript, and the CI you already run.

When model spend is the problem, the model-swap comparison on the same gold set shows where a cheaper model or prompt holds quality and where it does not.

Where it fits

After the audit, before the retainer

Most teams start with the Eval Audit, which ranks the failure modes the Sprint then builds checks for. If you already know your top failure modes, you can start here. Once the harness is live, the Evals Retainer keeps it true.

Evals don't make your model smarter. They tell you, on every change, whether your feature still does what you promised.

What it is not

Not a hosted platform or dashboard: everything lives in your repo and your team owns it
Not a replacement for your domain expert: they still label and sign off on what correct looks like
Not fine-tuning or model training

Know where your AI fails

Book a call and we will scope it on your real traces or workflow.