Eval Audit: a one-week trust check on your AI feature
A one-week trust check on one AI feature, run with your own domain expert on your real traces: where it fails, and how far you can trust your current evals.
From USD 8,000
- Price
- From USD 8,000, fixed scope
- Timebox
- 1 week
- Who it is for
- AI-native teams with a live or launching AI feature, real traces, and an owner who has to answer "does it work?"
What you get
- Failure modes ranked from your real traces
- A trust check of the evals and judges you have
- A starter gold set and draft scorers
- A model-swap cost comparison
- A ranked fix plan and a 60-minute readout
Not a build, not outsourced error analysis, not fine-tuning.
Your real traces, your domain expert, my read of both. You keep the gold set and the fix plan whatever you decide next.
Questions
Where should we start?
For most teams, with the 1-week Eval Audit on one AI feature. If you already have evals you no longer trust, the audit starts by checking them. If you already know your top failure modes, go straight to the Implementation Sprint.
Why does the audit need our domain expert?
Because the dataset is the product. Someone who knows what a correct answer looks like has to label real cases and sign off on the results. A dataset written without them only encodes a guess, and a gate built on it will be argued with or switched off. The audit is a trust check run with your expert, not error analysis done for you, so your team can continue afterwards.
When is an Eval Audit not the right move?
Not now if nobody can say what a correct answer looks like, if there are no real traces you can share (for example a vendor owns the AI), if the system is being rewritten so fast that a dataset would be out of date within days, if the output is deterministic and a unit test already covers it, or if you want a dashboard or a generic benchmark score. It is also not ML research or fine-tuning.
Can I see past work?
Past client work is under NDA, so I don't publish case studies, client names or numbers. What I can do is a short walkthrough on a few of your own traces or workflows, so you see the approach on your problem rather than someone else's.
More questions (4)
Will you sign our NDA? How do you handle our data?
Yes. I sign your NDA before I see any traces or documents. I work inside your environment or on a redacted sample you choose, and nothing leaves the access you give me.
What does the model-swap cost comparison show?
On the gold set built from your own cases, it compares the models you are considering on quality and on cost per case, so a cheaper model is chosen only where the scores show quality holds. It is part of the Eval Audit and the Implementation Sprint, and the Retainer includes one run a month.
How long does it take?
The Eval Audit takes 1 week from the day I have access. The Implementation Sprint takes 3 to 4 weeks. The Evals Retainer is monthly, with a 3-month minimum and then month to month.
What happens after the audit?
You keep the ranked failure modes, the gold set and the fix plan whatever you decide next. Your team can fix things on its own, or we move to the 3 to 4 week Implementation Sprint, and later the Evals Retainer.
Full scope and detail
A ranked answer to: where does it fail, and can we trust our evals?
- Failure modes found by reading real traces, ranked by how often they happen and what each costs you, using your own cost figures
- A trust check of the evals and judges you already have: how well they agree with your domain expert's labels
- A starter gold set of labeled cases and 2 to 3 draft scorers, each checked against those human labels
- A model-swap cost comparison on that gold set: quality and cost per case for the models you are considering
- A ranked fix plan with effort sizing, and a 60-minute readout
- A short page on what to show a customer or reviewer
- Price
- From USD 8,000, fixed scope
- Timebox
- 1 week from access, for one AI feature or workflow
- Who it is for
- AI-native teams, where an LLM or agent is the core of the product, with a feature live or about to launch, real traces, and someone who has to answer "does it work?" before a launch, a big customer or a review.
How the week works
Your traces, your domain expert, my read of both
After the NDA and access, I read real traces from your feature, not a made-up example. Your domain expert labels a set of real cases, so the checks test what matters to you and your team signs off on what correct looks like.
If you already have evals or an LLM judge, I check how far they agree with those human labels. If you have none, the starter gold set and draft scorers are the first version. Evals turn "it seems fine" into a test that passes or fails, with a trace that shows why.
Model-swap cost comparison
Which model is good enough, and what it costs per case
On the gold set built from your own cases, the audit compares the models you are considering on quality and on cost per case. A cheaper model is only recommended where the scores show quality holds.
What I need from you
Read access, and a domain expert for a few hours
I sign your NDA before I see any traces or documents. I work inside your environment or on a redacted sample you choose, and nothing leaves the access you give me.
- Read access to a sample of real traces, documents or workflow data (redacted is fine)
- One named domain expert who can give a few hours across the week to label cases and sign off on results
After the audit
Fix it yourselves, or build the evals into CI
You keep the ranked failure modes, the gold set and the fix plan whatever you decide next. Your team can fix things on its own, or we move to the Implementation Sprint, where the eval harness, scorers and CI gate go into your repo, and later the Evals Retainer.
What it is not
- Not a build: no CI gate and no tooling set up (that is the Implementation Sprint)
- Not outsourced error analysis: your domain expert is in the loop, and your team can continue afterwards
- Not fine-tuning, ML research, or a compliance certification or legal advice
Not for, or not now
- Nobody on your side can say what a correct answer looks like, or no domain expert can give a few hours across the week. The dataset is the product, and one written without your expert only encodes a guess.
- There are no real traces you can share, for example because a vendor owns the AI and you cannot see its inputs and outputs.
- The system is being rewritten so fast that any dataset would be out of date within days.
- The output is deterministic, and an ordinary unit test already covers it.
- You want a dashboard, or a generic benchmark score to put on a slide.
- You are looking for ML research, model training or fine-tuning.
Know where your AI fails
Book a call and we will scope it on your real traces or workflow.