Evals Retainer: keep your evals true as your product and models change
A capped monthly retainer that keeps your gold set, judges and CI gate true as your product and models change.
From USD 6,000/month · Capped at 16 hours a month
- Price
- From USD 6,000/month. Capped at 16 hours a month
- Timebox
- Monthly, capped at 16 hours
- Who it is for
- AI-native teams that already have evals, often from a Sprint, and need them to stay current.
What you get
- A monthly gold-set refresh and judge re-calibration
- Regression triage when a check starts failing
- One model-swap eval run a month
- A one-page quality card and a 30-minute call
Not on-call support, not a starting point, not feature development.
Evals go stale as traces and models change. The retainer is the monthly work of keeping them true.
Questions
What does the model-swap cost comparison show?
On the gold set built from your own cases, it compares the models you are considering on quality and on cost per case, so a cheaper model is chosen only where the scores show quality holds. It is part of the Eval Audit and the Implementation Sprint, and the Retainer includes one run a month.
How long does it take?
The Eval Audit takes 1 week from the day I have access. The Implementation Sprint takes 3 to 4 weeks. The Evals Retainer is monthly, with a 3-month minimum and then month to month.
What do you need from us?
Read access to a sample of real traces or workflow data (redacted is fine), one named domain expert who can give a few hours across the week to label cases and sign off on results, and repo and CI access where needed for the Sprint.
What does the Evals Retainer cover, and what doesn't it?
Up to 16 hours a month on keeping your evals true: a gold-set refresh from new traces, judge re-calibration against fresh human labels, regression triage, one model-swap run and a one-page quality card. It is not on-call support or feature development, and it needs an existing harness, so it follows a Sprint.
Full scope and detail
A gold set and gate that stay current, with a monthly quality card
- A monthly gold-set refresh from new traces, and judge re-calibration against fresh human labels
- Regression triage when a check starts failing, and one model-swap eval run a month
- A one-page monthly quality card: what moved, what is stale, what to fix
- A 30-minute monthly call; updated gold set and scorers in your repo
- Price
- From USD 6,000/month. Capped at 16 hours a month
- Timebox
- Monthly, up to 16 hours; 3-month minimum, then month to month
- Who it is for
- AI-native teams that already have evals, often because Skailex built them in an Implementation Sprint, and need them to stay current.
Why it exists
Evals go stale
A gold set is a snapshot. New traces show new failure modes, an LLM judge drifts from your experts' labels, and a model upgrade changes what passes. The retainer is the monthly work of keeping the gold set, the judges and the CI gate true.
How it works
Capped hours, written down
Each month has up to 16 hours: a gold-set refresh from new traces, judge re-calibration against fresh human labels from your domain expert, regression triage, one model-swap eval run, a one-page quality card and a 30-minute call. The term is monthly, with a 3-month minimum and then month to month.
Anything beyond the cap is scoped and agreed in advance, so the hours never grow quietly.
Where it fits
After the Sprint
The retainer needs a working harness, so it follows an Implementation Sprint, or evals you already have.
What it is not
- Not unlimited support or on-call: the cap is 16 hours a month, and anything beyond it is agreed in advance
- Not a starting point: it needs an existing harness, so it follows a Sprint
- Not feature development on your product
Know where your AI fails
Book a call and we will scope it on your real traces or workflow.