Engineering preview
Prove the cheaper model is good enough. Then change the routing.
Cost analysis tells you which workload is expensive. It does not tell you whether a cheaper model can do that workload. The control plane is the part that answers the second question before anything in production moves.
Where this stands today
- Runs locally: one command starts the web server, one starts the worker.
- 118 tests and an eight-step acceptance demonstration pass on a fresh database.
- No hosted deployment. Storage is a local SQLite file, which is not durable on serverless hosting.
- A first evaluation has run against real providers on a design partner's own question set.
- Results stay with the design partner. Nothing measured on someone else's workload is published here.
The loop
Four stages, and the fourth is the point.
01
Observe
Every request is recorded with the workflow it belongs to, the model actually used, per-attempt token usage and timing. Observed production traffic stores metadata only: prompt and response text is written solely when the calling application opts in per request, and is otherwise absent from the record.
02
Ontology
A workflow's requirements are written down and versioned: how often it must be correct, how often it must decline to answer, and which languages have to clear the bar on their own.
03
Enforce
A model becomes eligible only by passing a reviewed evaluation against that requirement version. That is evidence on the cases tested, not proof of production reliability. A policy then proposes a route, runs in shadow until someone enables it, and can be returned to observation in one action.
04
Observe again
The same ledger records what the change did, including every fallback and retry. Rollback is a mode change, not a redeploy.
How the numbers are kept honest
Measured costs, clearly marked estimates.
Token counts and timings are measured: they come from the provider's own response for each attempt. Money is not. A cost is computed from those tokens at a published rate card carrying its source and the date it was read, so it is an estimate of the charge rather than the invoice. The rules below are enforced in code and covered by tests, not described in a policy document.
Token counts and latency are measured from each provider response. Cost is computed from them at a dated rate card, so it is an estimate of the charge, never a reconciled invoice.
Every billable attempt is included. A retry that consumed tokens was paid for, and it appears in the total with a breakdown beside it.
An attempt whose token usage the provider did not report is charged at a conservative ceiling, never at zero.
When any attempt cannot be priced, the total is labelled a lower bound and no cost difference or savings percentage is derived from it.
Prices carry their source and the date they were read. A figure is only as current as that date.
A paid evaluation cannot start without an explicit total spending cap, checked before every attempt including retries.
A simulated run cannot establish that a model is eligible, however good its numbers look.
An unreviewed answer is never counted as a correct one.
What it will not do on its own
Observation is the default. Enforcement is a decision.
A workflow starts in observation mode and stays there. In shadow mode a policy records the route it would have taken and changes nothing. Routing only changes after someone activates enforcement, and every activation records who did it and why. Returning to observation is one action.
An eligibility decision always carries a reason, and overriding one requires a named person. Nothing is inferred from a model's name or price.
Evaluation is the deliberate exception to the storage rule above. An evaluation run keeps its dataset, the exact context sent, the system prompt and every raw answer, and the export carries all of them, because an answer nobody kept is an answer nobody can review. That material stays in the operator's own database and is never published here.
Access
Private access for early design partners.
The repository is available on request to design partners. If you run a workload where a cheaper model might be good enough and you need to prove it before switching, that is the conversation worth having.