Synthetic sample scorecard
The cheaper candidate lost.
A transparent demonstration of the measurement and rejection process using six invented scenarios and a local model. This is not a customer result or a savings claim.
Synthetic proof only. Six cases demonstrate the harness plumbing and acceptance logic. They do not establish production reliability, customer savings, or cloud-provider cost.
| Strategy | Automated checks | Total tokens | Average latency | Passed checks? |
|---|---|---|---|---|
| Verbose baseline | 6 / 6 | 1,200 | 2,876 ms | Yes |
| Shorter candidate | 5 / 6 | 904 | 1,552 ms | No |
Decision
Reject the shorter strategy.
The candidate used 296 fewer tokens and reduced average latency by 1,324 milliseconds, but failed a required scenario check. Passing all six automated checks is only a screening result, not proof of factual correctness or permission to deploy. Human semantic review and representative evaluation are still required.
What the harness demonstrated
- Token and latency measurement on identical cases.
- Automated checks for JSON structure, required keys, and scenario criteria.
- Retention of failed cases instead of removing them from the report.
- A lower-resource candidate can—and should—lose when quality regresses.
Limitation found
The rubric itself needed correction.
The first run used overly literal keyword requirements and rejected semantically correct answers. Those initial results were preserved. The rubric was revised and rerun. It still uses structural and keyword checks, which can miss incorrect reasoning or accept incomplete answers.
Before real customer proof
The evaluation set should expand to 20–30 representative cases, add current provider pricing, define business-specific scoring, and compare at least one multi-provider workflow using approved sanitized data.
Traceable demonstration
Inspect the source summary.
The figures above match the saved September 3, 2026 qwen2.5:7b synthetic run. This replaces an older displayed set of figures. The source summary retains the run date, model and limitations; no customer information is included.
Download the source summaryAgency reporting: a second example
Inspect a deterministic report that reconciles spend, flags missing conversion value, and withholds unsupported ROAS claims. It is an offline synthetic prototype—not a connected advertising service or proof of AI model quality.
Download the synthetic agency reportResidual Forge / founding cohort
Know which AI costs are buying quality—and which are just waste.
Tell us which AI workflow you run repeatedly and where cost, latency, or quality is creating friction.