The vending machine, the video game, and the pelican
In an architecture discussion, Patrick noticed that some apparently silly AI tests reveal what neat academic scores miss. A long-running vending-machine task exposes planning and adaptation. An old video game exposes perception, memory, and recovery. A pelican on a bicycle or a plate of spaghetti exposes spatial and physical consistency.
Start with the decision the model must support
Do not ask which model is best in general. Name the workflow, user, risk, time horizon, tools, allowed data, expected volume, and failure cost. A model that wins a reasoning benchmark may still fail when it must use your forms, recover from a tool timeout, or respect an approval boundary for six weeks.
Build a three-layer evaluation
- Unit cases: short examples with known answers and clear rubrics.
- Workflow simulations: realistic multi-step tasks with tools, incomplete information, and recovery paths.
- Shadow deployment: run beside the human process without taking action, then compare outcomes.
Measure behavior, not charisma
Track task success, unsupported claims, policy violations, recovery rate, tool-call accuracy, latency, token and vendor cost, human correction time, and consistency across repeated runs. Save prompts, outputs, traces, model versions, source data, rubric decisions, and reviewer disagreements.
Test the ugly edge cases
Include missing fields, contradictory instructions, stale documents, a failed API, duplicate events, malicious text inside a source file, long conversations, and a request the model must refuse or escalate.
Keep the benchmark alive
Every production failure should become a redacted regression case. Re-run the suite before model, prompt, tool, or policy changes. The goal is not a perfect score; it is a known operating envelope and a safe response when the system leaves it.
Software, links, and setup
- OpenAI evaluation guide.
- OpenAI Evals repository.
- Anthropic test and evaluation guidance.
- MLflow GenAI evaluation and monitoring.
- A versioned repository for cases, rubrics, traces, and reviewer decisions.
Prompt pack
Evaluation designer
Design an evaluation for this workflow: [workflow]. Identify users, decisions, failure costs, source-of-truth data, tool dependencies, approval boundaries, and expected volume. Propose unit cases, workflow simulations, and shadow-deployment measures. Every metric must connect to a real risk or outcome.Adversarial case generator
Create test cases for missing data, contradictions, stale sources, prompt injection inside documents, tool timeout, duplicate action, ambiguous identity, policy boundary, long context, and required escalation. Include expected behavior and evidence a reviewer should inspect.Failure-to-regression prompt
Convert this redacted production failure into a reusable regression case with setup, input, expected behavior, unacceptable behavior, scoring rubric, trace fields, and pass threshold. Remove personal or confidential data.What to remember
- Evaluate the workflow and risk, not a model in the abstract.
- Combine unit cases, realistic simulations, and shadow deployment.
- Measure recovery, correction time, cost, and policy behavior.
- Turn production failures into a living regression suite.
Then test it, document it, and make the next version easier for someone else to reproduce.