Choose a model with a job-specific scorecard

There is no useful “best model” without a job, operating environment, and failure budget. A scorecard turns model selection from a general reputation contest into a repeatable workflow decision.

The unit of selection need not be “one model for the product.” Extraction, judgment, tool use, and final drafting may have different demands. Tie each scorecard to one well-defined workflow step and include its validators and fallback route.

Separate gates from weighted preferences

First write non-negotiable gates: permitted data region, required interface, maximum response time, structured-output support, or deployment constraint. A candidate that fails a gate is not rescued by a high quality score.

Then weight the factors that can trade off:

• correctness on your cases;
• adherence to the output contract;
• tool-call reliability;
• latency distribution, not only the fastest run;
• full cost per accepted task;
• privacy and retention fit;
• failure clarity and ease of recovery;
• operational effort to monitor and change.

Weights must sum to a deliberate priority, not whatever benchmark data happens to be available.

Example: support routing

The task produces one category, urgency level, and confidence. Fast response and stable structured output matter more than long-form reasoning. Low-confidence routing to a person makes a smaller model viable.

A candidate that writes excellent explanations but sometimes breaks the schema can lose to a simpler model that keeps the queue moving and fails visibly.

Example: contract comparison

The workflow compares clauses against an approved playbook and cites both documents. Traceability, long-context behavior, and conservative uncertainty dominate. Latency matters, but a fast unsupported conclusion has negative value.

The scorecard may select a more capable model for the comparison step while keeping extraction and formatting on cheaper deterministic or smaller-model steps.

Test the workflow, not an isolated prompt

Use the same context assembly, tools, validators, retry policy, and fallback planned for production. Record version and configuration. Otherwise the scorecard measures a laboratory prompt that users will never run.

Run the same case more than once and retain every result rather than selecting the best attempt. Report blocking failures, accepted cases, human corrections, and cost per accepted task separately. Keep failed attempts in the total cost while counting only accepted tasks in the denominator. A candidate with a good average but intermittent unauthorized actions on identical input should not hide behind one composite score.

Count timeouts, rate limits, invalid structures, rejected tool calls, and empty outcomes as workflow results. Measuring successful responses alone systematically omits the failures that shape real operation.

Build a reviewable scorecard

Apply gates first and calculate preference scores second.

Review area	What to record
Candidate identity	Model, version, region, configuration, and test date
Hard gates	Pass or fail for each, with verification evidence
Task slices	Case count, risk, and relation to real work
Quality rules	Acceptance-contract rules used for judgment
Stability	Failure distribution across repeated runs
Operating measures	End-to-end latency, retries, tools, and human intervention
Cost	Cost per accepted task rather than per request
Recovery	Whether failure is visible, stoppable, switchable, and replayable
Decision	Approved scope, reserve candidate, open risk, and retest trigger

Tradeoff factors may use a shared five-level scale, but define each level for this task. A top structure-stability rating might mean every test passes both schema and semantic validation, not that a reviewer found the output “quite stable.” Weighted arithmetic organizes evidence; it never overrides a failed gate.

Handle ties and divide the work deliberately

When candidates are close, inspect whether the difference occurs in a decision-bearing slice. If not, select the option that is easier to observe, recover, and replace, or keep both in a controlled comparison. Do not invent irrelevant metrics merely to produce one winner.

A candidate may also win only one step. Reserve a capable model for complex evidence comparison, use a smaller model for low-risk routing, or use deterministic software for calculation and schema enforcement. Recalculate end-to-end errors, latency, and cost after splitting; a locally optimal component may worsen the full chain.

Decide when to retest

Retest after a material model version, prompt contract, source set, tool, or traffic change. Do not update the winner merely because a new model launches; update when evidence on the job changes.

Preserve prior scores so the reason for switching remains visible.

Common questions

Are public benchmarks useless? They can narrow a candidate list and indicate capability families. They cannot represent your inputs, tools, permissions, languages, or failure gates.

Is the model with the lowest listed price always cheapest? No. Include retries, long completions, tool calls, recovery, and human correction in cost per accepted task.

Should an updated model with the same name be retested? Treat a potentially changed version or behavior as a new candidate. Pin versions where possible and rerun the same evidence set.

Boundary

A scorecard makes priorities visible but does not remove judgment. Treat close scores as uncertainty, run more cases where the decision matters, and keep the selected model replaceable.