How to set AI classification thresholds with a human review queue

A classifier gives an application a score of 0.73. That number alone cannot decide whether to approve or reject it. First define the positive class, establish whether the score behaves like a probability, price the two kinds of mistake, and find out how many cases reviewers can finish before the deadline. Only then can you justify three routes: automatic approval, automatic rejection, and human review.

This page assumes that the task and candidate classifier have already been chosen. The model selection scorecard compares candidates; the evaluation-set guide chooses and labels representative cases. Here we hold one binary classifier fixed and design the probability-to-action layer. AI grader calibration asks whether rules or an LLM judge correctly assess a generated answer. A judge’s self-reported numeric score is not automatically a probability to which this page’s formulas apply.

Define the positive class, actions, and loss

Consider a fictional education discount for a software product. Define y=1 precisely as “eligible under the current policy,” and p=P(y=1 | inputs available now). Automatically approving an ineligible application is a false positive. Automatically rejecting an eligible application is a false negative. A reviewer may check additional evidence and then approve or reject. An automatic rejection needs a reason and an appeal route; a probability is not itself the policy.

Freeze the policy version, the time at which eligibility is judged, and the evidence each action may use. Missing required material, contradictory rules, an invalid score, or an out-of-scope application needs an explicit request-for-information or human route. A low score must not silently stand in for missing evidence. Subtract any mandatory reviews from available capacity before allocating ordinary cases by score.

The following losses are invented teaching assumptions, in incremental loss units relative to a correct action, not measured company costs: a false approval costs C_FP=12, a false rejection costs C_FN=8, and a review costs C_H=2.5. For this calculation only, assume a reviewer makes the right decision in time and a correct automatic decision has zero incremental loss. Then:

Action	Expected loss for an application with calibrated probability p
Automatically reject	L_reject=8p
Automatically approve	L_approve=12(1−p)
Send to review	L_review=2.5

Without review, approval and rejection have equal loss at p=12/(12+8)=0.60. This is not a universal 0.5 cutoff. Google’s classification guide points out that the default threshold may be unsuitable when class balance or error costs differ. With unlimited review capacity, review beats both automatic actions only when 8p>2.5 and 12(1−p)>2.5, or approximately 0.3125<p<0.7917. Specify a tie rule for equality before operating the policy.

Example one: ten known applications, three review slots

Now add a constraint: all ten scores in today’s batch are already known, and reviewers have three slots for that batch. For each case, find the cheaper automatic action, then calculate the value of review as min(8p, 12(1−p))−2.5. Under these simplifying assumptions, the largest positive values deserve the scarce slots first.

Calibrated p	Lowest automatic loss	Value of review	This batch’s route
0.10	0.80	−1.70	Automatically reject
0.20	1.60	−0.90	Automatically reject
0.35	2.80	0.30	Reject; review capacity binds
0.45	3.60	1.10	Reject; review capacity binds
0.52	4.16	1.66	Human review
0.60	4.80	2.30	Human review
0.68	3.84	1.34	Human review
0.75	3.00	0.50	Approve; review capacity binds
0.85	1.80	−0.70	Automatically approve
0.95	0.60	−1.90	Automatically approve

The three slots go to 0.60, 0.52, and 0.68. For this known set of scores, the route happens to fit two cutoffs: automatically reject if p<0.52; review if 0.52≤p≤0.68; automatically approve if p>0.68. The example’s expected loss with no review is 27.0. Replacing the three cases’ lowest automatic losses of 4.16+4.80+3.84=12.80 with 3×2.5=7.50 yields 27.0−12.8+7.5=21.7. These are calculated consequences of the assumptions, not observed error rates or savings.

The 0.35, 0.45, and 0.75 cases still have positive value of review. They merely lost the capacity contest. Nothing in this table proves that automatically handling them is safe. If a non-negotiable error limit forbids one of those routes, three review slots may be an infeasible plan; add capacity, narrow intake, or stop the affected automatic action instead of averaging the hard limit into a cost score.

Verify that the score is a probability

The multiplication above is valid only if p has a useful probabilistic meaning for the current task. scikit-learn separates estimating a class score from choosing an action at a threshold. It describes good calibration as a group property: among many cases given a probability near 0.8, roughly 80% should be positive. This does not guarantee that any one case is correct. A raw logit, similarity value, ranking score, or an LLM’s claimed “80% confidence” cannot be substituted for p without testing.

Give four data uses clear boundaries: (1) train the classifier; (2) fit the probability calibrator on cases disjoint from classifier training and inspect reliability; (3) select action cutoffs, cost assumptions, and capacity rules; (4) evaluate the frozen policy on a previously untouched holdout. With limited data, cross-validation can help within development, but preserve a final independent assessment; where time changes the case mix, include a later time window in that assessment. scikit-learn specifically warns against tuning a decision threshold on the same data used to train the classifier. Reusing the same cases to fit calibration, choose thresholds, and announce final performance makes the result optimistic.

Show the case count, mean predicted probability, and actual positive share for each reliability bin, including important task slices. Sparse positive labels or very narrow bins make a curve unstable. A single summary metric can hide local miscalibration. scikit-learn notes that Brier loss also reflects discrimination and inherent uncertainty; a lower Brier number alone does not establish better calibration. Recheck probability meaning after the eligibility policy, applicant mix, or input fields change.

Example two: a stream fills the review queue

0.52 and 0.68 are not recommended fixed cutoffs for a future stream. When all ten scores are known, you can choose the top three review values. In an online flow, the system cannot know whether a later request would deserve a slot more. Even with fixed cutoffs, tomorrow’s number of middle-band cases may greatly exceed three. Waiting to collect a batch also requires evidence that the delay fits the user deadline.

Suppose four applications arrive between 09:00 and 09:20 the next day with scores 0.54, 0.56, 0.58, and 0.60, while the on-duty team can finish only three reviews before a promised 18:00 deadline. Reusing the previous batch’s 0.52–0.68 review band sends all four to people. When the fourth arrives, the slots are full, yet this case is closest to the original automatic-decision crossover. The system should visibly mark it pending and immediately ask the queue owner for added capacity. If nobody can take it before the deadline, disclose the missed promise, escalate this application, and stop promising the same-day route to new applicants. It must not silently approve or reject the overflow case. This is another invented stress exercise, showing that cutoffs inferred from ten known cases do not guarantee review capacity in every future period.

Use a separate recent-like sample to estimate reviews per period, then replay actual arrival order, bursts, review duration, and deadlines. Name the queue owner, maximum backlog, tie handling, and action when capacity runs out: visibly leave the case pending for review within its permitted time, or narrow or pause the affected automatic workflow while adding review capacity. Never silently convert a full review queue into automatic approvals or rejections. If error costs or reviewer accuracy differ by case, the value ranking may no longer fit one global pair of cutoffs; write separately approved slice rules instead.

Real human review is neither error-free nor delay-free. A production loss ledger must include reviewer mistakes, requests for more information, waiting, appeals, and recovery. Give reviewers the policy version, evidence, and available actions. The human review checkpoint guide explains how to place a gate before the outcome becomes expensive to change.

Validate on untouched cases and after launch

Freeze the cutoffs and overflow route before opening the final holdout. At minimum, report automatic approvals and their false positives, automatic rejections and their false negatives, review volume and outcomes, estimated cost for the whole set and important slices, and queue peaks and deadline misses by period. State the denominator of every rate. Reviewed cases are not “automatic predictions that were correct,” and one accuracy number cannot hide a rare severe mistake.

After launch, independently audit authorized samples from both automatically approved and automatically rejected cases, and connect outcomes that arrive later. Looking only at the review queue or appeals creates selection bias; an automatic rejection may even prevent a positive outcome from becoming visible without independent adjudication. Watch positive prevalence, probability reliability, task slices, reviewer overrides, and queue peaks. The workflow drift watchboard connects change signals to narrowing, recalibration, or pausing a route. Revalidate after a policy, model, calibrator, cost, or traffic change rather than merely moving a cutoff.

Fill the threshold and routing sheet

The workbench below is fillable, copyable, and exportable. Begin with synthetic IDs or sanitized aggregates; it saves a draft in this browser. Another reviewer should be able to recalculate the routing from the recorded costs, scores, capacity, and evidence, then point out which cases retain risk after capacity runs out.

Boundary

This is a method for assigning actions in a defined binary task. It does not make eligibility policy or turn population-level calibration into a guarantee for an individual. High-impact, regulated, or hard-to-appeal decisions may be unsuitable for automatic rejection even if expected loss looks lower. If reference labels are disputed, scores cannot be calibrated, or reviewer capacity and quality are unknown, narrow automation and repair the evidence base first.