Confidence Scoring

Confidence scoring assigns a numeric or categorical reliability estimate to each agent output. Common approaches: ask the model to rate its own confidence ("how sure are you, 1-10?"), inspect token-level log probabilities (when the API exposes them), use a separate classifier model to score outputs, or compare against retrieval-grounded sources. The scores then drive routing decisions: high-confidence outputs go straight to the user, medium-confidence go to a self-correction loop, low-confidence escalate to human review. Confidence scoring is what makes human-in-the-loop and human-on-the-loop patterns workable at scale.

A persistent problem: model self-rated confidence is poorly calibrated. A model that says "I am 90% confident" is often wrong much more than 10 percent of the time on out-of-distribution inputs. Two practical fixes. First, calibrate against your own data: log model confidence vs actual correctness, then learn the mapping from raw score to true probability. Second, prefer behavior-based confidence (did the model reach a stable answer across multiple sampled runs? Did retrieval find supporting evidence?) over self-reported confidence. The behavioral signals are more reliable than the asked-for ones.

Add confidence scoring as soon as you have variable-stakes decisions where some need human review and some do not. The score is what routes work between automation and humans.

• Anthropic: Building Effective Agents
• OpenAI: Calibrated language models

Related Terms