Top-P (Nucleus Sampling)

Top-P (also called nucleus sampling, introduced by Holtzman et al. in 2019 and published at ICLR 2020) is an alternative to temperature for controlling output randomness. At each step the next token probabilities are sorted and the smallest set of tokens whose cumulative probability reaches P is kept. The rest are discarded and the kept probabilities are renormalized before sampling. With P at 0.9, only the tokens that together hold the top 90 percent of the probability mass can be chosen. Because the cutoff follows the shape of the distribution, the nucleus is small when the model is confident and large when many continuations are plausible, which a fixed top K cutoff cannot do. The aim is diversity without sampling the unreliable tail.

Provider support has changed, so check before you tune. OpenAI documents top_p as an alternative to temperature. Anthropic deprecated it: the Messages API reference says models released after Claude Opus 4.6 do not support setting top_p, and the documentation says temperature, top_p and top_k are not supported on Claude 4.7 and later models, where a non default value returns a 400 error. On those models you steer style with prompting instead.

def top_p_filter(probs: dict[str, float], p: float) -> dict[str, float]:
    kept, total = {}, 0.0
    for token, prob in sorted(probs.items(), key=lambda kv: kv[1], reverse=True):
        kept[token] = prob
        total += prob
        if total >= p:
            break
    return {token: round(prob / total, 3) for token, prob in kept.items()}


probs = {"the": 0.5, "a": 0.3, "an": 0.15, "zebra": 0.05}
print(top_p_filter(probs, 0.9))  # zebra is cut, the rest are renormalized

A pure Python illustration of the cutoff. Inference engines apply it to the full vocabulary before sampling.

Where the API still allows it, tune top P or temperature, not both. Values around 0.9 to 0.95 trim the unlikely tail while keeping variety. Setting temperature to 0 already picks the most likely token, so top P has no effect there. For extraction and tool calls, constrain the output with a schema instead of tuning sampling. On models that reject sampling parameters, leave them out.

• Holtzman et al. 2019: Nucleus Sampling paper
• OpenAI: API parameters
• Anthropic: API reference
• Anthropic: Using the Messages API

Related Terms