Human Preferences in Encoding Uncertainty During Parsing

When a parser is uncertain, there are multiple valid parses or low-confidence decisions. A key human-centered question is not only “how to compute uncertainty,” but “how humans want uncertainty encoded while parsing”—e.g., in UI, explanations, or interactive annotation. Human preferences typically trade off between:

• Accuracy of the uncertainty signal (does it reflect real likelihoods?)
• Actionability (can the user decide what to do next?)
• Cognitive load (is the uncertainty representation easy to interpret?)
• Calibration and trust (do confidence levels match observed correctness?)

In probabilistic parsing, model uncertainty is often represented as posterior probabilities over parses/labels, derived from probabilistic grammars or neural models with probabilistic outputs. Encoding uncertainty “during parsing” can mean showing it incrementally as the parse is constructed (early decisions), post-hoc (final parse with confidence), or as distributions (n-best lists, forest weights, entropy). Human preferences vary by task context: e.g., interactive annotation vs. explanation to end users.

Core concept map (from model uncertainty → user-facing representation):

Key keyword: probability of a parse/label given the observed sentence under a probabilistic model.
Key keyword: agreement between predicted confidence and empirical correctness.
Key keyword: scalar or structured metric (entropy, margin, variance) derived from a distribution.
Key keyword: compact structure representing many possible parses with weights.

Practical takeaway: Most human-friendly uncertainty encodings are not raw probabilities—they are summaries that help users decide and reduce confusion.

What “encoding uncertainty during parsing” can mean

Across NLP and human-computer interaction, “encoding uncertainty during parsing” commonly falls into four interface/annotation patterns:

1. Best-choice confidence: show the chosen parse and a confidence score (e.g., “high/medium/low”).
2. Alternatives with weights: show top-$k$ parses/edges with probabilities.
3. Structured uncertainty: highlight uncertain spans/relations in a parse tree/graph (forest-like visualization).
4. Interactive clarification: ask the user targeted questions when uncertainty crosses a threshold.

Human preference typically follows a hierarchy:

• When users mainly need correct final outputs, they prefer compact encodings (best parse + confidence).
• When users need diagnostics (why wrong, how to fix), they prefer richer encodings (alternatives, structured uncertainty).
• When the task is high-stakes or ambiguous, they prefer interactive clarification over passive probability display.

Key keyword: measure of uncertainty for a categorical distribution; higher means more spread.
Key keyword: difference between the top two probabilities; small margin implies ambiguity.
Key keyword: abstaining on low-confidence cases to improve accuracy on answered cases.
Key keyword: selecting the most informative clarification question.

Human preferences: why some encodings work better than others

Human interpretation of uncertainty is shaped by several psychological/behavioral factors:

• Numeracy and interpretability: Many users struggle to reason about precise probabilities, so discrete labels (“likely/unlikely”) and ranked alternatives are often preferred.
• Cognitive effort: Structured uncertainty (highlighting uncertain edges/spans) can be more informative but also more visually complex.
• Trust and calibration: If displayed confidence does not match real error rates, users either ignore the signal (overconfident wrong outputs) or distrust the system (underconfident correct outputs).
• Actionability under time pressure: In interactive parsing/annotation, users prefer uncertainty cues that minimize extra effort (e.g., only ask when necessary; present top-$k$ options only when they are plausible).

A useful mental model is to treat uncertainty encoding as a communication channel from model beliefs to human decisions. Good encodings maximize expected utility for the user, not necessarily the raw informativeness of the underlying distribution.

Key keyword: decision-theoretic criterion: choose the option maximizing expected user value.
Key keyword: mental effort required to interpret and act on information.
Key keyword: how well user trust tracks actual model reliability.