Confidence Calibration

v0.6.7

Large language models systematically produce overconfident extraction scores — raw confidence values cluster between 0.85 and 1.0 regardless of actual correctness. This makes confidence-based filtering unreliable.

DocIntel applies temperature scaling to calibrate raw LLM confidence scores before they reach the per-profile threshold filter.

The problem

Without calibration:

• A hallucinated entity gets confidence 0.88
• A correct entity gets confidence 0.92
• Your threshold of 0.75 lets both through
• The hallucinated entity pollutes your knowledge graph

How it works

The _calibrate_confidence() method applies a logit → scale → sigmoid transform:

calibrated = σ(logit(raw) / T)

Where:

• logit(p) = ln(p / (1 − p)) — converts probability to log-odds
• T = 2.0 — temperature parameter (halves the log-odds)
• σ(x) = 1 / (1 + e^−x) — sigmoid converts back to probability

Calibration table

Raw LLM score	Calibrated score	Effect
0.99	~0.95	High-confidence entities stay high
0.95	~0.82	Moderate reduction
0.90	~0.74	Significant spread
0.85	~0.68	Falls below 0.75 threshold → filtered out
0.80	~0.62	Large correction
0.70	~0.58	Aggressive correction for mid-range
0.50	0.50	Fixed point — no change

Where it runs in the pipeline

Pass 1: Entity extraction (parallel windows)
Pass 2: Relationship extraction (parallel windows)
    ↓
Fuzzy deduplication (rapidfuzz WRatio ≥ 88)
    ↓
★ Confidence calibration (T = 2.0)          ← HERE
    ↓
Per-profile confidence threshold filter
    ↓
EmitPayload → Purple8 Graph

Calibration runs after fuzzy dedup (so duplicate scores are already merged) and before the threshold filter (so thresholds operate on calibrated values).

Results

Expected Calibration Error (ECE)

ECE measures how well confidence scores predict actual correctness. Lower is better.

Domain	Before (v0.6.6)	After (v0.6.7)	Target
Invoices	0.30	0.13	0.08–0.20
Contracts	0.45	0.42	—
Scientific	0.22	0.18	—
BIM Models	0.38	0.32	—

Configuration

The temperature parameter T = 2.0 is currently hardcoded. Higher values produce more aggressive calibration:

T value	Behaviour
1.0	No calibration (identity)
1.5	Mild calibration
2.0	Default — halves log-odds
3.0	Aggressive calibration

TIP

The calibration is applied to both entity and relationship confidence scores. The fixed point at 0.50 means entities that the LLM is genuinely uncertain about are not affected.

Technical details

• Edge cases: Scores of exactly 0.0 and 1.0 are clamped to [1e-9, 1 − 1e-9] before logit to avoid infinity
• Idempotent: Applying calibration twice does further compress scores, so it is called exactly once
• 238 tests passing including 8 dedicated calibration tests