Lesson 5.5Lesson 29 of 30
Human review and confidence calibration
Design human review workflows and confidence calibration
Key points
- 1Aggregate accuracy hides weak segments: 97% overall can coexist with 80% on one supplier's invoice layout or on one field such as
tax_id. Break accuracy down by document type and by field before you reduce human review for anything. - 2Raw self-reported confidence is an unreliable proxy. Field-level confidence becomes a valid routing signal only after its thresholds are calibrated on a labelled validation set: you measure the real accuracy in each score band, per field, and set the threshold where it meets your target.
- 3Automated (high-confidence) output still needs measurement: a stratified random sample by document type, field and confidence band gives error rates for every segment and surfaces novel error patterns. Uniform random sampling barely touches rare document types.
- 4Reviewer capacity is limited, so route by need: fields below the calibrated threshold and extractions where sources are ambiguous or contradict each other go to humans first. Reviewing everything is not a design; it just moves the problem to people.
What the exam guide checks here
Objective wording from the exam guide (English).
- 5.5.K1Aggregate accuracy (97%) can hide weak document types or fields
- 5.5.K2Stratified random sampling measures error rates in high-confidence output
- 5.5.K3Field-level confidence calibrated on labelled validation sets routes review
- 5.5.K4Validate accuracy per document type and field before automating
- 5.5.S1Stratified sampling of high-confidence extractions for error rates and novel patterns
- 5.5.S2Analyse accuracy by document type and field before reducing review
- 5.5.S3Field-level confidence with thresholds calibrated on labelled data
- 5.5.S4Route low-confidence or contradictory-source items to humans first
Lock it in
Answer scenario questions on this topic. Every option comes with an explanation.
Practise 5.5 · 21 questions