Skip to main content

Methodology & Architecture

Paper: DocEng Short Paper - Cost-Aware Human-LLM Collaboration for Post-OCR Corrections
Authors: Stergios Konstantinidis, Hayman Lotfy, Michalis Vlachos (University of Lausanne)
Venue: ACM Symposium on Document Engineering (DocEng) - Short Paper
GitHub: https://github.com/Stergios-Konstantinidis/Cost-Aware-Human-LLM-Collaboration-for-post-OCR-Corrections
Overleaf: https://git.overleaf.com/6a2cb93de1bec86186ee8613

Methodology: Three-Tier Collaborative Routing & Safeguard

1. System Architecture

[ Raw OCR Transcription ]
          │
          ▼
 [ 54 Feature Extraction ] ── (OCR Confidence, Perplexity, Typographic Norms)
          │
          ▼
 [ Regression Router ] ────► Bypass Correction (CER = 0 / Clean)
          │
          ├───────────────► LLM Correction Pipeline (Gemini / GPT / Llama)
          │                        │
          │                        ▼
          │              [ Post-Correction Safeguard ]
          │                        │
          │                   Pass │  Fail (Regression detected)
          │                        │         │
          ▼                        ▼         ▼
[ Severe OCR Failures ] ───────► [ Human Review Queue (<5% Budget) ]
                                             │
                                             ▼
                                   [ Final Archive Text ]

2. Mathematical Formulation

Let $\mathcal{D} = {d_1, \dots, d_n}$ be a corpus of segments. Each segment has raw OCR text $d_i^{\text{raw}}$, potential LLM correction $d_i^{\text{corr}}$, and human ground truth $d_i^{\text{human}}$.

Decision Paths:

  • Class 0 (Bypass): No correction applied. Saves 100% token cost and prevents modernization.
  • Class 1 (LLM Correction): Sent to LLM with zero-shot / few-shot historical prompt.
  • Class 2 (Human Correction): Bypasses LLM directly to human annotators (e.g. for complete OCR hallucination or empty text $d_i^{\text{raw}} = \text{''}$).

Quality Metric & Benefit:

$$\Delta_i = CER(d_i^{\text{raw}}) - CER(d_i^{\text{corr}})$$

The router predicts $\hat{\Delta}_i$ using a fast linear regressor ($<0.1$s per batch) on 54 surface and statistical features.

3. Post-Correction Safeguard Mechanism

Even with optimal pre-routing, LLMs occasionally produce catastrophic errors on historical names and archaic terms. The Safeguard Classifier inspects the pair $(d_i^{\text{raw}}, d_i^{\text{corr}})$:

  • Measures edit distance, length divergence, vocabulary drift, and historical dictionary match.
  • If the safeguard predicts $CER(d_i^{\text{corr}}) > CER(d_i^{\text{raw}})$, the segment is intercepted before committing to the archive and routed to a human annotator.
Back to top