Skip to main content

Methodology & Architecture

Test📄 Callout:Paper: NativeCost-Aware HTMLHuman-LLM Collaboration for Post-OCR Corrections

👥 Authors: Stergios Konstantinidis, Hayman Lotfy, Michalis Vlachos • Department of Information Systems (DESI), HEC Lausanne, UNIL

🏛️ Venue: ACM Symposium on Document Engineering (DocEng) • Short Paper Track

🔗 Repositories: GitHub (Public) • Overleaf Project (6a2cb9...)

1. System Architecture: Three-Tier Collaborative Routing

Preserving digitized historical archives requires mitigating transcription degradation without incurring unsustainable computational inference costs or inducing catastrophic hallucination. Naive application of Large Language Models (LLMs) across entire archival collections results in prohibitive expenses, vocabulary modernization, and frequent regression of correctly transcribed rare historical entities.

Our architecture introduces a three-tier routing paradigm that dynamically allocates each text segment to one of three optimal treatment paths based on learned degradation profiles and empirical confidence safeguards:

Collaborative Human-LLM Post-OCR Correction Architecture

Figure 1: Collaborative Human-LLM Post-OCR Correction Architecture with Dynamic Confidence Routing and Post-Correction Safeguard Loop.

Class 0: Bypass Route (Direct to Archive)

Segments identified as already clean or high-confidence bypass LLM correction completely. Saves 100% token cost and eliminates any risk of stylistic hallucination or diacritic corruption.

Class 1: Automated LLM Correction

Moderately degraded segments are dispatched to historical prompting pipelines (Gemini / GPT / Llama) with few-shot archival examples and strict orthographic grounding constraints.

Class 2: Priority Human Verification (<5%)

Severe segmentation failures, empty OCR blocks, or segments intercepted by the post-correction safeguard are routed to professional archivists under a strict <5% human time budget.

2. Mathematical Formulation & Quality Metrics

Let D = {d1, …, dn} denote a digitized archival corpus. Each document segment di possesses an original raw OCR transcription diraw, a candidate machine correction dicorr, and a ground-truth transcription dihuman.

We quantify correction quality via the Character Error Rate (CER) reduction benefit Δi:

Δi = CER(diraw, dihuman) − CER(dicorr, dihuman)
    Positive Benefit (Δi > 0): The machine correction successfully lowered transcription noise and improved readability. Negative Regression (Δi < 0): The LLM degraded accuracy, hallucinated false cognates, or altered archaic syntax. The post-correction safeguard is working!designed to catch these regressions.

    3. Feature Extraction & LASSO Degradation Modeling

    Prior to invoking expensive generative models, the router extracts 54 lightweight features capturing multi-faceted document degradation without requiring ground truth annotations:

    Feature Family Extracted Indicators Routing Rationale Confidence Metrics Mean token confidence, lowest quartile confidence, token variance Low confidence triggers LLM or human verification Lexical & Orthographic Out-of-vocabulary ratio against 18th/19th c. Swiss French lexicons, symbol density Distinguishes historical spelling from OCR garbage characters Perplexity & Language Model CamemBERT cross-entropy loss, token log-likelihoods Captures ungrammatical syntax disruptions caused by broken line breaks Typographic & Layout Bounding box aspect ratio, average word length, uppercase-to-lowercase transitions Identifies broken column cuts and hyphenation splits
    LASSO Feature Selection Coefficients

    Figure 2: LASSO Feature Selection Coefficients identifying the strongest predictive drivers of post-OCR correction benefit (Δi).

    4. Safeguard Routing Curves & Verification Budget

    A critical vulnerability in all-LLM correction pipelines is the silent corruption of historical names and rare dates. To eliminate this risk, our framework deploys a lightweight Post-Correction Safeguard classifier evaluated on the pair (diraw, dicorr):

      Computes normalized Levenshtein distance, word count divergence, and proper noun preservation. If the safeguard predicts that the candidate correction introduced regressions (CER(dicorr) > CER(diraw)), the text is intercepted prior to writing to the database and escalated directly to human review.
      Empirical Safeguard Routing Curves

      Figure 3: Empirical Safeguard Routing Boundaries balancing Character Error Rate reduction against the strict human verification budget (<5%).

      Key Architectural Takeaways

        Cost Reduction: Bypassing 60%+ of clean segments cuts API inference expenditure by more than half. Guaranteed Quality: The dual-gate architecture (Router + Safeguard) prevents hallucinated modernizations from ever corrupting the archival repository. Human Efficiency: Prioritizing human review exclusively on high-uncertainty and safeguard-flagged segments maximizes archival accuracy per reviewer hour.
        Back to top