DocEng Short Paper

• Overview & Executive Summary
• Methodology & Architecture
• Dataset & Benchmark Results
• Codebase & Reproduction Guide
• Presentation slides

Overview & Executive Summary

1. Research Motivation & Core Problem

Historical document archives, such as digitized centuries-old Swiss newspapers, suffer from substantial OCR degradation caused by aged paper, bleed-through, complex multi-column typography, and antique fonts.

While modern Large Language Models (LLMs) excel at OCR correction, naive all-to-LLM pipelines suffer from three severe pitfalls:

1. Financial Cost: Processing millions of archive pages with commercial LLM APIs is economically non-viable.
2. Degradation of Clean Text: LLMs frequently "hallucinate" or modernize archaic French spellings on segments that were already correctly transcribed by the OCR engine.
3. Severe OCR Failures: On heavily damaged regions, LLMs fabricate plausible-sounding text instead of preserving historical fidelity.

This paper formulates OCR post-correction as an optimal budget allocation and routing problem uniting human domain experts and LLMs.

2. Key Contributions

• Three-Tier Human-LLM Collaboration: Explicit routing of each segment to either *(1) No Correction*, *(2) Automated LLM Correction*, or *(3) Human Review*.
• Regression-Guided Prioritization: Learned Lasso/Ridge router predicting expected Character Error Rate (CER) reduction \hat{Δ}_i before sending tokens to an LLM.
• Post-LLM Safeguard Layer: Classification mechanism detecting when LLM correction introduced regression or hallucination, diverting compromised outputs to human reviewers.
• Superior Budget Efficiency: Achieves 14% relative CER reduction over the All-LLM baseline with <5% human review budget.

Methodology & Architecture

1. System Architecture: Three-Tier Collaborative Routing

Preserving digitized historical archives requires mitigating transcription degradation without incurring unsustainable computational inference costs or inducing catastrophic hallucination. Naive application of Large Language Models (LLMs) across entire archival collections results in prohibitive expenses, vocabulary modernization, and frequent regression of correctly transcribed rare historical entities.

Our architecture introduces a three-tier routing paradigm that dynamically allocates each text segment to one of three optimal treatment paths based on learned degradation profiles and empirical confidence safeguards:

Figure 1: Collaborative Human-LLM Post-OCR Correction Architecture with Dynamic Confidence Routing and Post-Correction Safeguard Loop.

2. Mathematical Formulation & Quality Metrics

Let D = {d₁, …, d_n} denote a digitized archival corpus. Each document segment d_i possesses an original raw OCR transcription d_i^raw, a candidate machine correction d_i^corr, and a ground-truth transcription d_i^human.

We quantify correction quality via the Character Error Rate (CER) reduction benefit Δ_i:

• Positive Benefit (Δ_i > 0): The machine correction successfully lowered transcription noise and improved readability.
• Negative Regression (Δ_i < 0): The LLM degraded accuracy, hallucinated false cognates, or altered archaic syntax. The post-correction safeguard is designed to catch these regressions.

3. Feature Extraction & LASSO Degradation Modeling

Prior to invoking expensive generative models, the router extracts 54 lightweight features capturing multi-faceted document degradation without requiring ground truth annotations:

Figure 2: LASSO Feature Selection Coefficients identifying the strongest predictive drivers of post-OCR correction benefit (Δ_i).

4. Safeguard Routing Curves & Verification Budget

A critical vulnerability in all-LLM correction pipelines is the silent corruption of historical names and rare dates. To eliminate this risk, our framework deploys a lightweight Post-Correction Safeguard classifier evaluated on the pair (d_i^raw, d_i^corr):

• Computes normalized Levenshtein distance, word count divergence, and proper noun preservation.
• If the safeguard predicts that the candidate correction introduced regressions (CER(d_i^corr) > CER(d_i^raw)), the text is intercepted prior to writing to the database and escalated directly to human review.

Figure 3: Empirical Safeguard Routing Boundaries balancing Character Error Rate reduction against the strict human verification budget (<5%).

Dataset & Benchmark Results

Experimental Benchmark & Evaluation

1. Archival Corpus

Evaluated across 609 text segments spanning 45 digitized pages from 9 Swiss French-language historical periodicals (Canton of Vaud, 1733–1945):

2. Key Benchmark Results

Error Rate Comparison (Tesseract OCR Base):

• Raw OCR Baseline: CER = 6.30\%, WER = 14.20\%
• All-LLM Baseline (GPT-4o / Gemini 1.5 Flash): CER = 2.92\% (Cost: 100% token budget)
• Selective Pre-Routing (Ours): CER = 3.12\% (Bypasses 40% of all segments!)
• Collaborative Safeguard (<5% Human Budget): CER = 2.49\% (14% relative reduction over All-LLM)

Comparison with Routing Baselines:

1. Random Routing: Weakest; frequently sends clean text and degrades it.
2. OCR Confidence (Thresholding): Underperforms because Tesseract confidence scores poorly correlate with semantic error in archaic fonts.
3. ConfBERT: Good detection but computationally heavy.
4. Our Regression Router + Safeguard: Consistently establishes the Pareto frontier across all budget limits.

Experimental Evaluation & Visual Benchmark Plots

Historical Newspaper Segmentation Sample

Figure 1: Representative historical archival newspaper segment illustrating layout artifacts and OCR degradation.

CER Reduction Across Correction Strategies

Figure 2: Character Error Rate (CER) reduction across baseline OCR, All-LLM, ConfBERT, and Collaborative Routing.

OCR Engine Improvement Comparison

Figure 3: Relative error reduction across historical OCR engines (Tesseract, Abbyy, and modern HTR).

Strategy Performance Heatmap

Figure 4: Comprehensive performance heatmap comparing LLM prompt strategies across historical periodicals.

Codebase & Reproduction Guide

Codebase Structure & Execution

1. Directory Overview

• paper/: LaTeX source files, TikZ diagrams (overview_tikz.tex), figures, and presentation slides.
• code/plotting/: Scripts generating progressive performance curves and slide assets (generate_ppt_assets.py).
• code/evaluation/: Metric computation (CER, WER, edit distance, cost modeling).
• ppt assets/: Rendered publication assets (PDF/PNG curves for baseline, random, ConfBERT, ours, safeguard, oracle).
• DocEng presentation.pptx: Official presentation slide deck.

2. Key Commands

# Clone the repository
git clone https://github.com/Stergios-Konstantinidis/Cost-Aware-Human-LLM-Collaboration-for-post-OCR-Corrections.git
cd Cost-Aware-Human-LLM-Collaboration-for-post-OCR-Corrections

# Generate publication and presentation assets
python3 code/plotting/generate_ppt_assets.py

# Overleaf synchronization (pull before modify, push after)
git pull origin main
git push origin main

Presentation slides

Slide Deck & Presentation Breakdown

Slide 1: Title & Authors

Slide 01 / 10

Speaker Note: Cost-Aware Human-LLM Collaboration for Post-OCR Corrections in Swiss Historical Newspapers (DocEng 2026, 28.08.2026). Presented by Stergios Konstantinidis, Hayman Lotfy, Prof. Michalis Vlachos.

Slide 2: Motivation & Archival Scale

Slide 02 / 10

Speaker Note: Building RAG systems for historical archives across 300+ years of newspapers, millions of pages, and 12 TB of raw high-resolution scan imagery.

Slide 3: Why Document Cleaning is Not Straightforward

Slide 03 / 10

Speaker Note: Post-OCR correction is essential for downstream retrieval, but traditional tools neglect historical linguistic shifts and archaic orthography.

Slide 4: Heterogeneous Segment Utility

Slide 04 / 10

Speaker Note: Different segments benefit very differently: Segment D gains +2.0% CER improvement with zero overcorrections, while Segment B suffers -3.0% CER degradation due to 10 overcorrections!

Slide 5: Optimal Correction Prioritization

Slide 05 / 10

Speaker Note: Theoretical premise: What if we knew in advance which archival documents would benefit most from LLM correction?

Slide 6: The Real-World Dilemma

Slide 06 / 10

Speaker Note: In reality, ground truth is unavailable. Naively correcting all documents produces unpredictable degradation and wastes massive token budgets.

Slide 7: Our Contribution - Intelligent Routing Architecture

Slide 07 / 10

Speaker Note: A machine-learned Router evaluating expected CER benefit: If LLM is sufficient -> route to LLM; if degradation is too severe -> route to human review (<5%); otherwise leave the segment untouched.

Slide 8: Experimental Evaluation & Benchmark Curves

Slide 08 / 10

Speaker Note: Empirical results comparing baseline OCR, All-LLM, ConfBERT, and our Cost-Aware Routing framework across historical Swiss periodicals.

Slide 9: Key Takeaways & Research Streams

Slide 09 / 10

Speaker Note: Key contributions: 40% reduction in correction effort, negligible setup cost, and an effective human guardrail identifying segments requiring expert archivist attention.

Slide 10: Conclusion & Contact

Slide 10 / 10

Speaker Note: Wrap-up, open questions, and research collaboration contact information at UNIL-DESI.