Dataset & Benchmark Results
Paper: DocEng Short Paper - Cost-Aware Human-LLM Collaboration for Post-OCR Corrections
Authors: Stergios Konstantinidis, Hayman Lotfy, Michalis Vlachos (University of Lausanne)
Venue: ACM Symposium on Document Engineering (DocEng) - Short Paper
GitHub: https://github.com/Stergios-Konstantinidis/Cost-Aware-Human-LLM-Collaboration-for-post-OCR-Corrections
Overleaf: https://git.overleaf.com/6a2cb93de1bec86186ee8613
Experimental Benchmark & Evaluation
1. Archival Corpus
Evaluated across 609 text segments spanning 45 digitized pages from 9 Swiss French-language historical periodicals (Canton of Vaud, 1733–1945):
| Periodical | Era | Issues | Pages | Segments | Characteristics |
|---|---|---|---|---|---|
| La Revue | 1875–1945 | 4 | 5 | 139 | 19th/20th century serif, multi-column |
| Feuille d'Avis | 1762–1841 | 4 | 6 | 118 | Antique typography, long 's' ligatures |
| Mercure Suisse | 1733–1748 | 3 | 5 | 92 | 18th-century early modern French |
| Gazette de Lausanne | 1804–1920 | 5 | 8 | 145 | High density, complex layout |
| Others (Tribune, etc.) | 1880–1930 | 5 | 21 | 115 | Mixed commercial & political press |
2. Key Benchmark Results
Error Rate Comparison (Tesseract OCR Base):
- Raw OCR Baseline: $CER = 6.30%$, $WER = 14.20%$
- All-LLM Baseline (GPT-4o / Gemini 1.5 Flash): $CER = 2.92%$ (Cost: 100% token budget)
- Selective Pre-Routing (Ours): $CER = 3.12%$ (Bypasses 40% of all segments!)
- Collaborative Safeguard (<5% Human Budget): $CER = 2.49%$ (14% relative reduction over All-LLM)
Comparison with Routing Baselines:
- Random Routing: Weakest; frequently sends clean text and degrades it.
- OCR Confidence (Thresholding): Underperforms because Tesseract confidence scores poorly correlate with semantic error in archaic fonts.
- ConfBERT: Good detection but computationally heavy.
- Our Regression Router + Safeguard: Consistently establishes the Pareto frontier across all budget limits.
Experimental Evaluation & Visual Benchmark Plots
Historical Newspaper Segmentation Sample
Figure 1: Representative historical archival newspaper segment illustrating layout artifacts and OCR degradation.
CER Reduction Across Correction Strategies
Figure 2: Character Error Rate (CER) reduction across baseline OCR, All-LLM, ConfBERT, and Collaborative Routing.
OCR Engine Improvement Comparison
Figure 3: Relative error reduction across historical OCR engines (Tesseract, Abbyy, and modern HTR).
Strategy Performance Heatmap
Figure 4: Comprehensive performance heatmap comparing LLM prompt strategies across historical periodicals.