CIKM Short Paper: Knowing When to Correct
[!NOTE]Overleaf Project:Venue:git.overleaf.com/6a1d78e803cbdf7def32a839
ACMLastInternationalSynced:Conference2026-09-29on11:35:05Information and Knowledge Management (via PA Git Sync)CIKM)Source File:paper/main.tex
Knowing When to Correct: Cost-Aware LLM Routing for OCR Post-Correction in Historical Documents
Affiliation:
[!ABSTRACT] Historical digital archives present new challenges for retrieval systems becauseDepartment ofOCRInformationerrorsSystemsintroduced(DESI),byHECdegradedLausanne
GitHub Public: CIKM_public
Overleaf Project:https://git.overleaf.com/6a1d78e803cbdf7def32a839
Core Codebase:/Users/stergios/Documents/GitHub/Optimizing-LLM-based-OCR-Corrections
1. Problem Statement
At multi-million page
quality, old typefaces, and complex page layouts. Large language models (LLMs) offer an effective means of correcting transcription errors. However, applying LLM-based correction at scale can become prohibitively expensive. We investigate a selective, cost-aware strategy for LLM-assisted OCR post-correction, using a regression-guided router to predict when LLM correction is likely to improve transcription quality, thereby reducing unnecessary LLM processing and associated costs while preserving historically valid text. We evaluate this strategy on a dataset comprising historical Swiss newspapers published between 1733 and 1945. Our results show that selective correction can retain most of the quality gains of full correction while substantially reducing the LLM token cost. Using Tesseract OCR, sending all text segments for LLM correction improves CER from 6.3% to 2.9% (best approach). In contrast, our regression-guided router achieves CER of 3.1% (0.2% worse) while avoiding routing 40% of segments for LLM correction. Finally, we show that our regression-guided LLM router closely trails the decisions of an oracle that knows in advance which segments benefit from correction.
\city{Lausanne}
\country{Switzerland}}
\city{Lausanne}
\country{Switzerland}}
\city{Lausanne}
\country{Switzerland}}
\renewcommand{\shortauthors}{Konstantinidis et al.}
\begin{CCSXML}
<concept_id>10010405.10010497.10010504.10010505</concept_id>
<concept_desc>Applied computingDocument analysis</concept_desc>
<concept_significance>500</concept_significance>
<concept_id>10010147.10010178.10010179.10010182</concept_id>
<concept_desc>Computing methodologiesNatural language generation</concept_desc>
<concept_significance>300</concept_significance>
\end{CCSXML}
\keywords{OCR Correction, Large Language Models, Historical Newspapers, Cost-Effectiveness, Document Analysis, Selective Routing}
Introduction
Historical newspapers are valuable sources for historians, social scientists, and cultural heritage researchers. However, many digitized archives remain difficult to use because OCR systems often introduce transcription errors on historical material [piryani2025evaluating]. Old typefaces, damaged paper, ink bleed-through, irregular spelling, and complex layouts all contribute to high word-error-rates (WER) [holley2009good,alex2012digitised].
Large language models (LLMs) can correct many OCR errors after transcription using textual context. However, using an LLM for refining every OCR output can be costly at archive scale and may be unnecessary when the OCR text is already accurate. In some cases, correction can even introduce new errors by altering historically valid spelling or typography.
This paper addresses the problem of reducing the cost ofscale, LLM-based OCR post-correction while sacrificing as little transcription quality as possible. We study whether information available at OCR time, especially character-level confidence scores, can guide the selection of text segments that should be submitted to an LLM for correction. The proposed approach is evaluated on historical newspaper material from the Canton of Vaud, Switzerland, comprising nine publications issued between 1733 and 1945. Through this evaluation, we make two key contributions:
* First, we formulate cost-aware post-OCR correction as a lightweight binary routing problem and propose a regression-guided router that predicts the expected correction benefit of each segment relative to its token cost.
* Second, on historical newspaper archives across multiple OCR engines and LLMs, we demonstrate that our router closely trails an optimal oracle, capturing more than $96%$ of full-correction CER (character-error-rate) gains while reducing token costs by more than $40%$.
Related Work
Post-OCR correction is graduallyconstrained movingby fromcost:
Core accordingResearch to recent studies [10.1007/978-3-031-70442-0_13].Questions:
\medskip\noindent
\medskip\noindentLanguage Models.
Lately, LLMs are being increasingly used across various stages of the OCR and post-OCR pipeline, though not without challenges; recent work has shown that LLMs can improve post-OCR correction for historical newspapers [thomas2024leveraging].
However, evaluations across historical transcription benchmarks show that LLM performance varies substantially with model choice, prompt design, document type, language, and initial transcription quality [boros2024post].
Sohail et al. [sohail2024decipheringunderservedbenchmarkingllm] benchmarked multimodal models like GPT-4o for low-resource scripts, noting that while LLMs offer context-aware alternatives to conventional OCR, their performance remains sensitive to visual distortions and script complexity in zero-shot settings. Comprehensive benchmarks such as CC-OCR [yang2024ccocrcomprehensivechallengingocr] further emphasize the challenges of using LLMs for complex literacy tasks. Shi and Chen [10.1117/12.3068186] route OCR outputs when confidence falls below 0.95; however, raw confidence ignores text length and error repairability. Multi-LLM routers (e.g., RouteLLM [Ong2025RouteLLMLT, li2025costaware], PILOT [panda2025pilot], ParetoBandit [tabernermiller2026paretobandit]) dispatch queries between strong and weak LLMs, but still invoke an LLM for every request via dense embeddings. In contrast, our router decides whether each segment should be sent to an LLM for correction at all, using lightweight features that require no GPU-based inference at routing time.
\medskip\noindentHistorical Archives and Digital Humanities.
Historical document digitization presents challenges that standard OCR pipelines often fail to address. Backer and Hyman [backer-hyman-2025-bootstrapping] point out that WER and CER can understate how useful a system is for historical research, and they call for discipline-driven evaluations'' that match the needs of humanities scholars. A related problem is the *modernization trap* [hammarstrom2025investigation], where a cleaning step rewrites old spellings into their modern form. For example, the OCR may correctly read the historical French word *seroit*, but the cleaning step then changes it to the modern *serait* (*seroit* $\rightarrow$ *serait*). This kind of correction'' loses the original spelling and weakens the fidelity of the source.
Our approach, by combining domain-specific features with selective, cost-aware routing, aims to balance LLM benefits against the cost and fidelity requirements of large-scale historical archives.
Methodology
We approach OCR post-correction as a selective routing problem: rather than
sending every transcription to an LLM, we decide per segment whether
correction is likely to improve transcription quality, using only signals available before any correction is run (see Figure [fig:methodology]). This section formalizes that decision and its quality-cost trade-off. We refer to each unit of text that is routed as a whole as a segment.
\begin{figure}[ht]
\centering
\includegraphics[width=\linewidth]{paper/figures/overview.png}
\caption{Overview of our learning-driven, cost-aware correction methodology.}
\label{fig:methodology}
\vspace{-1.5em}
\end{figure}
\medskip\noindentSetup: correction utility.
Let $\mathcal{D} = {d_1, \dots, d_n}$ be a corpus of text segments. Each segment $d_i$ has a raw OCR transcription $d_i^{\text{raw}}$. During evaluation, we also obtain an LLM-corrected version $d_i^{\text{corr}}$, which allows us to measure how much the correction improves transcription quality. We score quality using the Character Error Rate $W(\cdot)$ against the ground truth. The routing problem is therefore to identify, before running the LLM, which segments are likely to benefit from correction. We formalize this decision through two quantities measured per segment after evaluation: the realized correction gain and the number of tokens that would need to be processed.
* the **benefit** of correction, i.e.\ the reduction in error it
yields,
\begin{equation}
\Delta_i \;=\; W\!\left(d_i^{\text{raw}}\right) - W\!\left(d_i^{\text{corr}}\right);
\end{equation}
* the **cost** of correction, $C(d_i) = |d_i|$, the segment length
in tokens, which is what LLM pricing scales with.
A positive $\Delta_i$ means correction helps; a negative value means it
hurts; for instance by normalizing historically valid spelling. The difficulty
is that $\Delta_i$ depends on $d_i^{\text{corr}}$ and so is unknown at the moment
we must decide whether to correct.
\medskip\noindentPredicting correction utility.
We therefore estimate the benefit from the raw text alone. A feature map
[
\phi: \mathcal{D} \to \Phi \subseteq \mathbb{R}^{54}
]
encodes each segment as a feature vector $\phi(d_i)$ in the feature space
$\Phi$; these are the 54 features comprising surface statistics, orthographic/lexicon features, OCR confidence, and document metadata derived from the text segment. A model $\mathcal{M}_\theta$ with learnable parameters
$\theta$ maps features to a predicted correction utility
$\hat{u}i = \mathcal{M}\theta(\phi(d_i))$ which can be instantiated in either of two forms:
* **Regression**, $\mathcal{M}_\theta: \Phi \to \mathbb{R}$,
predicting the expected error reduction directly,
$\hat{u}_i = \hat{\Delta}_i \approx \mathbb{E}[\Delta_i]$;
* **Classification**, $\mathcal{M}_\theta: \Phi \to [0,1]$,
predicting the probability that correction is net-positive,
$\hat{u}_i = P\!\left(\Delta_i > 0 \mid \phi(d_i)\right)$.
In both cases $\theta$ is fitted on segments with known $\Delta_i$, and a larger
$\hat{u}_i$ marks a stronger candidate for correction. Our implementation and experiments adopt the regression approach, using Lasso with cross-validated regularization selection (LassoCV) as the estimator. Comparative results for alternative regression models are available in our repository\footnote{https://github.com/Stergios-Konstantinidis/CIKM_public}.
\medskip\noindentCost-aware priority score.
Ranking by predicted benefit alone would over-spend on long segments. We instead
rank by benefit per unit cost:
\begin{equation}
\hat{s}i ;=; \frac{\mathcal{M}\theta(\phi(d_i))}{C(d_i)} ;=; \frac{\hat{u}_i}{|d_i|}.
\end{equation}
The intuition is that under a fixed token budget the relevant question is not
will this segment improve?'' but how much improvement do we get per token
spent?'' For the regression model, $\hat{s}_i$ reads directly as expected CER
reduction per token; for the classification model it acts as a cost-discounted
likelihood of benefit. Either way, prioritizing high-$\hat{s}_i$ segments favors
corrections that are both likely to help and are inexpensive (most quality under a given budget).
\medskip\noindentRouting policy and frontier.
Sorting segments by priority score yields a permutation $\pi$ of
${1,\dots,n}$ with
\begin{equation}
\hat{s}{\pi(1)} \geq \dots \geq \hat{s}{\pi(n)}.
\end{equation}
For a budget of $k$ corrections we route the segments in
[
R = {\pi(1),\dots,\pi(k)}
]
to the LLM and keep the rest as raw OCR. Equivalently, we correct every segment whose score exceeds a cutoff. Routing
replaces the raw text of each segment in $R$ with its correction, so the
residual CER (the unweighted mean of per-segment errors) is
\begin{equation}
\begin{aligned}
\widetilde{W}(k)
&= \frac{1}{n}!\left( \sum_{i \notin R} W!\big(d_i^{\text{raw}}\big)
+ \sum_{i \in R} W!\big(d_i^{\text{corr}}\big) \right) \[2pt]
&= \frac{1}{n}!\left( \sum_{i=1}^{n} W!\big(d_i^{\text{raw}}\big)
- \sum_{i \in R} \Delta_i \right)
;=; W_0 - \frac{G(k)}{n},
\end{aligned}
\end{equation}
where $W_0 = \tfrac{1}{n}\sum_{i} W!\big(d_i^{\text{raw}}\big)$ is the
uncorrected baseline CER and $G(k) = \sum_{i \in R}\Delta_i$ is the cumulative
gain. Note that this algebraic derivation is metric-agnostic: it holds for any evaluation metric $W(\cdot)$ that is macro-averaged over the corpus segments. Because $W_0$ and $n$ are constants, the residual error rate is a strictly decreasing
affine function of the gain, so maximizing $G(k)$ is exactly equivalent to
minimizing $\widetilde{W}(k)$. Sweeping $k$ from $0$ to $n$ traces a continuous
routing frontier of residual CER against token budget
(Figure [fig:routing_frontier]), along which any operating point can be
selected.
\begin{figure*}[t!]
\centering
\includegraphics[width=\textwidth]{results/figures/routing_frontier/lassocv/routing_frontier_0pp.png}
\caption{Routing frontier comparing four OCR-correction strategies: LLM Oracle, Spell-check baseline, ConfBERT [10.1007/978-3-031-70442-0_13], and our regression-guided router. AUC is annotated per curve. The secondary x-axis (top) reports cumulative tokens submitted to the LLM (Gemini~3 Flash).}
\label{fig:routing_frontier}
\end{figure*}
Experimental Results
\label{sec:experiments}
We structure the evaluation around three research questions:
* **RQ1:** How closely does our selective routing match the oracle,theoretical oracle ceiling, and does it improveoutperform onnaive confidence-basedconfidence routing?thresholding?
Problem Visual Illustration
\medskip\noindentCorpusFigure and1: annotation.
We use the corpus summarized in Table [tab:corpus]: 609 text segments from nine Swiss historical newspapers (1733--1945) in the Canton of Vaud archives. Three French speakers produced fidelity-oriented transcriptions preserving historical spelling and source anomalies. We report WER/CER after typographic normalization unifying character variants and spacing. Fitting the Lasso router on 54 features takes $<0.1$,s on a single CPU core.
\begin{table}[h]
\centering
\caption{Corpus composition by newspaper publication.}
\vspace{-\baselineskip}
\label{tab:corpus}
\resizebox{0.78\columnwidth}{!}{
\begin{tabular}{@{}lcccc@{}}
\toprule
Newspaper & Dates & Issues & Pages & Text \
& & & & segments \ \midrule
La Revue & 1875--1945 & 4 & 5 & 139 \
Feuille d'Avis & 1762--1841 & 4 & 13 & 131 \
Tribune de Lausanne & 1912 & 3 & 4 & 105 \
Nouvelliste Vaudois & 1822--1840 & 3 & 7 & 90 \
Petite Revue & 1943 & 1 & 1 & 46 \
Lausanne Artistique & 1926 & 1 & 1 & 32 \
Almanach & 1832 & 1 & 8 & 31 \
Estafette & 1862 & 1 & 1 & 19 \
Mercure Suisse & 1733--1738 & 2 & 5 & 16 \ \midrule
Total & 1733--1945 & 20 & 45 & 609 \ \bottomrule
\end{tabular}
}
\end{table}
\medskip
\noindentOCR engines and LLMs.
We run threeTypical OCR engines (PaddleOCR v3 [cui2025paddleocr], EasyOCR\footnote{https://github.com/JaidedAI/EasyOCR}, Tesseractv5 [tesseractOCR]) and six correction
LLMs spanning closed-source (Gemini3 Flash, Gemini3.1 Flash Lite, GPT-4o) [google2025gemini3flash,openai2024gpt4osystemcard] and
open-weight (Qwen2.5-72B, Llama3.3-70B, Gemma4-31B) [qwen2024qwen25,meta2024llama33,google2026gemma4] models, each with its
best prompt out of ten templates of increasing complexity. Because of space restrictions, the paper reports results using Tesseract OCR and Gemini~3 Flash; evaluations across additional engines, LLMs, and prompt ablations are available in our repository.
\medskip\noindentBaselines and protocol.
We compare our regression-guided router (ranking segments by predicted improvement per token $\hat{s}_i = \hat{\Delta}_i / |d_i|$) against: (1) Full LLM correction (correct all segments), (2) ConfBERT [10.1007/978-3-031-70442-0_13] (BERT with token-level confidence injection), (3) spell-checking (pyspellchecker), (4) random routing, and (5) an Oracle routing by true error reduction $\Delta_i$. The regression uses 54 features computed from raw OCR alone (37 surface statistics, 4 orthographic/lexicon features, 1 OCR confidence score, and 12 document metadata features; full list in the repository), evaluated with leave-one-out cross-validation (inner 10-fold CV for regularization). We summarize each strategy by the area under its frontier (AUC) of residual CER and WER against routing budget; lower is better.
\smallskip\noindent
RQ1: Routing Quality vs.\ Oracle.
Figure [fig:routing_frontier] shows the WER/CER routing frontiers. Across the entire budget, our router closely tracks the theoretical upper bound of the Oracle, achieving the lowest residual error curves and substantially outperforming ConfBERT, raw OCR confidence thresholding, and random routing.
Table [tab:routing_comparison] compares operating points across budget tiers. Full LLM correction establishes the quality ceiling but incurs maximum token spend. In contrast, dictionary spell-checking proves to be the worst intervention for character accuracy: while it marginally improves WER on common modern terms, it actively degrades CER by corrupting valid historical spellings. Random routing improves error rates only linearly with budget without prioritizing high-yield segments. ConfBERT improves upon random selection, but remains consistently inferior to our method. Across all operating points, our regression-guided router delivers the best performance, recovering nearly all achievable quality gains while substantially cutting token expenditures.
\smallskip\noindentAdditional OCR Engines. These results generalize across OCR engines: on both high-accuracy engines (PaddleOCR) and noisier baselines (EasyOCR), our router captures the vast majority of full LLM correction gains while routing only a subset of segments (results in the repository).
\begin{table}[h]
\centering
\caption{Routing comparison on Tesseract OCR, using Gemini~3 Flash for LLM correction. CER and WER report the corpus-level error rate after applying each method.}
\label{tab:routing_comparison}
\resizebox{\columnwidth}{!}{
\begin{tabular}{@{}l|cccc|c@{}}
\toprule
Method & CER & WER & Segs. & Token & CER \
& & & Corrected & Savings & Improv. \ \midrule
Baseline OCR (no correction) & 0.063 & 0.234 & 0% & 100% & 0% \
Spell-check & 0.068 & 0.228 & 100% & N/A & $-$6.80% \
\hline
Random routing (40% segments) & 0.050 & 0.179 & 40% & 60.0% & +21.2% \
ConfBERT (Top 40% segments) & 0.042 & 0.159 & 40% & 39.6% & +33.2% \
Our approach (Top 40% segments) & 0.033 & 0.120 & 40% & 65.2% & +48.4% \
\hline
Random routing (60% segments) & 0.043 & 0.152 & 60% & 40.0% & +31.7% \
ConfBERT (Top 60% segments) & 0.037 & 0.133 & 60% & 15.6% & +41.9% \
Our approach (Top 60% segments) & 0.031 & 0.107 & 60% & 41.0% & +51.2% \
\hline
Random routing (80% segments) & 0.036 & 0.124 & 80% & 20.0% & +42.3% \
ConfBERT (Top 80% segments) & 0.032 & 0.118 & 80% & 2.5% & +49.7% \
Our approach (Top 80% segments) & 0.030 & 0.100 & 80% & 11.8% & +52.1% \ \hline
Full LLM (all segments) & 0.029 & 0.097 & 100% & 0% & +52.9% \
\bottomrule
\end{tabular}
}
\end{table}
\smallskip\noindent
RQ2: Predicting CER Improvement.
RQ1 depends on ranking segments by predicted improvement, so we next ask how well
$\Delta_i$ can be recovered from raw OCR text alone, without access to ground truth.
Figure [fig:wer_prediction] plots predicted against actual $\Delta_i$ on
held-out segments (leave-one-out with 10-fold inner CV for
regularization); the regression attains a Spearman rank correlation $\rho = 0.580$ ($p < 10^{-53}$). The correlation is moderate, but the router needs to rank segments by expected benefit, not to predict $\Delta_i$ exactly. The RQ1 frontier confirms that this ranking is strong enough to beat the ConfBERT baseline and closely approach the oracle.
\begin{figure}[t!]
\centering
\includegraphics[width=0.7\linewidth]{paper/figures/predicted_vs_actual_cer_loo_cv10.pdf}
\vspace{-\baselineskip}
\caption{Predicted vs actual CER improvement $\Delta_i$ on held-out segments (leave-one-out).}
\label{fig:wer_prediction}
\vspace{-1.5em}
\end{figure}
\smallskip\noindent
RQ3: LLM Cost.
In budget-constrained deployments (Table [tab:routing_comparison]), routing is done as follows: segments are ranked by priority score $\hat{s}_i$ and the top-$k$ are selected to match the available token budget. In unconstrained settings without an explicit budget, cross-validation yields an optimal cutoff of $\hat{\Delta}_i > 0.03$ to guard against marginal regressions. Table [tab:llm_cost] compares full-correction costs against this unconstrained routing strategy across several LLMs. The Gemini Flash family provides the best quality/cost balance among the commercial models. Older premium models such as GPT-4o incur higher cost without appreciable quality gains; open-source models occupy an intermediate band at zero API cost.
\smallskip\noindentTraining Cost and Amortization.
The router setup requires minimal overhead on the seed corpus ($N=609$): $\sim$40 annotator-hours ($\sim$4,min/segment) and $<1$,s CPU fitting. Even spanning extensive evaluations across multiple OCR engines, prompt variations, and LLMs, our total exploratory API expenditure was $$16$ ($\approx$0.50$ for a single production setup), which is rapidly amortized by $>40%$ token savings at scale.
\begin{table}[h!]
\centering
\caption{LLM inference cost comparison on Tesseract OCR. Full correction (Cost (full)'') vs.\ our routing (Cost (routed)'').
Open-source models have $$0$ API cost.}
\label{tab:llm_cost}
\resizebox{\columnwidth}{!}{
\begin{tabular}{@{}l|rr|cc|rr@{}}
\toprule
LLM Model
& \makecell{CER\(full)}
& \makecell{WER\(full)}
& \makecell{CER\(routed)}
& \makecell{WER\(routed)}
& \makecell{Cost\(full)}
& \makecell{Cost\(routed)} \ \midrule
Gemini 3 Flash & 0.029 & 0.097 & 0.033 & 0.120 & $0.182 & $0.064 \
Gemini 3.1 Flash Lite & 0.040 & 0.109 & 0.048 & 0.154 & $0.087 & $0.031 \
GPT-4o & 0.049 & 0.128 & 0.046 & 0.154 & $0.738 & $0.259 \
Gemma 4-31B & 0.039 & 0.123 & 0.043 & 0.150 & $0 & $0 \
Qwen 2.5-72B & 0.058 & 0.208 & 0.060 & 0.169 & $0 & $0 \
Llama 3.3-70B & 0.137 & 0.230 & 0.072 & 0.180 & $0 & $0 \ \midrule
No correction& 0.063 & 0.234 & 0.063 & 0.234 & --- & --- \ \bottomrule
\end{tabular}
}
\end{table}
\vspace{-0.1em}
Conclusion
The indiscriminate use of LLMs can render large pipelines expensive. We presented a lightweight, cost-aware router for OCR post-correctiondegradation in historical archives.periodicals Therequiring feature-basedcost-aware regression model predicts when correction is beneficial: correcting 60% of segments brings CER within 0.2% of full correction and closely trails an oracle. This routing principle, of predicting the LLM utility before paying for inference, extends naturally to other cascade tasks such as post-correction of speech or translation.routing.
\smallskip\noindentLimitations and Failure Modes.
Our evaluation focuses on historical Swiss French newspapers (1733--1945). While surface and confidence features are language-agnostic, orthographic features require a target lexicon. Error analysis reveals two primary failure modes: (1)Hallucinatory Modernization, where archaic syntax or valid 18th-century spellings are routed and standardized to modern French by the LLM as shown in Figure [fig:sample_modernization]; and (2)Irrecoverable Degradation, where severe physical damage yields illegible text that neither OCR nor LLM can reconstruct (such as in Figure [fig:sample_degraded]), thus consuming tokens with little corresponding improvement.
\begin{figure}[t!]
\centering
\fbox{\includegraphics[width=\linewidth]{paper/figures/issue_7_page_2_region_4277.png}}
\vspace{-\baselineskip}
\caption{Sample segment: areas highlighted in blue indicate segments prone to the modernization trap, while areas highlighted in red indicate instances where an OCR machine would misinterpret the letter S as F.}
\label{fig:sample_modernization}
\end{figure}
\begin{figure}[t!]
\centering
\fbox{\includegraphics[width=0.99\linewidth]{paper/figures/issue_292_wide_plain.jpg}}
\vspace{-.6\baselineskip}
\caption{Scanned newspaper with irrecoverable degradation}
\label{fig:sample_degraded}
\end{figure}
Acknowledgments
Supported by Swiss National Science Foundation, Grant No. 237991.
GenAI Usage Disclosure
GenAI was used as a code assistant and for formatting and editing the final manuscript. Research ideas are the authors' own. The authors take full responsibility for this work.
\bibliographystyle{ACM-Reference-Format}
\bibliography{main}
