# BCUL new ingestion

# Surya and ocr

# BCUL Historical Newspaper Ingestion Pipeline (Surya Layout + Surya-Aware Quadrant Apple Vision OCR)

## 1. Overview & Architecture

The ingestion pipeline ([`data_ingestion/ingest_pipeline.py`](file:///Users/stergios/Desktop/DocEng/data_ingestion/ingest_pipeline.py)) processes historical Swiss newspaper PDFs from the BCUL Scriptorium archive (`67,493` issues across 26 titles, 1732–2000) into structured layout segments and OCR text stored in PostgreSQL (`article_segments` and `article_ocr`).

### Core Design Principles
1. **Decoupled Layout & OCR**:
   - **Layout Segmentation (`surya-ocr` `LayoutPredictor` on MPS/GPU)** detects structural blocks (`Text`, `SectionHeader`, `Title`, `PageHeader`, `PageFooter`, `Picture`, `Figure`, `Caption`, `Table`, `ListItem`, `Formula`, `Footnote`) at the semantic block/column level.
   - **Text Recognition (`ocrmac` / Apple Vision `VNRecognizeTextRequest`, `recognition_level="accurate"` on Apple Neural Engine / CPU)** extracts individual text lines with bounding boxes and confidence scores.
2. **Parallel Execution**:
   - On every page, Surya layout segmentation (MPS GPU) and full-page Apple Vision OCR (ANE/CPU) run concurrently via a 2-thread `ThreadPoolExecutor`.
3. **Adaptive Surya-Aware Quadrant Splitting (Fallback for Dense Broadsheets)**:
   - Full-page Apple Vision OCR internally downscales input images to ~2048px on the longest edge and caps line proposals at ~350–400 lines per call. On dense 19th/20th-century broadsheet pages (`40–120` text segments, `500–800+` lines of 5–6 pt type), a single full-page call can drop entire columns of text.
   - Whenever a dense page ($\ge 10$ `Text` segments) has $< 60\%$ of its `Text` segments populated by the initial full-page OCR pass, the pipeline automatically upgrades that page to **Surya-Aware Quadrant Splitting**, recovering **99–100%** of segments without cutting any Surya segment or word across quadrant boundaries.

---

## 2. Step-by-Step Page Processing Pipeline

```mermaid
flowchart TD
    A["PDF Issue (pypdfium2)"] --> B["Render Page Bitmap (scale = 2.0–2.5, 144–180 DPI)"]
    B --> C["Surya LayoutPredictor (MPS GPU)"]
    B --> D["Full-Page Apple Vision OCR (ANE/CPU)"]
    C --> E["Spatial Line-to-Segment Matching (Pass 1 + Pass 2)"]
    D --> E
    E --> F{"Dense Page Check:<br/>Text segs >= 10 AND<br/>Filled Text segs < 60%?"}
    F -- "No (99.4% of pages)" --> H["Multi-Column Reading-Order Line Sort"]
    F -- "Yes (Dense Broadsheet)" --> G["Surya-Aware 2x2 Quadrant OCR<br/>(4 Non-Splitting Cluster Crops)"]
    G --> H
    H --> I["Normalize Coordinates to 300 DPI Space"]
    I --> J["Bulk Upsert into PostgreSQL<br/>(article_segments + article_ocr)"]
```

### Step 1: PDF Rasterization
- Each page of the PDF is rendered to an RGB PIL image using `pypdfium2` (`pdf_page.render(scale=render_scale)`), and C-level PDF/bitmap handles are explicitly closed after each page to prevent memory leaks over multi-day runs.
- **Scale factor**:
  - Standard ingestion uses `render_scale = 2.0` (`144 DPI`).
  - Dense broadsheet repair ([`repair_dropped_ocr_pages.py`](file:///Users/stergios/Desktop/DocEng/data_ingestion/repair_dropped_ocr_pages.py)) renders at `render_scale = 2.5` (`180 DPI`).

### Step 2: Concurrent Layout Detection & Full-Page OCR
- **Surya Layout Detection**:
  - Runs `LayoutPredictor([page_img])`, returning a list of layout boxes $[x_1, y_1, x_2, y_2]$ in page pixel coordinates, each with a `label` and `confidence`.
- **Full-Page Apple Vision OCR**:
  - Runs `ocrmac.OCR(page_img, recognition_level="accurate").recognize()`, returning `(text, confidence, [nx, ny, nw, nh])` in normalized bottom-left coordinates, which are converted to top-left pixel coordinates $[lx_1, ly_1, lx_2, ly_2]$ via `ocrmac.convert_coordinates_pil`.

### Step 3: Two-Pass Line-to-Segment Matching (`match_ocr_lines_to_segments`)
Each OCR line is matched to a Surya segment using its center point $(c_x, c_y) = \left(\frac{lx_1 + lx_2}{2}, \frac{ly_1 + ly_2}{2}\right)$:
1. **Pass 1 — Tightest Enclosing Segment**:
   - Finds all Surya segments whose bounding box (expanded by a `6px` tolerance margin) contains $(c_x, c_y)$:
     $$(bx_1 - 6) \le c_x \le (bx_2 + 6) \quad \text{and} \quad (by_1 - 6) \le c_y \le (by_2 + 6)$$
   - If multiple nested/overlapping Surya boxes contain the line center, the line is assigned to the segment with the **smallest area** $\text{area} = (bx_2 - bx_1)(by_2 - by_1)$ so specific inner boxes (`SectionHeader`, `Title`, `Caption`) win over coarse outer containers.
2. **Pass 2 — Vertical Near-Miss Snapping**:
   - Any OCR line not matched in Pass 1 (e.g., first/last lines of a paragraph slightly outside Surya's tight crop) is checked against segments whose horizontal span contains $c_x$ within `20px`:
     $$(bx_1 - 20) \le c_x \le (bx_2 + 20)$$
   - Assigned to the vertically closest segment if vertical distance $\Delta y < 35\text{ px}$.

---

## 3. Surya-Aware Quadrant Splitting (`ocr_page_with_surya_quadrants`)

### Why Naive Grid Splitting Fails vs. Why Surya-Aware Quadrants Succeed

| Approach | OCR Calls / Page | Dense Broadsheet Recall | Boundary / Slicing Artifacts |
| :--- | :---: | :---: | :--- |
| **Full-Page (`scale=2.0–3.0`)** | `1` | `0%–60%` on dense pages | None, but Apple Vision hits its ~2048px internal downscale & ~350-line cap, dropping whole columns |
| **Naive $2 \times 2$ Grid (`W/2, H/2`)** | `4` | `~98%` | Cuts mid-page Surya segments and words in half across the $x=W/2$ and $y=H/2$ cut lines |
| **Per-Segment Cropping (`1 crop / box`)** | `50–120` | `~99%` | `10x` slower (`50–120` CoreML launches/page); clips ascenders/descenders if Surya box is 2px tight |
| **Surya-Aware $2 \times 2$ Quadrants** | **`4`** | **`100%` (`10/10` pages)** | **Zero split segments, zero split words, zero duplicate lines** |

### Algorithm
Given a page of size $(W, H)$ and $N$ Surya layout segments $S = \{s_1, \dots, s_N\}$:

1. **Partition Surya Segments into 4 Quadrant Clusters (`TL=0`, `TR=1`, `BL=2`, `BR=3`)**:
   - For each Surya segment $s_i$ with bounding box $[bx_1, by_1, bx_2, by_2]$, compute its center:
     $$c_x = \frac{bx_1 + bx_2}{2}, \qquad c_y = \frac{by_1 + by_2}{2}$$
   - Assign $s_i$ to exactly one quadrant cluster $G_k$ ($k \in \{0, 1, 2, 3\}$):
     $$k = \mathbb{I}\!\left(c_x \ge \frac{W}{2}\right) + 2 \cdot \mathbb{I}\!\left(c_y \ge \frac{H}{2}\right)$$
   - Every Surya segment belongs to **one and only one** cluster $G_k$.

2. **Compute Non-Splitting Outer Bounding Box for Each Cluster**:
   - Instead of cropping at $(W/2, H/2)$, compute the tight union bounding box of all Surya segments in $G_k$, padded by `12px`:
     $$gx_1 = \max\!\left(0,\; \min_{s \in G_k} bx_1 - 12\right), \qquad gy_1 = \max\!\left(0,\; \min_{s \in G_k} by_1 - 12\right)$$
     $$gx_2 = \min\!\left(W,\; \max_{s \in G_k} bx_2 + 12\right), \qquad gy_2 = \min\!\left(H,\; \max_{s \in G_k} by_2 + 12\right)$$
   - Because $[gx_1, gy_1, gx_2, gy_2]$ completely encloses every segment $s \in G_k$, **no segment or word is ever cut in half**, even when a segment straddles the horizontal or vertical midline of the newspaper page.

3. **Quadrant-Scoped OCR & Exclusive Matching**:
   - Crop `page_img.crop((gx1, gy1, gx2, gy2))` and run `ocrmac.OCR(crop_im, recognition_level="accurate").recognize()`.
   - Shift detected line coordinates back into full-page pixel space by adding $(gx_1, gy_1)$.
   - Match the OCR lines from crop $G_k$ **exclusively** against the Surya segments in $G_k$ (using Pass 1 + Pass 2 above).
   - Because lines from crop $G_k$ can only bind to segments in $G_k$, overlapping crop margins near the center of the page **never produce duplicate lines**.

---

## 4. Intra-Segment Multi-Column Reading Order (`_sort_lines_reading_order`)

Occasionally, Surya groups two adjacent narrow newspaper columns into a single wide `Text` segment. Simply sorting OCR lines top-to-bottom (`by y, then x`) would interleave lines across the two columns horizontally.

`_sort_lines_reading_order(lines, box_width)` prevents horizontal column interleaving recursively:
1. Computes the median line width $w_{\text{med}}$ of all OCR lines inside the segment.
2. If the segment is wide ($\text{box\_width} > 280\text{ px}$), has $\ge 6$ lines, and $w_{\text{med}} < 0.55 \times \text{box\_width}$:
   - Sorts all line centers $c_x$ horizontally and locates the largest horizontal gap $\Delta x_{\max}$ between adjacent line centers (the column gutter).
   - If $\Delta x_{\max} > 25\text{ px}$ at gutter position $x_{\text{split}}$, recursively sorts all lines with $c_x < x_{\text{split}}$ (left column top-to-bottom) followed by all lines with $c_x \ge x_{\text{split}}$ (right column top-to-bottom).
3. Within a single column, lines are sorted by vertical row bucket `(round(y1 / 8.0), x1)`.

---

## 5. Coordinate Normalization & Database Schema

### 300 DPI Coordinate Normalization
To ensure compatibility with the downstream `BCUL-historic-data` image renderer (`TARGET_DPI = 300`), all segment bounding boxes are normalized from `render_scale` pixel coordinates to `300 DPI` pixel coordinates before insertion:
$$\text{dpi\_scale} = \frac{300.0 / 72.0}{\text{render\_scale}}$$
$$\text{bbox}_{300\text{dpi}} = \bigl[\text{round}(bx_1 \cdot \text{dpi\_scale}, 2),\; \text{round}(by_1 \cdot \text{dpi\_scale}, 2),\; \text{round}(bx_2 \cdot \text{dpi\_scale}, 2),\; \text{round}(by_2 \cdot \text{dpi\_scale}, 2)\bigr]$$

### PostgreSQL Tables Written During Ingestion
1. **`article_segments`**:
   - `newspaper_issue_id` (`FK -> newspaper_issues.id`)
   - `page_number` (`1-indexed`)
   - `coordinates` (`JSON` string `[[x1, y1, x2, y2]]` in `300 DPI` pixel space)
   - `label` (`Text`, `SectionHeader`, `Title`, `PageHeader`, `Picture`, `Caption`, `Table`, etc.)
   - `method` (`'surya'`)
   - `title` (auto-populated from the first line for headers or single-line short segments)
   - `processor_host`, `processor_run_id`, `created_at`
2. **`article_ocr`**:
   - `article_segment_id` (`FK -> article_segments.id`)
   - `ocr_text` (newline-joined lines sorted in reading order)
   - `ocr_method` (`'apple'`)
   - `processor_host`, `processor_run_id`, `created_at`
   - Upserted with `ON CONFLICT (article_segment_id, ocr_method) DO UPDATE SET ocr_text = EXCLUDED.ocr_text`.

---

## 6. Retroactive Repair Utility ([`repair_dropped_ocr_pages.py`](file:///Users/stergios/Desktop/DocEng/data_ingestion/repair_dropped_ocr_pages.py))

For pages ingested prior to the Surya-Aware Quadrant upgrade where full-page OCR under-matched dense text:
- Queries PostgreSQL for pages where `text_segs >= 10` and `filled_text_segs / text_segs < max_fill_ratio` (ordered from lowest fill ratio `0%` upward).
- Reuses the **existing** `article_segments` bounding boxes in PostgreSQL (no Surya GPU re-run required).
- Renders the PDF page at `render_scale = 2.5` (`180 DPI`), runs Surya-Aware Quadrant OCR across the 4 segment clusters, and keeps `max(new_text, old_text)` by length per segment so no existing text is ever lost.
- Upserts the recovered `article_ocr` rows in place without altering `article_segments.id` foreign keys.