Building a RAG Ingestion Pipeline for Messy PDFs, Tables, and Scanned Documents

Messy PDFs break retrieval for a predictable reason: the text that reaches the index is not the text a human would read. Running headers and footers get glued into sentences, hyphenated line breaks split words across lines, tables collapse into columns of bare numbers, and scanned pages contribute nothing at all because there is no text layer to extract. The fix is a staged ingestion pipeline — extract, clean, normalise structure, transcribe, chunk — with a measurement at each stage instead of one hopeful check at the end.

This guide follows the pipeline as it is built in the Document AI lab, a 55-minute, five-step intermediate exercise that requires no setup, and pairs it with the ingestion failure modes catalogued in Anyscale’s data ingestion strategies. It assumes a reader who can follow Python but has not yet built a document pipeline.

What “messy” actually means in a document set

The lab’s corpus is deliberately small and deliberately awkward: a two-page tenant handbook with headers, footers and hyphenated line breaks, a weekly rent table, and a rent review letter that was printed, signed and scanned. That is four distinct failure modes across three documents, and each one needs a different remedy.

• Running headers and footers. Page furniture such as a document title or page number repeats on every page. Extracted naively, it lands in the middle of body text and pollutes any chunk it touches.
• Hyphenated line breaks. A word split as associ- at the end of one line and ation at the start of the next becomes two tokens that no query will match.
• Tables. A table row extracted as plain text is a sequence of bare values. Once separated from its header row, 120.00 no longer means anything.
• Scanned pages. A page that was printed, signed and photographed has no embedded text, so native extraction returns an empty string.

Anyscale’s documentation frames the same problem at a higher level, listing five common ingestion challenges: missing, inaccurate or stale content; poor OCR or challenging layouts; suboptimal chunking; semantic mismatch between queries and documents; and schema drift in structured data sources. Its recommended countermeasures are comprehensive data curation, automated cleaning scripts before indexing, human-in-the-loop validation, and update pipelines that regularly refresh and re-index content. The pipeline below is the mechanical part of that advice.

The five stages, and why each one is measured

The lab’s pipeline has five checked steps, and the framing is the most useful idea in it: at each stage you count how many known answer sentences survive. That turns ingestion from an act of faith into a regression test.

1. Read the pages of each PDF.
2. Strip running headers and footers.
3. Repair hyphenated line breaks.
4. Turn table rows into self-describing records.
5. Transcribe the scanned page with a vision model.

The end product is chunks whose evidence can be found and retrieved — not merely chunks that exist.

Stage 1: Read the pages, and keep the image path open

Native text extraction comes first. Use a PDF parser to read the embedded text when it exists, and fall back to OCR only for pages that have none; that ordering is explicit in Anyscale’s guidance and it saves both time and money on born-digital documents.

import os
import pypdf

def page_text(folder, doc, page):
    reader = pypdf.PdfReader(os.path.join(folder, doc))
    return reader.pages[page - 1].extract_text()

The same library exposes embedded images, which is how the lab reaches the scan. Its helper for that is short enough to quote directly:

def page_image(folder, doc, page):
    """The bytes of the first image on the page (a scan is one big JPEG)."""
    return pypdf.PdfReader(os.path.join(folder, doc)).pages[page - 1].images[0].data

Note the assumption baked into that docstring: the scan is one large JPEG covering the whole page. A page with a logo, a stamp and a photograph would return the wrong image, so this is a corpus-specific shortcut rather than a general rule.

Stage 2: Remove running headers and footers

The usual heuristic is repetition. A line that appears at the top or bottom of most pages is furniture, not content. The following is an illustrative implementation, not the lab’s code:

def strip_running_lines(pages, min_ratio=0.6):
    """Drop lines that repeat on most pages, usually at the same position."""
    counts = {}
    for lines in pages:
        for line in set(lines[:2] + lines[-2:]):   # top and bottom candidates
            counts[line] = counts.get(line, 0) + 1
    threshold = min_ratio * len(pages)
    return [[l for l in lines if counts.get(l, 0) < threshold] for lines in pages]

The tradeoff is obvious once stated: a genuinely repeated line — a standard clause, a recurring address — will also be deleted. Counting surviving answer sentences after this stage is what tells you whether the threshold is too aggressive.

Stage 3: Repair hyphenated line breaks

Joining a hyphen at end-of-line with the following word is a one-line regular expression in the common case:

import re

def join_hyphenated(text):
    # "associ-nation" becomes "association"; "rent-free" may be damaged
    return re.sub(r"(w)-n(w)", r"12", text)

This is a heuristic with a known false positive. Genuine compounds that happen to break across a line, such as rent-free, get fused into rentfree. A dictionary check or a rule that only joins when the following fragment is not itself a word reduces the damage; the lab’s check step is what confirms the rule is good enough for its corpus.

Stage 4: Make table rows self-describing

This is the stage most often skipped, and the one that pays off most in retrieval. A table extracted as text produces rows of values whose meaning lives entirely in the header row. If those values are chunked on their own, a question like “what is the weekly rent for a two-bedroom flat?” has nothing to match against.

The remedy is to repeat the column names inside every record, so each row carries its own context. An illustrative record from a rent schedule might look like this — the field names and values are examples, not measurements from the lab:

{
  "document": "weekly_rent_schedule.pdf",
  "page": 1,
  "table": "Weekly rent schedule",
  "property_type": "2 bedroom flat",
  "weekly_rent_gbp": 120.00,
  "service_charge_gbp": 8.50
}

Each record is now a self-contained unit that can be embedded, retrieved and quoted without dragging the header row along. It also gives you a natural chunk boundary: one row, one record.

Stage 5: Transcribe scanned pages with a vision model

For the scanned rent review letter, the pipeline switches from text extraction to image understanding. The lab’s step 4 defines a transcribe() function that takes the image bytes, wraps them in a base64 data URL of the form data:image/jpeg;base64,..., and sends one user message containing a transcription prompt plus that data URL to a vision model.

import base64

def transcribe(image_bytes):
    data_url = "data:image/jpeg;base64," + base64.b64encode(image_bytes).decode("ascii")
    # one user message containing TRANSCRIBE_PROMPT and the image data URL
    ...

Two practical points sit around this step. First, OCR quality depends heavily on the image: Anyscale recommends deskewing, denoising and deblurring before OCR, because garbled or out-of-order text collapses retrieval quality no matter what happens downstream. Second, this is the stage where a locally hosted vision model becomes attractive — scanned correspondence often contains personal data, and keeping the images on your own hardware avoids sending them to a third party, at the cost of throughput and, depending on the model, transcription accuracy. The lab’s step is checked, which is the right pattern: transcription output should be validated, not trusted.

Chunking so the evidence stays findable

Chunking is the point where the previous four stages either pay off or are wasted. Anyscale lists suboptimal chunking as a distinct ingestion failure, and the mechanism is easy to see: split a cleaned sentence in half and neither half answers the question it was extracted for. Because stages 2 and 3 produce coherent prose and stage 4 produces one record per row, the chunker can be simple — respect sentence and record boundaries rather than cutting at a fixed character count. The lab’s stated goal for this stage is chunks whose evidence can be found and retrieved, which is a better acceptance criterion than a target token count.

Measure survival, not vibes

The lab’s evaluation approach is the part worth copying into any pipeline. It starts from a set of questions with known answer sentences in the source documents, then counts how many of those sentences survive each stage. If stage 2 removes an answer sentence, the header-stripping rule is wrong. If stage 3 mangles one, the hyphen rule is wrong. If stage 4 loses a row, the table parser is wrong. Without that per-stage count, a retrieval failure six weeks later is unattributable — you cannot tell whether the document changed, the chunker regressed, or the model drifted.

Anyscale’s broader guidance complements this: automated cleaning scripts before indexing, human-in-the-loop validation to catch subtle inaccuracies that automation misses, and update pipelines that re-index time-sensitive content. A question set with known answers is the cheapest form of that human validation, because it converts a judgement call into a number.

Limits, tradeoffs, and unresolved cases

• Heuristics overfit to a corpus. Repetition-based header removal and end-of-line hyphen joining both work well on the lab’s three documents and can fail on a different set. Re-run the survival count whenever the corpus changes.
• Tables are not always row-shaped. Merged cells, multi-level headers and nested tables do not map cleanly onto one record per row. Those cases need either a layout-aware extractor or a human review pass.
• Transcription has a confidence problem. A vision model can produce fluent text that is subtly wrong — a misread digit in a rent figure is worse than an obvious OCR garble, because it looks correct. Anything financial or legally significant should be checked against the source image.
• Freshness and schema drift. Even a perfect one-off pipeline decays. Anyscale flags stale content and schema drift in structured sources as first-class ingestion risks, which means re-indexing and validation belong in the design from the start.

What to do next

Build the smallest version of this pipeline on three documents you already have: one with headers and footers, one with a table, one scanned. Write five questions whose answers you can point to in the source text, then count how many of those answer sentences survive each stage. That count is your baseline, and it will tell you which stage deserves your attention before you touch chunk sizes or embedding models. If you hit a layout or transcription case this guide does not cover, share the document type, the extraction tool and the exact output you got in the community — concrete failure cases are what make ingestion guidance better.