Why PDF text extraction is imperfect: columns, ligatures, scanned pages and hyphenation

Inside a PDF, text is a sequence of drawing operators in a content stream: set this font, move to this position, show these character codes. The codes are indexes into the font's encoding, not Unicode; a document is readable and copyable only because most fonts also ship a ToUnicode map that says which character each code represents. When that map is missing or wrong — common with old generators and subsetted fonts — the page looks perfect and the extracted text is nonsense.

There is no paragraph in that model at all. An extractor receives a list of text runs with coordinates and has to rebuild lines from their baselines, then paragraphs from the vertical gaps between lines, the indentation and the changes of font. DocDiffer does exactly this, and treats a run set in a different font — a bold heading, for instance — as the start of a new block rather than a continuation.

The order of text in a content stream is the order the generator happened to draw it in, which need not match the order a human reads it. Usually they agree; in two-column layouts, documents with margin notes, pull quotes or footnotes, they often do not. The classic failure is a two-column page extracted line by line across both columns, producing sentences that alternate between two unrelated paragraphs.

A reasonable defence is geometric: look for a vertical band down the page that no line of text crosses, and if there is one with enough lines on both sides, read the left column first and then the right. DocDiffer applies that test per page. It handles ordinary two-column documents and gives up gracefully on anything more complex, which is the honest trade-off — a wrong column guess is worse than none.

Running headers and footers are the same problem in miniature. They are drawn on every page, they are not part of any sentence, and in a comparison they generate a change for every page whenever a page count or a date in them shifts. Detecting them by position and repetition, and offering to ignore them, removes an entire class of false differences.

Fine typography merges certain letter pairs into single glyphs: fi, fl, ffi and friends. If the font's mapping is complete, they extract back as two or three characters; if it is not, “finalise” arrives as a word with a character missing or an exotic symbol in the middle. The same applies to typographic quotes, en and em dashes, and non-breaking spaces, which differ from their ASCII cousins at the byte level while looking nearly identical on the page.

Spaces are the subtler case, because a PDF need not contain any. A generator can position every word individually and never draw a space character at all. Extractors therefore infer word boundaries from the horizontal gap between runs relative to the font size — which is why text pulled from a heavily kerned or justified page sometimes arrives with a missing space, or an extra one inside a word.

Justified text breaks words across lines. Whether the hyphen is a real character, a soft hyphen or an artefact of the layout engine varies, and a naive extractor keeps it: “obli-” plus “gation” becomes “obli- gation”, a word that matches nothing in the other version. Rejoining is the right default when a line ends in a hyphen and the next begins with a lower-case letter.

The rule is right far more often than it is wrong, but it does misfire on genuine compounds broken at the hyphen — “cost-effective” split across a line loses its hyphen. This is one of several reasons that two PDFs of the same text, laid out at different line widths, are never quite byte-identical, and why comparing at word level with whitespace ignored gives a far more useful answer than comparing raw strings.

A PDF produced by a scanner contains an image of a page and no text at all. Nothing can be extracted from it, and a comparison tool should say so rather than report an empty document. The fix is OCR, which adds an invisible text layer over the image; the page then extracts, with the accuracy of the recogniser rather than of the original.

Treat an OCR layer as a good transcription, not as the document. Confusions between 1, l and I, between 0 and O, and between comma and full stop in figures are the normal failure modes, and they land precisely on the numbers a contract review cares about. If a comparison of two OCR'd scans shows an amount changing from 1,000 to 1.000, check the page image before you write the email.

Tables are the other structure PDF does not record. Visually a table is lines and text boxes; there is no notion of a row or a cell. Extracted column by column or line by line, a price table becomes a sequence of short paragraphs, and a comparison of it is only as meaningful as the order in which the cells came out.

Open the PDF in any reader, select all, copy, and paste into a plain text editor. What you see there is roughly what any extractor will see: if the text is garbled, out of order or absent, the problem is in the file, not in the comparison tool. This one test explains most surprising diffs.

Then look at the extraction statistics both versions report. Similar documents should yield similar paragraph and word counts; a version with half the paragraphs of its sibling has almost certainly been rebuilt differently, and comparing them will produce noise. Finally, switch on the ignore options — whitespace, punctuation and quote style, running headers — before deciding that a change is real.