Document Understanding

An invoice, a form, a scientific paper: the words matter and so does where they sit. A total in the bottom right of a table means something different from the same number in a paragraph, and a model reading the text in reading order has thrown away the layout that made it interpretable.

The fix is to give position on the page the same status as position in the sequence — a two-dimensional coordinate embedding alongside the ordinary one — so the model can attend to what is above a cell as easily as to what precedes it. Where the source has markup rather than pixels, the tree structure serves the same purpose and is cleaner, which is why HTML documents and scanned ones get different treatments rather than one pipeline.

Reading order is a lossy projection of a two-dimensional arrangement, and the loss is concentrated exactly where documents are most structured. A paragraph loses nothing; a table loses the column relationships entirely, because linearising a grid puts cells adjacent that were never near each other. So the share of a document whose meaning survives linearisation falls as the document becomes more tabular, which is the opposite of what a text pipeline assumes.