Read the full article: 7 OCR Quality Tests to Run Before Deploying a Document-Reading AI Agent
Discover more at AI Builds It: Easy Coding Tools
Excerpt:
Judge the Reader by Its Worst Page
A document-reading agent should be judged by the worst page it will ever be handed. Most get judged by an average instead. More often than not, that average comes from a tidy sample the vendor picked, which looks very little like the crumpled invoices and claims the agent will meet after launch. Do the sum for a dense page of two thousand characters at 99 percent character accuracy. About twenty of those characters will be wrong, and nothing in the figure says which ones. For an agent, that's the whole question.
Even where character accuracy looks respectable, the words that matter usually score lower. An assessment of the British Library's nineteenth-century newspaper archive found characters 83.6 percent right and whole words 78 percent, while the meaningful words people actually search for fell to 68.4 percent.
OCR errors don't land evenly, either. When researchers measured how OCR noise in digitized newspapers hurt the software that later analyzed the text, they found a one-character slip could split a sentence in two, even in text that was mostly correct. Place names were missed more often than people's names, with dates somewhere in between. Newspapers aren't invoices. Still, dates faring worse than names should worry anyone automating accounts payable, where a date and an amount are exactly what an agent acts on without a second look. Seven tests can show where a given reader's errors fall before the agent goes live, ranked here by what skipping each one is likely to cost.
1. Score Only the Fields an Agent Will Act On
Most of the money rides on this first test, which scores the reader only on the fields the agent will use. Start with two hundred documents from the inbox the agent will actually work. Each one gets its totals, dates, account numbers and names marked by hand. Then count exact matches per field. A total that's off by one decimal place gets no partial credit, since the agent will pay it in full. Agree the pass mark for each field before the results come in. Pass marks chosen after the fact have a suspicious habit of landing just under whatever the reader scored. Somewhere in the results there's often a field nobody asked about, a purchase order number or a tax ID, scoring far below the headline figure the vendor quoted.
An agent paying suppliers inherits the bank-details score, which might be 91 percent on a reader sold as 99.
Business cards show the problem in miniature. A card might hold a name, a title, a company, three phone numbers and an email address in about forty words, and nearly every one of those words belongs in a field. Plenty of designers print the mobile number in pale gray, too. Even when a reader transcribes the whole card without a single error, it can still file the fax number as the mobile. On a character score, that's a perfect read of a useless contact.
Shop for an OCR SDK for business card scanning and Apryse will walk you through which documents arrive and in what volume before it asks what the reading is supposed to produce. That last question seems to be the one agent projects skip most often. In agent work the answer fits on a sticky note, usually total, due date, payee and bank details. The first test scores those fields and nothing else.
Two people should mark each document without seeing each other's work. Ideally one of them comes from accounts payable and the other has never seen these invoices before. Before any scoring starts, they settle every disagreement. Why two? One tired marker who misreads a smudged 7 turns the answer key into one more unreliable reading. If one supplier sends forty percent of the invoices, roughly forty percent of the test set should come from that supplier. After that, freeze the set and leave it alone, so next year's release from the same vendor faces this year's pages.
2. Find Out Whether the Reader Can Admit Doubt
Recognition engines usually know when they're unsure of a word, and many attach a confidence value to each word they read. That number tends to get lost on the way to the agent, which ends up with a clean string of text and every hesitation sanded off. Ask the vendor whether the agent ever sees it. Next, damage a few documents on purpose. Put a coffee ring over one total and fold another page straight through an account number, then watch what the agent does with each. If the agent stops or asks somebody, it passes. One that carries on with a confident guess at the coffee-stained total has just shown you what production is likely to look like.
Set the review threshold field by field rather than page by page. Nobody gets hurt if a smudged delivery address waits for tomorrow's batch. Before an account number in the same state reaches a payment run, a person should look at it.
This test comes with a catch. Flag a third of the pages for review and watch what the reviewers are doing by the end of the month. Many will be clicking approve after a glance. In hospitals, clinicians see so many electronic warnings that they end up overriding the vast majority, critical ones included. The patient-safety research on alert fatigue has been tracking that habit for years. When scoring this test, put the good pages flagged by mistake right next to the damaged pages the agent caught. A reviewer who hasn't corrected a single field in a month has probably stopped reading.
3. Hidden Text Aimed at the Agent
Nikkei went looking last year and found hidden prompts in 17 research papers. Set in white text or in type too small to read, the prompts told any AI reviewer to praise the work. Anyone reading those pages normally saw nothing unusual, which was the point. On an invoice, the same trick could move money.
Security guidance files the trick under indirect prompt injection and counts any file an AI system reads, a resume as much as an invoice, as a possible way in for instructions nobody would notice. Those lines only reach the agent if the reader picks them up, which makes hidden text an OCR problem as much as a security one. As tests go, this one is cheap. Slip a line of near-white text onto a sample invoice telling the agent to update the payee's bank details. If the agent obeys, the pilot stops there. Bank details are the obvious target. In the FBI's 2025 complaint data, business email compromise, the scam where a hijacked or faked email gets money wired to the wrong account, accounted for about $3 billion in reported losses. Any change to a supplier's bank account, however official the document looks, deserves a phone call to the number already on file.
PDFs add a second trap. Underneath the printed page there can be a text layer that says something else entirely. Some readers trust that layer and never look at the picture at all. It takes about an afternoon to compare each test file's hidden layer with a fresh read of the page image. With the files open anyway, invisible text (white on white, or two-point type) can be stripped or flagged before the agent ever sees it.
4. Words in the Wrong Order
Every word on a two-column contract can come out right while the columns get spliced together line by line, into clauses no lawyer ever wrote. Tables are worse. On a long invoice the line items often wrap onto a second row, and the grand total may not turn up until page three. The agent has to work out which number is which from text that may not have arrived in reading order.
Margins cause trouble of their own. Say someone strikes out clause 7 and writes a new figure beside it in pen. A reader that drops the handwriting hands the agent the old clause as if nothing had changed. Worse still is a PAID stamp the reader misses, which can get the same invoice paid twice. Very little of this shows up on the leaderboards that rank AI models. Ten of your own multi-page invoices, read back in order, will tell you more in a few minutes.
With page numbers and positions attached to each value, a reviewer checking a flagged total can go straight to the right spot in a forty-page scan.
5. Formats That Change at the Border
A London supplier writing 03/04 means the third of April, as it would across most of Europe, and an agent used to American paperwork might schedule the payment for March. Amounts are no safer. A German invoice for 1.250 euros is asking for twelve hundred fifty of them. Read with American habits, the same figure is a euro and a quarter.
The body that maintains the metric system settled the argument in 2003 by accepting either a point or a comma as the decimal marker. Expect both in the same day's invoices.
Then there are accents. Núñez with the accents stripped off can be, as far as a supplier database is concerned, a different company. The ñ itself can be stored in two different forms, as a single character or as an n plus a separate accent mark. On screen the two look identical, yet they can still fail an exact match. Before any comparison, the names should be normalized, which takes a single line of code. Bilingual pages, the kind cross-border shipping produces by the box, need a few samples of their own in the test set.
Run the same invoice past the agent in three regional formats. If the dates and totals it reports don't match across the three versions, the reader is probably guessing, and an agent will pay on the guess. Recording the expected date and number format for each supplier gives the agent a rule to check its reading against.
6. Photograph Test Documents the Way Customers Do
Ranked by how often it happens, the bad photograph might deserve a higher place than sixth. Many of the agents that collect invoices and receipts for finance teams are likely to meet more ph
October 26 · 13 min
October 21 · 15 min
October 17 · 15 min
October 12 · 10 min
October 5 · 12 min
August 17 · 38 min