No matter where you’re at on your automation journey, at some point difficult data throws a wrench in the gears. Even very large automation projects with seemingly straight-forward requirements are halted with just a single pesky data source that seems untouchable.
The good news is that no data is out of reach, and if you are looking into intelligent document processing, this buyer’s guide will provide the resources you need to make the best decision possible.
This guide will include pricing as examples to help you ballpark pricing expectations.
Disclaimer: Before I answer the question “What is intelligent document processing,” it would be a disservice to you not to talk about the technology’s history.
With a better understanding of it’s history, you’ll be able to more accurately frame the range of possible solutions.
Understanding the technology’s roots will help you spot the fakes and technology over-promisers. Nobody wants a failed project because of mismatched expectations (not you or the vendor).
Intelligent document processing has two main predecessors: document imaging, and document capture.
The technology was first explored in the 1980’s and was mainstream by the mid ‘90s. Document imaging technology flatlined by 2010 as it was being replaced with the next best document processing technology – document capture.
Most notably, these solutions incorporated optical character recognition (OCR) software, the ability to create basic templates for data extraction, and point solutions like accounts payable automation and handwriting recognition for forms.
Document capture solutions have been around for two decades and include a few notable legacy vendors like:
But to talk about these vendors with any degree of accuracy is tricky because virtually all of them have changed names and hands multiple times (PSIGEN tops our list for the longest running capture vendor until their recent acquisition in 2021).
It is safe to say that almost all enterprise document capture software offerings are the result of acquisitions combined into newly-branded products. Whether this is an advantage or disadvantage, we leave you to decide.
Intelligent document processing (IDP) augments human understanding of unstructured and difficult data through data science tools like:
The best IDP platforms are configurable to process documents the same way that human operators would. They provide multiple methods to understand access document-based data so you get the flexibility to tackle a diverse range of document types.
An IDP system, once deployed, should expand beyond the initial requirement to every department that needs to process documents. IDP is industry-agnostic, and will process any kind of document and deliver data to virtually any workflow or downstream application.
Because the nature of document-based data integration hasn’t changed much over the years, IDP operates in three broad phases:
No IDP software can skip phases 1 or 2. Data processing requires accurate text recognition (document capture), and document capture requires robust document imaging.
What happens in phase 3 is just as important as the technology powering the first two phases of intelligent document processing.
Because intelligent automation or robotic processes automation projects require document images like invoices, financial statements, medical claims, etc., it is important to be able to handle both digitally-born and scanned documents that may or may not be text searchable (not all PDFs are text searchable!).
If you expect the IDP software to deliver data to your automation tools, it must be able to “read” the text on the document accurately and consistently. Reliability is crucial.
In general, document imaging technology involves the following:
A reinventing of document capture is the heart of intelligent document processing. As an IDP buyer or researcher, you need to understand the difference between legacy capture and the new approaches in IDP:
This is a solution to identify and extract document data from highly structured documents and forms that have extremely predictable layouts. No “A.I.” is needed or used in these solutions. A good example is a utility bill or internally created form. These are called “structured documents.”
If you control scanning quality, and the layouts of the documents you are working with almost never change, legacy document capture is still a viable option today. It works because all you must do is create a template for each document type that looks in the same spot on every document for the field you need to capture.
The template will be programmed where to look for information and what that information represents, e.g., invoice date, invoice number, supplier name, etc. But if the information you need is more complex; if the layouts change based on things out of your control, this method becomes extremely inefficient, and hardly worth the time and effort.
Guard yourself from marketing hype! If it sounds too good to be true, it probably is.
Any time artificial intelligence is brought up, condition yourself to call bullshit until it’s been proven to you.
Harsh, perhaps, but this will save you from failed projects or mismatched expectations.
As artificial intelligence, machine learning, and natural language processing continue to gain momentum, it is easy to have a fallacy in thinking: “Can’t A.I. just understand what’s on my documents?”
Steer clear of the implications that A.I.-based “intelligence” should, will, or can, work like human intelligence. For example, neural networks do not, in fact “copy” the structure of the human brain, as is often claimed.
A.I. doesn’t, and won’t ever, work like that that. There is absolutely zero evidence that a trained neural network, or any other A.I. system, has, or can have anything like generalized intelligence – absolutely none.
A.I.-based tools are incredibly helpful, but they’re only ever going to be part of a full solution. Intelligent document processing isn’t a narrow problem, and it’s never going to be. That’s why A.I. won’t save you!
OCR is responsible for transforming pixels on a page that represent the text we read with our eyes to text that software will work with to understand what is written.
OCR can also be a point of weakness in a solution. If the only OCR solution available belongs to a third party, consider what happens if that solution is discontinued or sold to a competitor (this has already happened!).
The same goes for proprietary OCR – will it remain supported, and is there a roadmap for making improvements?
Here are the critical things to find out about OCR:
Machine learning (ML) is a way to build systems that display artificial intelligence (remember, solving hard problems). Generally speaking, ML is a strategy that builds systems that:
Unsupervised machine learning should not be used in document-based A.I. projects because it:
If people ask about or promote unsupervised ML, they are talking about one of two different things:
The answer to both of these questions is “No!” Document A.I. doesn’t get better on its own, because a human-centric design of document processing systems achieves better outcomes than unsupervised ML in almost every use-case.
Nothing – not even Amazon, or Azure, or Watson – will save you from having to understand your own documents and data. If someone tells you otherwise, they’re lying.
And if someone you’re talking to believes that Grooper (or any other document processing system) will save your from your own document or data problems, they’re setting you up for failure.
If the IDP solution is using supervised machine learning, here’s what you need to know:
Natural language processing (NLP) finds paragraphs, sentences, or other language elements that convey specific meaning. NLP is responsible for context-based data capture which is important in creating an understanding of what specific dates, names, etc. on a document represent.
NLP is typically either “baked into” the solution and available at multiple points during processing or is essentially an add-on from an external library (not as useful).
Here are the types of questions to ask while evaluating natural language processing:
IDP solutions providers should have multiple use-cases with all types of documents. What you need to understand is what these different document types “look like,” and the significance of processing each one.
How a vendor replies to questions about these different document types will tell a great story about their overall document processing journey, and the power of their technology.
Structured documents are the easiest document types to process. As mentioned previously, they are documents with layouts that don’t change – even when new documents are received.
For structured document processing, you only need to ask how the system works with known documents where the formatting is sometimes off by a little bit. They should be able to explain how their data extraction models work when labels and values shift around a bit.
PRICE: Structured document extraction is the least expensive of all options. If this is all you need, expect to pay as little as $10,000 per year.
Semi-structured documents are some of the most difficult documents to process. Think of an invoice. It usually has a single-line header that shows where quantity, description, price, etc. values are to be found.
A good IDP solution will have no trouble finding each line-item, and all details regardless of layout and page breaks.
Now, think of that same invoice. But then add multiple headers, each with new data that changes how the information below is to be interpreted. This presents a very difficult problem because an understanding of many hierarchies of data must be built within the software to extract the correct data.
The most common example of difficult semi-structured documents is explanation of benefits forms for multiple patients used in Healthcare. Other examples include mortgage documents and financial statements.
The vendor’s approach to processing complex semi-structure forms should be different than that used to process simpler structured forms. Ask if they process explanation of benefits forms and how that works.
You’ll quickly gain an understanding of whether their tool is purpose-built for a few use-cases, or if it is built to handle multiple (any) document types.
PRICE: Semi-structured documents represent some of the most difficult document extraction use-cases imaginable. To build out robust explanation of benefits extraction, corresponding workflows, and receive training, expect solutions to start out in the $100,000 price range.
For easier semi-structured documents like invoices and mortgage documents, expect to pay the vendor around $50,000 - $90,000 to get the solution into production.
Unstructured documents rely mainly on machine learning, accurate OCR, and natural language processing. Unstructured documents are typically legal documents in the form of contracts. They likely contain:
Data extraction from these documents relies heavily on human validation. A good IDP software will provide a review module that shows operators where on the page extracted data was found, and a chance to review / modify the data.
PRICE: Success with unstructured document solutions depends on the availability of subject matter experts (SMEs). Because these documents are often somewhat universal within particular industries, it is not rare to find a solutions provider already using IDP who will offer a very low per-page price for data extractions. These can be as little as $.10 / page or as much as $.25 per page.
If you choose to start an unstructured document data integration project yourself, plan for licensing costs in the $40,000 range and training, and services in the $60,000 - $90,000 range.
In most situations, these text files contain a massive amount of information, and not all of it is needed. It will likely be in a format that is foreign to your back-end systems, so using IDP to extract the most important information, and then format it creates even more value for IDP in your organization.
As mentioned previously in the unstructured documents section, IDP platforms should provide a method for human review of all extracted data.
Another critical element of human-in-the-loop architecture is in the way the software is trained to work with your documents and data. The best approach to bringing an IDP system into production from proof-of-concept is to have subject matter experts intimately involved in the systems initial training and configuration.
They know the documents better than anyone and will quickly point out what data is important and how they find it on the documents. The software should be configured to mirror the way they think about the information contained on the documents.
Don’t neglect the critical step of how data will be moved from the IDP system to your workflows and downstream software applications. You may require real-time integration to robotic process automation bots, or you may need to move extracted data into a staging database for final review before integrating with line of business systems.
The IDP platform should provide multiple paths for data integration to meet the diverse needs of your organization.
Part of the IDP selection process must include your cloud strategy. What is your cloud strategy? Do you have one? If not, it’s OK, but if you do have one, be prepared to explain it and how new solutions needs to fit into it.
IDP platforms are delivered in many different models:
With a concisely stated cloud strategy you will be able to short list vendors much more quickly (and don’t be surprised if they challenge you to gain a better understanding of how their offerings might fit into your technology ecosystem).
Vendors have many ways of generating revenue from IDP software. Getting a clear picture of how pricing works initially, and in the future is important. Your goal is to gain an understanding of what present and future activities will affect pricing.
This isn’t necessarily good or bad; it just means the vendor must recoup costs on each piece of technology it has purchased. Oftentimes these solutions will be more expensive and complicated to price.
See Prices Based on the Structure of Data Being Processed
Perpetual licenses are becoming much less common as vendors shift to subscription-based pricing models. Perpetual licenses cost more upfront than subscription.
Perpetual Cons: While you may own the license, you also typically only own only a single version. You won’t gain access to new features and functionality provided by new software versions.
Subscription Pros: You pay only a flat fixed monthly fee for full access to the software. Upgrades and different support tiers are included, so there are no surprises to deal with down the road.
Subscription Cons: If you don’t take advantage of newly released features and functionality, you will eventually be paying more for the solution than if you were able to purchase a perpetual license.
Page volume pricing is typically offered in tiers, or buckets of pages. Because of the flexibility in IDP platforms, get a clear idea of what a “page” is, and all scenarios that will affect volume.
Page volume pricing has the benefit of allowing faster processing. These systems typically let you throw as much compute as you can afford at the IDP solution and it will scale to nearly endless speed and overall throughput.
Core-based pricing may provide unlimited page volume, but limits throughput, and increases the time it takes complete larger projects more quickly. By licensing additional computer cores, you will increase the total throughput of the IDP solution.
Per-feature pricing may be an option with both page volume and core-based license models. This pricing model is seen more often on IDP systems which have grown by acquisition.
When a new type of tool is needed, or advanced functionality, buying teams have the option of adding on additional functionality which incurs additional charges and fees. Ensure the demonstration or POC you were involved with is accompanied by accurate and transparent pricing for all required features / capabilities.
Because IDP solutions vary in scope and capability, delivery methods will vary significantly.
If an IDP platform solution has built-in invoice processing, for example, it will only really work on about 80% of your invoices. Because you are going to need more automation than just 80 out of 100 invoices, you will spend configuration time getting the platform tooled up to get the remaining 20% of your invoices.
This process may break part of what was working for the 80% and will require technical resources to get fully up and running.
It is our recommendation that for most use-cases, an IDP platform should be configured from the ground up on your documents. You most likely have enough specific extraction and workflow requirements that it makes sense to tailor the solution to your needs rather than re-work your processes due to software inflexibility.
Delivering a production-ready IDP solution can cost more than the software itself depending on how challenging your documents and workflows are. Do not let this deter you or put you back in the situation of manually extracting data!
Once the IDP system is fully up and running, you will reap the benefits of:
Keep in mind that your organization, processes, documents, expectations, and requirements are different from any other organization. The IDP vendor will need your input and your expertise to get the solution configured. Nobody knows your documents like you do, so your subject matter experts will be intimately involved with the IDP delivery team.
The more you invest in training, the more strategic you will be. All IDP vendors offer implementation, either through a partner, or through their own professional services teams.
Even if your initial use-case is something simple, like invoice processing, you will eventually discover new use-cases in other departments. If you have trained technical resources on how to configure and deliver the IDP platform, you’ll achieve:
Training resources should include an online wiki / user guides, tiered training programs (instructor-led and self-paced), a support community where users interact with each other, and regular demonstrations on how to use advanced functionality.
Intelligent document processing is primarily a machine that takes data in, and outputs structured data.
Because IDP solutions will create a pristine copy of original physical or scanned documents, you’ll want to store them somewhere for humans to look at later on. For this, you will need a content management system like:
These content management systems will all have different options for storing document metadata, approval workflows, storage limits, etc. You’ll need to factor in the cost of these solutions, or choose an IDP vendor who provides an all-in-one solution.
Outside of content management and workflow solutions, another popular complementary technology to IDP is robotic process automation. If you’ve made it this far in your search for information, then you’ve doubtless run across vendors like:
These are all great vendors offering solutions to automate tedious “star and compare” work between software systems that aren’t yet talking directly to each other.
IDP is a great fit to deliver structured data to RPA solutions. Buyer beware – just because a vendor’s portfolio includes “intelligent document processing” doesn’t mean that it is more than just document imaging or document capture. True IDP is incredibly complex and not typically offered within other automation solutions.
Above all, you must clearly define the value that automation will bring to your organization. If you are uncertain how to prove out the value, but know “there’s got to be a better way,” using a business analyst (BA) to uncover the value is a great place to start.
If your leadership demands a proof-of-concept (POC) before purchase, consider structuring the POC in a way that maximizes your investment. Obviously the POC will involve your documents.
Because IDP is not magic, and is in fact, quite difficult, choosing your most difficult documents for the POC may not be a good idea. Instead, you could start with a simpler automation that frees up staff time, while planning for a much more impactful project for the future.
Because IDP is all about intelligent automation, it needs to fit in with overall automation strategies and goals. One way we’ve seen organizations successfully transform business operations is by creating a series of goals. It may look like this: