“Intelligent document processing” (hereafter “IDP”) is an interesting name, because it indicates where we came from – unintelligent document processing. Document processing is, at its heart, the ability to decouple document-based information from its carrier medium and prepare it for integration with or export to other systems.
As the need to incorporate this type of information into automated workflows has grown, so has the need for being able to process a wide range of documents and respond intelligently to variations: in image quality, in layout, in reliability of information, and a number of other factors.
IDP concerns itself with being able to deal with these variations (in a single workflow where possible) and to alert the user when it encounters something outside of its ability to process automatically.
While the goal of IDP is simple to articulate in theory, in practice it can take a number of different forms.
The Benefits of Intelligent Document Processing
Generally, approaches to IDP combine algorithmic or artificial intelligence-based decisioning with human design and review. IDP systems need to be adaptable and robust; this may mean having a very large and well-trained neural net, or it may mean having a sophisticated framework for administrator- or programmer-developed models for document processing, whatever form those might take.
IDP systems also include an ability to validate or manipulate data according to predefined rules (be these financial, regulatory, or business), and external data sets. This means that IDP systems need to have both an ability to manipulate data once it has been decoupled from documents, and they need to be able to seamlessly integrate with outside database, content management, and line-of-business systems.
This is, in my mind, another differentiator between intelligent and unintelligent document processing: the former has an ability to contextualize the recovered information with respect to other enterprise systems, which is critical to integration with other enterprise workflows and business applications.
The ways in which intelligent document processing can complement enterprise applications and workflows is as myriad as the number of ways in which business can be done – but generally the automated availability of document-based information enriches business processes.
By this I mean that it does a number of things that make existing processes better:
It saves money on data entry, it makes critical information available faster, and it improves ability of business users to gain insight into and oversight of their mission-critical data.
Generally, intelligent document processing provides the most value where organizations need to process a large number of paper-based documents that are mission-critical and may have different origination sources or large variances in image quality or quality of information. For large organizations, AP invoice processing alone is a multimillion-dollar problem, and any increase in reliability and speed of processing can mean large savings.
IDP Challenges by Industry
Specific industries tend to have document processing problems that are dependent on how information flows into and out of the business – Oil and Gas, for example, typically deals with geographically disparate operations engaging multiple stakeholders at every level (drilling operators, lease and mineral rights holders, regulatory bodies, etc.) many of whom communicate exclusively via documents; because of this structure, the business is highly dependent on the ability to quickly and accurately process those documents.
Insurance and Healthcare payment processing organizations face similar challenges, as do other industries dealing with geographically and organizationally disparate stakeholders.
IDP challenges can occur at multiple levels: at the level of the document, at the level of the rules and validations, and at the level of the integration with external systems.
IDP Document Level Challenges
At the document level, challenges can involve poor image quality (which can mean low-resolution scans, unusual colors or document sizes, sticky notes or torn pages) to handwritten documents (especially challenging when the documents are historical in nature, as handwriting has evolved continuously with language) to the presence of less- or unstructured language (think a narrative description of a sale instead of a sale described in a form with prompts and blanks).
IDP Rules and Validation Challenges
At the rules and normalization level, challenges typically involve the relationship between the data and the rules being applied, or multiple competing sets of rules which can be conflicting or mutually exclusive.
Sometimes data comes in a very large number of forms and must be normalized, and sometimes it occurs in ways that make normalization difficult (e.g. being included in a “comments” field or database column) and sometimes it occurs (as in the case of oil and gas lease land descriptions) in different forms depending on geographical region.
IDP Integration Challenges
At the export and integration level, challenges frequently involve systems designed to make ingestion or export difficult (many legacy content management systems intentionally make retrieving data from the system difficult) or issues with standards compliance (although ODBC and CMIS are interchange standards for their relative industries, for example, many implementations of them are incomplete or only partially compliant, or require complicated configuration in the system they are used to connect to).
Unfortunately for IDP users and developers, the problem of document processing is essentially as large as the problem of language itself, and no amount of technical sophistication or computational resources will ever really “solve” it.
Just as no formal system capable of producing number theory will be able to prove all true theorems or disprove all false ones, no IDP system will ever be able to process documents perfectly: there will always be documents, data points, and cases beyond what the system is capable of handling.
This is, of course, true of humans as well: even experts will occasionally make incorrect decisions or be unable to decide, and more frequently need to ask for a second opinion or consult with colleagues.
