How to Transition to Electronic Records: Guide for Government Agencies

The US Federal Administration has set a course for achieving transformational change through modernizing electronic government.

This paper outlines how to become compliant.

What's in this Guide:

Background and outlook - why government agencies are moving to electronic records. There are two important presidential mandates addressing the requirement for change.

Challenges to overcome - why paper is a problem. Paper records cost money, hurt cross-agency collaboration, and add unnecessary risk. But, moving to digital records has it's own challenges.

Solution and opportunity overview - what moving to electronic records looks like, and the impact to both agencies and the public.

Deadlines - there are upcoming deadlines, starting December 2019. I'll guide your through what's required.

Required metadata - metadata is required on all electronic records. There are 15 required fields, and I describe each one.

How to get started - tips on how to approach the transition to electronic records, the skills you need, and how to bring key personnel together to ensure success.

Technology overview - what's involved in moving to electronic records. I'll overview everything from microform scanners, software choices, image processing, OCR, and storage options.

Vendor selection - how to find a good fit with potential vendors, and the importance of an accurate requirements list.

Additional resources - I've prepared a detailed list of websites to review for recommended products, software, vendors, and professional organizations.

Background and Outlook of Government Records and Documents

In March of 2018, then President Donald Trump issued a President’s Management Agenda[i] (PMA) establishing a sweeping vision to focus on improving the following six key areas:

Our country’s mission
Customer service
Stewardship
Information technology
Data transparency
The workforce of the 21^st century

In addition, three months later in June 2018, a proposal was made to transition federal agency business processes and recordkeeping to a fully electronic environment.

The proposal also ends the National Archives and Records Administration’s (NARA) acceptance of paper records by December 31, 2022 [ii].

Numerous Agency Data Improvements

This will improve the agency's: efficiency, effectiveness, and responsiveness to citizens by converting paper-based processes to electronic workflows, expanding online services, and enhancing management of government records, data, and information.

government-document-data The Federal Government spends hundreds of millions of taxpayer dollars and thousands of hours creating, using, searching, and storing paper records.

Storage facilities and handling electronic requests for paper records force citizens to conduct business where records can only be extracted manually - and at great expense.

Cost-effective solutions must be considered by every federal agency to transition to an electronic environment in support of this new reform plan.

All federal agencies are directed to comply with all records management, laws, and regulations. More specifically, agencies must:

Ensure that all federal records are created, retained, and managed in electronic formats with appropriate metadata; and
Consistent with records management laws and regulations, develop plans to close agency-operated storage facilities for paper and other, analog records, and transfer those records to Federal Records Centers operated by NARA or commercial storage facilities.

The Challenges of Paper-Based Records

Federal agencies spend billions of dollars supporting paper-based records practices.

Far too many government services still depend on paper. This use of paper causes increased costs for managing the movement, processing, and storage of huge volumes of paper records. Efforts to solve this problem have been inconsistent and ineffective agency-wide.

Antiquated, unsecure technology presents real risks for poorly digitized documents. And taking into account outdated duties being performed with outdated skillsets, it’s reasonable that countless hours in documenting and recording information are at risk.

Silos of information and recordkeeping hurt cross-agency collaboration. The resulting fragmentation of citizen services creates unnecessary delays, and excessive costs related to the delivery of basic services.

Another challenge is navigating the vendors, software, and technology available in today’s marketplace. With intelligent software solutions on the rise, and the existence of proven, legacy solutions – the choice isn’t an easy one.

The Solution - Move to Electronic Records

Federal agencies must adopt a comprehensive, lifecycle approach[iii] to records management. This includes stopping all paper processes to every extent possible and properly digitizing existing paper records.

The Electronic Records Mandate includes the oversight of records management to the NARA. This safeguards the authenticity of archival records, ensures compliance, and maintains record-keeping practices. Furthermore, the addition of strict metadata requirements ensures the plan for master data management stays intact.

This push for electronic records also creates a more nimble and effective approach to the technologies and skillsets needed to ensure the federal workforce meets current and future needs.

The Opportunity

Ultimately, adopting electronic records management will improve citizen services by preserving public access to records, and creating cost-saving efficiencies and processes. It will become easier for the public to connect with the federal government to apply for and receive needed services.

By digitizing records today, agencies also win. Decreasing records storage space opens up more real estate for workers and an opportunity to cut costs. Additionally, streamlining workflows which previously required accessing or creating paper records reduces stress and limits costly errors.

The Deadline for the Acceptance of Paper Records

December 31, 2020 marks the end of NARA’s acceptance of paper records. This forces agencies to direct attention and resources to maintain compliance. While many agencies, such as USCIS and the National Records Center have centralized millions of paper records into a single facility, there is still a demand for digitizing these records with the required metadata that enables search, retrieval, and integration with software applications.

Additional Deadlines

By December 2019, Federal agencies will manage all permanent electronic records in an electronic format.

By December 2022, Federal agencies will manage all permanent electronic records in an electronic format and with appropriate metadata.

By December 2022, Federal agencies will manage all temporary records in an electronic format or store them in commercial records storage facilities.

Beginning January 2023, All legal transfers of permanent records must be in electronic format, to the fullest extent possible, regardless of whether the records were originally created in electronic formats. All analog records must be digitized before transfer to NARA.

What is the Required Metadata?

Federal agencies are responsible for managing electronic records in accordance with NARA statutes and regulations[iv], and the Federal Records Act[v]. Metadata are the elements of information that answer the critical questions regarding a record. Essentially, the who, what, where, when, and why.

Metadata provide administrative, descriptive, and technical information regarding the structure, and content of the record. Additionally, metadata elements provide contextual information that explains how the records were created, used, managed, maintained, and how they relate to other records[vi]. There are currently 15 required metadata fields[vii].

Required Metadata Terms and Descriptions

The following is the minimal list of metadata terms necessary for describing permanent electronic records.

Identifier:FileName

The complete name of the computer file including its extension (if present).

Identifier:RecordID

The unique identifier assigned by an agency or a records management system.

Title

A name given to the record. Typically, a Title will be a name by which the record is formally known.

Description

A narrative description of the content of the record, including abstracts for document-like objects or content descriptions for audio or video records.

Creator

The agent primarily responsible for the creation of the record. Examples of a Creator include a person, an organization, or a service. Typically, the name of a Creator should be used to indicate the entity.

Date:CreationDate

The date that the file met the definition of a Federal record. If a file (such as a case file, database or spreadsheet) holds multiple records created at different times, then this element should note the date the file was originally created and the span of dates should be recorded in the element Coverage. Date may be used to express temporal information at any level of granularity. Best practice is to use an encoding scheme, such as the W3CDTF profile of ISO 8601[viii].

Rights:SecurityClassification

The classification allocated to the record indicating its official security status. The purpose of this qualifier is to facilitate proper and appropriate management of sensitive or security classified records.

Rights:PreviousSecurityClassification

The classification allocated to the record indicating its official security status prior to its current status. Many official documents have their security classification reduced over time. The ability to search on current and previous markings allows a user to locate records that have changed their classification.

Rights:AccessRights

Legal or other rights an individual has to access the record or that regulate the agency's right to release or provide access to the record. This repeatable field contains information documenting any restrictions related to information access that applies to the record. These are described in the General Restrictions section at 36 CFR 1235.20, and listed at 1256.40 through 1256.62[viiii].

Rights:UsageRights

Information about copyright and trademarks rights held in and over the record. Typically, rights information includes a statement about various property rights associated with the record, including intellectual property rights. Agencies should indicate any licenses obtained in the creation of the record.

Rights:RightsHolder

A person or organization, owning, or managing intellectual property rights relating to the record.

Coverage:SpatialCoverage

The geographic extent or scope of the content of the record. Spatial topic and spatial applicability may be a named place or a location specified by its geographic coordinates. A jurisdiction may be a named administrative entity or a geographic place to which the record applies. Where appropriate, named places can be used in preference to numeric identifiers such as sets of coordinates.

Coverage:TemporalCoverage

The temporal extent or scope of the content of the record. Temporal topic is a date or date range.

Relation:HasPart

A related record that is either physically or logically required in order to form a complete record. Best practice is to identify the related record by means of a string conforming to a formal identification system.

Relation:IsPartOf

A related record in which the described record is physically or logically included. Recommended best practice is to identify the related record by means of a string conforming to a formal identification system.

Where to Start with Automated Records

Begin exploring electronic records requirements, and related problems / concerns now. Don’t wait for a vendor to tell you they have the all-encompassing solution!

Successful agency-wide transition requires a great deal of knowledge, judgement, and wisdom on your part. It’s most likely that a combination of technologies and strategies will best support your agency and processes.

convert-government-documents Learn the basics of how automated records systems are created and how they function. When you have a more detailed understanding of data and information management principles and techniques, you’ll better understand how to extract the required metadata and integrate more accurate information.

Learn How Your Data Works and Is Being Used Currently

Your goal is to learn to speak the language of data and be able to perform basic records data modeling. It sounds complicated, but experienced vendors have the expertise to guide you in what you need to learn.

Identify key data and information management players within your agency:

Who manages your central databases?
Who’s involved with solving data and information management issues?
Who is responsible for risk assessment, management, and audits?

Begin forming partnerships with these individuals in order to connect the dots between these functions. They may not all coordinate today, and your success will depend on a program which leverage the expertise of each of these areas.

Key Technologies for Success

Successful electronic records conversion and management requires automating the collection of metadata. This is best achieved from the following core technologies:

High-Quality, High-Speed Scanners for Paper and Microform

There’s nothing better than having a high-speed scanner that produces excellent digital copies of records.

When owning a scanning center doesn’t make sense for your agency, look for vendors who will not only create digital images of your records, but also perform the metadata extraction and data integration with your systems. Whether you have microfilm, microfiche, or books of physical records, document scanning services with intelligent processing technology are becoming more and more cost-effective compared to in-house scanning.

Image Processing

Image processing does many things, including:

Enhances images displayed to users throughout record workflows
Removes image artifacts known to interfere with optical character recognition (OCR)
Provides crisp record versions for permanent archival
Analyzes page structure to assist with automations in downstream processes

Image processing applies to all forms of physical records, not just paper.

Image processing includes features like:

Halftone Removal

Dithering and other halftone patterns are a direct result of legacy document imaging platforms poorly converting color images to black and white.

These artifacts must be eliminated to prevent massive errors in OCR results, particularly with punctuation like periods and commas.

Border Removal

Borders have historically been very tricky to remove when the black region doesn’t extend all the way to the edge of a page. These need to be recognized and cleanly removed.

Artifact Removal

Images often contain artifacts that make the record difficult to view. Image processing will temporarily remove these non-text items like specks, pictures, hole punches, etc on a page to facilitate better OCR.

Line Detection and Removal

Lines are used all throughout standardized forms, table structures, and pages with “fill-in-the-blank” comb boxes to provide visual cues that increase legibility for readers. These lines, particularly the short, vertical ones, are commonly picked up by OCR engines as characters. Line detection features erase these lines.

Optical Character Recognition

Optical character recognition (OCR) is the electronic conversion of typed or printed text. If you are familiar with OCR, you may think converting paper to digital text is obvious, and does not need to be listed. Although OCR technology has been around for a very long time, there are recent, modern advances that make acquiring accurate metadata much easier.

Great OCR starts by removing everything on a page that isn’t text. This is done through image processing. Humans easily recognize characters on a page, and even words that are misspelled aren’t a problem. But when machines read records, the margin for error is much smaller.

For the most accurate OCR, look for these features:

Iterative OCR

This is a technique designed to capture text that the OCR engines miss the first time around. By digitally dropping out portions of the page where text is easily recognized, OCR can be run again with less distractions on the page. This makes subsequent OCR passes more accurate.

Cellular Validation

When text is printed in columns on a page, or when font sizes differ between columns, or the text is offset between columns, this presents a unique challenge for OCR engines. By identifying cells, or segments of data, and processing them independently of the other cells, cellular validation ensures high accuracy.

Layered OCR

Layered OCR is similar to iterative OCR, except that it uses multiple OCR engines. This is to overcome the problem of multiple font types. It’s easy for humans to read different fonts, but software needs help in this area. The layered OCR technique drops out accurate text, and intelligently chooses another type of OCR engine to read the remaining text. Multiple OCR engines can be cycled through, including those that read handwriting.

Spell Correction

OCR technology that uses tools like K-Means Clustering, text removal, and text correction provide extremely accurate OCR, which is necessary for automating metadata extraction. Spelling errors occur for many reasons, so planning to automate the correction saves time.

Document Management System

If you don’t already have a document management system, this will be the place to store your electronic records.

In order to ensure maximum accessibility to electronic records, choose a document management system which provides both integration with existing (or future) software applications and capabilities for managing required metadata.

A search for 'enterprise content management systems' will reveal top technologies available today (I've also listed some in the links below). More advanced systems include workflow management, collaboration tools, e-signature capability, email integration, and more advanced records searching.

While solutions exist for both cloud and on-premise electronic records storage, there are solid arguments for choosing either option. For most agencies, a blend of cloud and on-premises storage is likely the best option. They key is to consider scalability and long-term costs.

Records Classification

Classification is critical for extracting accurate metadata. Classification is a method of organizing and grouping records using machine learning models that are trained to recognize different records types. This may sound complicated, but today’s technology makes training machines (software) easy, and doesn’t require a data scientist.

Classification is performed using many different methods such as natural language processing, text-based rules, and by the visual layout of records. Now, even photographs and other media types can be classified.

Automating classification is absolutely critical. Without an understanding of what type of document the record is, data extraction software won’t know what information to expect on the record in order to create accurate metadata.

Data Integration

After all records information has been processed, it needs to be added back to the record as metadata. Also, the data in the record needs to be integrated into a document management system for easy retrieval and archival with NARA.

Data integration systems will migrate records between systems, generate the text files needed to search and discover the record, and use external databases to verify information found on the record.

Additionally, data integration tools create human-readable records. This includes creating PDF versions of records which are intended to be viewed on-screen or printed, if needed. The previously mentioned image processing techniques are an important part of creating a permanent document image, optimized for human viewing.

Vendor Selection

There are multiple technologies available for every stage of the electronic records journey. Vendor selection is an important consideration. A key component of establishing high-quality vendor relationships is the creation of a requirements list.

BIG TIP: A thorough requirements list is like turn-by-turn directions to get to a new destination.

Just like a single street rarely takes you to your destination, no single vendor will meet each requirement. Clearly establishing needs upfront ensures long-term success.

The very best way to ensure success with any vendor is through the help of a certified business analyst (BA). A BA has the experience to help guide the creation of a thorough requirements list and will foresee and mitigate future roadblocks and compatibility problems.

Many vendors provide an analyst as part of their services and organizations like IIBA provide BA resources and referrals for additional help.