Automated Data Harmonization (ADH) using Artificial Intelligence (AI)

Abstract

Organizations in the business of Information Services deal with very large volumes of data which is collected from a variety of proprietary, as well as, public sources in multiple languages with different formats, naming conventions, and context. Mapping such data into enterprise master data for reporting and prediction is an effort-intensive, time-consuming process which is prone to errors. Machines cannot match these sources and map to master data accurately. Enterprises are eager to automate the human intensive tasks of data harmonization so that their resources can focus on finding the insights to drive the business. We undertook one such automation initiative for a global Market Research Major (MRM) and achieved a significant level of success leveraging Artificial Intelligence (AI) techniques. The Automated Data Harmonization (ADH) solution has been a multi-step approach of Dictionary Matching, Fuzzy Text Similarity, and different Machine Learning techniques. It has been implemented on the Big Data stack for better performance and scalability. In order to streamline the overall business process, runtime rules and workflow has been implemented. The Proof of Concept has yielded an overall F-Score within the range of 82–93% depending on the variation of the dataset. The deployed version is continuing to deliver high accuracy and gained adoption as a core micro-service across the organization. The Business as Usual (BAU) cycle time has been reduced by 80% (from 14 days to 3 days). While the solution is unique and tailored to meet a set of specific business requirements, it can be extended for media metadata standardization across multiple devices, author name and citation resolution in scholarly journals, leads resolution in multi-channel marketing and ad campaigns etc.

Glossary

Agile

Software development methodology built on the principles of ability create and respond to change. Collaboration and the self-organizing team are key to agile development

Apache Spark or Spark

Low-latency processing framework for Big Data for interactive queries and stream processing

Docker

Open source tool designed to create, deploy, and run applications by using containers

Fair Scheduler

MapReduce Scheduler that provides a way to share large clusters

GET API

HTTP method to retrieve resource information

Hadoop

Open Source software framework that allows storage, management and massively parallel processing of Big Data

HBase

Open source non-relational database developed by Apache Software Foundation

HDFS

Distributed file storage data of Hadoop with high-throughput access to application data

Hive

Open source data warehouse software built on top of Apache Hadoop

Java

General purpose programming language

NoSQL

Database that provides a mechanism for storage and retrieval of data and modeled in non-relational form

Oozie

Workflow scheduling system to manage Hadoop jobs

Python

General purpose programming language

SAS

Statistical Analysis software to access, manage, analyze and report data to aid in decision-making

Scikit-learn

Open source software machine learning library for the Python programming language

Sqoop

Sqoop is a command-line interface application for transferring data between relational databases and Hadoop developed by Apache Software Foundation

YARN

Resource management solution of Apache Hadoop