How to Analyze 1 Billion CDRs per Sec on $200K Hardware

Abstract

Modern telecommunication systems generate large amounts of data such as details of network traffic and service usage (call detail records). Amongst other information, these huge databases contain the behavioral patterns of the company’s customers. By extracting this data, a telecommunications company (Telco) can better understand the needs of its customers.

Traditional database technology scales to hold vast amounts of data but has severe performance limitations when it comes to analyzing this data. Data mining tools, which often store data in a private representation, offer fast analysis on small data sets but generally do not scale beyond a few million rows.

This paper presents a scalable, parallel data analysis engine capable of processing tens of millions of rows per second per CPU. This technology enables knowledge workers to get sub-second responses to queries that would previously have taken minutes or even hours.