An On-Demand ELT Architecture for Real-Time BI
Online or real-time BI has remained elusive despite significant efforts by academic and industrial research. Some of the most prominent problems in accomplishing faster turnaround are related to the data ingest. The process of extracting data from source systems, transforming, and loading (ETL) it is often bottlenecked by architectural choices and fragmentation of the processing chain.
In this paper, we present a vision for a resource-efficient infrastructure that enables just-in-time processing with regards to data ingest. At the heart of our approach are (1) the converting of compute intensive parts of the ETL process into in-database processing and (2) the activating of the process on demand via a system of flexible views.
Our approach avoids processing of data that is not being accessed any time soon, scales effectively with the database system and avoids administration and management overhead.
KeywordsReal-time BI ETL ELT Materialized Views
Unable to display preview. Download preview PDF.
- 2.Andzic, J., Fiore, V., Sisto, L.: Extraction, Transformation, and Loading Processes. In: Wrembel, R., Koncilia, C. (eds.) Data Warehouses and OLAP: Concepts, Architectures and Solutions, pp. 88–110. IGI Global (2007)Google Scholar
- 3.Buchmann, A., Koldehofe, B.: Complex Event Processing. Information Technology 51(5), 241–242 (2009)Google Scholar
- 7.Greenplum Inc.: Technical Documentation, http://www.greenplum.com
- 8.Gupta, A., Mumick, I.S.: Materialized Views: Techniques, Implementations, and Application, 1st edn. MIT Press (1999)Google Scholar
- 9.Kimball, R., Caserta, J.: The Data Warehouse ETL Toolkit. John Wiley & Sons Inc. (2004)Google Scholar
- 10.Microsoft Corporation. Microsoft SQL Server, Technical Documentation, http://www.microsoft.com/sql
- 11.Oracle Corporation: Technet. Technical Documentation, http://www.oracle.com/technetwork/indexes/documentation/index.html
- 12.Quass, D., Widom, J.: On-Line Warehouse View Maintenance. In: Proc. of ACM SIGMOD Int. Conf. on Management of Data, pp. 393–404 (1997)Google Scholar
- 13.Vassiliadis, P., Karagiannis, A., Tziovara, V., Simitsis, A.: Towards a Benchmark for ETL Workflows. In: Proc. of Int. Workshop on Quality in Databases (QDB), pp. 49–60 (2007)Google Scholar