Advertisement

An On-Demand ELT Architecture for Real-Time BI

  • Tobias Freudenreich
  • Pedro Furtado
  • Christian Koncilia
  • Maik Thiele
  • Florian Waas
  • Robert Wrembel
Part of the Lecture Notes in Business Information Processing book series (LNBIP, volume 154)

Abstract

Online or real-time BI has remained elusive despite significant efforts by academic and industrial research. Some of the most prominent problems in accomplishing faster turnaround are related to the data ingest. The process of extracting data from source systems, transforming, and loading (ETL) it is often bottlenecked by architectural choices and fragmentation of the processing chain.

In this paper, we present a vision for a resource-efficient infrastructure that enables just-in-time processing with regards to data ingest. At the heart of our approach are (1) the converting of compute intensive parts of the ETL process into in-database processing and (2) the activating of the process on demand via a system of flexible views.

Our approach avoids processing of data that is not being accessed any time soon, scales effectively with the database system and avoids administration and management overhead.

Keywords

Real-time BI ETL ELT Materialized Views 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Agrawal, D., Abbadi, A., Singh, A., Yurek, T.: Efficient View Maintenance at Data Warehouses. SIGMOD Record 26(2), 417–427 (1997)CrossRefGoogle Scholar
  2. 2.
    Andzic, J., Fiore, V., Sisto, L.: Extraction, Transformation, and Loading Processes. In: Wrembel, R., Koncilia, C. (eds.) Data Warehouses and OLAP: Concepts, Architectures and Solutions, pp. 88–110. IGI Global (2007)Google Scholar
  3. 3.
    Buchmann, A., Koldehofe, B.: Complex Event Processing. Information Technology 51(5), 241–242 (2009)Google Scholar
  4. 4.
    Bruckner, R.M., List, B., Schiefer, J.: Striving Towards Near Real-time Data Integration for Data Warehouses. In: Kambayashi, Y., Winiwarter, W., Arikawa, M. (eds.) DaWaK 2002. LNCS, vol. 2454, pp. 317–326. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  5. 5.
    Chaudhuri, S., Dayal, U., Narasayya, V.: An Overview of Business Intelligence Technology. Communications of the ACM 54(8), 88–98 (2011)CrossRefGoogle Scholar
  6. 6.
    Colby, L.S., Griffin, T., Libkin, L., Mumick, I.S., Trickey, H.: Algorithms for Deferred View Maintenance. SIGMOD Record 25(2), 469–480 (1996)CrossRefGoogle Scholar
  7. 7.
    Greenplum Inc.: Technical Documentation, http://www.greenplum.com
  8. 8.
    Gupta, A., Mumick, I.S.: Materialized Views: Techniques, Implementations, and Application, 1st edn. MIT Press (1999)Google Scholar
  9. 9.
    Kimball, R., Caserta, J.: The Data Warehouse ETL Toolkit. John Wiley & Sons Inc. (2004)Google Scholar
  10. 10.
    Microsoft Corporation. Microsoft SQL Server, Technical Documentation, http://www.microsoft.com/sql
  11. 11.
    Oracle Corporation: Technet. Technical Documentation, http://www.oracle.com/technetwork/indexes/documentation/index.html
  12. 12.
    Quass, D., Widom, J.: On-Line Warehouse View Maintenance. In: Proc. of ACM SIGMOD Int. Conf. on Management of Data, pp. 393–404 (1997)Google Scholar
  13. 13.
    Vassiliadis, P., Karagiannis, A., Tziovara, V., Simitsis, A.: Towards a Benchmark for ETL Workflows. In: Proc. of Int. Workshop on Quality in Databases (QDB), pp. 49–60 (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Tobias Freudenreich
    • 1
  • Pedro Furtado
    • 2
  • Christian Koncilia
    • 3
  • Maik Thiele
    • 4
  • Florian Waas
    • 5
  • Robert Wrembel
    • 6
  1. 1.Technische Universität DarmstadtGermany
  2. 2.University of CoimbraPortugal
  3. 3.University of KlagenfurtAustria
  4. 4.Technische Universität DresdenGermany
  5. 5.Greenplum/EMCSan MateoU.S.A.
  6. 6.Poznań University of TechnologyPoland

Personalised recommendations