Abstract
In [12] we introduce a novel architecture for data processing, based on a functional fusion between a data and a computation layer. In this demo we show how this architecture is leveraged to offer significant speedups for data processing jobs such as data analysis and mining over large data sets.
One novel contribution of our solution is its data-driven approach. The computation infrastructure is controlled from within the data layer. Grid compute job submission events are based within the query processor on the DBMS side and in effect controlled by the data processing job to be performed. This allows the early deployment of on-the-fly data aggregation techniques, minimizing the amount of data to be transfered to/from compute nodes and is in stark contrast to existing Grid solutions that interact with data layers as external (mainly) “storage” components. By integrating scheduling intelligence in the data layer itself we show that it is possible to provide a close to optimal solution to the more general grid trade-off between required data replication costs and computation speed-up benefits. We validate this in a scenario derived from a real business deployment, involving financial customer profiling using common types of data analytics.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
The Condor Project. Online, at http://www.cs.wisc.edu/condor
The Global Grid Forum. Online, at http://www.gridforum.org
The Globus Alliance. Online, at http://www.globus.org
The Grid Physics Network. Online, at http://www.griphyn.org
The IBM DB2 Information Integrator. Online, at http://www.ibm.com/software/data/integration
The IBM DB2 Universal Database. Online, at http://www.ibm.com/software/data/db2
The IBM DB2 XML Extender. Online, at http://www.ibm.com/software/data/db2/extenders/xmlext
The Microsoft SQL Server. Online, at http://www.microsoft.com/sql
The Oracle Database. Online, at http://www.oracle.com/database
The Particle Physics Data Grid. Online, at http://www.ppdg.net
Ratner, J.: Human Factors and Web Development, 2nd edn. Lawrence Erlbaum Associates, Mahwah (2002)
Sion, R., Natarajan, R., Narang, I., Li, W.-S., Phan, T.: XG: A Data-driven Computation Grid for Enterprise-Scale Mining. In: Andersen, K.V., Debenham, J., Wagner, R. (eds.) DEXA 2005. LNCS, vol. 3588, pp. 828–837. Springer, Heidelberg (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Sion, R., Natarajan, R., Narang, I., Phan, T. (2006). XG: A Grid-Enabled Query Processing Engine. In: Ioannidis, Y., et al. Advances in Database Technology - EDBT 2006. EDBT 2006. Lecture Notes in Computer Science, vol 3896. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11687238_72
Download citation
DOI: https://doi.org/10.1007/11687238_72
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-32960-2
Online ISBN: 978-3-540-32961-9
eBook Packages: Computer ScienceComputer Science (R0)