XG: A Grid-Enabled Query Processing Engine

  • Radu Sion
  • Ramesh Natarajan
  • Inderpal Narang
  • Thomas Phan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 3896)


In [12] we introduce a novel architecture for data processing, based on a functional fusion between a data and a computation layer. In this demo we show how this architecture is leveraged to offer significant speedups for data processing jobs such as data analysis and mining over large data sets.

One novel contribution of our solution is its data-driven approach. The computation infrastructure is controlled from within the data layer. Grid compute job submission events are based within the query processor on the DBMS side and in effect controlled by the data processing job to be performed. This allows the early deployment of on-the-fly data aggregation techniques, minimizing the amount of data to be transfered to/from compute nodes and is in stark contrast to existing Grid solutions that interact with data layers as external (mainly) “storage” components. By integrating scheduling intelligence in the data layer itself we show that it is possible to provide a close to optimal solution to the more general grid trade-off between required data replication costs and computation speed-up benefits. We validate this in a scenario derived from a real business deployment, involving financial customer profiling using common types of data analytics.


Data Layer Query Processor Grid Schedule Grid Scheduler Hierarchical Grid 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    The Condor Project. Online, at
  2. 2.
    The Global Grid Forum. Online, at
  3. 3.
    The Globus Alliance. Online, at
  4. 4.
    The Grid Physics Network. Online, at
  5. 5.
    The IBM DB2 Information Integrator. Online, at
  6. 6.
    The IBM DB2 Universal Database. Online, at
  7. 7.
    The IBM DB2 XML Extender. Online, at
  8. 8.
    The Microsoft SQL Server. Online, at
  9. 9.
    The Oracle Database. Online, at
  10. 10.
    The Particle Physics Data Grid. Online, at
  11. 11.
    Ratner, J.: Human Factors and Web Development, 2nd edn. Lawrence Erlbaum Associates, Mahwah (2002)Google Scholar
  12. 12.
    Sion, R., Natarajan, R., Narang, I., Li, W.-S., Phan, T.: XG: A Data-driven Computation Grid for Enterprise-Scale Mining. In: Andersen, K.V., Debenham, J., Wagner, R. (eds.) DEXA 2005. LNCS, vol. 3588, pp. 828–837. Springer, Heidelberg (2005)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2006

Authors and Affiliations

  • Radu Sion
    • 1
  • Ramesh Natarajan
    • 2
  • Inderpal Narang
    • 3
  • Thomas Phan
    • 3
  1. 1.Stony Brook University 
  2. 2.IBM TJ Watson Research Lab 
  3. 3.IBM Almaden Research Lab 

Personalised recommendations