Implementing the Palomar Transient Factory Real-Time Detection Pipeline in GLADE: Results and Observations

  • Florin Rusu
  • Peter Nugent
  • Kesheng Wu
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8381)


Palomar Transient Factory is a comprehensive detection system for the identification and classification of transient astrophysical objects. The central piece in the identification pipeline is represented by an automated classifier that distinguishes between real and bogus objects with high accuracy. Given that the classifier has to identify the most significant transients out of a large number of candidates in near real-time, the response time it provides is of critical importance. In this paper, we present an experimental study that evaluates a novel implementation of the classifier in GLADE—a parallel data processing system that combines the efficiency of a database with the extensibility of Map-Reduce. We show how each stage in the classifier – candidate identification, pruning, and contextual realbogus – maps optimally into GLADE tasks by taking advantage of the unique features of the system—range-based data partitioning, columnar storage, multi-query execution, and in-database support for complex aggregate computation. The result is an efficient classifier implementation capable to process a new set of acquired images in a matter of minutes even on a low-end server. For comparison, an optimized PostgreSQL implementation of the classifier takes hours on the same machine.


parallel databases multi-query processing real-time classification 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Palomar Transient Factory (November 2013),
  2. 2.
    Law, N.M., et al.: The Palomar Transient Factory: System Overview, Performance and First Results. CoRR abs/0906.5350 (2009)Google Scholar
  3. 3.
    Bloom, J.S., et al.: Automating Discovery and Classification of Transients and Variable Stars in the Synoptic Survey Era. CoRR abs/1106.5491 (2011)Google Scholar
  4. 4.
    Grillmair, C.J., et al.: An Overview of the Palomar Transient Factory Pipeline and Archive at the Infrared Processing and Analysis Center. In: Astronomical Data Analysis Software and Systems XIX. ASP Conf. Ser., vol. 434, pp. 28–36 (2010)Google Scholar
  5. 5.
    Cheng, Y., Qin, C., Rusu, F.: GLADE: Big Data Analytics Made Easy. In: Proceedings of 2012 ACM SIGMOD International Conference on Management of Data, pp. 697–700 (2012)Google Scholar
  6. 6.
    PostgreSQL, (November 2013)
  7. 7.
    Python Programming Language (November 2013),
  8. 8.
    Cheng, Y., Rusu, F.: Astronomical Data Processing in EXTASCID. In: Proceedings of 2013 SSDBM Conf. on Sci. and Stat. Database Management, pp. 387–390 (2013)Google Scholar
  9. 9.
    Arumugam, S., Dobra, A., Jermaine, C., Pansare, N., Perez, L.: The DataPath System: A Data-Centric Analytic Processing Engine for Large Data Warehouses. In: Proceedings of 2010 ACM SIGMOD International Conference on Management of Data, pp. 519–530 (2010)Google Scholar
  10. 10.
    Rusu, F., Dobra, A.: GLADE: A Scalable Framework for Efficient Analytics. Operating Systems Review 46(1), 12–18 (2012)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Florin Rusu
    • 1
  • Peter Nugent
    • 2
  • Kesheng Wu
    • 2
  1. 1.University of CaliforniaMercedUSA
  2. 2.Lawrence Berkeley National LaboratoryBerkeleyUSA

Personalised recommendations