International Workshop on Databases in Networked Information Systems

DNIS 2014: Databases in Networked Information Systems pp 53-66

Implementing the Palomar Transient Factory Real-Time Detection Pipeline in GLADE: Results and Observations

  • Florin Rusu
  • Peter Nugent
  • Kesheng Wu
Conference paper

DOI: 10.1007/978-3-319-05693-7_4

Volume 8381 of the book series Lecture Notes in Computer Science (LNCS)
Cite this paper as:
Rusu F., Nugent P., Wu K. (2014) Implementing the Palomar Transient Factory Real-Time Detection Pipeline in GLADE: Results and Observations. In: Madaan A., Kikuchi S., Bhalla S. (eds) Databases in Networked Information Systems. DNIS 2014. Lecture Notes in Computer Science, vol 8381. Springer, Cham

Abstract

Palomar Transient Factory is a comprehensive detection system for the identification and classification of transient astrophysical objects. The central piece in the identification pipeline is represented by an automated classifier that distinguishes between real and bogus objects with high accuracy. Given that the classifier has to identify the most significant transients out of a large number of candidates in near real-time, the response time it provides is of critical importance. In this paper, we present an experimental study that evaluates a novel implementation of the classifier in GLADE—a parallel data processing system that combines the efficiency of a database with the extensibility of Map-Reduce. We show how each stage in the classifier – candidate identification, pruning, and contextual realbogus – maps optimally into GLADE tasks by taking advantage of the unique features of the system—range-based data partitioning, columnar storage, multi-query execution, and in-database support for complex aggregate computation. The result is an efficient classifier implementation capable to process a new set of acquired images in a matter of minutes even on a low-end server. For comparison, an optimized PostgreSQL implementation of the classifier takes hours on the same machine.

Keywords

parallel databases multi-query processing real-time classification 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Florin Rusu
    • 1
  • Peter Nugent
    • 2
  • Kesheng Wu
    • 2
  1. 1.University of CaliforniaMercedUSA
  2. 2.Lawrence Berkeley National LaboratoryBerkeleyUSA