Towards Scalable Architectures for Clickstream Data Warehousing

  • Peter Alvaro
  • Dmitriy V. Ryaboy
  • Divyakant Agrawal
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4777)


Click-stream data warehousing has emerged as a monumental information management and processing challenge for commercial enterprises. Traditional solutions based on commercial DBMS technology often suffer from poor scalability and large processing latencies. One of the main problems is that click-stream data is inherently collected in a distributed manner, but in general these distributed click-stream logs are collated and pushed upstream in a centralized database storage repository, creating storage bottlenecks. In this paper, we propose a design of an ad-hoc retrieval system suitable for click-stream data warehouses, in which the data remains distributed and database queries are rewritten to be executed against the distributed data. The query rewrite does not involve any centralized control and is therefore highly scalable. The elimination of centralized control is achieved by supporting a restricted subset of SQL, which is sufficient for most click-stream data analysis. Evaluations conducted using both synthetic and real data establish the viability of this approach.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
  2. 2.
    Oracle products & services,
  3. 3.
    Teradata: A division of ncr,
  4. 4.
    Androutsellis-Theotokis, S., Spinellis, D.: A survey of peer-to-peer content distribution technologies. ACM Comput. Surv. 36(4), 335–371 (2004)CrossRefGoogle Scholar
  5. 5.
    Chang, F., Dean, J., Ghemawat, S., Hsieh, W.C., Wallach, D.A., Burrows, M., Chandra, T., Fikes, A., Gruber, R.: Bigtable: A distributed storage system for structured data (awarded best paper!). In: OSDI, pp. 205–218. USENIX Association (2006)Google Scholar
  6. 6.
    Dean, J., Ghemawat, S.: Mapreduce: Simplified data processing on large clusters. In: OSDI, pp. 137–150 (2004)Google Scholar
  7. 7.
    Ghemawat, S., Gobioff, H., Leung, S.-T.: The google file system. In: SOSP, pp. 29–43 (2003)Google Scholar
  8. 8.
    Graefe, G.: Volcano - an extensible and parallel query evaluation system. IEEE Trans. Knowl. Data Eng. 6(1), 120–135 (1994)CrossRefGoogle Scholar
  9. 9.
    Isard, M., Budiu, M., Birrell, A., Fetterly, D.: Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks. In: EuroSys 2007, Lisbon, Portugal (March 2007)Google Scholar
  10. 10.
    Kossmann, D.: The state of the art in distributed query processing. ACM Comput. Surv. 32(4), 422–469 (2000)CrossRefGoogle Scholar
  11. 11.
    Kumar, R., Olston, C., Reed, B., Srivastava, U., Tomkins, A.: Research project: Pig,
  12. 12.
    Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the data: Parallel analysis with Sawzall. Scientific Programming Journal 13(4), 277–298 (2005)Google Scholar
  13. 13.
    Shen, K., Yang, T., Chu, L., Holliday, J., Kuschner, D.A., Zhu, H.: Neptune: Scalable replication management and programming support for cluster-based network services. In: USITS, pp. 197–208 (2001)Google Scholar
  14. 14.
    Venugopal, S., Buyya, R., Ramamohanarao, K.: A taxonomy of data grids for distributed data sharing, management, and processing. ACM Comput. Surv. 38(1), 3 (2006)CrossRefGoogle Scholar
  15. 15.
    Yu, C.T., Chang, C.C.: Distributed query processing. ACM Comput. Surv. 16(4), 399–433 (1984)MATHCrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Peter Alvaro
    • 1
  • Dmitriy V. Ryaboy
    • 1
  • Divyakant Agrawal
    • 1
  1., 555 12th Street, Suite 500, Oakland, CA 94607USA

Personalised recommendations