An Integrated Architecture for Real-Time and Historical Analytics in Financial Services
The integration of historical data has become one of the most pressing issues for the financial services industry: trading floors rely on real-time analytics of ticker data with very strong emphasis on speed, not scale, yet, a large number of critical tasks, including daily reporting and backtesting of models, put emphasis on scale. As a result, implementers continuously face the challenge of having to meet contradicting requirements and either scale real-time analytics technology at considerable cost, or deploy separate stacks for different tasks and keep them synchronized—a solution that is no less costly.
In this paper, we propose Adaptive Data Virtualization, as an alternative approach, to overcome this problem. ADV lets applications use different data management technologies without the need for database migrations or re-configuration of applications. We review the incumbent technology and compare it with the recent crop of MPP databases and draw up a strategy that, using ADV, lets enterprises use the right tool for the right job flexibly. We conclude the paper summarizing our initial experience working with customers in the field and outline an agenda for future research.
- 1.Garland, S.: Big Data Analytics: Tackling the Historical Data Challenge. Wired Magazine, Innovation Insights, October 2014Google Scholar
- 2.SAP HANA, May 2015. http://hana.sap.com/abouthana.html
- 3.Kx Systems, May 2015. http://kx.com
- 4.MemSQL, May 2015. http://www.memsql.com
- 5.New York Stock Exchange: Market Data, Data Products-Daily TAQ, May 2015. http://www.nyxdata.com/Data-Products/Daily-TAQ
- 6.Security Technology Analysis Center: STAC Benchmark Council-Various Benchmark Results, May 2015. http://stacresearch.com/research
- 7.Shasha, D.: KDB+ Database and Language Primer. Kx Systems, May 2005. http://kx.com/q/d/primer.htm
- 8.Soliman, M., et al.: Orca: a modular query optimizer architecture for big data. In: ACM SIGMOD Conference, May 2014Google Scholar
- 9.Informatica, May 2015. http://www.informatica.com
- 10.Amazon Redshift, May 2015. http://aws.amazon.com/redshift
- 11.IBM Netezza, May 2015. www.ibm.com/software/data/netezza
- 12.PostgreSQL, May 2015. http://www.postgresql.org