StreamMine3G: Elastic and Fault Tolerant Large Scale Stream Processing
During the past decade, we have been witnessing a massive growth of data. In particular the advent of new mobile devices such as smartphones, tablets and online services like Facebook and Twitter created a complete new era for data processing. Although there exist already well-established approaches such as MapReduce (Dean and Ghemawat 2008) and its open-source implementation Hadoop (2015) in order to cope with these large amounts of data, there is a recent trend of moving away from batch processing to low-latency online processing using event stream processing (ESP) systems. Inspired by the simplicity of the MapReduce programming paradigm, a number of open-source as well as commercial ESP systems have evolved over the past years such as Apache S4 (Neumeyer et al. 2010; Apache 2015) (originally pushed by Yahoo!), Apache Storm (2015) (Twitter), and Apache Samza (2015) (LinkedIn), addressing the strong need for data processing in near real time.
Since the amount of data...
- Apache s4 – distributed stream computing platform (2015). https://incubator.apache.org/s4/
- Apache samza – a distributed stream processing framework (2015). http://samza.incubator.apache.org/
- Apache storm – distributed and faulttolerant realtime computation (2015). https://storm.incubator.apache.org/
- Gu Y, Zhang Z, Ye F, Yang H, Kim M, Lei H, Liu Z (2009) An empirical study of high availability in stream processing systems. In: Proceedings of the 10th ACM/IFIP/USENIX international conference on middleware, Middleware ’09, New York, pp 23:1–23:9. Springer, New YorkGoogle Scholar
- Hadoop mapreduce open source implementation (2015). http://hadoop.apache.org/
- Hunt P, Konar M, Junqueira FP, Reed B (2010) Zookeeper: wait-free coordination for Internet-scale systems. In: Proceedings of the 2010 USENIX conference on USENIX annual technical conference, USENIXATC’10, Berkeley. USENIX Association, pp 1–14Google Scholar
- Hwang J-H, Balazinska M, Rasin A, Cetintemel U, Stonebraker M, Zdonik S (2005) High-availability algorithms for distributed stream processing. In: Proceedings of the 21st international conference on data engineering, ICDE ’05, Washington, DC. IEEE Computer Society, pp 779–790Google Scholar
- Martin A, Fetzer C, Brito A (2011) Active replication at (almost) no cost. In: Proceedings of the 2011 IEEE 30th international symposium on reliable distributed systems, SRDS ’11, Washington, DC. IEEE Computer Society, pp 21–30Google Scholar
- Martin A, Knauth T, Creutz S, Becker D, Weigert S, Fetzer C, Brito A (2011) Low-overhead fault tolerance for high-throughput data processing systems. In: Proceedings of the 2011 31st international conference on distributed computing systems, ICDCS ’11, Washington, DC. IEEE Computer Society, pp 689–699Google Scholar
- Martin A, Smaneoto T, Dietze T, Brito A, Fetzer C (2015) User-constraint and self-adaptive fault tolerance for event stream processing systems. In: Proceedings of The 45th annual IEEE/IFIP international conference on dependable systems and networks (DSN 2015), Los Alamitos. IEEE Computer SocietyGoogle Scholar
- Neumeyer L, Robbins B, Nair A, Kesari A (2010) S4: distributed stream computing platform. In: Proceedings of the 2010 IEEE international conference on data mining workshops, ICDMW ’10, Washington, DC. IEEE Computer Society, pp 170–177Google Scholar