Advertisement

Processing Partially Ordered Requests in Distributed Stream Processing Systems

  • Rijun Cai
  • Weigang WuEmail author
  • Ning Huang
  • Lihui Wu
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10048)

Abstract

In many application scenarios of distributed stream processing, there might be partial order relations among the requests. However, existing stream processing systems can not directly handle partially ordered requests, while indirect mechanisms are usually strongly coupled with business logic, which lack flexibility and have limited performance. We propose Pork, a novel distributed stream processing system targeting at partially ordered requests. In the experiments, the new system has achieved a parallelism and request throughput larger than the traditional mechanism in the presented example, and the performance overhead due to parallelism is considerably small. Then the scalability characteristic of the new system is discussed. What’s more, the experiment results also show that the new system has a more flexible load balancing ability.

Keywords

Processing Node Business Logic Dependency Group Partial Order Relation Message Queue 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

References

  1. 1.
    Apache Software Foundation: Apache Samza (2013). http://samza.apache.org/
  2. 2.
    Apache Software Foundation: Apache Flink (2015). http://flink.apache.org/
  3. 3.
    Ji, Y., Zhou, H., Jerzak, Z., Nica, A., Hackenbroich, G., Fetzer, C.: Quality-driven continuous query execution over out-of-order data streams. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, pp. 889–894. ACM (2015)Google Scholar
  4. 4.
    Kahn, A.B.: Topological sorting of large networks. Commun. ACM 5(11), 558–562 (1962)CrossRefzbMATHGoogle Scholar
  5. 5.
    Kreps, J., Narkhede, N., Rao, J.: Kafka: a distributed messaging system for log processing. In: NetDB (2011)Google Scholar
  6. 6.
    Lamport, L.: Time, clocks, and the ordering of events in a distributed system. Commun. ACM 21(7), 558–565 (1978)CrossRefzbMATHGoogle Scholar
  7. 7.
    Li, J., Tufte, K., Shkapenyuk, V., Papadimos, V., Johnson, T., Maier, D.: Out-of-order processing: a new architecture for high-performance stream systems. Proc. VLDB Endowment 1(1), 274–288 (2008)CrossRefGoogle Scholar
  8. 8.
    Meehan, J., Tatbul, N., Zdonik, S., Aslantas, C., Cetintemel, U., Du, J., Kraska, T., Madden, S., Maier, D., Pavlo, A., et al.: S-Store: streaming meets transaction processing. Proc. VLDB Endowment 8(13), 2134–2145 (2015)CrossRefGoogle Scholar
  9. 9.
    Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J.M., Kulkarni, S., Jackson, J., Gade, K., Fu, M., Donham, J., et al.: Storm@ Twitter. In: Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data, pp. 147–156. ACM (2014)Google Scholar
  10. 10.
    Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., Stoica, I.: Discretized streams: fault-tolerant streaming computation at scale. In: Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles, pp. 423–438. ACM (2013)Google Scholar

Copyright information

© Springer International Publishing AG 2016

Authors and Affiliations

  1. 1.School of Data and Computer ScienceSun Yat-sen UniversityGuangzhouChina
  2. 2.Guangdong Province Key Laboratory of Big Data Analysis and ProcessingGuangzhouChina

Personalised recommendations