Advertisement

The Art of Side Effects

  • Zubair Nabi
Chapter

Abstract

Spark Streaming applications by design are stateless and side-effect free: running the same application an infinite number of times results in the same behavior and output. Similar to functional programming, this simplifies debugging and reasoning about the state of a program, because input and output paths are deterministic. Although side-effect-free applications have many advantages, in distributed systems side effects cannot be completely avoided, especially when interfacing with external systems. For this reason, Spark Streaming provides a primitive called foreachRDD, which is the Swiss Army Knife of side effects for micro-batch processing. This chapter introduces design patterns for enabling side effects in Spark Streaming applications.

Keywords

Work Node Static Connection Connection Object Column Family Driver Program 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Copyright information

© Zubair Nabi 2016

Authors and Affiliations

  • Zubair Nabi
    • 1
  1. 1.LahorePakistan

Personalised recommendations