Deploying Mission-Critical Spark Applications on Kubernetes

Abstract

Before diving headfirst into what Kubernetes (K8s) is, and how Apache Spark fits into the distributed K8s ecosystem, it is important to first begin by stating simply that Kubernetes enables Apache Spark applications to run in isolation, pairing elastic scalability with the runtime consistency of containers, collocated in independent micro-environments called pods (which you’ll learn about soon). Ultimately, you can rely on consistent runtime environments, without having to deal with the pain and other hardships of multi-tenancy in a share-everything ecosystem. Rather, imagine each application running in its own isolated world, which at a high-level act similarly to the local environments you’ve run to power Spark applications using docker-compose on the local data platform we’ve been constructing throughout this book.