Bridging Spark SQL with JDBC

Abstract

In the last chapter, we looked at common patterns and techniques for harnessing the powerful core functionality available to us when transforming data using Spark SQL and the DataFrame APIs. While we certainly covered a lot of ground, we purposefully skipped over some of the more exciting capabilities available to us under the Spark SQL umbrella. Along that line, wouldn’t it seem to only make sense that we should be capable of connecting to and working directly with remote databases from the comfort of Apache Spark SQL? Additionally, wouldn’t it also be advantageous to use SQL's strongly typed semantics when reading data into Spark? Couldn’t we somehow also marry these rich type systems (inherent to Java/Scala) with both SQL and the strong internal typing mechanics of Apache Spark itself? Luckily, that is exactly what you will learn to do in this chapter.