Microsoft PolyBase Hadoop and Blob Storage Processing Big Data and Cloud Objects

  • Kevin Feasel

Your browser needs to be JavaScript capable to view this video

Try reloading this page, or reviewing your browser settings

You're watching a preview of subscription content. Log in to check access

This video dives into Microsoft SQL Server’s PolyBase feature for virtualizing data from Apache Hadoop and Azure Blob Storage into SQL Server. From this video you will learn how to configure SQL Server and PolyBase to connect to Azure Blob Storage, including how to make connections to public and secured blob containers. You’ll also learn how to make connections to Hadoop clusters and query data that is stored in Hadoop.

The video begins by showing how to read from and insert data into Azure Blob Storage. From there, you will learn how to connect PolyBase to a Hadoop cluster, whether in Cloudera’s distribution of Hadoop or the Hortonworks Data Platform. Once connected, you will learn how to query data in Hadoop using two methods: streaming and predicate pushdown. Finally, the video provides key tips on troubleshooting Hadoop integrations, including tips on where to look for log data as well as common configuration and data issues.

What You Will Learn

  • Connect to Azure Blob Storage and Hadoop clusters from SQL Server

  • Uncover configuration options needed to connect to a Hadoop cluster

  • Use predicate pushdown to create MapReduce jobs which offload work to a Hadoop cluster

  • Write data to Hadoop and Azure Blob Storage using T-SQL from SQL Server

  • Troubleshoot common errors when connecting to Hadoop or Azure Blob Storage

Who This Video Is For

Data platform specialists who need to work on multiple data platforms but want to use one unifying language to tie them together. Viewers should be familiar with SQL Server and the T-SQL language, and should be familiar with the content from the prior video in the series, PolyBase Concepts and Configuration.

This video dives into Microsoft PolyBase’s feature set for virtualizing data from Azure Blob Storage and Apache Hadoop into SQL Server. Learn to process large objects and big data from Azure Blob Storage and Hadoop clusters as if those objects and data were in SQL Server.

About The Author

Kevin Feasel

Kevin Feasel is a Microsoft Data Platform MVP and CTO at Envizage, where he specializes in data analytics with T-SQL and R, forcing Spark clusters to do his bidding, fighting with Kafka, and pulling rabbits out of hats on demand. He is the lead contributor to Curated SQL and author of PolyBase Revealed (Apress, 2019). A resident of Durham, North Carolina, he can be found cycling the trails along the triangle whenever the weather’s nice enough.

 

Supporting material

View source code at GitHub.

About this video

Author(s)
Kevin Feasel
DOI
https://doi.org/10.1007/978-1-4842-6584-0
Online ISBN
978-1-4842-6584-0
Total duration
1 hr 29 min
Publisher
Apress
Copyright information
© Kevin Feasel 2020

Related content

Video Transcript

[MUSIC PLAYING]

Hi, I’m Kevin Feasel. And this is Microsoft PolyBase, Hadoop and Blob Storage. In the prior video, Microsoft PolyBase Concepts and Configurations, we looked at how to install and configure PolyBase. Additionally, we covered some of the core ideas behind PolyBase including the trio of external objects.

In this video, we’re going to dive deeper into Hadoop and Azure Blob Storage as PolyBase data sources. We’ll learn where PolyBase does and does not work when connecting to Hadoop, understand the configuration settings needed to get everything working, and learn what to do when things break down. Join me as we join together two important data sources with SQL Server in Microsoft PolyBase, Hadoop and Blob Storage.