In this course data engineers access data where it lives and then apply data extraction best practices, including schemas, corrupt record handling, and parallelized code. By the end of this course, you will extract data from multiple sources, use schema inference and apply user-defined schemas, and navigate Databricks and Apache Spark™ documents to source solutions.
- Write a basic ETL pipeline using the Spark design pattern
- Ingest data using DBFS mounts in Azure Blob Storage and S3
- Ingest data using serial and parallel JDBC reads
- Define and apply a user-defined schema to semi-structured JSON data
- Handle corrupt records
- Productionize an ETL pipeline
- Course Overview and Setup
- ETL Process Overview
- Connecting to Azure Blob Storage and S3
- Connecting to JDBC
- Applying Schemas to JSON Data
- Corrupt Record Handling
- Loading Data and Productionalizing
- Capstone Project: Parsing Nested Data
Supported platforms include Azure Databricks, Databricks Community Edition, and non-Azure Databricks.
- If you're planning to use the course on Azure Databricks, select the "Azure Databricks" Platform option.
- If you're planning to use the course on Databricks Community Edition or on a non-Azure version of Databricks, select the "Other Databricks" Platform option.
The course is a series of seven self-paced lessons available in both Scala and Python. A final capstone project involves writing an end-to-end ETL job that loads semi-structured JSON data into a relational model. Each lesson includes hands-on exercises.