Course Duration
2 Days
Databricks
Authorized Training
IT
Course cost:
£1,500.00
IT Certification Overview
This course serves as an appropriate entry point to learn Advanced Data Engineering with Databricks.
Below, we describe each of the four, four-hour modules included in this course.
Newto Training Reviews
What Our Happy Alumni Say About Us
I had absolutely no experience in this field, but Newto Training recognised my potential and supported me every step of the way. The real-life scenarios were invaluable in making me job-ready.
The tutors at Newto are consistently patient, clear and more than willing to help. I couldn't have asked for a better learning environment.
From day one I felt welcomed and supported. The live sessions were interactive and the resources well thought-out. I passed my exam on the first attempt thanks to them.
I was nervous about switching careers but the practical approach and continual 1-to-1 support gave me the confidence I needed to succeed."
The training material was excellent, and the instructors really brought the subject to life with hands-on lab work. I feel fully prepared for the workplace."
What impressed me most was the flexibility and responsiveness of the support team. Whenever I had a question, they were there even outside of normal hours.
The course structure was superb. A great mix of theory and practice delivered at a pace that made sense even for beginners. Highly recommended.
Switching into IT felt overwhelming, but Newto Training made the journey manageable and quite enjoyable. Their coaching style is top class.
I particularly appreciated the way complex topics were broken down into manageable chunks, and the tutor actively encouraged questions at every stage.
The live-taught interactive classes set Newto apart from other providers. It wasn't just video-based learning and the real engagement made all the difference.
The cyber course exceeded my expectations. They've been on hand for everything and are constantly in touch with me
The resources provided were spot on and the teaching style made everything easy to understand. I particularly appreciated the extra help offered when needed.
Enrolling with Newto was one of the best decisions I've made. The job-readiness focus, strong teaching and excellent support all stood out.
I completed my SQL course and now feel confident in my skills. The real-world examples and supportive tutors made it all click.
I've just passed my AZ-900 exam thanks to the support received from Ken in the boot camp and his exam preparation sessions.
I've gained a qualification and the confidence to use it in real-world scenarios.
Their commitment to student success is obvious from the personalised guidance to the quality of delivery. I'd recommend them without hesitation.
Every tutor I encountered was professional, encouraging and keen to see me succeed. That personal touch made a big difference.
I'm now in a role I once thought was out of reach, thank you to the all the team
The bootcamps were excellent, we were given practical projects to complete and I feel so much more confident with my coding skills
I had absolutely no experience in this field, but Newto Training recognised my potential and supported me every step of the way. The real-life scenarios were invaluable in making me job-ready.
The tutors at Newto are consistently patient, clear and more than willing to help. I couldn't have asked for a better learning environment.
From day one I felt welcomed and supported. The live sessions were interactive and the resources well thought-out. I passed my exam on the first attempt thanks to them.
I was nervous about switching careers but the practical approach and continual 1-to-1 support gave me the confidence I needed to succeed."
The training material was excellent, and the instructors really brought the subject to life with hands-on lab work. I feel fully prepared for the workplace."
What impressed me most was the flexibility and responsiveness of the support team. Whenever I had a question, they were there even outside of normal hours.
The course structure was superb. A great mix of theory and practice delivered at a pace that made sense even for beginners. Highly recommended.
Switching into IT felt overwhelming, but Newto Training made the journey manageable and quite enjoyable. Their coaching style is top class.
I particularly appreciated the way complex topics were broken down into manageable chunks, and the tutor actively encouraged questions at every stage.
The live-taught interactive classes set Newto apart from other providers. It wasn't just video-based learning and the real engagement made all the difference.
I had absolutely no experience in this field, but Newto Training recognised my potential and supported me every step of the way. The real-life scenarios were invaluable in making me job-ready.
The tutors at Newto are consistently patient, clear and more than willing to help. I couldn't have asked for a better learning environment.
From day one I felt welcomed and supported. The live sessions were interactive and the resources well thought-out. I passed my exam on the first attempt thanks to them.
I was nervous about switching careers but the practical approach and continual 1-to-1 support gave me the confidence I needed to succeed."
The training material was excellent, and the instructors really brought the subject to life with hands-on lab work. I feel fully prepared for the workplace."
What impressed me most was the flexibility and responsiveness of the support team. Whenever I had a question, they were there even outside of normal hours.
The course structure was superb. A great mix of theory and practice delivered at a pace that made sense even for beginners. Highly recommended.
Switching into IT felt overwhelming, but Newto Training made the journey manageable and quite enjoyable. Their coaching style is top class.
I particularly appreciated the way complex topics were broken down into manageable chunks, and the tutor actively encouraged questions at every stage.
The live-taught interactive classes set Newto apart from other providers. It wasn't just video-based learning and the real engagement made all the difference.
The cyber course exceeded my expectations. They've been on hand for everything and are constantly in touch with me
The resources provided were spot on and the teaching style made everything easy to understand. I particularly appreciated the extra help offered when needed.
Enrolling with Newto was one of the best decisions I've made. The job-readiness focus, strong teaching and excellent support all stood out.
I completed my SQL course and now feel confident in my skills. The real-world examples and supportive tutors made it all click.
I've just passed my AZ-900 exam thanks to the support received from Ken in the boot camp and his exam preparation sessions.
I've gained a qualification and the confidence to use it in real-world scenarios.
Their commitment to student success is obvious from the personalised guidance to the quality of delivery. I'd recommend them without hesitation.
Every tutor I encountered was professional, encouraging and keen to see me succeed. That personal touch made a big difference.
I'm now in a role I once thought was out of reach, thank you to the all the team
The bootcamps were excellent, we were given practical projects to complete and I feel so much more confident with my coding skills
The cyber course exceeded my expectations. They've been on hand for everything and are constantly in touch with me
The resources provided were spot on and the teaching style made everything easy to understand. I particularly appreciated the extra help offered when needed.
Enrolling with Newto was one of the best decisions I've made. The job-readiness focus, strong teaching and excellent support all stood out.
I completed my SQL course and now feel confident in my skills. The real-world examples and supportive tutors made it all click.
I've just passed my AZ-900 exam thanks to the support received from Ken in the boot camp and his exam preparation sessions.
I've gained a qualification and the confidence to use it in real-world scenarios.
Their commitment to student success is obvious from the personalised guidance to the quality of delivery. I'd recommend them without hesitation.
Every tutor I encountered was professional, encouraging and keen to see me succeed. That personal touch made a big difference.
I'm now in a role I once thought was out of reach, thank you to the all the team
The bootcamps were excellent, we were given practical projects to complete and I feel so much more confident with my coding skills
Prerequisites
Participants should have:
- Experience using PySpark APIs for advanced data transformations.
- Familiarity with implementing Python classes.
- Experience using SQL in production environments such as data warehouses or data lakes.
- Hands-on experience with Databricks notebooks and cluster configuration.
- Understanding of Delta Lake table creation and manipulation with SQL.
These prerequisites can be met by completing the Data Engineering with Databricks and Apache Spark Programming with Databricks courses and by earning the Databricks Certified Data Engineer Associate and Databricks Certified Associate Developer for Apache Spark certifications.
If you do not have one or more of the pre-requisites QA recommends: (can be taken in either order)
Target audience
This course is ideal for:
- Data engineers aiming to optimise and scale data processing in Databricks.
- Big data professionals working with streaming and batch data.
- Cloud engineers focused on Delta Lake architectures and Structured Streaming.
- Individuals preparing for the Databricks Certified Data Engineer Professional exam.
Learning Objectives
Databricks Streaming and Delta Live Tables
This course provides a comprehensive understanding of Spark Structured Streaming and Delta Lake, including computation models, configuration for streaming read, and maintaining data quality in a streaming environment.
Databricks Data Privacy
This content is intended for the learner persona of data engineers or for customers, partners, and employees who complete data engineering tasks with Databricks. It aims to provide them with the necessary knowledge and skills to execute these activities effectively on the Databricks platform.
Databricks Performance Optimization
In this course, you’ll learn how to optimize workloads and physical layout with Spark and Delta Lake and and analyze the Spark UI to assess performance and debug applications. We’ll cover topics like streaming, liquid clustering, data skipping, caching, photons, and more.
Automated Deployment with Databricks Asset Bundles
This course provides a comprehensive review of DevOps principles and their application to Databricks projects. It begins with an overview of core DevOps, DataOps, continuous integration (CI), continuous deployment (CD), and testing, and explores how these principles can be applied to data engineering pipelines.
The course then focuses on continuous deployment within the CI/CD process, examining tools like the Databricks REST API, SDK, and CLI for project deployment. You will learn about Databricks Asset Bundles (DABs) and how they fit into the CI/CD process. You’ll dive into their key components, folder structure, and how they streamline deployment across various target environments in Databricks. You will also learn how to add variables, modify, validate, deploy, and execute Databricks Asset Bundles for multiple environments with different configurations using the Databricks CLI.
Finally, the course introduces Visual Studio Code as an Interactive Development Environment (IDE) for building, testing, and deploying Databricks Asset Bundles locally, optimizing your development process. The course concludes with an introduction to automating deployment pipelines using GitHub Actions to enhance the CI/CD workflow with Databricks Asset Bundles.
By the end of this course, you will be equipped to automate Databricks project deployments with Databricks Asset Bundles, improving efficiency through DevOps practices.
Advanced Data Engineering with Databricks Course Content
Databricks Streaming and Delta Live Tables
- Streaming Data Concepts
- Introduction to Structured Streaming
- Demo: Reading from a Streaming Query
- Streaming from Delta Lake
- Lab: Streaming Query Lab
- Aggregation, Time Windows, Watermarks
- Event Time + Aggregatios over Time Windows
- Lab: Stream Aggregation Lab
- Demo: Windowed Aggregation with Watermark
- Data Ingestion Pattern
- Demo: Auto Load to Bronze
- Demo: Stream from Multiplex Bronze
- Quality Enforcement Pattern
- Demo: Quality Enforcement
- Lab: Streaming ETL Lab
Databricks Data Privacy
- Regulatory Compliance
- Data Privacy
- Key Concepts and Components
- Audit Your Data
- Data Isolation
- Demo: Securing Data in Unity Catalog
- Pseudonymization & Anonymization
- Summary & Best Practices
- Demo: PII Data Security
- Capturing Changed Data
- Deleting Data in Databricks
- Demo: Processing Records from CDF and Propagating Changes
- Lab: Propagating Changes with CDF Lab
Databricks Performance Optimization
- DevOps Spark UI Introduction
- Introduction to Designing Foundation
- Demo: File Explosion
- Data Skipping and Liquid Clustering
- Lab: Data Skipping and Liquid Clustering
- Skew
- Shuffles
- Demo: Shuffle
- Spill
- Lab: Exploding Join
- Serialization
- Demo: User-Defined Functions
- Fine-Tuning: Choosing the Right Cluster
- Pick the Best Instance Types
Automated Deployment with Databricks Asset Bundles
- DevOps Review
- Continuous Integration and Continuous Deployment/Delivery (CI/CD) Review
- Demo: Course Setup and Authentication
- Deploying Databricks Projects
- Introduction to Databricks Asset Bundles (DABs)
- Demo: Deploying a Simple DAB
- Lab: Deploying a Simple DAB
- Variable Substitutions in DABs
- Demo: Deploying a DAB to Multiple Environments
- Lab: Deploy a DAB to Multiple Environments
- DAB Project Templates Overview
- Lab: Use a Databricks Default DAB Template
- CI/CD Project Overview with DABs
- Demo: Continuous Integration and Continuous Deployment with DABs
- Lab: Adding ML to Engineering Workflows with DABs
- Developing Locally with Visual Studio Code (VSCode)
- Demo: Using VSCode with Databricks
- CI/CD Best Practices for Data Engineering
- Next Steps: Automated Deployment with GitHub Actions
Upcoming Dates
Dates and locations are available on request. Please contact us for the latest schedule.
Advance Your Career with Advanced Data Engineering with Databricks
Gain the skills you need to succeed. Enrol in Advanced Data Engineering with Databricks with Newto Training today.
