Walnut DataTech
Create account

Data engineering with Spark and Airflow

Build scalable batch and streaming pipelines with Apache Spark, Delta Lake and Databricks, orchestrated by Airflow and modelled with dbt.

No reviews yetTaught by Walnut Jobs course team
Level
Advanced
Language
English
Duration
36 hours
Lessons
16

Last updated 22 Oct 2026

What you'll learn.

  • Design a lakehouse with bronze, silver and gold layers on open file and table formats.
  • Write efficient PySpark transformations and diagnose shuffles, skew and spills.
  • Use Delta Lake MERGE, time travel and table maintenance on Databricks.
  • Process streaming data with Spark Structured Streaming and checkpoints.
  • Orchestrate idempotent, backfillable pipelines with Apache Airflow.
  • Model, test and document warehouse transformations with dbt and deploy them through CI/CD.

How this course is taught.

Every module comes to you in four parts, so you can watch, read, discuss and check what you have learned.

  1. Quadrant I

    e-Tutorial

    Watch a short video lecture for each module, with full text notes beside it so you can review or translate at your own pace.

  2. Quadrant II

    e-Content

    Download or print the reading material for each module, with key terms, case studies and links to further resources.

  3. Quadrant III

    Discussion forum

    Ask questions and clear your doubts with other learners. The course team guides the discussion and answers questions.

  4. Quadrant IV

    Assessment

    Check your understanding with quizzes, short and long-answer practice and FAQs, then take the final test.

Curriculum.

5 modules · 16 lessons
1The modern data platform3 lessons · 46 mine-Content (notes)2Assessment1
  • Note: Batch, streaming and the lakehouse18 min
  • Note: File formats and table formats18 min
  • Quiz: Module 1 quiz10 min
2Processing data with Apache Spark4 lessons · 1 h 19 mine-Content (notes)3Assessment1
  • Note: Spark architecture and DataFrames22 min
  • Note: Joins, shuffles and performance tuning22 min
  • Note: Structured Streaming20 min
  • Quiz: Module 2 quiz15 min
3Delta Lake on Databricks3 lessons · 52 mine-Content (notes)2Assessment1
  • Note: Working in Databricks18 min
  • Note: Delta Lake operations: MERGE, time travel and maintenance22 min
  • Quiz: Module 3 quiz12 min
4Orchestration with Apache Airflow3 lessons · 54 mine-Content (notes)2Assessment1
  • Note: Airflow DAGs, operators and scheduling22 min
  • Note: Reliable pipelines: idempotency, backfills and data quality20 min
  • Quiz: Module 4 quiz12 min
5Transformation with dbt and platform operations3 lessons · 1 he-Content (notes)2Final test1
  • Note: Modelling and testing with dbt20 min
  • Note: CI/CD, cost and governance20 min
  • Final test: Final assessment20 min

Your certificate.

This is the certificate you receive when you complete the course and pass its final test.

Certificate No.

WDT-WJDES-2026-000000

Certificate of completion

Data Engineering with Spark and Airflow

This is to certify that

Learner Name

has successfully completed the course and passed its assessment, and is recognised as a

Data engineering with Spark and Airflow

5 modules16 lessons36 hours of learningOnline, on Walnut LMS

DD Month YYYY

Date of issue

Noida, India

Authorised Signatory

Walnut DataTech Private Limited

Verify this certificate at walnutdatatech.com/verify-certificate with certificate number WDT-WJDES-2026-000000. Issued on Walnut LMS.

Specimen — not a valid certificate
Data engineering with Spark and AirflowView full size

What you need to receive it.

Complete every lesson and pass the final assessment with at least 70%.

Every certificate carries its own number that anyone can check on the verification page.

About this course.

Data engineers keep the numbers flowing: from payment systems, apps and sensors into a lakehouse that analysts, data scientists and regulators can trust. Global employers ask for the same core stack again and again: Apache Spark for processing, Delta Lake or another open table format for storage, Airflow for orchestration and dbt for modelling in SQL.

This advanced course teaches that stack with production habits: partitioning and shuffle tuning, idempotent loads, incremental processing, data quality tests, CI/CD and cost control. Examples use PySpark and Databricks, and the concepts transfer to Amazon EMR, Google Dataproc, Azure Synapse and Microsoft Fabric.

Who this is for

  • Data and software engineers moving into big data roles
  • Analytics engineers who want to own pipelines end to end
  • Data scientists who need to productionise feature pipelines
  • Candidates applying for roles listing Spark, Databricks, Airflow or dbt

Skills this certificate proves: Apache Spark (Advanced), Airflow (Proficient), Databricks (Proficient), dbt (Working).

Linked to Walnut Jobs: complete this course to earn the certificate employers ask for.

Your instructor.

Walnut Jobs course teamJob-ready courses designed with hiring companies

Learner reviews.

No reviews yet

No reviews yet. Learners can review a course after completing half of it.

Questions about enrolling.

Is this course really free?

Yes. Create an account and enrol with one click. There's no payment step.

How long can I access this course?

You keep access for as long as the course is offered on Walnut LMS.

Do I get a certificate?

Yes. The certificate section above shows the certificate and sets out what you need to receive it.

Data and AI