---
title: "Discussion on \"Building Incremental Pipelines with Auto Loader and Delta Lake on Azure Databricks\" | Hashnode"
description: "Discussion on \"Building Incremental Pipelines with Auto Loader and Delta Lake on Azure Databricks\". A practical step-by-step guide to ingesting files incrementally without managing state yourself"
url: "https://hashnode.com/posts/building-incremental-pipelines-with-auto-loader-and-delta-lake-on-azure-databricks/6a799d3b450d127a5189ce05"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Building Incremental Pipelines with Auto Loader and Delta Lake on Azure Databricks](https://dataengineeringwithayush.hashnode.dev/building-incremental-pipelines-with-auto-loader-and-delta-lake-on-azure-databricks?utm_source=hashnode&utm_medium=feed)

A practical step-by-step guide to ingesting files incrementally without managing state yourself

Aug 10 · 7 min read · dataengineeringwithayush.hashnode.dev  
Read the article

-   [#auto-loader](https://hashnode.com/tag/auto-loader)
-   [#delta-lake](https://hashnode.com/tag/delta-lake)
-   [#apache-spark](https://hashnode.com/tag/apache-spark)
-   [#databricks](https://hashnode.com/tag/databricks)
-   [#data-engineering](https://hashnode.com/tag/data-engineering)

## Written by

[Ayush Kumar](https://hashnode.com/@ak2312)

@ak2312 · Bengaluru, India

  
[View profile](https://hashnode.com/@ak2312)

I'm a Senior Data Engineer at Mu Sigma, currently working on large-scale supply chain data pipelines for a global FMCG client. My day-to-day involves building production-grade PySpark pipelines on Azure Databricks, working with Delta Lake, Unity Catalog, and Azure Data Factory. I hold Databricks Data Engineer Associate and Professional certifications, along with DP-900 and AZ-900. I enjoy breaking down complex engineering problems into something that's actually useful to read and the kind of content I wish had existed when I was figuring things out myself.

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
