Warui Wanjiruwaruithemystery.hashnode.dev·Dec 5, 2023Navigating the Data Ingestion Landscape: A Comprehensive Guide, Part 1Introduction In the realm of data engineering, the process of data ingestion plays a pivotal role in the journey of information from source systems to storage. This comprehensive guide aims to delve into the intricacies of data ingestion, exploring k...42 readsData Ingestiondata solutions architecture
Stephen David-Williamsstephendavidwilliams.com·Mar 12, 2023Incremental ingestion with Databricks’ Autoloader via File notificationsWhat is Autoloader? Autoloader (aka Auto Loader) is a mechanism in Databricks that ingests data from a data lake. The power of autoloader is that there is no need to set a trigger for ingesting new data in the data lake - it automatically pulls new f...1 like·1.3K readsDatabricks
