samhita sarkarsamhita-sarkar.hashnode.dev·Nov 11, 2024Deciding When to Normalize or Denormalize Data for Best ResultsNormalisation and denormalisation are essentially design patterns in databases that determine how data is stored and used. Storage pattern : The key difference between normalisation and denormalisation is their storage patterns, which are essentially...Discuss·1 like·47 readsDatabases
Vishal Barvaliyavishalbarvaliya.hashnode.dev·Sep 17, 2024KPMG Pyspark interview questions for Data Engineer 2024.Image Source How do you deploy PySpark applications in a production environment? What are some best practices for monitoring and logging PySpark jobs? How do you manage resources and scheduling in a PySpark application? Write a PySpark job to per...Discusskpmg
Warui Wanjiruwaruithemystery.hashnode.dev·Dec 5, 2023Navigating the Data Ingestion Landscape: A Comprehensive Guide, Part 1Introduction In the realm of data engineering, the process of data ingestion plays a pivotal role in the journey of information from source systems to storage. This comprehensive guide aims to delve into the intricacies of data ingestion, exploring k...Discuss·41 readsData Ingestiondata solutions architecture
Kailash Sukumarankailashsukumaran.hashnode.dev·Sep 10, 2023Data Engineering Best Practices: Why Extract, Transform, and Load (ETL) Should Be DecoupledIn the ever-evolving world of data management and analytics, one principle remains steadfast: data is the lifeblood of informed decision-making. This data, often dispersed across various sources and formats, must be collected, transformed, and loaded...Discuss·93 readsdata
mani nekkalapudimaninekkalapudi.io·Jul 31, 2023Book Review- Fundamentals of Data EngineeringHi! Hope you’re doing well. Let me walk you through what’s going on in my head when I need to explain “What is Data Engineering? And What has been going on with it recently?”. Where do I start? And how to not kill an enthusiast or a friend with my te...Discuss·1 like·187 readsData Engineering data analysis
Warui Wanjiruwaruithemystery.hashnode.dev·Jul 20, 2023Unleashing the power of Data: Data Storage SystemsIntroduction As a data engineer, you'll encounter various data storage systems that play a crucial role in managing and accessing data. These storage systems exist at a higher level of abstraction than the raw storage ingredients, such as magnetic di...Discuss·10 likes·65 readsTechnical Tuesdaydata-engineering
Warui Wanjiruwaruithemystery.hashnode.dev·Jun 13, 2023Mastering the basics: The Data Science Hierarchy of NeedsWelcome back to our data engineering series! In our previous blogs, we explored the fascinating world of data engineering and its crucial role in shaping data-driven organizations. Today, we embark on a thrilling journey to unravel the secrets of the...Discuss·11 likes·146 readsTechnical Tuesdaydata
krishna chaitanyakrishnavulchi.hashnode.dev·Oct 29, 2022ETL Project for beginnersSimple ETL Web scrapping project where we extract data from a json, Transform the json data and Load it in one csv file Loading JSON file Transform JSON file Load json file into CSV Working json bank_data.json details Imports import glob import...Discuss·176 readsdata-engineering
