Shreyan Dasshreyandas.hashnode.dev·Jul 19, 2024Need for speed, but where do I store all of this data?Dear Diary, The most dreaded day in the life of any data engineer came to me a while ago — our TL sat us down to tell us that we had been billed an enormous amount for our cloud storage in the last quarter, and we needed to find a way to cut down sto...Kev and 1 other are discussing this2 people are discussing thisDiscuss·40 likesConfessions of a Data Engineerdata-engineering
Blaise Luisblaiseluis.hashnode.dev·Jul 17, 2024Step-by-Step ETL Process Guide: Tools, Tips, and Best PracticesETL Processes: Detailed Guide on Extract, Transform, Load (ETL) Processes, Tools, and Best Practices In today’s data-driven world, ETL (Extract, Transform, Load) processes are crucial for data integration, management, and analysis. This detailed guid...DiscussETL
Abhishek JaiswalforAbhishek Jaiswal's team blogdataplumbing.hashnode.dev·Jul 15, 2024Comprehensive Guide to DBT (Data Build Tool)What is dbt? dbt (Data Build Tool) is an open-source command-line tool that helps analysts and engineers transform data in their data warehouse. It allows you to manage your data transformations with SQL in a version-controlled and collaborative envi...Discuss·10 likesdata-engineering
Sudha Yadavsudhayadav.hashnode.dev·Jul 10, 2024AWS GlueIn the world of data ETL (Extract, Transform, Load) is a fundamental process that involves extracting data from various sources, transforming it into a suitable format and loading it into a data warehouse or other storage systems. AWS Glue is a fully...Discuss·2 likesAWS
Abhishek JaiswalforAbhishek Jaiswal's team blogdataplumbing.hashnode.dev·Jul 9, 2024The Rise of Zero ETL: Revolutionizing Data IntegrationIn the rapidly evolving landscape of data management, businesses are constantly seeking more efficient ways to handle and utilize data. One of the most groundbreaking advancements in this arena is the concept of Zero ETL (Extract, Transform, Load). T...Discuss·1 likeETL
Parth Ladthenavigatedata.com·Jun 22, 2024Avoid Repetitive Steps in MS Fabric Dataflow Gen2 with Custom FunctionsWhile working with Dataflow Gen2 or Power BI, If you keep finding yourself needing to apply the same transformations to different queries or values, making a custom function in Power Query can save you loads of time. It's like using SQL functions to ...DiscussMicrosoft FabricMicrosoft
Kunal GuptaforPeerDB Blogblog.peerdb.io·Jun 19, 2024PeerDB is now SOC 2 Type 2 CompliantAt PeerDB, security has always been a top priority. Our customers trust us with their critical data, and we are dedicated to upholding the highest standards of data protection and security. We are excited to announce that PeerDB has achieved SOC 2 Ty...Discuss·102 readsPostgreSQL
Sai SrirampurforPeerDB Blogblog.peerdb.io·Jun 13, 2024Overcoming Pitfalls of Postgres Logical DecodingAt PeerDB, we are building a fast and simple way to replicate data from Postgres to data warehouses like Snowflake, ClickHouse etc. and queues such as Kafka, Redpanda etc. We implement Postgres Change Data Capture (CDC) to reliably replicate changes ...Discuss·3.1K readsPostgreSQL
Ismail Khanismail-de.hashnode.dev·Jun 11, 2024Who is Data EngineerWe have a same definition with rephrased words for Data Engineering as " a professional responsible for designing, constructing, and maintaining systems and architectures that collect, store, process, and analyze large volumes of data. They create da...Discuss·33 readsdata-engineering
Brandon ClappProbrandonclapp.com·May 25, 2024Apache Airflow: The Key to Scheduled Data PipelinesIn the rapidly evolving landscape of data engineering, orchestrating and automating complex workflows has become a fundamental necessity. Businesses are increasingly dependent on data-driven insights, requiring robust systems to manage the seamless f...DiscussETLapache-airflow
