Nirja Modinirjamodi.hashnode.dev·Jul 22, 2024Working towards my GOAL :)Hello! Welcome to my first blog. After some recent reflection, I decided to start blogging. With some significant changes happening in my life, I thought it would be meaningful to share my journey rather than keeping it to myself :) Here are a few th...DiscussData Science
Alex Mercedalexmerced.hashnode.dev·Jul 12, 2024Databases Deconstructed: The Value of Data Lakhouses and Table FormatsCheckout out my Apache Iceberg Crash Course Get a free copy of Apache Iceberg the Definitive Guide Databases and data warehouses are powerful systems that simplify working with data by abstracting many of the inherent challenges, including: Storage...Discussdata engineer
Giang Ngogiangblackk.hashnode.dev·Jun 29, 2024Spark Connect - Streamline Apache Spark Data Pipeline Development in DagsterIntroduction In many years, the conventional way to develop data pipelines in Dagster with Apache Spark is submitting applications, which pose unique challenges to developers: Require a separate infrastructure (cluster of machines with a cluster man...Discussdagster
Shreyan Dasshreyandas.hashnode.dev·Jun 21, 2024Dear Diary, they call me a Data Engineer (but mostly it's just crunching numbers)Ah, Friday. The glorious day when the keyboard clicks slow down, the urgent Slack pings fade to a gentle hum, and I have a precious moment to reflect. Reflect on what, you ask? Well, on the air-quotes glorious, never-a-dull-moment life of a data engi...Discuss·10 likesConfessions of a Data Engineerdata engineer
Constantin Lungudatawise.dev·Jun 21, 2024Replicating datasets across regions in BigQuerySo this has been up for almost a year, but I've just found out that you can replicate datasets across regions in BigQuery. Obligatory remark that this is still in preview. This should help quite a bit if you're working with data distributed across mu...DiscussPractical BigQuerybigquery
Maximiliano Bertta Borgesmaximiliem.hashnode.dev·Jun 14, 2024Data Engineering: a new hopeSince 2011 my work area has been administrative, but I want to change that. At 2023 I started to walk into a beautiful way... Web Development. So, my vision of programming was changed. I really enjoyed that route, the classmates, the teachers, all of...Discussdata-engineering
Christopher Garzonchrisgarzon.hashnode.dev·May 30, 2024Stripe Data Engineer Interview GuideBECOME A DATA ENGINEER Stripe Interview Process The interview process at Stripe is designed to assess a candidate’s technical expertise, problem-solving skills, and alignment with the company’s values. It begins with an initial screening to verify yo...Discussstring
Data Senseisihan.hashnode.dev·May 4, 2024DBT for SQL Server? Yes possible.Installing DBT using SQL server is extremely simple. I had many clients coming to me and asking if they could use DBT with SQL Server on-prem or managed instances. They look surprised when I answered them, "Yes, why not". Now you might ask me, what i...Discuss·28 readsdbt
Alfin Rohadialfrode.com·May 2, 2024Everything you need to get started with data engineering (part 1)Not too long ago, data-driven companies started to use the phrase "data engineer" as the data has increase in velocity, volume, value, variety and veracity (also known as 5 V's in big data) that, in the nutshell, require a broader focus on sophistica...Gus Vieweg and 3 others are discussing this4 people are discussing thisDiscuss·17 likes·142 readsdata-engineering
Sai Madhavmadster.hashnode.dev·Apr 29, 2024Streamlining Time Series Data Analysis using Snowflake's ASOF JoinDealing with time series data is a common challenge for data engineers, especially when joining two tables where dates or timestamps do not align perfectly. For instance, Table 1 may have dates missing in Table 2 and vice versa. Typically, this probl...DiscussASOF
