Gabriela Caldasbyte-sizedjourneys.hashnode.dev·Nov 21, 2024Designing Data Pipelines for Success: Best Practices for Scalability and Data QualityIn today’s world, businesses need accurate and readily available insights to stay competitive. Data engineering plays a crucial role in creating the infrastructure that makes this possible. From building efficient data pipelines to ensuring data qual...data-engineering
Ankit Rajankitraj19.hashnode.dev·Nov 16, 2024From Basics to Brilliance: My Week 1 Journey with PostgreSQL for Data EngineeringIntroduction: Background As a data engineer working full-time in the industry, I’m always looking for ways to enhance my skills and stay ahead of the curve with the latest technologies. Recently, I decided to focus more on data engineering and dive d...data-engineering
pallavi chauhaninnovateitworld.hashnode.dev·Nov 12, 2024Data Science Career Paths: Which Role is Right for You?As the field of data science continues to evolve, organizations across industries are leveraging data to inform their decisions and drive growth. The surge in big data and machine learning has led to a growing demand for data science professionals, r...Data Science
samhita sarkarsamhita-sarkar.hashnode.dev·Nov 11, 2024Deciding When to Normalize or Denormalize Data for Best ResultsNormalisation and denormalisation are essentially design patterns in databases that determine how data is stored and used. Storage pattern : The key difference between normalisation and denormalisation is their storage patterns, which are essentially...1 like·54 readsDatabases
Gyuhang Shimplto001.hashnode.dev·Oct 24, 2024Data Engineer 가 알아야 할 Data Modeling (Korean)Dimensional Data Modeling 정의 Data Warehouse 설계에서 데이터를 구조화하는 중요한 기법 특히 OLAP (Online Analytical Processing) 시스템에서 주로 사용되며, 데이터를 분석하기 용이하게 구성하는 것이 목표 이 Modeling 은 크게 3 가지 구성 요소로 나눌 수 있습니다. Fact Table Dimension Table Summary Table Fact Table ...Fact Data Modeling
Arpit Tyagidataminds.hashnode.dev·Oct 24, 2024How to Transfer All SQL Database Tables to Azure Data Lake in One Go?Step 1: Create an instance of Azure Data Factory: Step 2: Set up Linked Services: Step 3: Create a dataset for both (source-SQL Database and destination-Azure Data Lake): Step 4: Build the Data Pipeline with the help of datasets: First I will us...10 likesAzure Data Factorydata-engineering
Gyuhang Shimplto001.hashnode.dev·Oct 22, 2024Data Engineer 가 알아야 할 SQL Part 1 (Korean)전제 ANSI SQL 을 어느정도 알고 있음. ANSI SQL 의 모든 Syntax 를 다루지는 않음. ANSI SQL 에서 사용하는 기본적인 Function 들에 대해서 지식을 가지고 있음. ANSI SQL SELECT 데이터를 조회할 때 가장 기본적인 구문입니다. -- Syntax SELECT column1, column2 FROM table_name; -- Example: 직원들의 이름과 나이를 추출합니다. SE...ANSI SQL
Indu Jawlacoders.hashnode.dev·Oct 14, 2024Diverse Career Paths in Data Analysis: Exploring Opportunities and Required SkillsData analysis is a rapidly growing field that plays a crucial role in decision-making processes across various industries. As organizations increasingly rely on data-driven insights, the demand for skilled data analysts has surged. This article will ...Quantitative analyst
KAPUPA HAAMBAYIdatasmithery.hashnode.dev·Oct 1, 2024Every Business Needs Data EngineeringIn today’s rapidly evolving digital landscape, data has become the lifeblood of modern business. Regardless of industry, the ability to harness, analyse, and leverage data efficiently is no longer a luxury—it’s a necessity. As businesses strive to re...10 likesdata-engineering
Vishal Barvaliyavishalbarvaliya.hashnode.dev·Sep 18, 2024Why Does the "Executor Out of Memory" Error Happen in Apache Spark?Apache Spark is a tool used to process large amounts of data. It’s fast, scalable, and great for big data tasks. However, sometimes when working with Spark, you might run into a common issue: the "Executor Out of Memory" error. If you've seen this er...9 likes#apache-spark
