---
title: "#azure-data-engineer articles | Hashnode"
description: "Explore 3 articles tagged #azure-data-engineer on Hashnode. Tutorials, guides, and how-tos from the community."
url: "https://hashnode.com/tag/azure-data-engineer"
---

Tag

# #azure-data-engineer

3 posts · 0 followers

[Venkatesh Marella](https://hashnode.com/@venkateshmarella) · [Apr 10, 2024](https://hashnode.com/posts/pyspark-read-large-csv-files-efficiently/67e149aba8f196367b3802cd)

### [PySpark: Read Large CSV files efficiently](https://data-engineer-solutions.hashnode.dev/pyspark-read-large-csv-files-efficiently?utm_source=hashnode&utm_medium=feed)

Scenario You have a large CSV file (100GB+ of data) with millions of records. Loading the file without optimization causes memory issues and slow performance. Solution: Use Partitioning & Parquet for Faster Processing Step 1: Read the Large CSV in Py...

1 min read · [data-engineer-solutions.hashnode.dev](https://data-engineer-solutions.hashnode.dev/pyspark-read-large-csv-files-efficiently?utm_source=hashnode&utm_medium=feed)

[Venkatesh Marella](https://hashnode.com/@venkateshmarella) · [Apr 6, 2024](https://hashnode.com/posts/pyspark-removing-duplicates-in-large-datasets/67e148b26881289477bd45e6)

### [PySpark: Removing Duplicates in Large Datasets](https://data-engineer-solutions.hashnode.dev/pyspark-removing-duplicates-in-large-datasets?utm_source=hashnode&utm_medium=feed)

Scenario Your dataset contains duplicate customer records. You need to remove duplicates based on the latest timestamp. Solution: Use dropDuplicates() & Window Functions Step 1: Sample Data from pyspark.sql.functions import col from pyspark.sql impor...

1 min read · [data-engineer-solutions.hashnode.dev](https://data-engineer-solutions.hashnode.dev/pyspark-removing-duplicates-in-large-datasets?utm_source=hashnode&utm_medium=feed)

[Sanjay Rathore](https://hashnode.com/@SanjayRathore) · [Jun 26, 2023](https://hashnode.com/posts/what-is-microsoft-azure-autoscaling/64994f2999738bee5f2eed35)

### [What is Microsoft Azure Autoscaling?](https://sanjayrathore.hashnode.dev/what-is-microsoft-azure-autoscaling?utm_source=hashnode&utm_medium=feed)

Microsoft Azure Autoscaling is a feature provided by the Azure cloud platform that allows automatic adjustment of computing resources based on the workload demands of an application or service. Autoscaling helps optimize resource utilization and ensu...

2 min read · [sanjayrathore.hashnode.dev](https://sanjayrathore.hashnode.dev/what-is-microsoft-azure-autoscaling?utm_source=hashnode&utm_medium=feed)
