---
title: "Discussion on \"From 3,600 to 400 API Calls: Optimizing PySpark on AWS Glue with the Yield Pattern\" | Hashnode"
description: "Discussion on \"From 3,600 to 400 API Calls: Optimizing PySpark on AWS Glue with the Yield Pattern\". How I used Python generators and RDD mapPartitions to enrich data from 2 APIs and write 9 staging tables — in a single Spark job."
url: "https://hashnode.com/posts/optimizing-pyspark-on-aws-glue-with-the-yield-pattern/69aa922d78c5adcd0ee668b2"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [From 3,600 to 400 API Calls: Optimizing PySpark on AWS Glue with the Yield Pattern](https://oorucelik.hashnode.dev/optimizing-pyspark-on-aws-glue-with-the-yield-pattern?utm_source=hashnode&utm_medium=feed)

How I used Python generators and RDD mapPartitions to enrich data from 2 APIs and write 9 staging tables — in a single Spark job.

Mar 6 · 9 min read · oorucelik.hashnode.dev  
Read the article

-   [#pyspark](https://hashnode.com/tag/pyspark)
-   [#aws-glue](https://hashnode.com/tag/aws-glue)
-   [#python](https://hashnode.com/tag/python)
-   [#mappartitions](https://hashnode.com/tag/mappartitions)
-   [#data-pipeline](https://hashnode.com/tag/data-pipeline)
-   [#etl](https://hashnode.com/tag/etl)
-   [#api-integration](https://hashnode.com/tag/api-integration)
-   [#aws-step-functions](https://hashnode.com/tag/aws-step-functions)
-   [#spark](https://hashnode.com/tag/spark)

## Written by

[Ömer Oruç ÇELİK](https://hashnode.com/@detail-pipelines)

@detail-pipelines · Istanbul, Turkey

  
[View profile](https://hashnode.com/@detail-pipelines)

BI Developer turned Data Engineer | Building pipelines that power decisions

Detail-oriented computer engineer with a self-taught mindset. After 4 years of SAP BI, settling back in my engineering chair. Throughout my career, I've built forecasting ETL and dashboards, designed countless reports and supported ETL systems for enterprise companies across the Middle East and Europe with DataIku, SAP Analytics Cloud, SDI(Smart Data Integrator) on SAP HANA. Now I'm focused on engineering with data platforms, building pipelines, modeling and visualizing datasets. So far I've worked with; PySpark, DAX, SQL, AWS, Microsoft Fabric and Power BI. I've always had the desire to share what I know but the problem was, not everyone around me are interested in what I'm about to teach them lol. So here we are, starting this blog for the people who would appreciate .I know I'll keep on growing, so why not share along the way with a community where we can grow together! Have a great harvest of knowledge!

## Responses

Markdown supported · ⌘↵ to post

No responses yet.
