---
title: "Discussion on \"Ingesting every partner's file format without rewriting code each time\" | Hashnode"
description: "Discussion on \"Ingesting every partner's file format without rewriting code each time\". Every new data partner sends a different file: different columns, different order, different format. Hard-code a parser per partner and your codebase balloons and your pipeline cannot scale. Worse, when a file breaks at three in the morning"
url: "https://hashnode.com/posts/ingesting-every-partners-file-format-without-rewriting-code-each-time/6a53c3cec3b619f075b78e85"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [Ingesting every partner's file format without rewriting code each time](https://sapotacorpvn.hashnode.dev/ingesting-every-partners-file-format-without-rewriting-code-each-time?utm_source=hashnode&utm_medium=feed)

Every new data partner sends a different file: different columns, different order, different format. Hard-code a parser per partner and your codebase balloons and your pipeline cannot scale. Worse, when a file breaks at three in the morning

Jul 12 · 7 min read · sapotacorpvn.hashnode.dev  
Read the article

-   [#aws](https://hashnode.com/tag/aws)
-   [#data-engineering](https://hashnode.com/tag/data-engineering)
-   [#data-ingestion](https://hashnode.com/tag/data-ingestion)
-   [#aws-glue](https://hashnode.com/tag/aws-glue)
-   [#airflow](https://hashnode.com/tag/airflow)

## Written by

[SapotaCorp](https://hashnode.com/@sapotacorpvn)

@sapotacorpvn · Hanoi, Vietnam

  
[View profile](https://hashnode.com/@sapotacorpvn)

## Responses

Markdown supported · ⌘↵ to post

No responses yet.