Tag feed

#delta-loading

2 posts0 followers

KBKiran Bhandarikiranbhandari.hashnode.dev

Delta Loading Using AWS Glue Job Bookmark and Dynamodb

Aug 17, 2024 · 3 min read · There are few approaches to achieve the delta loading in AWS. I will be discussing more about the two option here; Using Glue Job bookmark Using dynamodb to read unprocessed file Glue Job Bookmark Using job bookmark, AWS glue tracks data that has...

Join discussion

KBKiran Bhandarikiranbhandari.hashnode.dev

0

Incremental Loading using Spark Partition

Jun 11, 2024 · 3 min read · We can achieve the incremental loading using the spark data partition. I will be demonstrating with a simple example here, and you can you according to the business need. Let's consider the following data frame.(This could be your csv, or any formatt...

Join discussion

#delta-loading

Search Hashnode

#delta-loading

Delta Loading Using AWS Glue Job Bookmark and Dynamodb

Incremental Loading using Spark Partition