Kiran Bhandarikiranbhandari.hashnode.dev·Jan 16, 2024Incremental Loading using Spark PartitionWe can achieve the incremental loading using the spark data partition. I will be demonstrating with a simple example here, and you can you according to the business need. Let's consider the following data frame.(This could be your csv, or any formatt...#incremental-loading
Kiran Bhandarikiranbhandari.hashnode.dev·Jan 15, 2024Delta Loading Using AWS Glue Job Bookmark and DynamodbThere are few approaches to achieve the delta loading in AWS. I will be discussing more about the two option here; Using Glue Job bookmark Using dynamodb to read unprocessed file Glue Job Bookmark Using job bookmark, AWS glue tracks data that has...49 readsDelta Loading,
