When AWS Glue Couldn’t Read My S3 Files: Solving the Small Files Problem
If you are working with AWS Glue and Amazon S3, you may eventually come across a problem that doesn't look serious at first:
There are too many small files in S3.
A file that is only a few KB doesn'
rahul-dataeng.hashnode.dev13 min read
Rahul R
Data Engineer
The most common problem in data engineering.