From DynamoDB to TimestreamOnce upon a time… In 2018, we began collecting data with our sensors and persisted them in a DynamoDB. This decision was sufficient until recently. Even when Timestream was released in 2020 we thought: “Dang, Timestream is exactly made for our Use Ca...Oct 19, 2022 · 5 min readtechwood.hashnode.dev
How to collect and visualize data lineage in an AWS-based data lakeIn this article, we will go over a reference architecture for a Spark/Glue-based Data lake in AWS, we will discuss on the possiblities how to collect data lineage in this architecture and then describe how to visualize and interact with the collected...Jan 13, 2022 · 6 min readtechwood.hashnode.dev
Why metadata is key in data lakesAutomating data collection and centralization is great - but how can you keep up with context information of and interdependencies between the datasets? This is where metadata (in particualr data lineage - describing the origins and transformation hi...Jan 11, 2022 · 7 min readtechwood.hashnode.dev
