ML Researcher
Nothing here yet.
Sep 18, 2024 · 5 min read · This article solves an arguably niche issue where we wish to log both to the mlflow tracking server as well as other sources, such as a live terminal or some cloud storage destination. To be concrete, the aim of this post is to detail the setup proce...
Join discussion
Jan 31, 2024 · 8 min read · Delta tables carry many unique, useful properties. This post deconstructs three core concepts used by Delta tables for query runtime and space optimization. Compaction This is the most basic form of space optimization. When large amounts of data is s...
Join discussion
Aug 5, 2023 · 5 min read · This is something that I attempted to find a guide for online, but it appears that the niche nature of pyspark (and the lack of practice for testing data-centric operations) results in no clear one-stop location for figuring out how to do this (appar...
Join discussion
Jun 25, 2023 · 13 min read · This post covers the Delta Lake, which is an open-source format extending parquet files for ACID transactions. More specifically, this covers how to work with Delta tables using the pyspark and native Delta APIs in python. Delta tables can be thought...
Join discussion
Apr 17, 2023 · 4 min read · This post is going to walk through how to go about writing (and formatting) excel files using Kedro and openpyxl in Python. For the uninitiated, Kedro is an open-source framework for creating data-science code. This is not meant to be a comprehensive...
Join discussion
