AAbstract Algorithms·Apr 19Spark DataFrames and Spark SQL: Schema, DDL, and the Catalyst OptimizerCatalyst turns your DataFrame ops into an optimized physical plan — understanding it explains why column pruning and predicate pushdown are free24 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Kafka and Spark Structured Streaming: Building a Production PipelineKafka delivers events, Spark processes them — offset management, schema evolution, and exactly-once delivery determine production reliability23 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Watermarking and Late Data Handling in Spark Structured StreamingLate events are inevitable in distributed systems — watermarks tell Spark exactly when it is safe to finalize a window without waiting forever27 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Spark Structured Streaming: Micro-Batch vs Continuous ProcessingTreats streams as unbounded tables — the execution model determines whether your pipeline hits milliseconds or minutes.27 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Shuffles in Spark: Why groupBy Kills PerformanceA Spark shuffle moves every matching key across the network — understanding when and why is the difference between a 2-minute and 2-hour job31 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Broadcast Joins vs Sort-Merge Joins in SparkSpark picks your join strategy automatically — knowing when it picks wrong and why saves hours of debugging slow join stages26 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Functional Python: map, filter, itertools, and functoolsPython's functional tools are not just map/filter — itertools and functools turn complex data transformations into composable one-liners29 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Python OOP: Classes, Dataclasses, and Dunder MethodsPython OOP isn't Java OOP — from __init__ to dataclasses to __slots__, here's what actually matters22 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Caching and Persistence in Spark: Storage Levels and When to Use ThemCaching in Spark is lazy and partition-level — understanding how StorageLevel and eviction work is the difference between a speedup and an OOM24 min read·abstractalgorithms.hashnode.dev
AAbstract Algorithms·Apr 19Partitioning in Spark: HashPartitioner, RangePartitioner, and Custom StrategiesPartition count and strategy determine whether your Spark job scales linearly or falls apart at 10x data — here is what happens under the hood26 min read·abstractalgorithms.hashnode.dev
