--- title: "#intermediate articles | Hashnode" description: "Explore 103 articles tagged #intermediate on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/intermediate" --- Tag # #intermediate 103 posts · 3 followers [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-dataframes-sql-catalyst-optimizer/69e437889f1ae2c7968c205f) ### [Spark DataFrames and Spark SQL: Schema, DDL, and the Catalyst Optimizer](https://abstractalgorithms.hashnode.dev/spark-dataframes-sql-catalyst-optimizer?utm_source=hashnode&utm_medium=feed) Catalyst turns your DataFrame ops into an optimized physical plan — understanding it explains why column pruning and predicate pushdown are free 24 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-dataframes-sql-catalyst-optimizer?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-kafka-structured-streaming-pipeline/69e437a6b8882bee29d621f8) ### [Kafka and Spark Structured Streaming: Building a Production Pipeline](https://abstractalgorithms.hashnode.dev/spark-kafka-structured-streaming-pipeline?utm_source=hashnode&utm_medium=feed) Kafka delivers events, Spark processes them — offset management, schema evolution, and exactly-once delivery determine production reliability 23 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-kafka-structured-streaming-pipeline?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-watermarking-late-data-handling/69e437f990c8f3c452e31ff0) ### [Watermarking and Late Data Handling in Spark Structured Streaming](https://abstractalgorithms.hashnode.dev/spark-watermarking-late-data-handling?utm_source=hashnode&utm_medium=feed) Late events are inevitable in distributed systems — watermarks tell Spark exactly when it is safe to finalize a window without waiting forever 27 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-watermarking-late-data-handling?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-structured-streaming-micro-batch-continuous/69e437ed6e738bcf4b61fc37) ### [Spark Structured Streaming: Micro-Batch vs Continuous Processing](https://abstractalgorithms.hashnode.dev/spark-structured-streaming-micro-batch-continuous?utm_source=hashnode&utm_medium=feed) Treats streams as unbounded tables — the execution model determines whether your pipeline hits milliseconds or minutes. 27 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-structured-streaming-micro-batch-continuous?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-shuffles-groupby-performance/69e437d76e738bcf4b61fc32) ### [Shuffles in Spark: Why groupBy Kills Performance](https://abstractalgorithms.hashnode.dev/spark-shuffles-groupby-performance?utm_source=hashnode&utm_medium=feed) A Spark shuffle moves every matching key across the network — understanding when and why is the difference between a 2-minute and 2-hour job 31 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-shuffles-groupby-performance?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-broadcast-joins-vs-sort-merge-joins/69e437716e738bcf4b61fc26) ### [Broadcast Joins vs Sort-Merge Joins in Spark](https://abstractalgorithms.hashnode.dev/spark-broadcast-joins-vs-sort-merge-joins?utm_source=hashnode&utm_medium=feed) Spark picks your join strategy automatically — knowing when it picks wrong and why saves hours of debugging slow join stages 26 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-broadcast-joins-vs-sort-merge-joins?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/python-functional-map-filter-itertools-functools/69e43717f98b558fec73a30d) ### [Functional Python: map, filter, itertools, and functools](https://abstractalgorithms.hashnode.dev/python-functional-map-filter-itertools-functools?utm_source=hashnode&utm_medium=feed) Python's functional tools are not just map/filter — itertools and functools turn complex data transformations into composable one-liners 29 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/python-functional-map-filter-itertools-functools?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/python-oop-classes-dataclasses-dunder-methods/69e4373f9f1ae2c7968c2059) ### [Python OOP: Classes, Dataclasses, and Dunder Methods](https://abstractalgorithms.hashnode.dev/python-oop-classes-dataclasses-dunder-methods?utm_source=hashnode&utm_medium=feed) Python OOP isn't Java OOP — from \_\_init\_\_ to dataclasses to \_\_slots\_\_, here's what actually matters 22 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/python-oop-classes-dataclasses-dunder-methods?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-caching-persistence-storage-levels/69e4377cf98b558fec73a31c) ### [Caching and Persistence in Spark: Storage Levels and When to Use Them](https://abstractalgorithms.hashnode.dev/spark-caching-persistence-storage-levels?utm_source=hashnode&utm_medium=feed) Caching in Spark is lazy and partition-level — understanding how StorageLevel and eviction work is the difference between a speedup and an OOM 24 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-caching-persistence-storage-levels?utm_source=hashnode&utm_medium=feed) [Abstract Algorithms](https://hashnode.com/@abstractalgorithms) · [Apr 19](https://hashnode.com/posts/spark-partitioning-hash-range-custom/69e437bef98b558fec73a322) ### [Partitioning in Spark: HashPartitioner, RangePartitioner, and Custom Strategies](https://abstractalgorithms.hashnode.dev/spark-partitioning-hash-range-custom?utm_source=hashnode&utm_medium=feed) Partition count and strategy determine whether your Spark job scales linearly or falls apart at 10x data — here is what happens under the hood 26 min read · [abstractalgorithms.hashnode.dev](https://abstractalgorithms.hashnode.dev/spark-partitioning-hash-range-custom?utm_source=hashnode&utm_medium=feed)