Spark Connect - Streamline Apache Spark Data Pipeline Development in Dagster
Jun 29, 2024 · 3 min read · Introduction In many years, the conventional way to develop data pipelines in Dagster with Apache Spark is submitting applications, which pose unique challenges to developers: Require a separate infrastructure (cluster of machines with a cluster man...
Join discussion

