Giang Ngogiangblackk.hashnode.dev·Jun 29, 2024Spark Connect - Streamline Apache Spark Data Pipeline Development in DagsterIntroduction In many years, the conventional way to develop data pipelines in Dagster with Apache Spark is submitting applications, which pose unique challenges to developers: Require a separate infrastructure (cluster of machines with a cluster man...Discussdagster
