Niels Humbeckbuilding-data-products.com·Dec 18, 2024Mastering Data Pipelines: The Secret to Fast and Reliable Data OperationsIn today’s data-driven world, data pipelines are the backbone of efficient and scalable DataOps. These pipelines are vital for managing both data and code, automating complex workflows, and minimizing manual data handling. Data pipelines can be descr...Building Data Products with DataOps Methodologydataops
Tobias SchmidtforAWS Fundamentalsblog.awsfundamentals.com·Sep 13, 2024AWS Step Functions: Standard vs. Express ExplainedAWS Step Functions is an amazing orchestration service that enables you to design and manage complex workflows by coordinating various AWS services. It simplifies building and running multi-step applications and ensures that each step happens in the ...11 likes·1.4K readsaws stepfunctions
Ken Brewerkenbrewer.com·Aug 12, 2024Nextflow, nf-core, Seqera and more...I am excited to announce that starting next week I will be joining Seqera as a Senior Developer Advocate supporting the Nextflow, nf-core and Seqera Platform user communities. For this role, I'm going to be relocating to the San Francisco Bay area to...123 readsseqera
Jalel TOUNSIsecondbrain.hashnode.dev·Apr 11, 2024Engineering Flow, Mastering the symphony and Orchestrating AI agentsIn the realm of artificial intelligence, the orchestration and collaboration of AI agents play a pivotal role in achieving optimal performance and efficiency. Flow engineering involves designing the workflow of AI agents, determining how they operate...11 likes·31 readsEngineering Flow
Jose Rodrídguezproductiveblog.hashnode.dev·Mar 17, 2024Is Workflow Automation The Next Big Trend?In an era where efficiency is not just a goal but a necessity, workflow automation emerges as a beacon of innovation, transforming the very fabric of business operations. But what is propelling this significant shift towards automated processes? The ...the next big thing
Victor Ndutidatacurious.hashnode.dev·Mar 4, 2024Crafting a Basic Data Pipeline with Airflow.From setup to mastery: A Guide to Crafting Your Inaugural DAG In a previous blog post, we explored the fundamental concepts of Apache Airflow—a versatile workflow management platform that empowers users to orchestrate complex data pipelines with eas...1 like·322 readsapache
Smritismritisatyan.hashnode.dev·Aug 19, 2023‘Hello World’ in FlyteIn my previous article on Flyte, I listed out the salient features of Flyte, which can help you decide if Flyte is the right orchestration platform for you. In this article, you will understand how tasks and workflows in Flyte can be used to implemen...mlops
Preeti samuelkamilapreetisamuel.hashnode.dev·May 23, 2023OrchestrationOrchestration is the process of automating and coordinating complex workflows and systems, often involving multiple independent components or services. These workflows can involve the integration of various software systems, databases, services, and ...Microservices
Junho Kimjunojunho.hashnode.dev·Apr 30, 2023When to use ? Orchestration vs Event Driven ArchitectureThis year, I am a position driving medium size product to satisfy this year’s customer requests. During the design phase, I realized that I need one technical architecture to connect multiple components and let them work closely together. Two popular...43 readsarchitecture
Andrew Sharifikiaalireza-sharifikia.hashnode.dev·Mar 17, 2023DataOps: Apache Airflow - BasicIntroduction Apache Airflow is an open-source platform for authoring, scheduling, and monitoring data and computing workflows. It was developed by Airbnb and is now under the Apache Software Foundation.It uses Python to create workflows that can be e...98 readsDataOpsairflow
