--- title: "#datapipelines articles | Hashnode" description: "Explore 31 articles tagged #datapipelines on Hashnode. Tutorials, guides, and how-tos from the community." url: "https://hashnode.com/tag/datapipelines" --- Tag # #datapipelines 31 posts · 0 followers [ZyVOP](https://hashnode.com/@zyvop) · [Oct 21](https://hashnode.com/posts/kafka-connect-and-kafka-streams-moving-and-processing-data-96n2z/6ad86eafca7d264cbfa3d2b0) ### [Kafka Connect and Kafka Streams: Moving and Processing Data](https://blog.zyvop.com/kafka-connect-and-kafka-streams-moving-and-processing-data-96n2z?utm_source=hashnode&utm_medium=feed) Move data in and out of Kafka without writing custom code, then transform and aggregate it in flight with Kafka Streams. 9 min read · [blog.zyvop.com](https://blog.zyvop.com/kafka-connect-and-kafka-streams-moving-and-processing-data-96n2z?utm_source=hashnode&utm_medium=feed) [vikas sharma](https://hashnode.com/@vikasty) · [Oct 12](https://hashnode.com/posts/best-data-lake-platform-designing-reliable-data-infrastructure/6acc79401c114b6b99a932a3) ### [Best Data Lake Platform: Designing Reliable Data Infrastructure](https://vkdigiservices.hashnode.dev/best-data-lake-platform-designing-reliable-data-infrastructure?utm_source=hashnode&utm_medium=feed) How to Build a Flexible Data Environment That Supports Analytics and Future Growth 8 min read · [vkdigiservices.hashnode.dev](https://vkdigiservices.hashnode.dev/best-data-lake-platform-designing-reliable-data-infrastructure?utm_source=hashnode&utm_medium=feed) [Lakshmi Mahabaleshwara](https://hashnode.com/@lakshmi-mahabalesh) · [Oct 6](https://hashnode.com/posts/how-to-encrypt-pii-in-data-pipelines-while-keeping-it-searchable/6ac4810141f9b6b6ff5bf1cd) ### [How to Encrypt PII in Data Pipelines While Keeping It Searchable](https://freecodecamp.org/news/how-to-encrypt-pii-in-data-pipelines-while-keeping-it-searchable?utm_source=hashnode&utm_medium=feed) At Intuit, my team built a data pipeline that processed TurboTax e-filing data. This data needed to be made available for dashboards, analytics, and other downstream use cases. One of the biggest chal 16 min read · [freecodecamp.org](https://freecodecamp.org/news/how-to-encrypt-pii-in-data-pipelines-while-keeping-it-searchable?utm_source=hashnode&utm_medium=feed) [Micheal Mileba](https://hashnode.com/@dev-mileba) · [Jul 24](https://hashnode.com/posts/building-a-sales-data-pipeline-in-pure-python-no-pandas-a-campusmart-case-study/6a62af34e6646c4a0f885383) ### [Building a Sales Data Pipeline in Pure Python (No Pandas): A CampusMart Case Study](https://dev-mileba.hashnode.dev/building-a-sales-data-pipeline-in-pure-python-no-pandas-a-campusmart-case-study?utm_source=hashnode&utm_medium=feed) How I ingested, validated, and reconciled messy CSVs into clean reports, and the subtle bugs I found along the way 12 min read · [dev-mileba.hashnode.dev](https://dev-mileba.hashnode.dev/building-a-sales-data-pipeline-in-pure-python-no-pandas-a-campusmart-case-study?utm_source=hashnode&utm_medium=feed) [Muhammad Affan](https://hashnode.com/@maffant20) · [Apr 28](https://hashnode.com/posts/why-api-data-pipelines-fail-silently-in-production/69f0d6b010a70b3335d0bab4) ### [Why API Data Pipelines Fail Silently in Production](https://maffant20.hashnode.dev/why-api-data-pipelines-fail-silently-in-production?utm_source=hashnode&utm_medium=feed) A practical guide to detecting errors in API pipelines 13 min read · [maffant20.hashnode.dev](https://maffant20.hashnode.dev/why-api-data-pipelines-fail-silently-in-production?utm_source=hashnode&utm_medium=feed) [Ankit Raj](https://hashnode.com/@ankitsrivastava) · [Apr 25](https://hashnode.com/posts/7-skills-that-make-data-engineering-feel-less-hard/69ec4a648831b5effccfc9fe) ### [7 Skills That Make Data Engineering Feel Less Hard](https://ankit-data-engineering.hashnode.dev/7-skills-that-make-data-engineering-feel-less-hard?utm_source=hashnode&utm_medium=feed) Most data engineering problems are not what they seem and it took me time to realize what actually matters 3 min read · [ankit-data-engineering.hashnode.dev](https://ankit-data-engineering.hashnode.dev/7-skills-that-make-data-engineering-feel-less-hard?utm_source=hashnode&utm_medium=feed) [Vishal Uttam Mane](https://hashnode.com/@vishal-uttam-mane) · [Apr 1](https://hashnode.com/posts/data-engineering-trends-from-etl-to-elt-and-the-modern-data-stack/69cd02e2b504c4ed0a802db4) ### [Data Engineering Trends: From ETL to ELT and the Modern Data Stack](https://vishal-uttam-mane-data-engg.hashnode.dev/data-engineering-trends-from-etl-to-elt-and-the-modern-data-stack?utm_source=hashnode&utm_medium=feed) A modern deep dive into the evolution of data engineering, highlighting the shift from ETL to ELT and the rise of the modern data stack. 3 min read · [vishal-uttam-mane-data-engg.hashnode.dev](https://vishal-uttam-mane-data-engg.hashnode.dev/data-engineering-trends-from-etl-to-elt-and-the-modern-data-stack?utm_source=hashnode&utm_medium=feed) [Ankit Raj](https://hashnode.com/@ankitsrivastava) · [Mar 28](https://hashnode.com/posts/building-an-event-driven-dataflow-pipeline-with-google-cloud-storage/69c760dd299af73526fa958d) ### [Building an Event-Driven Dataflow Pipeline with Google Cloud Storage](https://ankit-data-engineering.hashnode.dev/building-an-event-driven-dataflow-pipeline-with-google-cloud-storage?utm_source=hashnode&utm_medium=feed) How a simple file upload to a GCS bucket can automatically start a Dataflow transformation pipeline. 5 min read · [ankit-data-engineering.hashnode.dev](https://ankit-data-engineering.hashnode.dev/building-an-event-driven-dataflow-pipeline-with-google-cloud-storage?utm_source=hashnode&utm_medium=feed) [Ankit Raj](https://hashnode.com/@ankitsrivastava) · [Mar 13](https://hashnode.com/posts/how-to-connect-external-applications-to-bigquery-using-google-cloud-service-accounts/69b491f9210c74252f9b4e52) ### [How to Connect External Applications to BigQuery Using Google Cloud Service Accounts](https://ankit-data-engineering.hashnode.dev/how-to-connect-external-applications-to-bigquery-using-google-cloud-service-accounts?utm_source=hashnode&utm_medium=feed) A practical guide to securely allowing external tools, APIs, and analytics platforms to access BigQuery and Google Cloud Storage without using personal credentials. 6 min read · [ankit-data-engineering.hashnode.dev](https://ankit-data-engineering.hashnode.dev/how-to-connect-external-applications-to-bigquery-using-google-cloud-service-accounts?utm_source=hashnode&utm_medium=feed) [Mohammed Anwar](https://hashnode.com/@manwar) · [Jan 26](https://hashnode.com/posts/building-a-production-grade-user-validation-layer-with-pydantic/69778146c3e59972ab4d922f) ### [Building a Production-Grade User Validation Layer with Pydantic](https://building-pydantic-user-validation.hashnode.dev/building-a-production-grade-user-validation-layer-with-pydantic?utm_source=hashnode&utm_medium=feed) When working on the backend with Python as your language and needing strict validations in pipelines or data validation before feeding data into the database, you understand how challenging it can be. Validation is one of those problems that looks si... 5 min read · [building-pydantic-user-validation.hashnode.dev](https://building-pydantic-user-validation.hashnode.dev/building-a-production-grade-user-validation-layer-with-pydantic?utm_source=hashnode&utm_medium=feed)