Anuj Syalanujsyal.com·Dec 8, 2024FeaturedMastering Data Quality in ETL Pipelines with Great ExpectationsIn the world of data engineering, ensuring data quality is paramount. From business analysts relying on dashboards to C-level executives making strategic decisions, and data scientists training machine learning models — everyone depends on the qualit...4 likes·44 readsData Science
Ilham Oulakbirensuringdataqualityandgovernance.hashnode.dev·Sep 23, 2024Best Practices for Data Engineers: Ensuring Data Quality and GovernanceIntroduction: In the world of data engineering, ensuring the quality and governance of data is as important as building robust pipelines and scalable architectures. Without proper governance and quality measures, the data you work with can lead to in...2 likes·32 readsData Tools
Mutabazi NobleforAirQo Engineeringairqoengineering.hashnode.dev·Dec 12, 2022Data Unit Testing With Great Expectations and Cloud FunctionsAt AirQo, we collect large amounts of data every day from our low-cost monitors deployed throughout the African continent, with the majority being in Uganda, as well as data from our partners such as Kampala Capital City Authority that are streamed e...1 like·244 readsData Sciencegreat-expectations
Itay Braunpostgres.hashnode.dev·Dec 3, 2022Preventing Data Quality problems from hitting the Production environment using Great ExpectationsBy Itay Braun, CTO, Metis In this post you'll learn how to apply data quality tests using an open-source platform called Great Expectations. Why Should I care? Data quality is important because it determines the accuracy and reliability of the inform...1 like·62 readsgreat-expectations
DataChefforDataChef's Blogblog.datachef.co·Apr 28, 2022How to Use S3 for Storing Great Expectation ResultsIntroduction Great Expectations(GE) allows us to oversee data pipelines and helps us to create data quality checks. In our previous post on running GE on EMR, we explored how to set up GE for your data pipeline and validate your data sources. Recentl...138 readsAWS
DataChefforDataChef's Blogblog.datachef.co·Jan 28, 2021How to run Great Expectations on EMRWhat is Great Expectations Great Expectations is a great tool to validate the quality of your data and can be configured against a number of data sources, including BigQuery, MySQL, Snowflake and Athena. It helps you to test data instead of code. Let...106 readsemr
