Constantin Lungudatawise.dev·12 hours agoRaising ERRORS in BigQueryDoes anyone have interesting use cases for the ERROR function in BigQuery? If you like BQ errors so much that you've decided to create your own, or if you're debugging with dirty data, maybe check it out. If will raise an error that you specify w...DiscussPractical BigQuerybigquery
Constantin Lungudatawise.dev·Apr 22, 2024Using INSTR in BigQueryIf you ever need to do something different based on the existence of a particular substring in hashtag#BigQuery, take a look at the INSTR function. It returns the 1-based index of the first occurrence of a substring (1 or more characters) in another ...Discuss·40 readsPractical BigQuerybigquery
Alex Mercedtechblog.alexmerced.com·Apr 21, 2024A Deep Intro to Apache Iceberg and Resources for Learning MoreFor a long time, siloed data systems such as databases and data warehouses were sufficient. These systems provided convenient abstractions for various data management tasks, including: Storage locations and methods for data. Identification and recog...Discussapacheiceberg
jeeva Bcanopy.hashnode.dev·Apr 21, 2024How to install Airflow on an Ubuntu machine?Before performing the operation, we need to ensure that our Ubuntu webserver is up to date with the desired packages. The code provided below will help us install the necessary packages in Linux. In addition to this, we need to install any other pack...Discuss#ApacheAirflow
Constantin Lungudatawise.dev·Apr 19, 2024Cross-dataset Foreign Key referencing in BigQueryIn one of my previous posts I've written about the then-newly-added Primary Key/Foreign Key constraints in BigQuery. While they are not enforced like traditional RDBMS, they can still provide an improvement to query performance. They have one importa...DiscussPractical BigQuerybigquery
Constantin Lungudatawise.dev·Apr 19, 2024Using STRUCTS for quick analysis in BigQueryI've posted earlier about STRUCTS in BigQuery, here's how I use it from time to time to help me debug and analyze data a bit faster. Since changing filter values for different test cases / observations you are interested about can be a headache (espe...DiscussPractical BigQuerySQL
Christopher Garzonchrisgarzon.hashnode.dev·Apr 16, 202410 Best ETL Tools 2024ETL tools automate processes, improve data accuracy, and generate valuable insights. This article reviews the top 10 ETL tools of this year, focusing on their distinctive features, scalability, ease of use, and overall performance. It is intended for...DiscussETL Tools
Christopher Garzonchrisgarzon.hashnode.dev·Apr 16, 2024Data Engineering: Incremental Data Loading StrategiesIncremental data loading is an approach to data integration that transfers only the new or changed records from one database or data source to another, rather than moving the entire data set. This method is especially beneficial in environments where...Discussincremental data loading
Christopher Garzonchrisgarzon.hashnode.dev·Apr 16, 2024Data Engineer Academy ReviewThe Data Engineering Academy takes pride in presenting the transformational story of Vish, who navigated the complexities of a shifting job market to become adept in data engineering, commencing an exciting new chapter in his professional life. Vish...Discusscareer advice
Christopher Garzonchrisgarzon.hashnode.dev·Apr 16, 2024Data Engineer Academy Succes StoriesThe Data Engineering Academy takes pride in presenting the transformational story of Vish, who navigated the complexities of a shifting job market to become adept in data engineering, commencing an exciting new chapter in his professional life. Vish...Discussinterview preparations