Constantin Lungudatawise.dev·20 hours agoConstructing STRUCTS in BigQueryAfter my previous STRUCTS in BigQuery post, I could not have skipped to mention number of options when it comes to construct one. You can choose whether to provide field type, field name or both. There's 3 main ways:- via tuple ('a', 1) - BQ crea...DiscussPractical BigQuerybigquery
Constantin Lungudatawise.dev·21 hours agoUnderstanding STRUCTS in BigQueryI've previously did a short intro post about ARRAYS in BigQuery, but I do see from time to time people that are just getting started become confused about how are they different from STRUCTS and when should we use them. Let's clarify this. STRUCT...Discuss·52 readsPractical BigQuerybigquery
navinkumarnotes123.hashnode.dev·Mar 18, 2024How to decide bucket count in hiveSteps Calculate Expected Bucket Size: Divide the table size by the block size on Hadoop to get an initial estimate. Expected Bucket Size = Table Size / Block Size on Hadoop Find the Nearest Power of 2: Take the base-2 logarithm of the ini...DiscussHivehive
Kiran Reddybhanukiranreddy.hashnode.dev·Mar 14, 2024Mastering Window Functions in SQL ServerUnderstanding Window Functions in SQL Server In the realm of database operations, while regular aggregate functions dominate by working on entire tables through the GROUP BY clause, a less-explored yet highly potent tool exists—the window functions. ...Discuss#windowfunction
Saurav Rajrajsaurav.hashnode.dev·Mar 14, 2024Tokyo Olympics Data Engineering and Analysis using Microsoft AzureThis project deals with Tokyo Olympics 2021 dataset. This project involves understanding the data architecture, creating the ETL pipeline, and finally analysing the data. The project is based off Darshil Parmar video on YouTube. This contains the det...DiscussData Science
Kiran ReddyforDatabricks - PySparkdatabricks-pyspark-blogs.hashnode.dev·Mar 13, 2024Mount Points in DatabricksWhat is DBFS? DBFS stands for Databricks File System. It's a distributed file system that's part of the Databricks Unified Data Analytics Platform. DBFS provides a scalable and reliable way to store data across various Databricks clusters. It's desig...Discuss·10 likesmountpoint
Anuj Syalanujsyal.com·Feb 26, 2024How I Passed Databricks Data Engineer Professional ExamIntroduction You've aced the Databricks Data Engineer Associate Exam – congrats! Or maybe, you’re just curious as you heard someone becoming a Certified Data Engineering Professional. Guess it’s time to set your data engineering career in motion. But...Discuss·224 readsDatabricks
Daniel Odanai.hashnode.dev·Feb 17, 2024Building Data Pipelines: Ingesting Data on AWSIn today's Data Engineering exploration, lets dive into creating data pipelines on AWS. Our focus will be on the ingestion to buffer (Kinesis) phase. The data used for this project is from UCI machine learning repository. Overview Key Steps Lambda ...Discuss·26 readsdataengineering
Constantin Lungudatawise.dev·Feb 16, 2024Watch out when using SAFE_CAST in BigQueryHere's an interesting situation I've seen with BigQuery. Say a source system provides JSON events with timestamps at microsecond grain (6 decimals, so something like 2024-01-01 14:00:00.123456). This is cast using SAFE_CAST into a proper TIMESTAM...Discuss·55 readsPractical BigQuerybigquery
Sweta_Sarangiswetasarangi.hashnode.dev·Feb 16, 2024Roles in dataCrafting a narrative with data is a journey that typically doesn't initiate with oneself. The data's origin is crucial, and the effort to make it usable often extends beyond individual capabilities, especially within an enterprise context. Contempora...DiscussData Science
