Chetan Sharmathechetansharma.com路Sep 24, 2023Transferring On-Premise Data to S3Problem Statement We will read data from an on-premise DBeaver Server database and automatically load it to Amazon object storage S3 using Python. We will use the AWS S3 API to achieve this. The idea here is that we have data in our premise database....Discuss路42 readsdatabasemigration
Lee Gainesleeg.hashnode.dev路Sep 18, 2023What is a dataset?Dataset is a word I often run into when reading things about AI, machine learning, data engineering, etc. I don't fully grasp what a dataset actually is. Maybe I'm overthinking it, or perhaps it's just semantics, but let's try to clarify it. This is ...Discussdataset
Elvis Davidtechlake.dev路Sep 18, 2023Real-time data ingestion to Databricks Delta Lake using Redpanda (Kafka Connect)Introduction We are in the era where Real-time data ingestion has become one of the critical requirement for many organizations which are seeking to tap the value from their data in Real-time and near Real-time. Real-time data ingestion has grown to ...DiscussData EngineeringData Science
Elvis Davidtechlake.dev路Sep 18, 2023Scala for Data Engineering: Harnessing the Power of Functional ProgrammingYou want to write a program that handles data. Which language should you choose? Introduction In the dynamic world of data engineering, where processing and managing vast amounts of data have become paramount, programming languages that offer flexibi...DiscussData EngineeringScala
Rita Okonkworitaokonkwo.hashnode.dev路Sep 17, 2023Mastering Efficient Iteration Techniques for Pandas DataFrames馃挕 To fully benefit from this article, it's important to have a basic understanding of Python, Numpy arrays, and Pandas DataFrames. Recently, I undertook a data manipulation task that involved using DataFrames for Data Engineering purposes. Whether...Discuss路10 likes路133 readsdata-engineering
VanshSutariyavanshsutariya.hashnode.dev路Sep 14, 2023Data : There are Billions of data generating daily.SO , without data we are nothing , nothing will happen without ,whatever we do like chatting , buying something online calling , for our every such activities we are generating data , so today we goona talk about data , how its important and what we ...Discussdata
Richmond Damaghateryx.hashnode.dev路Sep 12, 2023Fundamental Snippet: Database Backup and RecoveryDatabase backups are crucial for safeguarding data, ensuring business continuity, and meeting regulatory requirements. They provide a safety net to recover from various types of data loss scenarios, helping organizations maintain data integrity and o...Discussdata-engineering
Bikram Chatterjeebikramchatterjee.hashnode.dev路Sep 12, 2023Creation of Generic Data Pipeline 馃捇馃搷 Introduction: In this post, we will see how to create a generic data pipeline. 馃摵 Problem Statement: Suppose we need to pull data from 5 different tables, we can create 5 copy activities. But what if we need to pull data from 100 different tables,...Discussdata
Vignesh Kumar Muralivigneshkmr.hashnode.dev路Sep 12, 2023How is Data Engineering is Helping Operating System Across The GlobeData engineering has emerged as a critical force in enhancing operating systems worldwide, playing a pivotal role in refining their functionality and bolstering their performance on a global scale. Operating systems serve as the fundamental software ...Discussdata-engineering
Emmanuel A. Frankfrankolanari.hashnode.dev路Sep 10, 2023Demystifying Data EngineeringLast year, I embarked on a journey to learn data analysis, and it wasn't long before I encountered the complex world of data engineering. The intricacies of data engineering felt like an impenetrable maze to me, especially as a beginner. However, my ...Discuss路70 readsdata-engineering
