@chetucity

Integrating Airflow with AWS: Extracting Data through Apache Airflow And Store It In S3

Nov 14, 2023 · 2 min read · Goals and Achievements Install Apache Airflow in EC2 instance. Create a virtual environment and install all dependencies. Establish a connection between the EC2 instance and Visual Studio. Login into the airflow server. Build a DAG that fetches ...

Project :How to automatically copy data from AWS S3 – Lambda events

Nov 11, 2023 · 3 min read · Hocalwire Labs Private Limited 1. Please establish two folders within the S3 Bucket, which should be named as follows: - "original" - "compressed" 2. It is requested that you develop a Lambda function to execute the subsequent actions: a) Upon the ...

🚀Data Integration and Category Prediction

Nov 11, 2023 · 6 min read · Flow Chart : ✔Project Execution Flow Create an S3 Bucket for Raw Data --> ✔Load data using the CLI commands -->✔Create a Crawler and Crawl Open the Data --> ✔Create Glue Catalog -->✔ Error with the Json Format(Serde) -->✔ Pre-processing the data --> ...

Transferring On-Premise Data to S3

Sep 24, 2023 · 5 min read · Problem Statement We will read data from an on-premise DBeaver Server database and automatically load it to Amazon object storage S3 using Python. We will use the AWS S3 API to achieve this. The idea here is that we have data in our premise database....

A Beginner's Guide to Setting Up RDS PostgreSQL and Connecting with DBeaver(Privetly through SSH tunnel)🌐

Sep 23, 2023 · 4 min read · Step 1: Create Database in AWS RDS Choose the Standard Create option and the database you want to integrate with. I'll choose PostgreSQL for this hands-on. Do read the information provided on the right side. Choose the PostgreSQL version you want t...