Nnavinkumarinnotes123.hashnode.dev路Nov 29, 2025 路 2 min readData Engineering RoadmapIntroduction Many people step out of the Data Engineering learning journey because everything feels confusing, slow, and overwhelming. But the real problem isn鈥檛 the tools, it鈥檚 the lack of a clear purpose and a clear path. To solve this problem, I c...00
Nnavinkumarinnotes123.hashnode.dev路Nov 28, 2024 路 1 min readIncremental Load in Data bricks part -1Use cases Suitable if the pipeline runs infrequently. Assume the scenario if files are loaded in same directory everyday. If not used processed and yet to process folder. Steps to implement List the current files in directory create database if...00
Nnavinkumarinnotes123.hashnode.dev路Jul 14, 2024 路 4 min readList comprehension vs Generator expressionsyntax List comprehension [expression for item in iterable if condition] Generator expression (expression for item in iterable if condition) only difference is [] & () How Generator Expression Works: Initialization: When you create a generator...00
Nnavinkumarinnotes123.hashnode.dev路Jul 9, 2024 路 1 min readconnect Postgres SQL in pythonPre-requisites: Postgres database Python step -1 pip install psycopg2-binary step -2 postgres credentials host='localhost' dbname='practice_db' username='postgres' password=123456 port=5432 step -3 import the pscopg2 module and use the connect ...00
Nnavinkumarinnotes123.hashnode.dev路Jun 1, 2024 路 2 min readJava JDBCpackage jdbc; import java.sql.*; public class PostgreSQLCRUDExample { // JDBC URL, username, and password static final String JDBC_URL = "jdbc:postgresql://localhost:5432/practicedb"; static final String USERNAME = "postgres"; static...00