How to Supercharge Your Streaming Data Pipeline in PythonStreaming data processing has come a long way, so why stick to old methods and not use modern practices. Let me share my fresh perspective that can help you solve your problem. Inspiration from Batch Processing Batch Processing shines with below (tho...1MJun 8, 2025 路 7 min readimportidea.dev
Why Using SQL for Data Application Development Might Be a MistakeData is a crucial component of any modern application or service, and how you process it can determine the success or failure of your app. While data is often stored in databases, relying on SQL queries for processing can be problematic. Let me expla...1RMay 17, 2025 路 7 min readimportidea.dev
From Failure to Flow: How I Used Polars to Conquer Memory Issues in Our Data PipelinesEver been bogged down by data pipelines crashing due to memory issues? It's a frustratingly common problem in data engineering projects. This post chronicles my experience of identifying and resolving memory bottlenecks in our data processing using t...47HMSApr 26, 2025 路 8 min readimportidea.dev
How to effectively work with Databases in PythonIntroduction The age-old debate on the use of Raw SQL v/s ORM is still very much alive in today鈥檚 world. Let鈥檚 see some of the comparing points AspectRaw SQLORM (Object-Relational Mapping) Ease of UseRequires knowledge of SQL syntax and databa...Dec 26, 2024 路 7 min readimportidea.dev
Streamlining Your Databricks Environment SetupI'm pretty sure that if you're using Databricks to run your PySpark job, these might be your typical steps: Design and develop business logic. A notebook that performs all the business logic. Running that notebook using Databricks Workflow. This...Nov 28, 2024 路 4 min readimportidea.dev
