importidea.devHow to Supercharge Your Streaming Data Pipeline in PythonStreaming data processing has come a long way, so why stick to old methods and not use modern practices. Let me share my fresh perspective that can help you solve your problem. Inspiration from Batch Processing Batch Processing shines with below (tho...Jun 8, 2025路7 min read
importidea.devWhy Using SQL for Data Application Development Might Be a MistakeData is a crucial component of any modern application or service, and how you process it can determine the success or failure of your app. While data is often stored in databases, relying on SQL queries for processing can be problematic. Let me expla...May 17, 2025路7 min read
importidea.devFrom Failure to Flow: How I Used Polars to Conquer Memory Issues in Our Data PipelinesEver been bogged down by data pipelines crashing due to memory issues? It's a frustratingly common problem in data engineering projects. This post chronicles my experience of identifying and resolving memory bottlenecks in our data processing using t...Apr 26, 2025路8 min read
importidea.devHow to effectively work with Databases in PythonIntroduction The age-old debate on the use of Raw SQL v/s ORM is still very much alive in today鈥檚 world. Let鈥檚 see some of the comparing points AspectRaw SQLORM (Object-Relational Mapping) Ease of UseRequires knowledge of SQL syntax and databa...Dec 26, 2024路7 min read
importidea.devStreamlining Your Databricks Environment SetupI'm pretty sure that if you're using Databricks to run your PySpark job, these might be your typical steps: Design and develop business logic. A notebook that performs all the business logic. Running that notebook using Databricks Workflow. This...Nov 28, 2024路4 min read
