--- title: "srkhan2809 (@shahida-khan) | Hashnode" description: "Lead Data Infrastructure Architect | Databricks, PySpark & Cloud DW. Read the latest articles by srkhan2809 on Hashnode." url: "https://hashnode.com/@shahida-khan" --- @shahida-khan # srkhan2809 @shahida-khan · Dubai, UAE · Joined April 2026 Lead Data Infrastructure Architect | Databricks, PySpark & Cloud DW [Website](https://www.toptal.com/developers/resume/shahida-r-khan?preview) ### [Data Modeling Explained: A Principal Engineer's Guide](https://core-de-platform.hashnode.dev/data-modeling-principal-engineer-guide?utm_source=hashnode&utm_medium=feed) Learn how data modeling works, explore its major types, trade-offs, real-world examples, and how Principal Engineers choose the right model. 24 min read · [core-de-platform.hashnode.dev](https://core-de-platform.hashnode.dev/data-modeling-principal-engineer-guide?utm_source=hashnode&utm_medium=feed) [Nov 2](https://hashnode.com/posts/data-modeling-principal-engineer-guide/6aedb978ee7a0111f6da72a1) ### [Spark Structured Streaming: A Principal Engineer's Guide](https://core-de-platform.hashnode.dev/spark-structured-streaming-principal-engineer-guide?utm_source=hashnode&utm_medium=feed) It was 2 AM. The pipeline was green, but the numbers were wrong. Imagine you're building a data platform for a financial services company. Every second, thousands of transactions are happening. Customers are making payments, transferring money, placing orders, and checking account balances. The business wants to see these transactions almost immediately. So, you build a Kafka-to-Spark streaming pipeline. Everything looks good. Events are flowing, transformations are working, and Delta tables are getting populated. Then production happens. A transaction that occurred at 10:00 AM arrives at 10:15 AM. Another transaction arrives twice. A particular customer generates 50,000 events while most customers generate fewer than 10. Your aggregation starts consuming more memory. The processing time increases. The streaming query falls behind. And the business dashboard is now showing numbers that nobody trusts. The pipeline is technically running. But the data platform is failing. This is where understanding Spark Structured Streaming becomes important. Streaming is not just about reading data continuously. It is about handling late events, duplicates, state, failures, recovery, and data correctness without losing control of the system. Let's understand how Spark handles all of this, starting from the basics and gradually moving towards production architecture. 15 min read · [core-de-platform.hashnode.dev](https://core-de-platform.hashnode.dev/spark-structured-streaming-principal-engineer-guide?utm_source=hashnode&utm_medium=feed) [Oct 28](https://hashnode.com/posts/spark-structured-streaming-principal-engineer-guide/6ae85012a01cc8295170e1ff) ### [The Load Balancer That Kept Choosing a Broken Server](https://de-system-designs.hashnode.dev/load-balancer-keeps-choosing-broken-server?utm_source=hashnode&utm_medium=feed) Why least-connections load balancing can send more traffic to a failing server, and how health checks and outlier detection prevent it. 17 min read · [de-system-designs.hashnode.dev](https://de-system-designs.hashnode.dev/load-balancer-keeps-choosing-broken-server?utm_source=hashnode&utm_medium=feed) [Oct 23](https://hashnode.com/posts/load-balancer-keeps-choosing-broken-server/6af82de44569fbbba71094f6) ### [Databricks Features That Matter: A Staff Engineer's Guide](https://core-de-platform.hashnode.dev/databricks-features-staff-engineer-guide?utm_source=hashnode&utm_medium=feed) The platform is evolving fast. Here’s how to separate genuinely useful engineering capabilities from features that look impressive in a demo. There’s a particular kind of excitement in data engineering. A new feature launches. The demo looks fantastic. Someone shares a benchmark. Someone else announces that their architecture is now “AI-native.” Before the week is over, your backlog has acquired another migration project. Meanwhile, the production pipeline is still late, the cloud bill is doing cardio, and Finance has three different definitions of revenue. Welcome to data engineering. Databricks has been evolving well beyond its original identity as a managed Spark platform. Its newer capabilities span data ingestion, declarative pipelines, governance across engines, real-time processing, AI-assisted development, and conversational analytics. The 2025–2026 Lakeflow and Unity Catalog announcements make that direction particularly clear. But the interesting question isn't what has Databricks launched? It's this: Which capabilities change the way we build reliable data platforms, which problems do they actually solve, and when should we deliberately choose not to use them? Let's explore that through a problem most data teams will recognize. 23 min read · [core-de-platform.hashnode.dev](https://core-de-platform.hashnode.dev/databricks-features-staff-engineer-guide?utm_source=hashnode&utm_medium=feed) [Oct 19](https://hashnode.com/posts/databricks-features-staff-engineer-guide/6adb1ad27a096292d0555e0c) ### [Crypto Finance Data Modeling: A Staff Engineer's Field Guide](https://core-de-platform.hashnode.dev/crypto-finance-data-modeling?utm_source=hashnode&utm_medium=feed) Your data model isn't just a way to organize tables. In a financial system, it is part of how you prove what happened to someone's money. A badly designed model can make a successful trade look like a failed one, turn a refund into revenue, double-count trading volume, or report a wallet balance that never actually existed. And the most dangerous part? The dashboard can still look beautiful. I've seen teams spend weeks optimizing Spark jobs, partitioning Delta tables, tuning joins, and reducing query latency. All useful work. But before touching any of that, there's a more fundamental question worth asking: Are we even representing the business correctly? Because a query that returns the wrong answer in 200 milliseconds is not an optimization. It's a faster way to lose trust. Let's take a practical look at data modeling through the lens of a crypto trading platform. We'll examine the business requirements, the modeling choices, the architecture, the trade-offs, and the decisions a Staff Engineer should be making before the first production table is created. 29 min read · [core-de-platform.hashnode.dev](https://core-de-platform.hashnode.dev/crypto-finance-data-modeling?utm_source=hashnode&utm_medium=feed) [Oct 17](https://hashnode.com/posts/crypto-finance-data-modeling/6ad5fbd143247129ad7ddec7)