#experimentation articles

Mmayaanderssonllmasajudge.hashnode.devAug 12 · 5 min read

Run forty experiments against one eval set and you will find an improvement that is not there

You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an

0

RPRudrendu Paulfreecodecamp.orgAug 11 · 37 min read

Product Experimentation at Scale: How Airbnb, Netflix, Lyft, and Uber run Causal Inference on LLM-Based AI Features

Causal inference for LLM-based AI features is no longer theoretical. Airbnb, Netflix, Lyft, and Uber have published detailed engineering blog posts describing exactly how they measure the causal impac

0

RPRudrendu Paulfreecodecamp.orgAug 11 · 23 min read

Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications

Your AI product shipped an agent-mode opt-in six months ago. You ran a propensity analysis, adjusted for engagement tier and query confidence, and reported a clean +8 percentage-point lift in task com

0

RPRudrendu Paulfreecodecamp.orgJul 29 · 21 min read

Product Experimentation with Instrumental Variables: Unconfounding LLM Routing Decisions in Python

For data science leaders and product managers who are overseeing multi-model gateways, the standard regression approach to measuring model quality is fundamentally flawed. You're running a causal infe

0

RPRudrendu Paulfreecodecamp.orgJul 23 · 22 min read

Product Experiment Counterfactual Methods for Estimating the Effects of AI Prompt Engineering

Imagine your team deployed Prompt A globally two weeks ago. Tight deadlines and high confidence meant the rollout hit 100 percent of users without any A/B testing, shadow traffic, or holdout groups. W

0

AJAman Jaincurious-pm.hashnode.devJul 17 · 8 min read

From Building a GPT Model to Experimenting With One

Over the last few articles, I built the transformer block, assembled the complete GPT architecture, loaded pretrained GPT-2 weights, fine-tuned the model for classification and instruction tasks, and

0

RPRudrendu Paulfreecodecamp.orgJul 15 · 20 min read

Product Experimentation with Regression-Based Causal Inference: Estimating LLM Feature Impact with Python and statsmodels

A randomized A/B test is the cleanest form of product experiment available. The coin flip that splits users between the new prompt template and the control removes every possible confounder by constru

0

RPRudrendu Paulfreecodecamp.orgJul 9 · 24 min read

Product Experimentation with Uplift Modeling: Targeting Your LLM Feature Rollout to Users Who Actually Benefit (Python Implementation)

Your LLM product experiment just came back positive, with a promising 8-percentage-point lift in task completion. You ship the feature and leadership celebrates. Three months later, the core metric ha

0

RPRudrendu Paulfreecodecamp.orgJul 2 · 21 min read

Product Experimentation: Stop Early Without P-Hacking Using mSPRT and Sequential Testing in Python

Your AI product experiment reaches statistical significance on day 14 of a planned 30-day run, measuring a causal inference question: did the LLM-based feature genuinely improve outcomes? Every produc

0

RPRudrendu Paulfreecodecamp.orgJun 30 · 28 min read

Product Experimentation for LLM Platforms: Switchback Designs When User Randomization Breaks Market Equilibrium in Python

Your team ships an intelligent query-routing feature for an LLM SaaS platform. The feature reads each incoming request in real time and decides whether to send it to the fast standard model or the mor

0

#experimentation

#experimentation

Explore Hashnode

Trending tags this week

Run forty experiments against one eval set and you will find an improvement that is not there

Product Experimentation at Scale: How Airbnb, Netflix, Lyft, and Uber run Causal Inference on LLM-Based AI Features

Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications

Product Experimentation with Instrumental Variables: Unconfounding LLM Routing Decisions in Python

Product Experiment Counterfactual Methods for Estimating the Effects of AI Prompt Engineering

From Building a GPT Model to Experimenting With One

Product Experimentation with Regression-Based Causal Inference: Estimating LLM Feature Impact with Python and statsmodels

Product Experimentation with Uplift Modeling: Targeting Your LLM Feature Rollout to Users Who Actually Benefit (Python Implementation)

Product Experimentation: Stop Early Without P-Hacking Using mSPRT and Sequential Testing in Python

Product Experimentation for LLM Platforms: Switchback Designs When User Randomization Breaks Market Equilibrium in Python

#experimentation

Search Hashnode

#experimentation

Explore Hashnode

Trending tags this week

Run forty experiments against one eval set and you will find an improvement that is not there

Product Experimentation at Scale: How Airbnb, Netflix, Lyft, and Uber run Causal Inference on LLM-Based AI Features

Product Experimentation with Doubly Robust Estimation: When Both Your Models Are Wrong in LLM Applications

Product Experimentation with Instrumental Variables: Unconfounding LLM Routing Decisions in Python

Product Experiment Counterfactual Methods for Estimating the Effects of AI Prompt Engineering

From Building a GPT Model to Experimenting With One

Product Experimentation with Regression-Based Causal Inference: Estimating LLM Feature Impact with Python and statsmodels

Product Experimentation with Uplift Modeling: Targeting Your LLM Feature Rollout to Users Who Actually Benefit (Python Implementation)

Product Experimentation: Stop Early Without P-Hacking Using mSPRT and Sequential Testing in Python

Product Experimentation for LLM Platforms: Switchback Designs When User Randomization Breaks Market Equilibrium in Python