Tag feed

#statistics

828 posts215 followers

Explore Hashnode

Alternatives

Trending tags this week

DDataModelLabdatascienceclearly.hashnode.dev6d ago · 8 min read

Logistic Regression Assumptions Explained: Why They Matter and What to Do When They Fail

INTRODUCTION: What is Logistic Regression? Logistic regression is often one of the first classification algorithms we learn. It is simple, interpretable and widely used—but fitting the model is not en

0

MMohitmmonthlydatascience.hashnode.devAug 23 · 7 min read

Correlation Is Pairwise. Multicollinearity Isn't

Almost every regression project I've seen in the last decade starts the same way. Load the data, df.corr(), plot the heatmap, scan for red squares, drop one variable from every pair above 0.8. Then we

0

Mmayaanderssonllmasajudge.hashnode.devAug 17 · 9 min read

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

TL;DR: Judge-human agreement is almost always reported as one number over a whole validation set. That number is dominated by the easy cases, because most examples are not close to your decision bound

0

Mmayaanderssonllmasajudge.hashnode.devAug 12 · 5 min read

Run forty experiments against one eval set and you will find an improvement that is not there

You iterate on a prompt, run the eval set, keep the best variant. Repeat for a quarter. The number that survives is the maximum of forty noisy measurements, and the maximum of noisy measurements is an

0

Mmayaanderssonllmasajudge.hashnode.devAug 7 · 10 min read

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

A team I compared notes with recently runs continuous evals on production traffic: an LLM judge scores a sample of traces, the scores land on a dashboard, and a monitor pages when the hourly pass rate

0

Mmayaanderssonllmasajudge.hashnode.devAug 6 · 12 min read

Upgrading the judge ends one score series and starts another

There is a mature literature on what happens when you swap one measuring instrument for another, and it is not in machine learning. The standard treatment is Bland and Altman, "Statistical methods for

0

Mmayaanderssonllmasajudge.hashnode.devAug 5 · 15 min read

A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.

Most of what I have written about LLM judges argues in one direction: your improvement is probably not real. Small eval sets, selection across many experiments, position bias, clustered examples. All

0

BMBillie Mbilliem.hashnode.devAug 4 · 7 min read

The LLM was better at building a solver than playing the game

I did not expect to be the difficult opponent. I had only started watching Dota 2 occasionally again during the previous six months or year. I am not especially knowledgeable about it. When I asked a

0

Mmayaanderssonllmasajudge.hashnode.devJul 28 · 6 min read

Your eval's confidence interval assumes independent examples. Yours are clustered.

Every binomial confidence interval you have ever computed on an eval pass rate, Wald, Wilson, Clopper-Pearson, all of them, rests on one assumption: each example is an independent draw. Most eval sets

0

Mmayaanderssonllmasajudge.hashnode.devJul 22 · 8 min read

An LLM judge is a biased instrument, not a measurement

Last month I shipped an eval that ranked two prompt variants. Variant A won by four points. A teammate reran the same eval the next morning and Variant B won. Same model, same judge, same test set. Th

0

#statistics

Search Hashnode

#statistics

Explore Hashnode

Trending tags this week

Logistic Regression Assumptions Explained: Why They Matter and What to Do When They Fail

Correlation Is Pairwise. Multicollinearity Isn't

A judge that agrees with your humans 92 percent of the time can be at 60 percent where the gate actually decides

Run forty experiments against one eval set and you will find an improvement that is not there

Your eval monitor fired on four days this week. At your sample size, that was the most likely count

Upgrading the judge ends one score series and starts another

A noisy judge does not just add error bars. It shrinks the effect you are trying to measure.

The LLM was better at building a solver than playing the game

Your eval's confidence interval assumes independent examples. Yours are clustered.

An LLM judge is a biased instrument, not a measurement