I built an LLM eval framework from scratch. Here is what I wish I had bought instead.

One weekend I wrote an LLM eval framework in about two hundred lines of Python. It demoed beautifully. I felt clever. Six months later that same framework was a mess. Three different judge models with