Anyone who reviews AI output by eye should see your ACL example. Both versions look equally tidy, and only the order of two lines decides whether Telnet on port 23 gets through. I keep hitting the same thing with LLM evals, where a requirement nobody wrote down passes every check. In your workflow, would the engineer write "deny Telnet" as a Batfish policy check before asking the AI for a config, or pull it from the prompt afterwards?
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Anyone who reviews AI output by eye should see your ACL example. Both versions look equally tidy, and only the order of two lines decides whether Telnet on port 23 gets through. I keep hitting the same thing with LLM evals, where a requirement nobody wrote down passes every check. In your workflow, would the engineer write "deny Telnet" as a Batfish policy check before asking the AI for a config, or pull it from the prompt afterwards?