Encephos

Thanks for the great feedback and for checking it out! You were spot on about the token overhead and the pandas comparison, so I actually went ahead and ran benchmarks on a corpus of 30 complex Wikipedia tables. Here is exactly how it shakes out: Token Overhead: The dito trick is surprisingly cheap. It only adds a 1.5% overall token overhead compared to leaving cells empty, mostly because span continuations only make up about 3% of all cells in the wild. But you're right about the tradeoff - on heavily spanned tables, overhead can spike to ~55%. (Also, dumping to JSON is brutal; it burns 78% more tokens than Markdown!). Vs. Pandas: pandas.read_html is about 2.9x faster, which makes sense given its C-backend. They agree on spans 90% of the time, but html-table-rescuer wins on messy data. Pandas throws a ValueError if it hits something like colspan="abc", whereas this tool just survives it and keeps parsing. Validation: 100% agree with you. Null-density and type checks absolutely belong downstream in the RAG pipeline. The parser's only job is to faithfully report what the markup says. I've pushed all these benchmarks (scripts and data) to the repo. Thanks again for the push to get these numbers fully documented!

I'd love to test the limits of html-table-rescuer. What is the absolute worst, most malformed HTML table structure you've ever had to parse? Drop a snippet or a link below and let's see if the tool survives it.