---
title: "Discussion on \"My unpublished benchmark post said 10x. The honest number is 1.2x to 1.8x.\" | Hashnode"
description: "Discussion on \"My unpublished benchmark post said 10x. The honest number is 1.2x to 1.8x.\". An old Foxstash draft claimed it beat hnswlib by 2x. Its own table showed it losing 3x on one thread. Speed only means something at the same recall, on the same threads, on the dataset the label names."
url: "https://hashnode.com/posts/matched-recall/6ad8b7601fd070640089c8e6"
---

[Feed](https://hashnode.com/feed) /

Discussion

# [My unpublished benchmark post said 10x. The honest number is 1.2x to 1.8x.](https://blog.narcolepticfox.com/matched-recall?utm_source=hashnode&utm_medium=feed)

An old Foxstash draft claimed it beat hnswlib by 2x. Its own table showed it losing 3x on one thread. Speed only means something at the same recall, on the same threads, on the dataset the label names.

Oct 21 · 5 min read · blog.narcolepticfox.com  
Read the article

-   [#rust](https://hashnode.com/tag/rust)
-   [#performance](https://hashnode.com/tag/performance)
-   [#benchmarking](https://hashnode.com/tag/benchmarking)
-   [#open-source](https://hashnode.com/tag/open-source)
-   [#databases](https://hashnode.com/tag/databases)

## Written by

[Brock Claussen](https://hashnode.com/@claus)

@claus · Florida, US

  
[View profile](https://hashnode.com/@claus)

Founder of Narcoleptic Fox. Marine veteran. Senior engineer learning AI/ML and writing about it. Building Mousing.ai, Reqforay,.ai, and foxstash.

## Responses 2

Markdown supported · ⌘↵ to post

[c-wind](https://hashnode.com/@c-wind)

[Oct 23](https://hashnode.com/posts/matched-recall/6ad8b7601fd070640089c8e6/comment/6adb5ea7699e191b20128847)

The trap that is easy to miss is the one pointed the other way: hnswlib's Python query uses every core by default, so timing that against a single-threaded Rust loop made Foxstash look about 11x slower. Same class of mistake as the draft's batch-API headline, just with the sign flipped.

The surviving table is the part I would actually cite. On SIFT1M, SQ8 is 1.20x hnswlib and 1.30x faiss at 99.5% recall@10, while uncompressed Foxstash is 0.88x hnswlib. That matches the latency story: 87.1 ns per distance down to 67.3 because SQ8 keeps the 8-bit codes in the block the search reads, not because the graph got smarter. The fifth rule belongs in the code, not the comment above it. The harness swept hnswlib and faiss and pinned Foxstash, so the published numbers survived a sweep that was accidentally rigged against you.

[Brock Claussen](https://hashnode.com/@claus)

[Oct 23](https://hashnode.com/posts/matched-recall/6ad8b7601fd070640089c8e6/comment/6adb5ea7699e191b20128847#reply-6adb8d674854b822eedc868a)

Thanks for reading it this closely. You're right that the thread trap cuts both ways, and I think that direction is the more dangerous one. A number that makes you look worse doesn't get questioned, so nobody goes looking. The sweep was the same: rigged in our favor, I'd have caught it; rigged against us, it just sat there. Have you hit the corpus-shape trap yourself? That's the one that still bothers me most, because the number it produced looked completely normal.