Comment by Ahmet Özel on "Semantic Search Over Transcripts in Python: What Embeddings Fix and What They Don't"

The 'reduce churn' example is a useful demonstration of vocabulary coverage rather than a general failure of semantic embeddings. In the LSA version, churn has no TF-IDF feature at all, so the query effectively becomes reduce before the projection; the pretrained swap tests a materially different source of knowledge.

I would print the recognized and discarded query terms next to each result list. Then compare 'reduce churn' with 'reduce cancellations' and a no-overlap paraphrase under both embeddings, leaving the relevance labels fixed. That would make it clear which gains come from corpus vocabulary, the learned projection or pretrained semantics, rather than treating vector search as one interchangeable method.

Thanks Ahmet, this is a good way to split it. I built your version and added it to the post as an update, with the code and the real output. The right answers stay fixed for all three queries, and the script prints which query words the model used.

"reduce churn": only "reduce" is used, so keyword and LSA both find 0 of 3. "reduce cancellations": both words are used. Both find 1 of 3, but LSA ranks the right paragraph first and keyword ranks it second. "subscriber attrition": no word is in the transcript, so both return nothing.

So in this small test, vocabulary changes the result the most, and the learned projection changed the order once and the count never. I have not run the pretrained half yet, so the post says "not run" and gives no numbers for it. If you run it on your side, I would like to compare.

I also added a line near the top that says the vector search in this post is a small LSA stand-in, so the title is not read as a claim about all embeddings.

Thanks for running that comparison and keeping the pretrained half explicitly unmeasured. The printed query vocabulary now makes the zero-result cases explainable, and the unchanged hit count with a different rank is a useful distinction.

I have not run the pretrained version, so I do not have results to compare. For that extension, I would preserve the same corpus, relevance labels and cutoff, and report ranking changes separately from newly retrieved relevant paragraphs. The zero-vector case should remain explicit rather than producing arbitrary nearest neighbours.

Search Hashnode