Search posts, tags, users, and pages
There's a gap between the Quant UX book and my R for Marketing Research book (although it's filled in the Python for Marketing Research book) ... and that concerns hash tables, also known as associative arrays, lookup tables, and dictionaries. What's...
quantuxblog.com10 min read
Markdown supported · ⌘↵ to post
Good question and there are many ways to solve the word counting problem! I didn't intend so much to solve it as to demonstrate hash tables.
As for a tidyverse solution, the thing I would watch out for — both in practice and in an interview — is the scalability and performance. As I linked in the previous comment thread, hash tables have nearly constant time needed per-element, regardless of the number of elements. Data frames, lists, and many other kinds of solutions require more and more time to look things up as the object gets larger.
For many applications the performance differences don't matter. However, in an interview situation, I would expect follow-up questions about the pros and cons of any particular solution.
For tidyverse, I might ask about the footprint size and the multi-year code stability of a solution used in production. (The goal of a question like that is to probe how much a candidate understands about the tradeoffs of code. We say a lot more about that kind of question in the book.)
FWIW, the least complex solution to the specific problem in R given the assumption of a "words" vector is: table(words).
However, for that answer, I would do exactly what my interview did at the time, and say, "OK, but assume you have to do this in base R and there is no table() [or tidyverse] function!"
Is there a reason you prefer to use a new.env() rather than a list (which is how I would have done it)?
Great question, and good to hear from you! Lists and environments are extremely similar (thanks to the functional, Scheme-like nature of R) so either should be possible. If I get a chance, I'll see how much difference appears when written with lists.
IMO environments are a slightly better match for hash tables because they are exactly hashed in their implementation and unordered (whereas lists have sequential elements) and they work by reference (instead of copying on change, thus can be performance enhancing). I'm not sure whether there are different scale issues vs. lists, but wouldn't expect any (because of the underlying Scheme-like nature, as above).
Still, the old joke about economists applies to most things in R: if you ask a question of 4 R experts, you'll get 5 different answers :)
Looking to see if anyone has compared these, I found that Jeffrey Horner has: jeffreyhorner.tumblr.com/post/114524915928/hash-t…
The takeaway from that is that environments (using their default behavior of being hash tables under the hood) are substantially more efficient.
Environments/hashes have nearly constant speed (close to O(1)) as the size grows, but lists take linearly increasing time (close to O(n)) for lookups.
Riccardo Melani
Hello and thanks for this amazing blog! Assuming we have that character vector (words) containing the words to count (which would need to be generated following some text mining I guess?), how about this in tidyverse: wordcounter <- data.frame(word= words) %>% count(word)