Comment by Thomas Hettig on "Individual Scores in Choice Models, Part 1: Data & Averages"

Thanks Chris Chapman for this great blog article!

In your Quant UX book, you recommend using a higher number of tasks per participant to achieve precise individual-level estimates. However, in this blog, you used only 6 tasks for the MaxDiff analysis, which, according to your book, would primarily yield precise sample-level estimates.

I’m curious: How much of an impact does a relatively low number of tasks have on the precision of individual-level estimates?

Can you elaborate when I should definitely follow the textbook recommendation of 12-15 tasks (depending on the specific MaxDiff)? I would think that a segmentation study needs more accuracy then a simple analysis if there is an item that is highly important to some survey participants?!

Yes, generally speaking, segmentation benefits from higher per-respondent precision. The usual recommendation is to have 3 exposures of each item to each respondent. So, with N=14 items, 5 at a time, that would be 14x3 / 5 = 8.4 (or 8) screens.

OTOH, respondent attention lags over time, and there is some evidence that preferences do not change much after 6-8 tasks. At least not unless attention is refreshed, e.g., with an interstitial task or screen.

In the present case, I went with 6 screens x 5 tasks / 14 items = 2.1 average item presentations per respondent. But the goal here was mostly to do high-level summaries. (In a later post, I look at segmentation as a kind of side analysis.)

As for individual-level precision and segmentation, it also depends whether you want to do LCA (estimating segments together with utilities) or post-hoc clusters (utilities first, then segments). LCA is preferable to post-hoc clustering in the case of smaller numbers of per-respondent tasks. HTH!

Thanks for your reply Chris Chapman! I didn’t know that LCA could estimate segments alongside utilities.

I’d like to apply the methods you described in this blog to a current project. My survey includes some additional questions besides the MaxDiff tasks, and I don’t want to stretch participants’ patience too much. I’m wondering if it’s feasible to show each item an average of only two times per respondent and still obtain informative individual-level results. Or do you think this would be a “bad” idea and that it’s better to avoid analyzing individual scores in this case?

The answer is probably, “It depends,” but if I’m not conducting a segmentation study and just want to identify whether some items seem particularly important to certain respondents, could the following principle apply? “Our job is to help make decisions under uncertainty. We aim to take better bets. Sometimes it’s acceptable to show items only twice on average, if that’s the best we can reasonably do.”

Thomas Hettig That is exactly what I would do — run the survey with the best experience I can to get good data from a good respondent experience, and not worry about whether I have some "right" number of observations.

The other thing I would say is this: for any single individual, the specific estimates in that case are imprecise, but the overall averages & estimated numbers of individuals preferring one thing or another will be precise. Unless you are specifically targeting individuals, that is good enough for almost everything (some forms of segmentation being the exception).

Cool! Thank you so much Chris!

Search Hashnode