I’ve generally taken a pretty dim view of focus groups, which seem to be used mostly to detect and amplify unthinking prejudices. But, I have to admit, that probably is due at least in part to the fact that my prejudices aren’t very close to those of the median focus group participant. So, it wasn’t until I saw focus group results matching my own thoughts that I paid any attention to the question of whether the results actually meant anything.
According to the SMH report, participants in two Ipsos focus groups almost uniformly saw Morrison’s response to the bushfires as “pathetic” and “lacking empathy”. In each case, eight of nine respondents gave such negative views. Ipsos also concluded there was “little confidence” Labor leader Anthony Albanese would have provided better leadership, with descriptions like “weak” and “bland” being offered. Regular readers won’t be surprised to learn that I agree with both judgements.
But how much can we learn from an exercise involving only eighteen participants? Surprisingly, the answer is, quite a lot. Use q to denote the proportion of the population who approve of Morrison
If the population from which the group was drawn was evenly divided between approvers and disapprovers, the chance of getting results like this would be tiny. This table (look at N=18, x = 16, p 0.5) gives it at 0.000584. A classical hypothesis test would reject the null hypothesis (or, in the usual jargon, find a statistically significant effect) for any chance below 0.05. Using a more sensible, Bayesian approach, with just about any prior distribution, the updated estimate of for the distribution of q would lie entirely below 0.25 (25 per cent approval). We could do something similar for Albanese, but the exact numbers aren’t given.
These numbers are substantially worse than the approval/disapproval numbers given by Newspoll. So what is going on?
Read More »