Optimum Blending of Panels and Social Network Respondents
Understand the Differences
Social Network respondents are different from those now drawn from double opt-in panelized respondents. These differences are inherent in their reasons for being online. Those who are using the social network arena to communicate with others, obtain news, or be entertained are likely to be different from others who are participating only for online purchases, doing their banking, or searching for the best deal available for an airline ticket. We must establish methods for blending this new wave of respondents into existing panels, while maintaining consistent results. Users of these panels must be assured that the addition of any new source, including Social Networks, will not introduce instability to the samples and increase the variability in their data.
While we attempt to control for differences in our respondents via demographic quotas, it’s clear that individuals from Social Networks are considerably different. When examining education (Figure 1), among social networked individuals, with identical Sex x Age x Income distributions, we find a far less educated population than the ones derived from a typical online panel.
Standards: Minimum Measurable Difference
The minimum measurable difference is a means of determining the threshold at which we begin to detect statistical difference at an a value level so low that it represents a conservative measure of similarity. Anything below the minimum would be considered to be an undetectable difference, allowing the statement that “as we fail to detect difference, we can declare the two populations statistically similar in the metrics that we are evaluating.”
|An absence of detectable difference implies similarity|
Social media participants represent a large potential opportunity to source respondents for market research purposes. By virtue of their difference and abundance, we must find ways to include them in our online research. We have proposed a conservative and measured way of including these new sources in a granular fashion. Their inherent difference within each demographic cell dictates the maximum blending percentage we feel can comfortably be added to a host population of online panel respondents.
There are two measurement issues; first is how to measure differences between the two panels and second what is the largest acceptable difference. Since this is a simple (linear) mixture, the acceptable maximum ratio would be equal to the largest acceptable difference divided by measured difference between the data sources to be blended. The measured difference is taken as the “Root Mean Squared Difference”.
The management of online samples is shifting from quota fulfillment to a concern for total sample frame. This type of approach is sensitive to the overriding philosophy that those who use these samples must be confident that the change that they see in their data is real and not an artifact generated by shifts in the constituent elements of the sample source being employed. Sample providers have a responsibility to be transparent about their sample frame. It is only through clarity that research practitioners can understand how to interpret their data and it is only through that clarity that end users will know what reliance to place upon it.
Optimum Blending Solutions
CASRO Online Conference 2011: Optimum Blending of Panels and Social Network Respondents