Susan Frede, VP Research
November, 2008
Online research has become an accepted and widely used research methodology because of its benefits of lower cost and faster turnaround. As with any research methodology, it is important that online research produce stable, consistent results from test to test. Given the quick adoption of the online methodology in the U.S. concerns have emerged regarding the stability of online data and the ability to replicate results especially with concept tests. The goal of this research is to assess the stability of online research for collecting concept test data using the MySurvey Consumer Panel.
Background
Lightspeed Research has fielded two separate tests to address the question of stability:
- August 2006 Test – 10 concepts, both branded and unbranded, were tested from a variety of industry sectors. Nearly 6000 interviews were conducted with approximately 300 per concept per point in time.
- July/August 2007 Test – 20 branded consumer package goods concepts were tested. Nearly 17,000 interviews were conducted with approximately 400 per concept per point in time.
In both studies, each concept was tested monadically at two points in time with one week between the test and retest to minimize any potential impact from seasonality and market changes. The demographic universes were selected as appropriate for the product target market. Key concept measures included Purchase Intent, Uniqueness, Value, Liking, Believability, Quantity and Frequency. Respondents also were asked a number of category habit and brand usage questions.
Concept Purchase Intent
Purchase intent is the most important measure in concept testing. In both tests results are generally consistent between the two time periods, indicating stable results (see Table 1). Of the 60 statistical comparisons for purchase intent, 56 or 93% are not statistically different at the 95% confidence level (see Appendix for data). The small percentage of difference is well within the error expected from a probabilistic sample.
Other Key Measures
Similar to purchase intent, there are few statistically significant differences between the two time periods for the other key measures (see Table 2). Of the total comparisons 92% (380 out of 412) are not significantly different at the 95% confidence level. No sample is a perfect representation of the population of interest, which leads to some fluctuation in scores from sample to sample (random sampling error). Again, the small percentage of difference is well within the error expected from a probabilistic sample.
Demographics
Balancing outgoing samples on key demographic variables helps ensure representative return samples. In both tests, the return samples line up well between the waves on demographics. There are a few statistical differences, but no sample has differences across all the demographic variables and none is so skewed that key measures are impacted.
Category & Brand Usage/Purchase
Category and brand usage/purchase differences between samples can dramatically impact results. For example, having more users of a category in a sample can inflate key measure scores. Having samples that are demographically balanced reduces the potential for differences in category and brand usage/purchase thus producing more stable concept results. There are three category/brand usage differences between the waves: (See Appendix for data.) Those in wave 1 for Concept 10 are significantly more likely to be brand users. However, this does not impact key measure results because the concept is for a new variant of the product that current users are not likely to use. Those in wave 1 for Concept 20 are significantly more likely to be brand users. This does cause slightly, but not significantly higher purchase intent scores in wave 1. Those in wave 1 for Concept 24 are significantly more likely to be brand users. This is driving the purchase intent score differences for Concept 24.
It is important to examine all samples carefully to make sure category habits and brand usage fall within an acceptable range. When there are differences, weighting can bring imbalances in line.
Conclusions
This research demonstrates that Lightspeed Research produces stable concept test results with few statistically significant differences between time periods on key measures when utilizing the MySurvey Consumer Panel as a sample source. This research also points to the importance of careful evaluation of demographics, habits, and brand usage/purchase for each sample to identify any variations that could impact key measures.
Lightspeed Research provides stable concept test results because of our panel and sample management practices, which allow us to build samples that are representative of the population of interest on key demographic variables. Balancing on these key demographic variables reduces the chance of having category and brand usage differences between samples, which can drive differences in concept evaluations. The best way to guarantee stable results is to use professionally managed samples.
About Lightspeed Research
Lightspeed Research (www.lightspeedresearch.com) is the market researcher’s choice for digitally accessing and deriving insight from consumer opinions and behaviors whenever, wherever and in whatever segments needed. The industry’s most thorough panelist pre-screening process and large global pool delivers business-ready results quickly and cost-effectively. From proprietary online access panels to specialty panels, custom panels and innovative mobile surveys, Lightspeed Research offers the industry’s highest-quality and most complete combination of qualitative and quantitative online research. This is backed by an expert client operations team that provides a range of data collection services, from sample management and survey design to programming and reporting. Part of Kantar, a division of WPP, Lightspeed Research serves clients and cultivates online panelists across the Americas, Europe and Asia Pacific.
Susan Frede is the VP of Research at Lightspeed Research. She has worked in the research field for 23 years, has published numerous research-on-research papers and is a well-respected speaker at key industry events. Some of the topics she has recently explored include questionnaire length, best practices for online research, suspicious and professional respondents and data stability.
You can contact Susan at sfrede@lightspeedresearch.com
