· Panels regularly subcontract sample to other
panels, and don’t tell clients.
· The frequency cap for survey invitations seems
to be increasing – panels now typically invite their panelists at least weekly,
and sometimes multiple times per day, to take surveys.
· Respondents can become trained to understand
questionnaire scales, and their reaction change as a result of taking surveys
in the past.
· Panels willingly field terribly designed
surveys.
· It’s easy to join upwards of a dozen panels.
I was fortunate to be in on the ground floor of online
research back in the 90s, and to that end, I’m a bit jaded – nothing in the
report was really news to me. Many of
these talking points have been around for at least a decade, but there are far
more online research players today – many with dubious business
practices. It takes a lot more work to monitor it now, and the responsibility
lies with all of us. I educate my vendors (online sample providers or otherwise)
all the time, and I expect them to do the same for me.
It’s easy to think of sample as a commodity, but it’s
not. Panels have reputations, strengths and weaknesses, and built in
biases that can work to your benefit or detriment, depending on the study. Here
are some of the strategies I’ve used for minimizing the negative issues
outlined in the report.
1. Dig
beyond the bid. Ask questions of panel partners. Find out how they
recruit, how they frequency cap, what their invitations look like, etc.
2. Join
their panels. Be a spy. Make sure they practice what they preach.
Call them out on it when they don’t.
3. Tell
them upfront that they can’t outsource/subcontract, or must get approval from
you first. Don’t assume that quality panels don’t do this. Most of them do –
out of necessity.
4. Bring
in multiple sample partners on a study yourself. Tell them you’re doing
it, and who you’re using. Then compare them. It puts them on good
behavior and gives you insight on how each panel operates.
5. Pretest/soft
launch every study. Check data. Look for inconsistencies.
Look for places with heavy dropout. Adjust course accordingly before full
fielding. We’re in the business of market research, not emergency medicine,
the extra day this takes will be immaterial 99% of the time.
6. Be wary if they don’t ask you
questions. Good panels will have AMs who take your surveys, give you
feedback, provide insight into incentive necessity, and generally demand that
you don’t abuse their panelists. Bad panels will say yes to whatever
ridiculous request you have, usually because they’re desperate for business
and/or have inexperienced employees who don’t know better. It’s a good
litmus test to know if you should turn and run.
7. Don’t
just go with the cheapest sample bidder. There’s usually a reason they’re
cheaper, and it’s probably not one you’ll want to deal with on the backend.
8. Every
now and then, put a question on the survey asking what panels people belong
to. This gives you an idea of where there’s overlap, and how much the
heavy panel subscribers differ from people on only 1 or 2 panels. (I’ve seen
this matter a lot, and I’ve seen it matter not at all.)
9. Make
sure you and they can set and track quotas. Monitor it. This avoids
the dubious statistical manipulations that try to project 10 people to
100.
10. Add
past participation screeners on surveys where you fear a recent participation
effect. This isn’t always relevant, but as pointed out in this report, it
matters for normative testing. Well-designed panels will also be able to
target their invitations to only reach people who have not recently
participated in (your survey/a similar survey). That doesn’t mean
respondents haven’t taken it through another panel, but it’s an extra line of
defense.
The world of the RDD gold standard is long gone. But online panels can provide brilliant,
thoughtful, and valuable insights if you manage the research correctly.
No comments:
Post a Comment