The Survey Research Prisoner's Dilemma: Who's Going to Put the Brakes on Bad Survey Design?

Not too long ago, a Survey Monkey link came up in my Facebook feed.  As you may have deduced, I run a company that (among other things) specializes in survey research, so I clicked the link out of curiosity. Here’s what I saw on the splash page:

By all practical indications, this survey is not intended for me.  It’s targeting community organizers and neighborhood council members, people who volunteer their time for the city’s neighborhoods, and who probably know a thing or two about what gets prioritized, and what doesn’t.

I’m not one of those people.

I read the intro and instructions out of curiosity, expecting to get screened out. But I wasn’t screened out.  Instead, I began to receive questions that I’m uniquely unqualified to answer. Questions that are setting out to prioritize the budget of Los Angeles, the 2^nd largest city in the United States.

This survey underscores a giant flaw in the direction in which survey research is heading: everyone is a survey writer these days.  A few choice shortcomings of this particular survey:

·         There’s no screener.  Anyone can take the survey.  Including me.  Or, more amusingly, someone who does not actually live in the city of Los Angeles.  It doesn’t know I live in LA, because right now, at this very moment, I’m in Ohio. And my IP address will indicate that.

·         It’s not gated.  I took the whole survey. And then I was able to take it again.  Remember now, the topic is the city budget.  I can think of a few stakeholders who’d want to weigh in on that more than once.  Ballot box stuffing, anyone?

·         The survey is unprotected.  Most people are probably too lazy to take this whole thing, but that doesn’t matter – they can still aimlessly click through and screen cap it.  For a political office, this survey represents a potential PR nightmare.  (To which I am actively contributing, guilty as charged.)  My fair research colleagues -- this is how your weakly worded questions make the news.  I’ve had journalists screen capture and post my surveys in their columns before (such is the trade hazard of conducting web intercept surveys on news websites).  And that was stressful even when the questions were well-written, and the journalist was reasonably kind.

It’s tempting to use these self-serve tools to save money; I myself have gone that route when a “real” budget was not in the cards. But when we get to the point that major financial decisions are made based on research conducted by non-researchers using highly flawed methodologies – we need to start taking action to stop it.  Weak research from a select few gives our entire industry a bad reputation.  It causes million dollar mistakes.  And that’s a much bigger deficit than what it would have cost to conduct this survey the right way, with the right professional discipline.

The mayor’s office survey is hardly an isolated incident.  The Grey Matter online research report I mentioned in my last blog post alludes to questionnaire design issues with a few amusing anecdotes that it found in its research.  Among my favorites were the hilariously broad question, “What characters, movies, or brands do you like?”, the grammar cringe worthy, “Of LCD TV, that you currently own, what type does you mainly watch?”, and a straight guy being asked how he dresses his husband.

A lot of talk goes into other issues in questionnaire design – long surveys, question styles that can be taken on mobile devices, and so on.  These are all important issues to sort out, but what about the greater issue at stake here:  the one of amateurs taking on a business function that could lead to major business mistakes?   

I’m pained by what I see as the Survey Monkeyization of survey research. (And I say this as someone with friends – people I genuinely like and respect – who work there.)  Like any professional discipline, good questionnaire design takes training and practice.  And online research has made it easy for weak designs to infiltrate the industry, annoy respondents, decrease response rates, and lead to the wrong insights. It makes everyone skeptical of our trade. It devalues what we do.

It’s time we all take action to stop the madness.  Non-researchers - if you've made it this far, thank you for taking the time to understand that asking questions to a convenience sample of your friends or alumni list does not make you a professional researcher any more than blogging makes me a professional publicist.  Spread that word, and seek out the expertise of people who know the pros and cons of a rating vs. a ranking and how to handle challenges like gender response bias. Fellow researchers – speak up when you see these egregiously bad surveys; lend your insights to those who lack your experience.  And if you work for a panel, screen the surveys before you set them live – don’t just move forward with something that feels wrong in your gut.  In the economic game of market research, that lands us in that ugly quadrant where nobody wins.

-Kerry