Good idea, Bad Execution.Where's the Research?

Research is often about finding ways to maximize ROI. Today's tale addresses the opposite: research as a mechanism for preventing the wrong decision.

You may have heard the hubbub surrounding the new UC logo, designed to represent the multi-campus collective UC system.

When NPR, The Atlantic, and Christian Science Monitor all cover the release of a logo - that’s right, a logo - it’s either brilliant or a disaster. And if you're reading the media coverage, this seems the sort of train wreck that makes you wonder if the design team grew cannibas in their office and baked it into free team lunches.

Here is the logo. Take a moment to distill your impression of what it communicates, then read on.

Spoiler alert: The above image is supposed to represent a book (top half) combined with a U (University), with an ever-evolving and flexible C for California. Did you get that out of it? Me neither. I thought maybe the C was fading under its colossal state debt, or sinking into the ocean after an earthquake. I really want to finish coloring in the C.

In the interest of fair disclosure, I am a UCLA alum and a masters swimmer.  So when I first saw this logo, my immediate reaction was, “Why is the logo an image of California drowning in a wave pool? And why is it in UCLA colors? Cal alums must be pissed.”  It turns out, the logo is flexible to varying shades of blue and yellow. And the C is neither fading nor drowning (in water or debt), but rather “evolving.” Copy that.

It turns out I was not the only one who saw the logo and thought it was some combination of awful and perplexing. Someone hated it so much they started a Change.org petition to get rid of it. Within 48 hours, the petition’s garnered more than 52,000 signatures, and a healthy dose of comments from alumni wondering what on earth the UC Board of Regents was thinking. Consensus seems to be that not only is this logo not particularly dignified (an expectation of a highly regarded university system), it looks like child’s play. Literally. Nobody seems to mind the idea of a branding exercise for the whole UC system. Everyone seems to mind that what they came up with looks like playground art, or a Toys R Us ad.

It also turns out: designers hate it too.  A design site gave an oddly favorable review to this design execution, which was then met with amusingly unfavorable feedback from the design community.

http://www.underconsideration.com/brandnew/archives/ic_uc_we_all_c_for_california.php

Among my favorite responses:

•          “I didn't know that University of California was a children's television network.”

•          “I thought at first it was a rebranding for a candy store: cute and cuddly colors and shapes. Not the kind of image I would want to be portrayed if I were an institute of higher learning.”

•          “I suppose any institution could use a little "approachable" brand image, but try not to do it without losing any sense authority and/or respect. Maybe start by avoiding rounded font for your symbol?”

•          “You're kidding right? Seriously. This is a joke right? It's a whale tale [sic] with a bicycle tire. It's goofy as hell.”

To be fair, there are executions of this logo that I don’t hate. For example, it looks kind of okay if it's really small and not multi-color.

But then they had to go and turn the UC system into what your typical 6 year old will likely mistake for an ice cream truck:

Snow cone anyone?

Some may defend this entire exercise by pointing out that there’s a great story behind all of this creativity. There’s even a video on the above-linked review, explaining the root of the brand exercise. (Note: The logo made more sense once I watched the video. Unfortunately, if you need a video to explain your logo, the logo isn’t working.) Vanessa Kanan Correa, Creative Director at the University of California, comments on the branding objectives:

“Evoking California — and its innovative spirit — was a critical driving factor in the aesthetics across all the visual elements. We felt that what makes the University of California special is that, well, we’re in California. And what happens here impacts the world. So, we built something that was optimistic, experimental, pioneering, and bold… qualities that reflect California itself.”

I don’t disagree that California is innovative, even special. But while our state is certainly a hub for technology, media and entertainment, let’s not let our egos get the best of us. People change the world in other states. Find me a New Yorker who thinks Californians are more bold than New Yorkers, and I will give you a dollar.

Still, I like the idea that a logo can be “optimistic, pioneering, and bold.”  It's just that nobody seems to be taking any of that away from this logo.

Which brings me to the real question at stake here: What on earth was the UC staff thinking, and how did they come up with a logo that has been met with reactions that range from neutral to “I’ll start giving money again when they get rid of this nonsense”? Jason Simon, the University of California Director, Marketing Communications, wrote the following message to Change.org, implying that “testing” was actually done on this logo.

"The new mark was created as a part of our broader efforts to build awareness and support for all the things that UC does to make California (and by extension the world) better. What we have tried to do is to create a mark that is iconic, flexible, and solid enough that it works to represent the UC system as a whole. The mark can be used in a combination of the various UC blues and golds as well as in a multitude of applications. Seals are wonderful and carry a legacy and tradition. They also signify bureaucracy, staidness, and other not-so-great characteristics. Much of this was evident in the testing and discussion we did as part of the process."

I get the philosophy; I daresay I even like it. I also get that there is close to zero chance this logo passed through any wave of consumer testing.

Creative comrades, “testing” is not a discussion among your design team. It’s not even a “which do you like better” scenario that you bounce around the office, or throw out to your personal mailing list. “Testing” requires understanding the implications of every reaction. What do the colors evoke? What does the font or the style convey?  How do imagery impressions align with expectations of the UC system? What are the opportunities for integrating the imagery; what are the limitations? Who are the naysayers, and do they matter significantly to the UC system?

It seems apparent that most of these questions were left unanswered. If they’re saying that testing was done, then it was really subpar testing. But what I think is that they probably tested their ideas, and never actually tested their executions of those ideas.  I often tell clients, I don’t believe in concept testing unless you a) have something to show people, or b) test the ideas, develop something, and then test the executions. Otherwise it’s just ideation, and that’s not the same thing.

This logo did get one thing quite right: blue and yellow do evoke sky and sun – very California. Alas, it got most things very wrong: bubbly font doesn’t convey university, and a book that looks like a whale tail or a wave is easily confused with oceans instead of education. Not to mention, in this modern world of Kindles and iPads, choosing the symbol of a book to represent brand pillars such as “experimental” and “pioneering” is a questionable strategy. One could argue that this logo aligns with SoCal, but what about Silicon Valley, or that part of Mojave where they launch rocket ships, or that neighborhood south of LAX where they build them?

Real ad testing, when done correctly, doesn’t just tell you what to do. It tells you what not to do.  The reality is, had UC come up with something lackluster, nobody would have noticed. It’s because they came up with something bad that everyone did. Let this be a lesson. When you’re prepared to budget an 11 person design team for an initiative, allegedly commissioned over a 3 year period, you’re recklessly wasting money if you haven’t budgeted for research. Really solid, thorough, multi-phase research. 

Research isn’t just a tool for identifying how to make money, it’s a tool for preventing costly and public mistakes. I’ll wager a guess that the head UC publicist, not to mention every development director at every UC university, is currently wishing someone had set aside that investment for this endeavor.

-Kerry

ETA: As of 12/14/12, use of this UC monogram has been suspended. Let us hope that the upcoming "re-evaluation" is more thoroughly vetted this time around. 
http://blogs.kqed.org/newsfix/2012/12/14/uc-suspends-use-of-new-logo/

The Exclusivity of Fact

As the campaign trail draws to a close and the election nears, it’s become common knowledge that today’s political candidates fudged more than a few details in their speeches and debates.  (I won’t call out a particular candidate, as even if one is more egregious than another, both sides are objectively guilty of such behavior.)

Let’s not pretend that this year’s political candidates are the first to jump on the Hyperbole Train, or even the Fiction Express. Their behavior nonetheless begs the question:  In this age of transparency, why do politicians still lie right to the camera, when there is a near certainty they’ll be caught?

A look at the changing landscape of news sheds some light. Maybe you missed the memo, but breaking news is owned by the web and mobile these days, amplified further by social platforms.  And with that comes a marked decrease in turnaround time. Breaking news used to be “a day later.” Then it became “later that day.” Then “later that hour.” These days, it’s right now. If you take the time to check facts and don’t report the story immediately, someone else will report it first. Worse, that someone may or may not be a legitimate journalist.  We can call it sexy names like citizen journalism and crowdsourcing, but that doesn’t change the practical reality that amateurs now compete with the professional journalists. The very existence of digital platforms, and the reporting immediacy that ensues, challenges a previously presumed component of journalistic integrity: truth.

In other words, politicians lie because they can. Our world may be transparent, but for every fact checker who points out a fallacy, there are dozens, maybe even thousands, of amateurs who already tweeted their support of that fallacy hours earlier.

Compounding this is the changing role of TV news. With digital platforms taking over the breaking news role, TV has become a platform for depth and feature reporting. Some of the blame for today’s entertainment-driven sensationalist approach to TV news arguably goes to Fox News, the first of the TV nets to really good at entertainment and really blasé about accuracy. But they’re not alone. The Daily Show, not intentionally setting out to be a news destination, got really good at being one anyway. MSNBC then followed. And eventually, infotainment became the way of most television news. Personality began to trump professionalism; entertainment trumped accuracy.

Of course it’s not really “ok” that fact checking has become a separate discipline, rather than part of the fabric of news reporting and campaign politicking. On the consumer front, we’re tired of having to look up the facts. Time is a scarcity in life, sometimes our biggest one. The more that news organizations skip fact checking, and the more that politicians hyperbolize the truth, the more we will deprioritize them in frustration.

Of course, as a researcher, what perhaps confounds me most is that the pollsters haven’t curtailed this. Political researchers further the partisan divide and sweep the real issue under the rug, by asking about abortion, gay marriage, the economy, and healthcare. Maybe it’s time to ask voters this: at what point will you become so tired of the exaggerations and fact-shrouding, that you disengage from politics altogether?  

There’s nothing quite like a universal crisis to unify a country, and there’s no crisis quite like a public that is so fed up, they stop caring. Voter turnout is projected to be lower this year than in the past two Presidential elections. For the benefit of society, and the future of government, I hope those projections are wrong. But if they're right, may it be an eye opening call to action to the media and politicians alike that truth remains as critical as ever.

What's In A Name

Well my fair blogosphere friends, tomorrow marks the first anniversary of Research Narrative. And to celebrate, this blog is going to be a little longer than, and little different from, the others.  Like many of the media clients who’ve helped me turn Research Narrative from wishful thinking to rewarding reality, I’ll begin with a story.

Recently, I spoke on a panel at Digital Hollywood, attempting to give context to the array of prognostications that take place at such a conference. Somewhere in between my overview of platform adoption and a recap of digital vs. traditional advertising economics, a content creator on the panel leaned into his microphone and interrupted me. “Can I just say something?” he asked rhetorically. “None of this matters if the content isn’t strong.”

Although I think he expected me to engage in a verbal battle of commerce vs. creation, I took a tactic that caught him by surprise. I agreed. That’s not to say that platform and advertising don’t matter -- producing to a platform that has limited ROI or reaches the wrong audience can kill great content and waste millions of dollars. But I absolutely concur with my fellow panelist’s broader point, which was that content, business, and technology have to coexist and collaborate.

His comment reminded me that I spend much of my time talking about the research side of media research, so I want to take some time to talk about the content side of it.

As the above encounter suggests, media research is sometimes mistaken for the robotic dream crushing of content creators. At the risk of brazenly representing my fellow media research comrades, I don’t think any of us got into this line of work to crush dreams and commoditize storytelling. We got into it because we believe that when great strategic thinking is combined with great creative thinking, magic can happen.

When I think about the questions I set out to answer as a researcher and consultant, they are often fundamentally about content.  One that came up repeatedly at Digital Hollywood was, “How can a single creative concept exist on many platforms?” What does the digital version look like, versus the traditional (e.g. TV, radio, film) version?  As an amateur dancer, I love to point people to The LXD. What started out as a small online video became an online series, and is now a full-on dance company performing in commercials, on the Oscars, at TED, and at fundraisers. Given that the creator is Jon Chu, I imagine it’s a matter of time before it becomes a full length superhero film. I also love using TMZ as a great multiplatform example. The TV show is built to catch your attention and make you laugh. The website does too, but if they didn’t break stories first and update with the urgency of a nuclear war, the site would probably be a flop.

Another question I love addressing is, “How can consumer adoption of new platforms create opportunities for new types of content, attracting new audiences?” I turn inward to myself for an example of this: I prefer music radio, I almost never listen to talk or news radio. But I like public radio’s coverage of news. I like the questions they ask, the interesting topics they address, the industry experts they interview. These days, with Facebook and Twitter, mobile apps and transcripts, I don’t have to change my radio behaviors to consume public radio. Instead, I read it. I am, by definition, the growth opportunity that people vaguely speak about.

Then there are the creative questions. One that often rings familiar to TV producers and writers is what to do when a cast change is impending. “What personalities or characters are driving a series, and why? What happens if one of those personalities/actors leaves the show for a new endeavor?” How can it evolve? Can it evolve?  Some shows can move past a major cast change, and some shows can’t.  I’ve seen research projects funded entirely on the back of getting ammunition to fire on-air talent, only to discover that the talent under question was too central to the program the creators had fashioned.  If the talent goes away, the entire premise of the show goes away.

Which brings to mind another common content question that I see a lot in television.  “What’s the fundamental promise of a show? How are we executing on that, and where can we better deliver upon that promise?” Just this week I worked on a series that viewers deemed “mindless.” They meant that as a positive. Sometimes a TV show works simply because it’s a fun, simple escape from the daily grind. The producers went into the evening concerned that they might have to tackle the budgetary consideration of additional set time, in order to infuse more “meat” into the series. They came out of it realizing that all viewers wanted was to laugh out loud after a long day, and that pretty much any modification necessary could be done with a few post-production edits.

Many content creators also strive to infuse relatable elements into their creations – a character, a scene, a line of dialogue, etc.  A question I often ask them to consider is, “What elements are relatable to audiences, and when might we be getting just a little too close to home for “entertainment”?”  I recently made the following case to a colleague: it’s one thing for a character to get laid off in Season 3, struggle for a couple episodes, and then find a new job. It’s quite another to build a show or movie around the premise of a character losing his/her job. There’s relatable, and there’s deflating. Research can help to define that line.

I’ve been talking a lot about entertainment content, but all of this applies to news content as well. Ask yourself this:  “What makes a news story compelling?”’ One of my favorite talking points at cocktail hours is the following: a good interviewer question is the most salient trigger of “intelligent coverage” that is recognized by news consumers. That nugget of wisdom came from an online survey. I encourage you now to pay attention to how often you cringe at “stupid questions” asked by interviewers, and how much more interested you become when an interviewer asks exactly what you were wondering.

Indeed, the underlying questions in media research require as much appreciation for the creative process as they do skill in data analysis and distilling business insight. If my fellow conference panelist had offered me the opportunity, I would have pointed out that I’ve been an artist for twice as long as I’ve been a researcher.  To know me personally is to know that I play several musical instruments and was, at earlier stages in life, the conductor my college pep band and editor of my graduate school paper. I spend several hours a week at a dance studio, and serve on the school board at an arts academy. When I talk about casting, storytelling, editing, character development, or the mood created by a music score, I’m not just drawing on professional experience, I’m drawing on life experience.  Research and creativity are not mutually exclusive, but rather, a potent combination.

I often say that my career lies at the intersection of commerce and creativity.  My trade is research. But my trade is also storytelling, whether it be a comedic story, dramatic story, real life story, news story, or research story.  And ladies and gentlemen, to bring this all full circle, that is why my business is called Research Narrative.  

Thank you everyone, from the bottom of my heart, for making that vision not just a reality, but a success.

We’re Actively Media Multi-Tasking, But Are We Good At It?

It’s a familiar scene. I’m sitting on my couch, smartphone in hand, laptop by my side, a 50 inch Samsung flatscreen TV catching me up on a backlog of DVR recordings. An email pops up on my phone, and I turn to the laptop to respond – it’s faster to type that way. Then while I’m there, I check Facebook. Oh look, an interesting article. I read it. It links me to a video. Instinctively, I grab the remote control and pause the DVR so I can hear the video on my laptop. Which inevitably leads me to another video, and another. And suddenly an hour has flown by, and I’ve only watched 10 minutes of that show on my DVR, and I don’t remember even a minute of it. Rewind, restart, replay.

Media-multitasking has been a topic of discussion among media professionals for years. TV in particular has long been a companion tool – one that co-exists not only with other media, but with activities ranging from eating to cleaning to parenting. I broach the subject of media-multitasking not because it's new, but because it's increasingly relevant for programmers and marketers alike. My own in-home ethnography is not universally projectable, but it does illustrate the very human limitation of being able to digest only so much information at once.

Earlier this month, The Hollywood Reporter referenced a recent Harris Interactive research study in which 63 percent of adults claimed that they look at online content while watching TV.  (http://www.hollywoodreporter.com/news/digitas-research-tv-viewers-online-celebrities-upfrontsl-308202).  The article posited that, “Overall, the nation seems increasingly comfortable with doing several things at once.

In fact, the reality is that while we may be comfortable with media-multitasking, academic research suggests that we might not actually be good at it.

A 2011 study conducted by researchers at Boston College found that when consumers watch TV in front of their computer, they change focus from one device to the other roughly every 14 seconds.  That’s approximately 240 fixation changes every hour. It seems that when given simultaneous media choices, we may generally lack the ability to focus on just one for any extended period of time: the study reported that the average visual gaze lasted 5.3 seconds on the computer and just 1.8 seconds on the TV. Only 7.5% of all computer gazes and 2.9% of all glances at the TV lasted longer than a minute.

The results might bear particular concern to TV programmers, who rely on viewers to remain loyal and engaged week after week (or day after day, in the case of syndicated series and news): 68% of participants’ time was focused on the computer, with just 31% of the time focused on the TV.

Perhaps the most interesting part of the study is that participants believed they were far more attentive than they actually were. On average, they estimated that they changed their visual fixation about 15 times in half an hour. In reality, that number was closer to 120.  We think our attention spans can last a couple minutes, but when we start media-multitasking, we’re far more inclined to stay focused for only a few seconds. (Read the whitepaper here: http://online.liebertpub.com/doi/pdfplus/10.1089/cyber.2010.0350)

So while it’s true that media-multitasking is escalating and can indeed be used to drive cross-platform engagement, it’s also true that this escalating behavior creates distraction. Even if the ratings indicate that audiences are still with you, their attention and loyalty might very well be gravitating somewhere else. Consumers might be comfortable with that, but programmers and advertisers shouldn’t be.

The Survey Research Prisoner's Dilemma: Who's Going to Put the Brakes on Bad Survey Design?

Not too long ago, a Survey Monkey link came up in my Facebook feed.  As you may have deduced, I run a company that (among other things) specializes in survey research, so I clicked the link out of curiosity. Here’s what I saw on the splash page:

By all practical indications, this survey is not intended for me.  It’s targeting community organizers and neighborhood council members, people who volunteer their time for the city’s neighborhoods, and who probably know a thing or two about what gets prioritized, and what doesn’t.

I’m not one of those people.

I read the intro and instructions out of curiosity, expecting to get screened out. But I wasn’t screened out.  Instead, I began to receive questions that I’m uniquely unqualified to answer. Questions that are setting out to prioritize the budget of Los Angeles, the 2^nd largest city in the United States.

This survey underscores a giant flaw in the direction in which survey research is heading: everyone is a survey writer these days.  A few choice shortcomings of this particular survey:

·         There’s no screener.  Anyone can take the survey.  Including me.  Or, more amusingly, someone who does not actually live in the city of Los Angeles.  It doesn’t know I live in LA, because right now, at this very moment, I’m in Ohio. And my IP address will indicate that.

·         It’s not gated.  I took the whole survey. And then I was able to take it again.  Remember now, the topic is the city budget.  I can think of a few stakeholders who’d want to weigh in on that more than once.  Ballot box stuffing, anyone?

·         The survey is unprotected.  Most people are probably too lazy to take this whole thing, but that doesn’t matter – they can still aimlessly click through and screen cap it.  For a political office, this survey represents a potential PR nightmare.  (To which I am actively contributing, guilty as charged.)  My fair research colleagues -- this is how your weakly worded questions make the news.  I’ve had journalists screen capture and post my surveys in their columns before (such is the trade hazard of conducting web intercept surveys on news websites).  And that was stressful even when the questions were well-written, and the journalist was reasonably kind.

It’s tempting to use these self-serve tools to save money; I myself have gone that route when a “real” budget was not in the cards. But when we get to the point that major financial decisions are made based on research conducted by non-researchers using highly flawed methodologies – we need to start taking action to stop it.  Weak research from a select few gives our entire industry a bad reputation.  It causes million dollar mistakes.  And that’s a much bigger deficit than what it would have cost to conduct this survey the right way, with the right professional discipline.

The mayor’s office survey is hardly an isolated incident.  The Grey Matter online research report I mentioned in my last blog post alludes to questionnaire design issues with a few amusing anecdotes that it found in its research.  Among my favorites were the hilariously broad question, “What characters, movies, or brands do you like?”, the grammar cringe worthy, “Of LCD TV, that you currently own, what type does you mainly watch?”, and a straight guy being asked how he dresses his husband.

A lot of talk goes into other issues in questionnaire design – long surveys, question styles that can be taken on mobile devices, and so on.  These are all important issues to sort out, but what about the greater issue at stake here:  the one of amateurs taking on a business function that could lead to major business mistakes?   

I’m pained by what I see as the Survey Monkeyization of survey research. (And I say this as someone with friends – people I genuinely like and respect – who work there.)  Like any professional discipline, good questionnaire design takes training and practice.  And online research has made it easy for weak designs to infiltrate the industry, annoy respondents, decrease response rates, and lead to the wrong insights. It makes everyone skeptical of our trade. It devalues what we do.

It’s time we all take action to stop the madness.  Non-researchers - if you've made it this far, thank you for taking the time to understand that asking questions to a convenience sample of your friends or alumni list does not make you a professional researcher any more than blogging makes me a professional publicist.  Spread that word, and seek out the expertise of people who know the pros and cons of a rating vs. a ranking and how to handle challenges like gender response bias. Fellow researchers – speak up when you see these egregiously bad surveys; lend your insights to those who lack your experience.  And if you work for a panel, screen the surveys before you set them live – don’t just move forward with something that feels wrong in your gut.  In the economic game of market research, that lands us in that ugly quadrant where nobody wins.

-Kerry

10 Steps to Managing Online Panels

Grey Matter Research recently released a report entitled, “More Dirty Little Secrets of Online Panels,” where some clever industrial espionage revealed a range of inconsistencies and questionable business practices on the part of online panels.  (Email me if you want a copy: kerry@researchnarrative.com).  Among the key findings:

·       Panels regularly subcontract sample to other panels, and don’t tell clients.

·       The frequency cap for survey invitations seems to be increasing – panels now typically invite their panelists at least weekly, and sometimes multiple times per day, to take surveys.

·       Respondents can become trained to understand questionnaire scales, and their reaction change as a result of taking surveys in the past.

·       Panels willingly field terribly designed surveys.

·       It’s easy to join upwards of a dozen panels.

I was fortunate to be in on the ground floor of online research back in the 90s, and to that end, I’m a bit jaded – nothing in the report was really news to me.  Many of these talking points have been around for at least a decade, but there are far more online research players today – many with dubious business practices. It takes a lot more work to monitor it now, and the responsibility lies with all of us. I educate my vendors (online sample providers or otherwise) all the time, and I expect them to do the same for me.   

It’s easy to think of sample as a commodity, but it’s not.  Panels have reputations, strengths and weaknesses, and built in biases that can work to your benefit or detriment, depending on the study. Here are some of the strategies I’ve used for minimizing the negative issues outlined in the report.

1.      Dig beyond the bid.  Ask questions of panel partners. Find out how they recruit, how they frequency cap, what their invitations look like, etc.

2.      Join their panels. Be a spy.  Make sure they practice what they preach.  Call them out on it when they don’t. 

3.      Tell them upfront that they can’t outsource/subcontract, or must get approval from you first. Don’t assume that quality panels don’t do this. Most of them do – out of necessity.

4.     Bring in multiple sample partners on a study yourself.  Tell them you’re doing it, and who you’re using.  Then compare them.  It puts them on good behavior and gives you insight on how each panel operates. 

5.     Pretest/soft launch every study.  Check data.  Look for inconsistencies.  Look for places with heavy dropout.  Adjust course accordingly before full fielding.  We’re in the business of market research, not emergency medicine, the extra day this takes will be immaterial 99% of the time.

6.     Be wary if they don’t ask you questions.  Good panels will have AMs who take your surveys, give you feedback, provide insight into incentive necessity, and generally demand that you don’t abuse their panelists.  Bad panels will say yes to whatever ridiculous request you have, usually because they’re desperate for business and/or have inexperienced employees who don’t know better.  It’s a good litmus test to know if you should turn and run.

7.     Don’t just go with the cheapest sample bidder.  There’s usually a reason they’re cheaper, and it’s probably not one you’ll want to deal with on the backend.

8.     Every now and then, put a question on the survey asking what panels people belong to.  This gives you an idea of where there’s overlap, and how much the heavy panel subscribers differ from people on only 1 or 2 panels. (I’ve seen this matter a lot, and I’ve seen it matter not at all.)

9.     Make sure you and they can set and track quotas.  Monitor it.  This avoids the dubious statistical manipulations that try to project 10 people to 100. 

10.  Add past participation screeners on surveys where you fear a recent participation effect.  This isn’t always relevant, but as pointed out in this report, it matters for normative testing. Well-designed panels will also be able to target their invitations to only reach people who have not recently participated in (your survey/a similar survey).  That doesn’t mean respondents haven’t taken it through another panel, but it’s an extra line of defense.

The world of the RDD gold standard is long gone.  But online panels can provide brilliant, thoughtful, and valuable insights if you manage the research correctly. 

The Latest Kodak Moment

A fellow member of LinkedIn’s Next Gen Market Research community recently posed the question, “With Kodak blowing up, I'm thinking about why a company with such a strong tradition for MR failed so miserably? What responsibility do researchers have for influencing the business?”

Several people gave thoughtful responses regarding the role of research at Kodak and in large organizations in general.  Yet all were rooted in the underlying assumption that if research identifies a digital opportunity, then digital must be the solution.  Case closed.  To this native Rochesterian, that myopic viewpoint ignores the nuance of conducting business in my hometown – and the critical factors that one must acknowledge in order to understand what really happened over at Kodak.

I am a vocal advocate for (and creator of) business-influential research; I would love it to be the case that strong market research could have prevented Kodak’s current woes.  And I am not ruling out the possibility that somewhere in the walls of Kodak Park, there were misguided or misused research reports, researchers with inadequate influence, or executives with a rampant distrust of research they needed to heed. 

The thing is, it didn’t take market research to see what was happening in the photo/imaging space 10-15 years ago. It took a trip to Best Buy. Kodak wasn’t ignorant to digital cameras, they invented them.  In 1975.  (If you don’t believe me, look it up on Wikipedia.)

The scenario at Kodak runs deeper than market projections and customer satisfaction.  And it’s incumbent upon us as researchers to understand all of the business dynamics, not just the consumer-related ones.  When I think about the sustainability of Kodak, a few points of consideration immediately come to mind:

1) A full scale change in a company’s entire business model requires visionary and powerful leadership.  And the courage to take the bullets when you shred apart a dying but familiar cash cow operation, lay-off thousands of hard working employees, and make the other thousands of loyal employees unhappy by forcing change.  Not too many CEOs want to be the person to spawn that blood bath, and then clean it up.

2) Stockholders think short term. Film had a huge profit margin, and Kodak had nearly universal market share in the US.  Digital has minimal profit margins, and some big competitors that make Kodak a more minor player.  I’m going to wager an educated guess that stockholders were not lining up telling Kodak to make a change that would render the company unprofitable for a few years, in order to be sustainable in the long run.  That is not typically the way the stock market works with companies that have been around for 100 years.

3) Kodak employed tens of thousands of manufacturing workers in the US.  I have to imagine that there was some political pressure to save their jobs.

4) Rochester itself is arguably in a talent pool crisis, as alternative employment opportunities are limited. I went to one of the best public high schools in the country, 20 minutes from Kodak HQ.  Dozens of my friends went off to Harvard, Michigan, Dartmouth, Penn, Cornell, Stanford... and almost none of that intellectual firepower came back to Rochester.  The Wall Street Journal makes a very convincing argument that Kodak was never going to turn around and innovate in Rochester, and as much as I have a soft spot for my birthplace, I agree.  (You can find the article here:  http://online.wsj.com/article/SB10001424052970204124204577153053662634584.html?mod=googlenews_wsj)

Now put yourself back in Kodak's shoes:  a) needing to entirely reinvent a 100+ year old company, b) either moving locations altogether or staying in a market that struggles to attract talent and has to compete with the likes of Silicon Valley, and c) knowing that any change means eliminating of tens of thousands of local jobs and training the remaining tens of thousands on totally new skills.

In hindsight, it seems clear that Kodak needed to run with digital technology when its engineers first invented it.  But in practicality, it’s not too difficult to see why they didn’t. 

So what responsibility do researchers have for influencing the business?  As researchers, we can and should strive to be heard, and to influence business decisions – otherwise our function lacks purpose.  But if executive management is to trust us and consider us part of their team, we also need to understand the full spectrum of what it is we’re influencing.  In influencing the long term success of a company, we might also be influencing the short term loss of jobs, the deteriorating vitality of a city, and the declining value of stockholder portfolios.  With the responsibility of success comes the responsibility of failure, and the humility to understand that external powers of influence will sometimes win out in spite of our efforts.