Did this survey make you think less of us?
The Magnetic Group - Ziggy | July 13, 2010 Not enough companies understand this simple fact: Surveys are another form of customer communication.
Bad surveys demonstrate poor communication skills and create negative impressions for the parent brand.
So I’m always stunned when I see large, well-known companies field shoddy, poorly crafted surveys that besmirch their image.
Recently, The Weather Channel used an online survey tool to conduct some web research. But survey tools are like any other tool – its success depends a little on the tool, but mostly on the skill of the person using it.
In this case, the person using the tool needed more skill.
One of the early questions on the survey asks the respondent to rate several vacation spots. (See pic below.) However, the final vacation spot listed is, “None of these,” which makes no sense given how this question is formatted.
Okay, just a little oversight. Happens to everyone. But surely the survey won’t make the respondent actually provide a rating for “None of these,” right? I mean, surely someone tested the survey before it hit the field, right?
Wrong. Note the error message below (in red) when I tried to advance without stating my opinion on “None of these.”
And this wasn’t an isolated incident. Other questions demonstrate this same construction error (see pic below).
Finally, the survey also made the unfortunate mistakes of (1) being far too long and (2) punishing respondents for telling the truth.
For every vacation spot in which you expressed any interest, you were “rewarded” by having to fill out this nightmarishly long series of questions (see below).
If you’re like me, you look at all of those options and immediately think:
“No way. I volunteered for this to help you out, I told you that a few of those vacation destinations sounded appealing, and now you’re going to punish me by making me read and answer all of this for every destination?!?!? No way !!!”
I’m sure I wasn’t the only one who just blew through the rest of the survey to get it over with, making the entire experience a waste of everyone’s time and The Weather Channel’s money.
In the end, every person who took this survey ended up thinking that the administrator of the survey was incompetent and unrealistic about their expectations of our time.
And those impressions reflect on The Weather Channel’s overall brand, whether (couldn’t resist) they like it or not.
Reader Comments (4)
The problem with online survey tools such as Survey Monkey etc is, while they're quite cleverly designed, they put the art of survey writing in the hands of non-professionals. Clearly somebody at the Weather Channel doesn't know the difference between rows and columns - and doesn't understand the consequences when they route, skip or branch questions.
As a professional researcher I'm in two minds about easily accessible survey tools. I rather like the idea that the art of listening should be open to everyone - and if you can write a great survey, then great - why pay for somebody like me to do it for you? But the fact is, most people cannot write great surveys - and don't understand the psychology of how people answer surveys.
Thanks for posting your article. One thing I do with online surveys is post my contact details and email link. Oh yes, I get whacked for this and that - but every criticism helps me hone my craft. In balance I get more thank you emails than I get negatives. But a shame that somebody at Weather Channel didn't put their name and face to the questionnaire. They might have got immediate feedback, fixed their remedial problems and relaunched with much greater success.
Great thoughts, Duncan. I agree with you; online survey tools are great, but in many ways they put tools in the hands of people lacking the skill to use them. Sort of like giving me a circular saw and some wood...if you think I'm going to be able to make furniture, you're crazy.
More likely, I'm going to lose a hand. Be well!
-Z
Amen!
Although it is disheartening to see that the Web is rife with downright shoddy surveys, it is even more so to see them from reputable brands. As you so graciously allow, this might have been written by someone with little or no questionnaire writing training or experience who was simply told to "go do a survey."
The downside, of course, is data and information that is simply not credible.
As I have learned and observed, no data or information is better than bad data or information.
The professional market research community understands this for the most part, although there is a growing faction that simply dismisses the fundamentals of statistical probability and is forging ahead with "market research" tools and processes that are unrecognizable within the standards of fundamental solid research.
However, the issue is not really theirs; it is ours. How do we as an industry explain why the fundamentals are so critical?
Hopefully, we are groping toward an answer.
Thankfully, there are still many clients who do understand and value the fundamentals.
And thanks to you for pointing out the issue.
Regards, Dr. Bob
Dr. Bob,
Very much appreciate your determination to educate businesses on the importance of better research. As you say, it's an issue that all of us are facing and must confront together.
In my experience, the business-types often pulling the budgetary strings in these matters believe the opposite of you; they think that ANY data is better than no data.
This leads to absurd decisions by the poor marketing research staff working under them.
And to me, that's the real challenge...influencing the power brokers who know nothing about, and don't care to know anything about marketing research. Because I know that they have people internally who want things done well and embrace the arguments that you, and Duncan, and I are making. But their directors just won't listen to them.
So I think we have some allies in that regard, which is a start.
Be well!
-Z