Why IT surveys can’t be trusted for strategic decisions

Or, as they’re being surveyed by a prestigious analyst firm, they don’t want to admit they have no idea what the question means. Or, if they do, they’re embarrassed to admit that even though the analysts tell them that if they don’t follow this latest industry trend they’ll be out of business, following it just isn’t in the cards this year.

For the most part, survey value comes down to this: You think your company should be doing something. Someone’s survey associates a big bar with that subject. A big bar looks important. But really, using a survey to justify a course of action is little more than playing follow-the-leader.

Whom you measure matters more than what they say

Surveys also fail to reduce our uncertainty when they aren’t accompanied by an account of who responded to it — not only which companies or types of company, but also the specific job title or titles. After all, ask a CIO what they plan to spend on next year and compare it to what information technology the CEO or chief marketing officer plan to pay for and it’s far from guaranteed their responses will sync up.

Error bars offer little more than false precision

Yes, survey perpetrators are getting better about letting us know their survey’s sample size. But does anyone have the time and energy to use this information to compute error bars?

Even if you did, the thing about error bars is that — speaking of uncertainty and the reduction thereof — error bars have an interesting property: They reduce our uncertainty about how certain survey results are.

Error bars are a useful remedy for how so many surveyors indulge themselves in the sin of false precision. They might, that is, “inform” their audience that 53.7% of respondents say they’re doing something or other. This is a bit like the officials at a football game unpiling the stack of players who are all trying to shove the football in a favorable direction, placing the ball in what seems to be a fair spot, then bringing out the chains and measuring, with micrometer precision, whether the team on offense earned a first down.