It's a well known "fact" that 73% of all statistics are made up on the spot. ~Author Unknown…

Used wisely, research can be a powerful tool for understanding, whether that is understanding of a market, of people's preferences and affinities, or the kind of raw research that uncovers the laws of science.

But when the statistics that it generates are used without due diligence, as a tool to inflate an argument rather than to inform and educate, the research becomes something else. The Internet (particularly since Google turned search engines into a powerful automated indexing tool) make it easier than ever to put the cart before the horse and simply to find the numbers that back up a theory, rather than use information and knowledge to inform and build it.

He uses statistics as a drunken man uses lampposts - for support rather than for illumination. ~Andrew Lang

There's an interesting presentation that's been doing the rounds for a while called "Shift Happens." If you've not already seen it, then it's worth watching this video which is (I believe) the most up to date version of it in it's original format. (By "original format", I mean in the format by the person who first put the presentation together. It's not his first version, but there are several versions out there; some credit their influences better than others.)

It's definitely an interesting, entertaining and thought provoking video. The problem is that it isn't necessarily believable- not that this is a complaint about the presentation, as for it's original intended audience (a small and relatively private group working within a school), and as a thought-provoking piece designed to raise questions about the direction that the world of education is heading over the next few years and the challenges that will have to be addressed, it does the job excellently.

But it's precisely because it does such a good job that it's then been shared. Which is where some of the problems begin…

One problem is that at the time of writing this, it's about three years old— a fact that you probably wouldn't deduce from the video itself, as the references to "now" and "today" (especially given the context) makes it feel very current. It's a neat trick, and actually one often used by those chain emails that use social engineering tactics to prompt you to forward them to your friends and help them win free money/avoid being bombed/warn them about a computer virus that might wipe your mobile phone etc. etc. (If you find yourself about to forward one on, please check snopes.com first.)

For example, this video on Audience Science's website may look a little familiar in certain places.

(AudienceScience Website.)

Many of the "Shift happens" slides are in there— but no longer being presented as a thought-provoking, questioning exercise; in a different context, they have become forecasts and predictions for a completely different industry.

Any statistic without a source and a date is meaningless conjecture, and by reciting them in any kind of meeting or presentation, you are simply offering a golden opportunity to anyone who wants to make you look stupid by asking you a perfectly simple, innocent question; "where does that figure come from?"

A statistic with a source, but without a sample size is better, but still fairly meaningless; the same "simple, innocent" question becomes a slightly more pointed and barbed question— but still carries the power to transform anything you say from the wisdom of a learned expert into the ramblings of an ill-informed fool.

There are two kinds of statistics, the kind you look up and the kind you make up. ~Rex Stout, Death of a Doxy

Statistics may be defined as "a body of methods for making wise decisions in the face of uncertainty." ~W.A. Wallis

Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: "There are three kinds of lies: lies, damned lies, and statistics." Mark Twain, autobiography, 1904 (but there is no actual record of this under Disraeli's authorship)

Then there are the kind of second-hand statistics that do actually cite some kind of source, such as a Daily Mail story by Baroness Susan Greenfield, mentioning an Ofcom report which found that "99% of children and young people use the internet."

Ofcom is indeed an authoritative source for this kind of information, making it a very compelling argument. It's a shame that she doesn't say exactly which Ofcom study found this "fact." Reading these kinds of reports and understanding these particular numbers happens to be the kind of thing that I have to know about for a living, and 99% is a very high number indeed. So, given that Ofcom's Communications Market Report for 2008 found that boys aged 12-15 were the most likely to own or be able to access an internet connection out of boys and girls across three age groups (5-7. 8-11 and 12-15), yet only 81% of them had access to an internet connection, I personally find it very hard to put faith in her claim. And as a result, I find myself questioning all the other claims she makes (especially as in this instance, she is quoted as an authority figure as a neuroscientist, but without any kind of scientific evidence to back up her claims.)

Then there is the man who drowned crossing a stream with an average depth of six inches. ~W.I.E. Gates

The average human has one breast and one testicle. ~Des McHale

A man may have six meals one day and none the next, making an average of three meals per day, but that is not a good way to live. ~Louis D. Brandeis

Say you were standing with one foot in the oven and one foot in an ice bucket. According to the percentage people, you should be perfectly comfortable. ~Bobby Bragan, 1963

Sometimes, there are statistics that are cited without an understanding of what they mean.

There are a number of credible sources that will tell you that mobile phones have over 100% penetration in the UK. (I'm not citing sources here because I've honestly lost track of how many times I've heard or read this over the last year or two.)

What the research is saying is that there are more active mobile phones than people in the UK.

What the snappy soundbite is actually saying is that there are more people with mobile phones in the UK than there are people. The simple fact is that as long as there is a single person in the UK without a mobile phone (there are actually something like 8 million), there isn't 100% penetration- let alone "more than" 100%.

The problem is that giving someone a "bad statistic"— that is, a number without a source (or an unreliable source) is effectively handing them a loaded gun with which to shoot down your argument. Which is fine if the numbers are there to illustrate, rather than prove your point.

I could prove God statistically. Take the human body alone - the chances that all the functions of an individual would just happen is a statistical monstrosity. ~George Gallup

Torture numbers, and they'll confess to anything. ~Gregg Easterbrook

Then there are the "8 out of 10 cats prefer..." The kind of "studies" which, in the world of science, are clearly laughable, but in the world of marketing and advertising it becomes a slightly different story. Sure, it's no longer carrying the implications of life, death and billions of dollars worth of pharmaceutical patents and healthcare commissions, but for those who want to use market research for something other than generating a press release- for illumination rather than support- it makes it more difficult to find elements of illumination from within the mountains of "facts."

But perhaps the biggest problem is the kind of research that's only cited to tell a story— and not necessarily the story that the actual numbers are telling.

Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital. ~Aaron Levenstein

Do not put your faith in what statistics say until you have carefully considered what they do not say. ~William W. Watt

When a study is conducted to look at a specific audience or activity, it isn't unusual for a selected sample of people who aren't representative of the greater population to be used. For example, when looking at advertisers opinions about the future of the industry, Advertising Age conducted a study asking 100 advertising executives about Google's position in the Internet advertising market. But by the time the numbers found their way to the Phorm newsletter, the context had been dropped, and it was now saying that […]65 per cent of people said that Google dominates the online ad market and that there are ‘limited choices and price options for online advertisers.’ Some 85 per cent said it's important that a strong competitor emerges to challenge Google's dominance.

Not 65% of advertisers— who happen to be directly threatened by Google's dominance of the rapidly growing online ad market— but 65% of "people." Which tells quite a different story (especially in the context of what amounts to a press release from a company claiming to redistribute the wealth of Google's advertising to even the smallest of website owners— while turning a profit for the ISPs to boot.)

So, my call to anyone who finds themselves copy & pasting a percentage into a document, presentation or email is to consider whether you have the answers to the following questions;

Where did the numbers come from? An independent study, or a business with a vested interest in what they are reporting?

How were the numbers arrived at? Focus groups, one-on-one interviews, or self-completion questionnaire? Are they claimed or observed behaviour? Are they asking what people remember doing, or asking people to monitor what they do and report back? All these factors make a massive difference to the way people's behaviour will be reported.

What is the base? Not just how many people were questioned, but who were they- all ages, or over 15? If they were nationally representative, by what criteria were they weighted or filtered? (In particular, was the questionnaire performed online— as most research is now performed, due to the massively reduced costs involved? If so, then a fairly large proportion of the population will have been filtered out right there.)

Finally, make sure that your numbers are there for a reason. Whatever the story that you are trying to tell is, it should stand up without them; statistics should be there as the icing on the cake. Not a crutch that your story is leaning on.