, , Drinking Red Bull boosts job performance
Math-myopia and Prince Harry: do these numbers from Red Bull about performance enhancements make sense?

Will drinking Red Bull and smoking cigars boost our productivity at work?
Will sleep deprivation increase the number of mistakes we make?

This post addresses these questions, as well as how math-myopia affects love for metrics and statistics about sports, dieting, work injuries and so forth.

This blog entry is part of our series on business analytics and big data

If you read German, check out our series on political campaigning and the usefulness of polling (US presidential election).

[su_box title=”Table 1: Keep the numeracy problem in mind” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1.1 Making a good guess: Avoid base rate neglect

When we combine two pieces of information, we tend to ignore one of them completely. This phenomenon is called “base rate neglect”. For example, the base rate tells us how many people are affected by bowel cancer (6 out of 100), or how many have a fatal injury at work (i.e. 142 deaths per year in the UK – a rate of 0.46 deaths per 100,000 workers).

Knowing the base rate helps put things in perspective.

1.2 Our culture makes it acceptable to say, “I do not do numbers.”

Imagine a study that presents a test that is 75 percent accurate. In 25 percent of the cases where the test predicts a self-reported injury at work will happen, it does not. This is called a false positive.

What is the chance that the person has a work-related accident? Intuitively we might say that in 75 percent of cases a fatal accident will occur. However, the correct answer is, ‘we do not know’ – unless we have the base rate.

To illustrate, if we test 100 people and 4 come out positive, 3 were rightfully identified to likely have a work-related accident and 1 was wrongfully identified. But wait! Of the 96 others, 24 (= 25 percent) will have a false positive. This means we predict a non-fatal work injury, but they will not have one.


Unfortunately, in a culture where Prince Harry can publicly state that he may not have the math skills to be an air ambulance helicopter pilot, he is likely to ignore the base rate…

The base rate is a good way to start if we want to forecast something or put test results in perspective (see Tables 2 and 3 below).

Below we illustrate this a bit more with an example based on a May 2015 Financial Times article, which nicely illustrates how things can be misconstrued by journalists.

To reduce this risk, we must go to the trouble and check the numbers.

Financial Times gets the ball rolling

[su_custom_gallery source=”media: 1986″ limit=”7″ link=”image” target=”blank” width=”531px” height=”130px”]

Great headline. Unfortunately, the FT journalist fails to refer us readers to the original study from which she got these numbers.

I left a comment, asking author and Employment Correspondent Sarah O’Connor for help.

[su_custom_gallery source=”media: 1984″ limit=”7″ link=”image” target=”blank” width=”441px” height=”234px”]

My comment (see above image) did not get published, nor did I get an answer from the journalist regarding my query.

So as a paying subscriber, should I trust these claims? Might it be wiser to go and check?

You guessed, I continued digging myself. An interesting journey that took me 28 minutes…

The Daily Mail

I found an article from the Daily Mail (see image below).

But it referred to an earlier article by Mail staff referring to a study by Uppsala University (Sweden) researchers.

References? None whatsoever!
[su_custom_gallery source=”media: 1987″ limit=”7″ link=”image” target=”blank” width=”492px” height=”368px”]

So where did the Financial Times’ (FT) journalist find the information if not from the Daily Mail?

Huffington Post links to FT

The Huffington Post managed a link to the FT article from which it had copied. In other words, to avoid copyright infringement the journalist had done a fast re-write. The content was the same as in the FT article using different wording.

[su_custom_gallery source=”media: 1988″ limit=”7″ link=”image” target=”blank” width=”471px” height=”337px”]

So the Huffington Post failed to add substance to what was posted in the FT. What now?

Vitality Health Life – study cannot be found

You might suggest as a good next step to go and check whether the sponsor of a study might offer the full report. That is what we did.

Unfortunately, the sponsor’s website did not make it easy – it failed basic usability requirements. After some digging we found something, but it did not link to the original or complete report either.

You got that right, the sponsor did not provide the full report. Just a bit of information and nothing more. Real bummer.

[su_custom_gallery source=”media: 1985″ limit=”7″ link=”image” target=”blank” width=”490px” height=”349px”]

Do another organic search

Maybe a search with different keywords could help? Read on and find out. Incidentally, why not subscribe to this blog’s newsletter right now?

[su_box title=”Table 2: 6 things we must do to make sure the numbers add up” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

2.1 Go search for the original (see organic search data in image below)

This got me the link to the Rand Corporation. Here something was written up about the study. But once you have the original study, read it carefully, including the method section.

2.2 Read the method and result section carefully

To illustrate, I recently read:

Murray, Sarah (2016-02-26). Frustrated US workers go it alone. Freelancing. Work is becoming more flexible but less secure. Financial Times, FT Executive Appointments. Employment Global Best Practice. Retrieved, Feb. 27, 1026 from

I went to the original study, Freelancing, and found another study where the authors do not provide a methodology section.

After reading such a study, make sure the following four questions are answered and if not, don’t make decisions based on such work:

2.2.a — How was the sample selected? No information given in the report!
2.2.b –What kind of survey was used? No information given!
2.2.c — Were participants interviewed using an online survey? No information!
2.2.d — Was a combination of landline and cellphone random digit dialing samples used to get responses through interviews? No information!

If the report does not provide information regarding questions 2.2.a to 2.2.d, should I trust it?

Put differently, as a shareholder or tax assessor, would you trust the company’s financial statements if answers to such auditor questions were missing?

Certainly not, so why should you trust such numbers for an opinion poll? I rest my case!


[su_custom_gallery source=”media: 1999″ limit=”7″ link=”image” target=”blank” width=”413px” height=”498px”]

Rand Corporation

A short description is offered and at the bottom a link to the report. Another page opens with another description about the study. Eureka – I can finally download the report.

[su_custom_gallery source=”media: 1989″ limit=”7″ link=”image” target=”blank” width=”489px” height=”268px”]

The report itself is very interesting. On page 11 it introduces the reader to the concept of abseenteeism and presenteeism.

– Absenteeism refers to the measure of days absent from work
– Presenteeism refers to the measure of reduced productivity while at work (e.g., due to headache, flu, etc.).

On page 12, it goes on to say, “The instrument consists of six questions with a recall time frame of seven days. The questions ask whether the respondent is employed; the number of hours missed from work; the number of hours actually worked; and the degree to which the respondent feels that a health problem has affected productivity while at work and affected their ability to do daily activities other than work. WPAI-GH outcomes are expressed as impairment percentages, where higher percentages indicate greater impairment and lower productivity. We use the following three work-related impairment percentages calculated on the basis of the WPAI-GH scale

– Per cent work time missed due to ill-health (absenteeism),
– Per cent impairment while working due to ill-health (presenteeism),
– Per cent overall work impairment due to ill-health (absenteeism and presenteeism).”

Hafner, Marco; van Stolk, Christian; Saunders, Catherine, L; Krapels, Jochim; Baruch, Ben (May 22, 2015). Health, wellbeing and productivity in the workplace. A Britain’s Healthiest Company summary report. Retrieved, May 31, 2015 from

What is the problem with the Rand study?

The survey depends on how well subjects recollect facts from last week. But do seven days in a person’s year accurately reflect the status of their health? Additionally, does it make a difference if we collected these data in July, October, December or February of the year we studied?

Finally, large companies are over-represented in this sample. Moreover, companies with under 50 employees – over 70 percent of British firms – could not participate.

So is this a great study? It is very interesting, but the journalists’ interpretation of these data far exceeds what the authors infer from their own data.

By the way, there is research that is far better suited than the above to learning how sleep deprivation can affect job performance or studying math.

[su_custom_gallery source=”media: 1993″ limit=”7″ link=”image” target=”blank” width=”445px” height=”179px”]

Lack of sleep increases risk of failure in school

Uppsala University to the rescue

Olga E. Titova, et al., (2014) Associations of self-reported sleep disturbance and duration with academic failure in community-dwelling Swedish adolescents. Sleep Medicine doi: 10.1016/j.sleep.2014.09.004 Retrieved May 31, 2015 from…… (click on citation to get study link since it is too long to post here).

The study included 20,000 adolescents aged 12 to 19. This longitudinal study was conducted from 2005 to 2011. About 30 percent of participants reported regular sleep problems.

The study found that if you have less than seven hours of sleep, data indicate an increased risk of failure in school.

The group also found in a previous study that going without a night of sleep increased toxic substances in the brain. Possible increased risk of Alzheimer’s, Parkinson’s and multiple sclerosis was reported. Other prior research shows how the brain uses sleep to cleanse itself.

Interesting Reading

2015-07-20 – One night of sleep loss can alter clock genes in your tissues
2015-07-13 – Sleep loss makes memories less accessible in stressful situations
Three studies show that teens should decrease screen time before going to bed

Bottom line: Show me the data…

[su_box title=”Table 3: Does the story meet 2 critical benchmarks?” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. Does the article provide the reader with a link to the original study discussed?

Every journalist does some research before writing their story. If the original material is available online, why not reference it? Saves your reader time and gives them a chance to read up on this interesting topic.

Hence, the printed article should provide a link to the original source(s). At the very least, it should refer to the online version of the article where links to the original sources are provided.

2. Does the study report provide the reader with a method section?

Explain succinctly how you did the study, such as:

“The analysis in this report is based on telephone interviews conducted February 18-21, 2016 among a national sample of 1,002 adults, 18 years of age or older, living in the continental United States (501 respondents were interviewed on a landline telephone, and 501 were interviewed on a cellphone, including 312 who had no landline telephone)….

A combination of landline and cellphone random digit dialing samples were used… Interviews were conducted in English and Spanish. Respondents in the landline sample were selected by randomly asking for the youngest adult male or female currently at home. Interviews in the cell sample were conducted with the person who answered the phone, if that person was an adult 18 years of age or older.”

Great read: PEW – information about our survey methodology

For those like Prince Harry, who claims, “I can’t do maths,” the above paragraph can be a lifesaver.


If you read an article like the one for this post, better check the number to see if the headline by the journalist can be justified from the study’s results. Very likely it cannot, so putting decreasing amounts into content seems to only viable strategy left.

And to answer our question in the title: No study shows drinking #RedBull boosts job performance.

However, it does increase your daily sugar intake significantly, which is probably not a good thing.

Join the conversation

  1. Do you have an example of how mathematics phobia is affecting basic mastery of mathematics skills?
  2. Do you have a good example of a sponsored study that addresses some of the issues outlined here?
  3. How do you make good guesses about things that affect your decision-making (i.e. invest my money here or there…)?

Of course, I will answer you in the comments. Guaranteed.

Can we trust these numbers?

Interesting reads point out that trust is learned more than inherited. Trust is socially received and transmitted.

The truth about trust
Van Lange, P. A. M. (2015). Generalized trust: Four lessons from genetics and culture. Current Directions in Psychological Science, 24, 71–76.

Doing an opinion poll: Interviewing subject at door

The United States presidential election of 2016 is scheduled for Tuesday, November 8. In the meantime, we know that in almost every race for 36 years, the eventual nominees have won either Iowa or New Hampshire.

While Hillary Clinton won by a razor slim margin in Iowa, Bernie Sanders took home New Hampshire by a wide margin. With the Republicans, it seems increasingly likely that Trump could win the nomination.

This blog entry is part of our series on business analytics and big data

CLICK - Outside chance - the primary contest is about to get serious.

Mr Sanders promises free education in public universities. He wants to have government rather than private insurers to pay health care bills.

This could cost US $14 trillion over a decade, and would result in new taxes, costing most workers 8.4 percent of their income.

Workers might approve of such changes. But Mr Sander’s plans would have no chance of making it past Congress, even in one with a majority of Democrats.

Nevertheless, he won 60 percent of the vote in New Hampshire.

This represented one of the biggest victories in a contested Democratic primary.

Of course, polls play an important roll during any election, including US Presidential (see chart). But can we trust these polling data?

As the above graphic shows, polls gave Governor Kasich less than 5 percent. Nonetheless, he raked in 16 percent of votes in the New Hampshire primary of the Republican party. Trump hoovered up 35 percent of the vote as predicted by pollsters (see above graphic from The Economist, January 30, 2016, p.17).

The above illustrates that polling is a tough job, especially if one intends to get it right. In 2014, pollsters struggled with these challenges during the Scottish referendum and Swedish elections. Both times they got it wrong.

The same happened during the UK elections on May 7, 2015. The final polls showed Labour and the Conservatives neck-and-neck at 34 percent.

But when the final numbers where in, David Cameron’s Tories ended up 7 percent ahead of Ed Miliband’s Labour party. As pollsters have pointed out, however, they got the numbers right for the smaller parties. Too bad, their prediction was outright wrong for those fighting for the post of running the country, i.e. becoming Prime Minister…

So if you see a statement like the one below, will you believe it?

Polls indicate, Hillary Clinton is leading Bernie Sanders by 30 points in South Carolina

We discuss this below in more detail.

1. Getting the right sample size is very expensive

In early states such as Iowa and New Hampshire pollsters conducted more polls for 2016 than in 2012, when the last presidential race was happening. But the average sample size has fallen. According to the Financial Times, for New Hampshire the averages look as follows:

2012 – average pool of Republican voters interviewed was 590
2016 – average is 490

Of course, the margin of error of plus or minus 5 percent are common with such polls. Governor Kasich was predicted to get less than 5 percent. With a margin of error of 5 percent he could have gotten 10 percent but he actually got 16 percent.

In this case, the pollsters were vastly off, illustrating that sample size matters. In other words, does polling 490 Republicans justify the conclusions drawn by the authors?

[su_custom_gallery source=”media: 2470″ limit=”7″ link=”image” target=”blank” width=”529px” height=”322px” Title=”National Statistical Service of Australia – calculating sample size – it works” alt=”National Statistical Service of Australia – calculating sample size – it works”]

National Statistical Service, Australian Bureau of Statistics – Sample size calculator

[su_box title=”Polls are getting ever more costly thus end up with smaller samples” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. 2016 US Presidential Election

Pollsters now make 30-35 calls to complete a single interview.

Ten years ago, a pollster called ten people to get one to participate in the poll or study.

To interview 1,000 voters — only 400 of which may be likely to vote Republican — the pollster now has to dial up to 35,000 numbers.

2. UK polling

In the UK, today it takes about 15,000 phone calls to get 1,000 interviews.

About 20 years ago, 2,000 calls were needed to get 1,000 replies.


The above shows, depending upon the country, we may need to make anywhere from 10,000 to just about 40,000 calls to end up with a sample of 1,000 respondents.

But in the case of an election, we also need to make sure that those who answer are also those who will go cast their vote. This challenge is discussed below.

2. Selecting the wrong sample is a growing risk

Polls are conducted over two or three days, meaning we try to interview those who are more easily contacted. This may happen over the internet or via phone, but such work makes it hard to arrive at a sample surveyed that is representative of all voters.

[su_custom_gallery source=”media: 2469″ limit=”7″ link=”image” target=”blank” width=”528px” height=”317px” Title=”National Statistical Service of Australia – calculating sample size – Youth Barometer UBS” alt=”National Statistical Service of Australia – calculating sample size – Youth Barometer UBS”]

National Statistical Service of Australia – calculate your sample size required for the relative standard error you desire

For instance, May 2015 British election data indicate that opinion polls failed to reach the harder to find Tory voters. In turn, estimates of the vote share attributed to Labor were skewed.

According to the British Social Attitudes survey, Labour was six points ahead among respondents who answered the door on the first visit. However, looking at those that required three to six home visits to be interviewed, the Tories enjoyed an 11-point advantage. Adjusting for social class and age, first time respondents are less conservative. “Busy” respondents are more likely to be so – but harder to chase for pollsters.

Hence, for pollsters it is not easy to get those “busy” people that might vote for a particular party as outlined below. Another challenge is that more and more people no longer have a landline. They can only be reached by mobile number (see also FT mentioning Martin Boon, director of ICM research). Because of this trend, in the US polls are increasingly conducted using mobile phone numbers to call.

3. Getting likely voters is a challenge for pollsters

Selecting those that are likely to vote for your political poll (see graphic below) is important. Finally, the views of voters and nonvoters are often very different, as was the case in 2014 in the US mid-term elections.

[su_custom_gallery source=”media: 3274″ limit=”7″ link=”image” target=”blank” width=”757px” height=”394px” Title=”The Pew Research Center: distinguishing likely voters from nonvoters is critical for pollsters.” alt=”The Pew Research Center: distinguishing likely voters from nonvoters is critical for pollsters.” ]

Pew Research (2016-01-07). Can likely voter models be improved? Section 2: Measuring the likelihood to vote

As well, for the 2015 UK elections, under-30s generally lean left, but very often fail to turn out on polling day. The pollsters, however, reached an atypical group of youngsters, who were unusually engaged with politics and committed to voting.

Another factor that can bias polls is that they are often based on internet polls. These tend to use volunteers to sign up to online panels. They may be drawn at random to participate. Nevertheless, the underlying group of people is self-selecting. Thus, data collected using a sampling method known as random digit dialing or “RDD” results in better data sets (i.e. random-probability samples representative of the population).

As this shows, getting those to participate in a poll that are likely to vote continues to be a challenge.

Sign up for our newsletter. Get the next post about big data, business analytics via email

Bottom line

Here are three challenges pollsters will continue to grapple with and we voters must keep in mind when studying poll results.

[su_box title=”3 challenges for pollsters” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. Polls are expensive!

Ever fewer people want to participate in consumer surveys or election polls. Hence, it takes thousands of calls to get a useful sample. In turn, inferences can be made from such a sample about the election outcome.

But just the calls may take about 400 hours. In turn you get 1,000 replies and still have to pay for data analyses and a write-up. A costly exercise that is not getting cheaper.

2. Getting a representative data set is nearly impossible

As the 2015 UK election illustrated, it takes a greater effort to get Tory voters than others.

Cultural differences may also limit the applicability of Pew Research’s findings in the US whereby “…better-educated people tend to be more available and willing to do surveys than are those with less education.”

However, the current solution of using online non-probability survey panels makes polling results less accurate.

3. Identifying and polling likely voters is hard

It is tough to get accurate readings to predict election outcomes by using self-selected online panels.

YouGov knows this is a sensitive issue and does not publish any answers to research methodology questions on its blog. For instance, the voters over 70 who broke heavily for the Tories were not reflected in YouGov’s online panels used for predicting the 2015 UK elections.

Those under 30 years of age are less likely to cast their ballot. If your panel or poll includes those overeager millennials, who were unusually engaged with politics, your findings will be skewed.

Polling models will continue to be improved, but surprises will also continue to happen.

InterestingWhile Apps and Web respondents do not differ in their type of responses, the response rate is lower using mobile apps to collect data.


Nevertheless, polls are the best way of trying to figure out what the election result is going to be next time an election, primary or referendum comes around. So go and vote.

Join the 3,000+ organizations using the DrKPI Blog Benchmark to double reader comments in a few months while increasing social shares by 50 percent – register now!

Mr Sander’s foreign policy promise boils down to one thing: he will not start a foreign war. The rest is a replay of 1960s student radicalism. He intends to convert the US into a Scandinavian social democracy. Mr Trump would make America great again. He thinks the US has been screwed by its allies. If he needs expertise, since he has none, he will hire experts. In the meantime his solution is to “bomb the shit out of ISIS.” That is about it for his foreign policy.

Unfortunately, both Sanders and Trump are arch-fantasists with the ability to recruit voters. Both will spell extremism as far as US foreign policy is concerned. The system, dominated by the Democrats and Republicans, has always rejected the political extremes. The nation has benefited from a deep political stability, which has contributed greatly to its economic strength and global power. If America’s immunity to extremism is ending, the whole world will feel the consequences. Not a good prospect for anyone, is it?

This post: Yes Virginia, pollsters really are wrong

Join the conversation

  1. Do you have an example of a great poll / study?
  2. I refused participating in a poll / consumer survey last week. You?
  3. Do you think pollsters will get it right in the 2016 US elections?
  4. US Primaries: Clinton and Trump or Sanders and Trump? What do you think?

Of course, I will answer you in the comments. Guaranteed.

Data, and by extension data analytics, are becoming increasingly important for business. At the same time, the data deluge makes making sense of it all a bigger challenge every day.

Here are three trends to should keep in mind for 2016.

1. Don’t shoot from the hip

Numbers are becoming more popular for most people, but the more numbers we get the more useless most of these seem to be. Especially if they are drawn out of a hat; why would you take that into consideration in your decision-making process?

Polling your audiences is fine.
But that is not a statistic that adds up exactly to something like 97%, is it?
Or are you keeping tallies of your straw polls and then doing the statistics?

Comment by DrKPI on Adrian Dayton’s Clearview Social blog

[su_custom_gallery source=”media: 2884″ limit=”7″ link=”image” target=”blank” width=”792px” height=”473px” Title=”Adrian Dayton Clearview Social claims 93% of lawyers… based on straw polls – should we trust this, does it make a difference?” alt=”Adrian Dayton Clearview Social claims 93% of lawyers… based on straw polls – should we trust this, does it make a difference?”]

View slide on Flickr – measure-for-impact – DrKPI

Google Flu Trends is an example that illustrates this problem further. For instance, it:

– looks at historical data – descriptive analytics and research, and
– tries to predict what might happen – predictive analytics – with the help of a model that was developed.

The results are supposed to help us better understand how the flu will spread next winter. Unfortunately, in the Google flu trends versus National Institutes of Health (NIH) challenge, the winner is? NIH! Google estimates are simply far off from the actual data the NIH produces for policy makers and health professionals.

2. Bad data result in bad decisions

Publishing rankings or product tests is popular. Since some readers devour such rankings, publishers can sell more copies, which keeps advertisers happy.

A real win-win situation, right? Not so. Wrong decisions can result in outcomes that are not desirable. For instance, attending the wrong college or polluting more than the test results indicate (think Volkswagen and #dieselgate) is not something we want.

Lucy Kellaway felt so incensed about the ever growing acceptance of making errors in corporate circles, that she wrote:

…I would be exceedingly displeased to learn that the bankers to whom I was handing over a king’s ransom were being taught that errors were perfectly acceptable.

This mistake-loving nonsense is an export from Silicon Valley, where “fail fast and fail often” is what passes for wisdom. Errors have been elevated to such a level that to get something wrong is spoken of as more admirable than getting it right.

By collecting data and using flawed methods we produce rankings or test results that will can seriously hurt people. For instance, when drug certification tests are done improperly and the regulator has no idea, unknown side effects can kill people.

Using the wrong test results to approve or certify a car can result in dismal effects as well. Volkswagen is accused of manipulating tests, and the public got more pollution than it bargained for. VW is working on fixing the 11 million vehicles affected by the diesel cheat, but this will not un-do the damage to the firm’s reputation and our health.

3. Check before you trust the method used

It is always wise to take 5 minutes to do an acid test with any study report we see, such as:

– what does the methodology tell us (e.g., we asked university deans to rank their competitors); and

– does the measure or measures used make sense (e.g., one question about how university developed / improved study programs – result = ASU is more innovative than Stanford or MIT… who are you kidding?).

The Art Review publishes an annual ranking of the contemporary art world’s most influential figures. In short, it helps if you live in London or New York so the Art Review editors or journalists are aware of who you are.

I asked for an explanation of how these numbers develop:

Dear Sir or Madam
I would like to know more about the methodology you used for the ArtReview’s Power 100 List.
Can you help… this would be great to use with my students in a class.
I could not find anything on the website that I could show my students.
Professor Urs E. Gattiker, Ph.D.

14 days later I got an answer from the makers of the ranking:

Subject: Re: Message from user at

We are not following a grid of criteria per se, and the list emerges from a discussion between a panel of international contributors and editors of the magazine, who each advocate for the people they feel are most influential in their region. The influence of the selected people on the list is based on their accomplishments in the past 12 months. I have attached here the introduction to the Power 100, which might help you in defining our approach.
I hope that helps,
Best, Louise

A grid of criteria, what is that? Of course, the office clerk answering me has no clue about research methodology used, as the answer indicates. One could start believing that this Top Art list came from a discussion or using a straw poll. Totally chaotic approach.

You can view the attachment that explains this sloppy method below.

[embeddoc url=”” download=”all” viewer=”google”]

Download the ArtReview criteria with this link.

A friend of mine smiled, and said:

For me this is a great list, Urs. Those on the list rarely if ever represent value for money for serious art collectors. Instead you get buzz and have to pay for their image. The list tells me who we do not need to work with. We use other experts. These give us more value for money. They help us to complement our award-winning collection.

[su_custom_gallery source=”media: 2881″ limit=”7″ link=”image” target=”blank” width=”780px” height=”479″ Title=”Sound research takes plenty of resources” alt=”Sound research takes plenty of resources”]

Bottom line

We all know that data quality is important and frequently discussed. In fact, the trustworthiness of data directly relates to the value it can add to an organisation.

As the image above suggests, doing quality research takes a decent method that results in data that permits careful analysis. Sloppy data are cheap to get, but dangerous if used in decision-making. Such findings are neither replicable nor likely valid.

However, we are increasingly required to present findings in order to attract more readers. Some master this very well like Inc. Another example of theirs I came across was:

Though truly quantifying “best” is impossible, the approach Appelo’s team used makes sense, especially when you read the books that made the list.

The 100 Best Business Books of 2015 by Jeff Haden

And here’s the methodology:
The purpose of our work was to find out which people are globally the most popular management and leadership writers, in the English language.
Step 1: Top lists
With Google, we performed a lot of searches for “most popular management gurus”, “best leadership books”, “top management blogs”, “top leadership experts”, etc. This resulted in a collection of 36 different lists, containing gurus, books, and blogs. We aggregated the authors’ names into one big list of almost 800 people.
Step 2: Author profiles
Owing to time constraints, we limited ourselves to all authors who were mentioned more than once on the 36 lists (about 270 people), though we added a few dozen additional people that we really wanted to include in our exploration. For all 330 authors, we tried to find their personal websites, blogs, Twitter accounts, Wikipedia pages, Goodreads profiles, and Amazon author pages.

So you defer to 36 people and their lists and include those that are mentioned more than once. Fine, if that does then not include the ones you believe should be on the list because you read these books and liked them, no worries. You add a few dozen people (60) and voilà, you have 330 authors (how they ranked them is totally unclear, but interesting – blog reputation, Twitter followers, etc.).

[su_box title=”3 checks you should undertake before accepting a study’s findings” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. Evidence-based management and policy advice

A sloppy method is like following no method.
Can you find a method section, and does the method make sense to you? For example, did the study use a long-form questionnaire to get employment data? Or was it just based on scans of Internet job boards? If the latter, the problem lies with double counting when relying on websites or job search engines.

If the method section does not instill you with confidence that it was done properly, watch out. And, most importantly, don’t complain about a study before you read it carefully!

Interesting read: CRDCN letter to Minister Clement – Census long-form questionnaire (July 9, 2010) explains why Statistics Canada needs to get the funds to collect data for the census to provide evidence-based policy data.

2. Minestrone: Great soup but wrong research method

So the study has a decent method section that makes sense and explains things accurately. What are the chances that somebody else could follow the methodology and get the same result?

To illustrate, if it was done the same way I put together a Minestrone (Italian vegetable soup), you can forget it. I take whatever vegetables are in season, plus, each family’s soup is seasoned differently, guaranteed. This neatly illustrates the fact that if no systematic method is used, it is not science. For the soup this means it turns out different each time anyone makes it.

Without a recipe or method followed, you cannot repeat the performance or generalise from your findings.

3. Buyer beware: Click biting studies using navel gazing metrics

Usage of Sainsbury’s #ChristmasIsForSharing being higher than John Lewis’ #ManOnTheMoon by just 4% is interesting. However, Social Bro’s verdict is based on 50 votes (26 versus 24) from a Twitter poll. In turn, the analytics company uses this data to decide on 2015’s Most Creative Christmas Campaigns. What? Are they real, is their analytics work also that sloppy?

Apparently, even analytics companies like Social Bro have to defer to such navel gazing metrics to get more traffic. Such samples are neither representative nor big enough to draw any inferences.

Just because something is interesting or suggests it is a bit better based on 3 more votes on Twitter, does not mean you should invest your hard earned cash that way. Investing your marketing dollar based on such nonsense is plain dumb.

What is your take?

– what will you change in your data #analytics and #analysis work in 2016?
– what is your favourite example for 20015, illustrating GREAT analytics work and research?
– how do you deal with this data deluge?
– what would you recommend to a novice (ropes to skip)?

More insights about analytics, analysis and big data.