EU Referendum, European fallout, broken utopia, like fake online reviews: IT CANNOT BE TRUE, CAN IT? | Urheber: Miriam Dörr| Fotolia #111044349

Summary: What does the Brexit crisis have in common with fake online reviews?
What role can data analytics and analysis play in this saga?
This post provides guidance for depending on hashtags and online reviews.

Pollsters and punters had the Remain campaign winning the Brexit vote by a hair. But the headlines this and last week speak to a different tune.

The Pound Sterling fell as traders prepared for a cut in the interest rate. Some felt that Britain was starting to imitate Greece and called it Britain’s Greek tragedy…

So are we analysts and number crunchers to blame? Here are some things the Brexit crisis has taught us big data pundits.

1. Hashtags do not win elections

Groups tapped into social media in the hope of persuading the young to join in (or out). Stars joined campaigns and let themselves be used as messengers in videos. Nevertheless, all this failed to sway enough voters to go to the polls.

Just about two weeks before the referendum was held Thursday 2016-06-23, Adam & Eve DDB tried its luck with an online video campaign featuring celebrities and swear words, including this example:

Nevertheless, featuring Keira Knightly failed to give it much traction on YouTube. Maybe just another example for how advertising professionals know far less than they claim about what it takes to make a viral video.

Inspiring young people to vote in the EU referendum proved tough in the UK. But other countries have similar experience. For instance, in Switzerland younger people tend to vote less often than their elders.

Brandwatch did a study about hashtags for the Brexit campaign. Unfortunately, data are sketchy. Did they include the hashtag #voteremain that was apparently used 600,000 times in their analysis? And what about others?

#Voteexit was used over a million times, but was that the only Brexit hashtag? What about those social media users that used several hashtags in their tweets for one campaign?

♥ Curious? Join 1500 other subscribers to this blog’s newsletter and read on!

Besides, using a hashtag does not mean my tweet is positive toward that particular side of the campaign. And does voicing our opinion using social media mean we go to the trouble of voting?

Nor has social media been known to change people’s firm opinions, so who cares about the social media echo chamber. It is the voting booth that counts, stupid.

2. Fake online reviews and your cash register

Like hashtags fail to necessarily sway a large group of voters, so do fake online reviews. But some attribute increasing importance to them. In fact, Social Bites won awards and was feted by social media ‘influencers’.

Unfortunately, it all turned out to be a fake used by Mark Cowper to show how much social network content is fake (see image below). The Twitter account has now been closed.

In particular, this highlights how influencers let themselves be swayed and thereby provide their fans or followers with useless information or fake content / reviews.

Read: Fake online reviews

What is really sad is that the campaign grabbed a lot of attention on social networks and in the media.

But can we conclude that all this hype would have resulted in more sales? The site never got launched, but this could well indicate that buzz has little to do with your bottom line.

Social Bites won awards and was feted by social media ‘influencers’ but it was a fake.

Social Bites won awards and was feted by social media ‘influencers’ but it was a fake.

Nevertheless, fake online reviews are becoming a plague. So much so that the UK government decided to investigate online reviews late last year.

Read the interesting press release here: Press Release – UK Competition and Markets Authority (CMA) takes enforcement action against fake online reviews

March 2016, the authority published some guidance for businesses on what is okay under the law. Check out the report here: Guidance Online reviews: letting your customers see the true picture – see image below.

What do you need to do if you are a business whose products are being reviewed?

What you need to do if you are a business whose products are being reviewed?

3. Should I put trust in reviews from associates?

Mark Cowper’s experiment nicely illustrates the problem of how sharing of fake reviews and wrongful information affect one’s online reputation.

In turn, he wanted to illustrate that we need a review platform that provides us with product reviews from our trusted network.

The idea is great. Getting recommendations from my trusted network sounds like a classic word-of-mouth marketing approach.

We love to hear our friends’ experiences before we buy (see screenshot below – find and save recommendations from your trusted network).

Recomazing, a social network that enables people to review, share and find reviews from their real-life network of friends and relatives.

Recomazing, a social network that enables people to review, share and find reviews from their real-life network of friends and relatives.

Unfortunately, the Recomazing network is a perfect example of sunk costs. I have already invested in having an online presence on such platforms as Twitter, Google+, Flickr, Facebook, WhatsApp, Snapchat, Instagram and Tencent.

So starting anew on Recomazing also means some switching costs.

  • Is it worth it?
  • Who has the time to join another network, never mind write and read product reviews about things one will never buy?
  • Do we have the time – or is it just so entertaining (e.g., like watching a video) that we do not mind spending time on a network about product reviews?

Besides, while this may work in OZ (Australia and New Zealand), hardly any of my friends in Europe use it. So am I wasting my time joining? Probably.

Watch the short video below and you’ll see what I mean.

♥ Curious? Join 1,500+ other subscribers to this blog’s newsletter! ♥

[su_box title=”3 takeaways: Use product reviews wisely” box_color=”#86bac5″ title_color=”#ffffff”]

1. Hashtags help but may not make your cash register ring

Do we use actionable metrics, i.e. metrics which we can take action on?

If one million users use a hashtag, will it result in Trump or Clinton winning the election?
As the Brexit referendum illustrates, hashtags and social media activity did not swing the Remain vote enough to make a difference at the polls.

2. Do we understand the metrics behind the analysis?

If we analyse the numbers, were the online reviews from real or fake customers?
Are those people regulars or Chinese tourists that failed to understand the customer survey they were handed in German?

And before I forget: Have we even thought about the possibility of errors in our data set due to e.g., sampling bias, response bias or sampling error?

3. Wait a while before asking your out-of-town customers for feedback

I remember having dinner with my family at a restaurant in Amsterdam. Before we even got the bill, we were invited to rate them. The waiter brought us an iPad where we could enter our review right away.

It’s better to wait and then send your client a short survey (5 questions max). Or invite them to write a review / testimonial for your webpage. And yes, writing the review on a platform like TripAdvisor might not hurt.

Research demonstrates that giving your customer four weeks before asking for feedback is a smarter approach if you want better reviews.

Interesting read: How to boost online ratings legally

More interesting reads – misinformation tends to ripple… facts remain scarce

a) Word-of-mouth marketing can make a difference (in German)
b) Data analytics: Lessons learned from Ebola
c) Can infographics show you the money?
d) Scottish referendum: A false sense of precision?
e) Data analytics: UPS or Apple?

4. Have your say – join the conversation

Source: Fake reviews? Lessons from Brexit

What is your opinion?

  • How do you decide what to buy?
  • Do you have a circle of friends that write online reviews regularly?
  • When was the last time you shopped for a brand or stayed at a hotel because of a review?
  • Do you remember last time your friend recommended a product based on their great experience (e.g., running shoe, coffee maker or going to shop at a store with knowledgeable and friendly staff) and you took their advice?

The author declares that he had no conflict of interest with respect to the content, authorship or publication of this blog entry (i.e. I neither own any of these brands’ products nor are they our clients).


Update 2016-03-17
Yesterday American doctors informed us that our budget should also include an item for follow-up treatment.
By now we have raised some money and are continuing to also raise funds to be able to pay for subsequent check-ups and treatment. We can do this with your help. Thank you.

Update 2016-03-15
#helpilayda we have collected more than €420’000
we stop at €800’000 ==> depending on where in the USA the therapy will happen, it will cost more or less.

Update 2016-03-14
#helpilayda we have collected more than €300’000 

Update 2016-03-13
#helpilayda we collected over €200,000 so far – a first in the D-A-CH region of countries – social media can make something happen.

Our campaign to secure the funding needed to save Ilayda Yildiz’s life is in full swing. Our German blog entry, #helpilayda – Crowdfunding Kampagne: Rettet das Leben von Ilayda Yildiz, was launched successfully last Friday.
Ilyada’s Blog
Facebook page
Donate now please – it makes a REAL difference

This blog entry is part of our series on viral marketing and word-of-mouth marketing – WOMMA

Who and what is this story about?

[su_custom_gallery source=”media: 3448″ limit=”7″ link=”image” target=”blank” width=”780px” height=”410px” Title=”This picture(right-hand side) shows (from left to right) Nuhhaci Yildiz, Hülya Yildiz, Ilayda Yildiz and her sister” alt=”This picture  (right-hand side) shows (from left to right) Nuhhaci Yildiz, Hülya Yildiz, Ilayda Yildiz and her sister”]

The above picture shows (from left to right) Nuhhaci Yildiz, Hülya Yildiz, Ilayda Yildiz and her sister.

At the center of this story is Ilayda Yildiz (second from right in the picture above). She was born December 17, 2005 in Singen, Germany, a community on the Swiss border. On February 27, 2012, shortly after turning six, Ilayda Yildiz’s parents were informed that preliminary tests suggested their child had leukemia. Additional tests revealed it to be acute lymphocytic leukemia (ALL), also known as acute lymphoid leukemia or acute lymphoblastic leukemia. The majority of leukemias diagnosed in childhood are ALL.

About 75 percent of all childhood leukemias are ALL (see New York Times – Leukemia In-Depth Report very nicely structured, plenty of facts, numbers and diagrams).

For 20 percent of those suffering from ALL, chemotherapy will not help. This is what happened to Ilyada Yildiz, who is now 10 years old, and has been fighting her disease since 2012. Estimates suggest:

6,000 people in the US (National Cancer Institute),
1,500 in Germany, and
150 in Switzerland die annually because they suffer from a chemotherapy-resistant type of leukaemia.

But thanks to a new therapy, 92 percent of these patients can recuperate fully.

Unfortunately, this therapy is not covered by German or Swiss health insurance. The result is that those patients – primarily kids – die.

Just imagine what would happen if the treatment were covered:

Every year up to about 1350 of 1500 patients in Germany or 130 of 150 patients in Switzerland and thousands more in France, Spain, Italy and the Netherlands could be cured!

[su_custom_gallery source=”media: 3457″ limit=”7″ link=”image” target=”blank” width=”780px” height=”410px” Title=”Ilayda backstage in Stuttgart with Violetta from the Disney Telenovela” alt=”Ilayda backstage in Stuttgart with Violetta from the Disney Telenovela”]

The above image shows Ilayda backstage at Violetta – the Disney Channel Telenovela where a talented teenager returns to her hometown in Buenos Aires after living many years in Europe.

Where does this leave Ilayda?

Like 20 percent of leukaemia patients, Ilayda suffers from a type that is described as chemotherapy-resistent.

Her only option is a personalized treatment called T cell therapy.

This is a process whereby the patient’s own T-cells are removed and genetically engineered to attack his or her cancer cells. The modified t-cells are called CD19-chimeric antigen receptor T cells or CD19-CAR T cells (information from the US National Cancer Institute). These modified cells are then reinjected into the patient. In various tests, the “CAR T-cell therapy can help patients that suffer under acute lymphoblastic leukemia (ALL),”  (see ASH 2014, Abstract of study 382).

This treatment has been tested and successfully administered to several patients. The Children’s Hospital of Philadelphia has conducted tests under the auspices of the US FDA (Federal Drug Administration) for CAR T-Cell therapy; they are leaders in the field.

92 percent of 39 kids treated using CAR T-Cell therapy showed no evidence of cancer one month after treatment. – Children’s Hospital of Philadelphia

How YOU can make a REAL difference in Ilayda’s life

We decided to try to raise the money to save Ilayda Yildiz’s life, so we launched the #helpilyada #crowdfunding #campagin.

We have all heard about crowdfunding campaigns to launch new products or influence the Presidential campaign. We also have seen others raise €100,000 or more for a child.

So, here is the poster for our campaign – you are welcome to use it to share good thoughts!

[embeddoc url=”” viewer=”google”]

Download the poster (745 KB PDF) for Crowdfunding Campaign #helpilayda: Please print and share

3. We need €823,471 to save Ilayda Yildiz’s life

So we need people like you. People who care about others and are willing to do some good. It will not cost you a fortune, and some will give more than others. No matter what, every donation is generous and every single dollar counts towards helping to pay for Ilayda’s treatment.

Crowdfunding is part of fundraising, but crowdfunding focuses on one project with a time limit.

In the case of the #helpilyada crowdfunding campaign we want to raise enough money to pay for Ilayda to receive life-saving CAR T-cell therapy at The Children’s Hospital of Philadelphia (CHOP for short).

For instance, we need resources to pay for Ilyada and her family to get to Philadelphia (US) and back. Her parents and sister will have to accompany her to provide the support she needs to keep up her spirits.

So if you know an airline that could help, please reach out to us here through the blog or via the contact form. We will get in touch with you like Speedy Gonzales.

But real cash is necessary to pay for the laboratory and genetic engineering work at the hospital to produce and pay for the CD19-chimeric antigen receptor T-cells, or CD19-CAR T-cells for short.

The procedures and required hospital stay must be financed as well.

At this stage the family has raised €90,650 (2016-02-03), so there is still a long way to go – another €700,000. Ilayda’s parents are at their limit. Their financial resources are tapped out, and the emotional rollercoaster hasn’t helped.

Likes on Facebook are great. Using our hashtag #helpilayda helps too. But what we need most is donations from the heart!



Even Christiano Ronaldo (Real Madrid) has joined the fund raising efforts for Ilayda Yildiz. He wishes her great success.


Your donation can be the last bit we need to make it happen and save Ilayda from a life cut indecently short.

[su_box title=”Table – this is how you can help Ilayda today” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]1. Money matters.

In this case it is money that will make it possible for Ilayda to get the life-saving treatment, so please give now:

Here is your chance to be part of the group that support Ilayda’s fight for survival – do it with a smile.

2. How can we get airline tickets, hotel stays and so forth to make this happen?

We need your help to connect with those who can provide tickets. Can you make use of your contacts?

Ilayda’s parents and her sister (what a trooper she is!) will have to travel with her, so Ilayda has the emotional support she needs to keep fighting.

What about Novartis, can you help us get Ilayda into their European trials? Anybody have an idea, maybe Susan Longman who is Head, Drug Regulatory Affairs, Europe and Great China at Novartis Pharma AG?

3. Will we get the crowdfunding or is it a pipedream?

We do not know if we will succeed with this crowdsourcing project. But we feel it is worth every hour of our time to help save another child’s life.

We need your support and help to make this happen. Your comment here, and your Like is an important step. But please do not stop there, go beyond and become one of our donors.


4. Can Novartis help?

Novartis plays a big part in these trials, as David Lebwohl and Usman (Oz) Azam (both Novartis employees) point out. David Lebwohl also states that:

The treatment doesn’t exist until we transform the patient’s own cells into CTL019 cells and deliver them back to the patient.

According to this blog entry, the company is looking at doing cross-country trials, in turn bringing CTL019 clinical trials across borders…

Apparently, European health authorities have helped Novartis start clinical trials across the Atlantic, as mentioned by Eric Couture, head of Regulatory in the Cell and Gene Therapies Unit.

5. What can you do?

We want this story to go viral and hope that with this crowdsourcing effort we secure the funds needed to save Ilayda’s life, BUT after reading Ilayda’s blog entry, we need your help:

– What skills, talents, know-how, contacts can you offer to help save Ilayda’s life?

– As of 2016-02-03, we have raised €90,650, but we still need YOUR CONTRIBUTION to get another €709,350 to make this happen.

6. More interesting information

Crowdfunding campaign: Save Ilayda

Update 2014-03-07 DrKPI – #helpilayda – CyTRAP Labs GmbH donated €750  – Please contribute as much as you can and help us save Ilayda Yildiz’s life

[su_custom_gallery source=”media: 3428″ limit=”7″ link=”image” target=”blank” class=”alignleft” width=”780px” margin=”2px” height=”407px” Title=”#helpilayda Crowdfunding campaign – save Ilayda’s life” alt=”#helpilayda Crowdfunding campaign – save Ilayda’s life”]

More information about the CAR T-cell therapy

Ilaydas blog

NIH – US National Cancer Institute – CAR T-Cell Therapy: Engineering Patients’ Immune Cells to Treat Their Cancers

Brower, Vicki (April 1, 2015). The CAR T-Cell Race. Retrieved March 2, 2016 from

President Obama’s “Moonshot” to Cure Cancer

Orphan Healthcare – Foundation for Rare Diseases

Your donation will help save Ilayda’s life – donate now (trust account)

Please, don’t forget to to ask your employer to match your donation. This will help your donation go much further! Thank you. 

The authors declare that they had no conflict of interest with respect to the content, authorship or publication of this blog entry.

Drinking Red Bull boosts job performance
Math-myopia and Prince Harry: do these numbers from Red Bull about performance enhancements make sense?

Will drinking Red Bull and smoking cigars boost our productivity at work?
Will sleep deprivation increase the number of mistakes we make?

This post addresses these questions, as well as how math-myopia affects love for metrics and statistics about sports, dieting, work injuries and so forth.

This blog entry is part of our series on business analytics and big data

If you read German, check out our series on political campaigning and the usefulness of polling (US presidential election).

[su_box title=”Table 1: Keep the numeracy problem in mind” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1.1 Making a good guess: Avoid base rate neglect

When we combine two pieces of information, we tend to ignore one of them completely. This phenomenon is called “base rate neglect”. For example, the base rate tells us how many people are affected by bowel cancer (6 out of 100), or how many have a fatal injury at work (i.e. 142 deaths per year in the UK – a rate of 0.46 deaths per 100,000 workers).

Knowing the base rate helps put things in perspective.

1.2 Our culture makes it acceptable to say, “I do not do numbers.”

Imagine a study that presents a test that is 75 percent accurate. In 25 percent of the cases where the test predicts a self-reported injury at work will happen, it does not. This is called a false positive.

What is the chance that the person has a work-related accident? Intuitively we might say that in 75 percent of cases a fatal accident will occur. However, the correct answer is, ‘we do not know’ – unless we have the base rate.

To illustrate, if we test 100 people and 4 come out positive, 3 were rightfully identified to likely have a work-related accident and 1 was wrongfully identified. But wait! Of the 96 others, 24 (= 25 percent) will have a false positive. This means we predict a non-fatal work injury, but they will not have one.


Unfortunately, in a culture where Prince Harry can publicly state that he may not have the math skills to be an air ambulance helicopter pilot, he is likely to ignore the base rate…

The base rate is a good way to start if we want to forecast something or put test results in perspective (see Tables 2 and 3 below).

Below we illustrate this a bit more with an example based on a May 2015 Financial Times article, which nicely illustrates how things can be misconstrued by journalists.

To reduce this risk, we must go to the trouble and check the numbers.

Financial Times gets the ball rolling

[su_custom_gallery source=”media: 1986″ limit=”7″ link=”image” target=”blank” width=”531px” height=”130px”]

Great headline. Unfortunately, the FT journalist fails to refer us readers to the original study from which she got these numbers.

I left a comment, asking author and Employment Correspondent Sarah O’Connor for help.

[su_custom_gallery source=”media: 1984″ limit=”7″ link=”image” target=”blank” width=”441px” height=”234px”]

My comment (see above image) did not get published, nor did I get an answer from the journalist regarding my query.

So as a paying subscriber, should I trust these claims? Might it be wiser to go and check?

You guessed, I continued digging myself. An interesting journey that took me 28 minutes…

The Daily Mail

I found an article from the Daily Mail (see image below).

But it referred to an earlier article by Mail staff referring to a study by Uppsala University (Sweden) researchers.

References? None whatsoever!
[su_custom_gallery source=”media: 1987″ limit=”7″ link=”image” target=”blank” width=”492px” height=”368px”]

So where did the Financial Times’ (FT) journalist find the information if not from the Daily Mail?

Huffington Post links to FT

The Huffington Post managed a link to the FT article from which it had copied. In other words, to avoid copyright infringement the journalist had done a fast re-write. The content was the same as in the FT article using different wording.

[su_custom_gallery source=”media: 1988″ limit=”7″ link=”image” target=”blank” width=”471px” height=”337px”]

So the Huffington Post failed to add substance to what was posted in the FT. What now?

Vitality Health Life – study cannot be found

You might suggest as a good next step to go and check whether the sponsor of a study might offer the full report. That is what we did.

Unfortunately, the sponsor’s website did not make it easy – it failed basic usability requirements. After some digging we found something, but it did not link to the original or complete report either.

You got that right, the sponsor did not provide the full report. Just a bit of information and nothing more. Real bummer.

[su_custom_gallery source=”media: 1985″ limit=”7″ link=”image” target=”blank” width=”490px” height=”349px”]

Do another organic search

Maybe a search with different keywords could help? Read on and find out. Incidentally, why not subscribe to this blog’s newsletter right now?

[su_box title=”Table 2: 6 things we must do to make sure the numbers add up” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

2.1 Go search for the original (see organic search data in image below)

This got me the link to the Rand Corporation. Here something was written up about the study. But once you have the original study, read it carefully, including the method section.

2.2 Read the method and result section carefully

To illustrate, I recently read:

Murray, Sarah (2016-02-26). Frustrated US workers go it alone. Freelancing. Work is becoming more flexible but less secure. Financial Times, FT Executive Appointments. Employment Global Best Practice. Retrieved, Feb. 27, 1026 from

I went to the original study, Freelancing, and found another study where the authors do not provide a methodology section.

After reading such a study, make sure the following four questions are answered and if not, don’t make decisions based on such work:

2.2.a — How was the sample selected? No information given in the report!
2.2.b –What kind of survey was used? No information given!
2.2.c — Were participants interviewed using an online survey? No information!
2.2.d — Was a combination of landline and cellphone random digit dialing samples used to get responses through interviews? No information!

If the report does not provide information regarding questions 2.2.a to 2.2.d, should I trust it?

Put differently, as a shareholder or tax assessor, would you trust the company’s financial statements if answers to such auditor questions were missing?

Certainly not, so why should you trust such numbers for an opinion poll? I rest my case!


[su_custom_gallery source=”media: 1999″ limit=”7″ link=”image” target=”blank” width=”413px” height=”498px”]

Rand Corporation

A short description is offered and at the bottom a link to the report. Another page opens with another description about the study. Eureka – I can finally download the report.

[su_custom_gallery source=”media: 1989″ limit=”7″ link=”image” target=”blank” width=”489px” height=”268px”]

The report itself is very interesting. On page 11 it introduces the reader to the concept of abseenteeism and presenteeism.

– Absenteeism refers to the measure of days absent from work
– Presenteeism refers to the measure of reduced productivity while at work (e.g., due to headache, flu, etc.).

On page 12, it goes on to say, “The instrument consists of six questions with a recall time frame of seven days. The questions ask whether the respondent is employed; the number of hours missed from work; the number of hours actually worked; and the degree to which the respondent feels that a health problem has affected productivity while at work and affected their ability to do daily activities other than work. WPAI-GH outcomes are expressed as impairment percentages, where higher percentages indicate greater impairment and lower productivity. We use the following three work-related impairment percentages calculated on the basis of the WPAI-GH scale

– Per cent work time missed due to ill-health (absenteeism),
– Per cent impairment while working due to ill-health (presenteeism),
– Per cent overall work impairment due to ill-health (absenteeism and presenteeism).”

Hafner, Marco; van Stolk, Christian; Saunders, Catherine, L; Krapels, Jochim; Baruch, Ben (May 22, 2015). Health, wellbeing and productivity in the workplace. A Britain’s Healthiest Company summary report. Retrieved, May 31, 2015 from

What is the problem with the Rand study?

The survey depends on how well subjects recollect facts from last week. But do seven days in a person’s year accurately reflect the status of their health? Additionally, does it make a difference if we collected these data in July, October, December or February of the year we studied?

Finally, large companies are over-represented in this sample. Moreover, companies with under 50 employees – over 70 percent of British firms – could not participate.

So is this a great study? It is very interesting, but the journalists’ interpretation of these data far exceeds what the authors infer from their own data.

By the way, there is research that is far better suited than the above to learning how sleep deprivation can affect job performance or studying math.

[su_custom_gallery source=”media: 1993″ limit=”7″ link=”image” target=”blank” width=”445px” height=”179px”]

Lack of sleep increases risk of failure in school

Uppsala University to the rescue

Olga E. Titova, et al., (2014) Associations of self-reported sleep disturbance and duration with academic failure in community-dwelling Swedish adolescents. Sleep Medicine doi: 10.1016/j.sleep.2014.09.004 Retrieved May 31, 2015 from…… (click on citation to get study link since it is too long to post here).

The study included 20,000 adolescents aged 12 to 19. This longitudinal study was conducted from 2005 to 2011. About 30 percent of participants reported regular sleep problems.

The study found that if you have less than seven hours of sleep, data indicate an increased risk of failure in school.

The group also found in a previous study that going without a night of sleep increased toxic substances in the brain. Possible increased risk of Alzheimer’s, Parkinson’s and multiple sclerosis was reported. Other prior research shows how the brain uses sleep to cleanse itself.

Interesting Reading

2015-07-20 – One night of sleep loss can alter clock genes in your tissues
2015-07-13 – Sleep loss makes memories less accessible in stressful situations
Three studies show that teens should decrease screen time before going to bed

Bottom line: Show me the data…

[su_box title=”Table 3: Does the story meet 2 critical benchmarks?” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. Does the article provide the reader with a link to the original study discussed?

Every journalist does some research before writing their story. If the original material is available online, why not reference it? Saves your reader time and gives them a chance to read up on this interesting topic.

Hence, the printed article should provide a link to the original source(s). At the very least, it should refer to the online version of the article where links to the original sources are provided.

2. Does the study report provide the reader with a method section?

Explain succinctly how you did the study, such as:

“The analysis in this report is based on telephone interviews conducted February 18-21, 2016 among a national sample of 1,002 adults, 18 years of age or older, living in the continental United States (501 respondents were interviewed on a landline telephone, and 501 were interviewed on a cellphone, including 312 who had no landline telephone)….

A combination of landline and cellphone random digit dialing samples were used… Interviews were conducted in English and Spanish. Respondents in the landline sample were selected by randomly asking for the youngest adult male or female currently at home. Interviews in the cell sample were conducted with the person who answered the phone, if that person was an adult 18 years of age or older.”

Great read: PEW – information about our survey methodology

For those like Prince Harry, who claims, “I can’t do maths,” the above paragraph can be a lifesaver.


If you read an article like the one for this post, better check the number to see if the headline by the journalist can be justified from the study’s results. Very likely it cannot, so putting decreasing amounts into content seems to only viable strategy left.

And to answer our question in the title: No study shows drinking #RedBull boosts job performance.

However, it does increase your daily sugar intake significantly, which is probably not a good thing.

Join the conversation

  1. Do you have an example of how mathematics phobia is affecting basic mastery of mathematics skills?
  2. Do you have a good example of a sponsored study that addresses some of the issues outlined here?
  3. How do you make good guesses about things that affect your decision-making (i.e. invest my money here or there…)?

Of course, I will answer you in the comments. Guaranteed.

Can we trust these numbers?

Interesting reads point out that trust is learned more than inherited. Trust is socially received and transmitted.

The truth about trust
Van Lange, P. A. M. (2015). Generalized trust: Four lessons from genetics and culture. Current Directions in Psychological Science, 24, 71–76.

Doing an opinion poll: Interviewing subject at door

The United States presidential election of 2016 is scheduled for Tuesday, November 8. In the meantime, we know that in almost every race for 36 years, the eventual nominees have won either Iowa or New Hampshire.

While Hillary Clinton won by a razor slim margin in Iowa, Bernie Sanders took home New Hampshire by a wide margin. With the Republicans, it seems increasingly likely that Trump could win the nomination.

This blog entry is part of our series on business analytics and big data

CLICK - Outside chance - the primary contest is about to get serious.

Mr Sanders promises free education in public universities. He wants to have government rather than private insurers to pay health care bills.

This could cost US $14 trillion over a decade, and would result in new taxes, costing most workers 8.4 percent of their income.

Workers might approve of such changes. But Mr Sander’s plans would have no chance of making it past Congress, even in one with a majority of Democrats.

Nevertheless, he won 60 percent of the vote in New Hampshire.

This represented one of the biggest victories in a contested Democratic primary.

Of course, polls play an important roll during any election, including US Presidential (see chart). But can we trust these polling data?

As the above graphic shows, polls gave Governor Kasich less than 5 percent. Nonetheless, he raked in 16 percent of votes in the New Hampshire primary of the Republican party. Trump hoovered up 35 percent of the vote as predicted by pollsters (see above graphic from The Economist, January 30, 2016, p.17).

The above illustrates that polling is a tough job, especially if one intends to get it right. In 2014, pollsters struggled with these challenges during the Scottish referendum and Swedish elections. Both times they got it wrong.

The same happened during the UK elections on May 7, 2015. The final polls showed Labour and the Conservatives neck-and-neck at 34 percent.

But when the final numbers where in, David Cameron’s Tories ended up 7 percent ahead of Ed Miliband’s Labour party. As pollsters have pointed out, however, they got the numbers right for the smaller parties. Too bad, their prediction was outright wrong for those fighting for the post of running the country, i.e. becoming Prime Minister…

So if you see a statement like the one below, will you believe it?

Polls indicate, Hillary Clinton is leading Bernie Sanders by 30 points in South Carolina

We discuss this below in more detail.

1. Getting the right sample size is very expensive

In early states such as Iowa and New Hampshire pollsters conducted more polls for 2016 than in 2012, when the last presidential race was happening. But the average sample size has fallen. According to the Financial Times, for New Hampshire the averages look as follows:

2012 – average pool of Republican voters interviewed was 590
2016 – average is 490

Of course, the margin of error of plus or minus 5 percent are common with such polls. Governor Kasich was predicted to get less than 5 percent. With a margin of error of 5 percent he could have gotten 10 percent but he actually got 16 percent.

In this case, the pollsters were vastly off, illustrating that sample size matters. In other words, does polling 490 Republicans justify the conclusions drawn by the authors?

[su_custom_gallery source=”media: 2470″ limit=”7″ link=”image” target=”blank” width=”529px” height=”322px” Title=”National Statistical Service of Australia – calculating sample size – it works” alt=”National Statistical Service of Australia – calculating sample size – it works”]

National Statistical Service, Australian Bureau of Statistics – Sample size calculator

[su_box title=”Polls are getting ever more costly thus end up with smaller samples” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. 2016 US Presidential Election

Pollsters now make 30-35 calls to complete a single interview.

Ten years ago, a pollster called ten people to get one to participate in the poll or study.

To interview 1,000 voters — only 400 of which may be likely to vote Republican — the pollster now has to dial up to 35,000 numbers.

2. UK polling

In the UK, today it takes about 15,000 phone calls to get 1,000 interviews.

About 20 years ago, 2,000 calls were needed to get 1,000 replies.


The above shows, depending upon the country, we may need to make anywhere from 10,000 to just about 40,000 calls to end up with a sample of 1,000 respondents.

But in the case of an election, we also need to make sure that those who answer are also those who will go cast their vote. This challenge is discussed below.

2. Selecting the wrong sample is a growing risk

Polls are conducted over two or three days, meaning we try to interview those who are more easily contacted. This may happen over the internet or via phone, but such work makes it hard to arrive at a sample surveyed that is representative of all voters.

[su_custom_gallery source=”media: 2469″ limit=”7″ link=”image” target=”blank” width=”528px” height=”317px” Title=”National Statistical Service of Australia – calculating sample size – Youth Barometer UBS” alt=”National Statistical Service of Australia – calculating sample size – Youth Barometer UBS”]

National Statistical Service of Australia – calculate your sample size required for the relative standard error you desire

For instance, May 2015 British election data indicate that opinion polls failed to reach the harder to find Tory voters. In turn, estimates of the vote share attributed to Labor were skewed.

According to the British Social Attitudes survey, Labour was six points ahead among respondents who answered the door on the first visit. However, looking at those that required three to six home visits to be interviewed, the Tories enjoyed an 11-point advantage. Adjusting for social class and age, first time respondents are less conservative. “Busy” respondents are more likely to be so – but harder to chase for pollsters.

Hence, for pollsters it is not easy to get those “busy” people that might vote for a particular party as outlined below. Another challenge is that more and more people no longer have a landline. They can only be reached by mobile number (see also FT mentioning Martin Boon, director of ICM research). Because of this trend, in the US polls are increasingly conducted using mobile phone numbers to call.

3. Getting likely voters is a challenge for pollsters

Selecting those that are likely to vote for your political poll (see graphic below) is important. Finally, the views of voters and nonvoters are often very different, as was the case in 2014 in the US mid-term elections.

[su_custom_gallery source=”media: 3274″ limit=”7″ link=”image” target=”blank” width=”757px” height=”394px” Title=”The Pew Research Center: distinguishing likely voters from nonvoters is critical for pollsters.” alt=”The Pew Research Center: distinguishing likely voters from nonvoters is critical for pollsters.” ]

Pew Research (2016-01-07). Can likely voter models be improved? Section 2: Measuring the likelihood to vote

As well, for the 2015 UK elections, under-30s generally lean left, but very often fail to turn out on polling day. The pollsters, however, reached an atypical group of youngsters, who were unusually engaged with politics and committed to voting.

Another factor that can bias polls is that they are often based on internet polls. These tend to use volunteers to sign up to online panels. They may be drawn at random to participate. Nevertheless, the underlying group of people is self-selecting. Thus, data collected using a sampling method known as random digit dialing or “RDD” results in better data sets (i.e. random-probability samples representative of the population).

As this shows, getting those to participate in a poll that are likely to vote continues to be a challenge.

Sign up for our newsletter. Get the next post about big data, business analytics via email

Bottom line

Here are three challenges pollsters will continue to grapple with and we voters must keep in mind when studying poll results.

[su_box title=”3 challenges for pollsters” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. Polls are expensive!

Ever fewer people want to participate in consumer surveys or election polls. Hence, it takes thousands of calls to get a useful sample. In turn, inferences can be made from such a sample about the election outcome.

But just the calls may take about 400 hours. In turn you get 1,000 replies and still have to pay for data analyses and a write-up. A costly exercise that is not getting cheaper.

2. Getting a representative data set is nearly impossible

As the 2015 UK election illustrated, it takes a greater effort to get Tory voters than others.

Cultural differences may also limit the applicability of Pew Research’s findings in the US whereby “…better-educated people tend to be more available and willing to do surveys than are those with less education.”

However, the current solution of using online non-probability survey panels makes polling results less accurate.

3. Identifying and polling likely voters is hard

It is tough to get accurate readings to predict election outcomes by using self-selected online panels.

YouGov knows this is a sensitive issue and does not publish any answers to research methodology questions on its blog. For instance, the voters over 70 who broke heavily for the Tories were not reflected in YouGov’s online panels used for predicting the 2015 UK elections.

Those under 30 years of age are less likely to cast their ballot. If your panel or poll includes those overeager millennials, who were unusually engaged with politics, your findings will be skewed.

Polling models will continue to be improved, but surprises will also continue to happen.

InterestingWhile Apps and Web respondents do not differ in their type of responses, the response rate is lower using mobile apps to collect data.


Nevertheless, polls are the best way of trying to figure out what the election result is going to be next time an election, primary or referendum comes around. So go and vote.

Join the 3,000+ organizations using the DrKPI Blog Benchmark to double reader comments in a few months while increasing social shares by 50 percent – register now!

Mr Sander’s foreign policy promise boils down to one thing: he will not start a foreign war. The rest is a replay of 1960s student radicalism. He intends to convert the US into a Scandinavian social democracy. Mr Trump would make America great again. He thinks the US has been screwed by its allies. If he needs expertise, since he has none, he will hire experts. In the meantime his solution is to “bomb the shit out of ISIS.” That is about it for his foreign policy.

Unfortunately, both Sanders and Trump are arch-fantasists with the ability to recruit voters. Both will spell extremism as far as US foreign policy is concerned. The system, dominated by the Democrats and Republicans, has always rejected the political extremes. The nation has benefited from a deep political stability, which has contributed greatly to its economic strength and global power. If America’s immunity to extremism is ending, the whole world will feel the consequences. Not a good prospect for anyone, is it?

This post: Yes Virginia, pollsters really are wrong

Join the conversation

  1. Do you have an example of a great poll / study?
  2. I refused participating in a poll / consumer survey last week. You?
  3. Do you think pollsters will get it right in the 2016 US elections?
  4. US Primaries: Clinton and Trump or Sanders and Trump? What do you think?

Of course, I will answer you in the comments. Guaranteed.

Update 2015-01-17: WEF Davos 2016: Top 100 CEO bloggers
Social media has been around for a while. Yes, various social network platforms have come and gone, like Bebo. Founded in 2006 to compete with Facebook, it was relaunched in December 2014, but has not really been heard from since.

Is social media for breakfast pictures or milestones?

Seems like a valid question, but this is also a time of growing digital fatigue.

We can share anything online. Unfortunately, this takes away opportunities for having a real conversation with a real person.

How does WEF Davos cope with its fans’ digital fatigue? Yes, WEF Davos continues to tweet furiously and madly update Facebook.

BUT, is anybody listening; do you care? We investigated this a bit further. See also:

WEF Davos in DrKPI’s Benchmark Test: Survey says…!
WEF Davos 2015: Top 100 CEO bloggers
Facebook, viral marketing or #wef15 – why benchmark?

We all know that one of most CEOs’ undoubted skills is burnishing their own profile

For starters, the World Economic Forum provides CEOs a great podium to push their pet projects.

For instance, in 2015 Eric Schmidt talked about his Google Search and #endtrafficking project. But just like the Google Flu Trends and earthquake monitoring fads, Mr Schmidt has now moved on from #endtrafficking.

Last year, Marissa Mayer from Yahoo even managed to curate an image as a fashionista, hanging out with Anna Wintour, the editor of American Vogue. Anna Wintour is definitely not in Davos this year. Marissa Mayer might make it to Davos, unless she is ousted by her board of directors beforehand.

But how well does WEF Davos foster dialogue? Does it go beyond those attending?

1. Houston, we have a problem

Klaus Schwab states in interviews that WEF Davos’ strength lies in bringing together varying opinions. In turn, he feels that one also needs to listen to those with different opinions.

With social media, this means you must monitor your blog’s commenting system carefully. While any comment should be reviewed before being published to avoid spam, this work should be done quickly.

Incidentally, WEF Davos rarely if ever gets a blog comment. This would suggest that its resonance with the public (i.e. readers) and delegates is not that wonderful. In such a situation we must monitor comments carefully, release them in a timely fashion, and most importantly, answer the questions raised by our readers.

But as the example below shows, there seems to be a bug in the system, which fails to publish reader input. The comment was left here: WEF provides the hashtags
[su_custom_gallery source=”media: 1294″ limit=”7″ link=”image” target=”blank” width=”530px” height=”750px” Title=”If you do not publish the few comments you get, are you engaging in the conversation or still following Web 1.0 = broadcasting?” alt=”If you do not publish the few comments you get, are you engaging in the conversation or still following Web 1.0 = broadcasting?”]

Mr Schwab talks about engaging and dialoguing often. I am pretty certain that he is very serious about this. Unfortunately, his social media and content marketing staff seem to fail him.

But DrKPI is ready to come to their rescue if they would just ask. Besides, this entry shows that WEF is failing the engagement challenge, but its staff seem to know that better than anyone else. That is to say, whenever they talk about their social media efforts, you would think everything is just peachy-keen.

Of course, arrogance or too much self-confidence is probably a form of ignorance… or vice versa?

Whatever it is, Davos 2016 must do better in this regard. Otherwise, the attention it gets through traditional media will not be carried further by the people – an important factor in getting things done after the meeting closes.

[su_box title=”3 things we learned from WEF Davos about Social Media” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

When these three things happen, be quick and fix the problem – or face the consequences. Whatever you do, DON’T continue in the same vein!

1. Comment ended up in the spam inbox and got lost, deleted or whatever: Possibly an explanation, BUT not an excuse!

2. You just ignore the comment: If you want dialogue, this is a non-starter!

3. You delete the comment: A HUGE no-no if you listen to Klaus Schwab, who wants to foster dialogue and have different opinions be heard!

Whatever the reason, none of the above is excusable. Surely, it stops any conversation from ever starting.

Experience: I left a comment (see above). It is still in the list of ‘to be moderated’ comments on the Disqus platform, which is what WEF uses.

Of course, any true blogger could have told the WEF folks, as I did a while back, that using Disqus is not a great move. That is, if you want to foster more dialogue.

But WEF probably does not know any better.


2. We all make mistakes, but failing to learn is plain stupid

For the last eight to ten years, experts have pointed out that in Web 2.0, broadcasting is out. Instead, engagement with your fans, clients, enemies, and so forth, is in.

“Mastering the Fourth Industrial Revolution” is the theme for WEF Davos 2016, in addition to other interesting and very important topics. The WEF has used social media extensively for quite a few years.

But has WEF continued fine-tuning its social media use effectively?

I began monitoring these efforts in 2008. They were quite good then, and got better by 2011, but since…

If you look at either the 2016 agenda pages or WEF events the comments are always closed when the blog entry is posted (see below).

Do we have dialogue here? Or is WEF Davos 2016 once again following the classic broadcasting model?
Tweet updates come fast and furious, you can barely keep up. Over the months leading up to this year’s meeting, no dialogue has happened. Neither on its blog about Davos, nor any other channel.

Is this how one overcomes the audience’s possible social media fatigue to foster engagement?
[su_custom_gallery source=”media: 2960″ limit=”7″ link=”image” target=”blank” width=”780px” height=”766px” Title=”#WEF #WEFdavos Why talk is cheap. #bigfail: Plenty of broadcasting but ZERO dialogue with and engagement from readers and attendees” alt=”#WEF #WEFdavos Why talk is cheap. #bigfail, plenty of broadcasting but ZERO dialogue with and engagement from readers and attendees”]

We are barely a week out from WEF Davos 2016. Unfortunately, people are not discussing the event. “How can that be?” I ask. Or is the theme too abstract and buzzword like for many of us?

Mind you, cities with few or no cars will certainly look very different from what we experience now, no matter what city we find ourselves in. So the theme chosen by the WEF Davos 2016 organisers seems an important one and timely as well.

So why such paltry resonance on its blogs and social networks? What might be going wrong?

[su_box title=”WEF Davos 2016: 3 simple checks that show the problem” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

It is not what you show or want me to believe (also called impression management). Instead the focus is on what you do. For instance:

– How well does WEF Davos manage to engage its readers on its blogs? Check: Reader Comments.

– Do your moderators do a good job and release the comments? Check: Cases I know – NOPE not done well.

– Do content authors / posters write replies that give readers added value? Maybe, but we cannot find ONE REPLY!

Incidentally, WEF Davos rarely if ever gets a blog comment. This would suggest that its resonance with the public (i.e. readers) and delegates is not that wonderful.

What do you think is the reason for this?

Does no one care?


Davos 2015 had a set of hashtags. For Davos 2016, 10 days before the event I still could not find a blog entry on WEF telling me what to use. Seems unfortunate.

What is your take?

– What do you do to fight the possible social media fatigue of your target audience?
– What would you recommend WEF Davos do to foster greater engagement on their conference blog?
– How do you deal with this data deluge from Davos via Facebook and Twitter accounts?
– What would you recommend to a novice (ropes to skip)?

Incidentally, if you look at the tweets this last weekend (January 9 and 10, 2016) and WEF’s Twitter account, there is no apparent strategy regarding content or getting people hyped for WEF Davos 2016. What is going on, is a robot posting? Just watching the updates go by makes me feel dizzy.

Data, and by extension data analytics, are becoming increasingly important for business. At the same time, the data deluge makes making sense of it all a bigger challenge every day.

Here are three trends to should keep in mind for 2016.

1. Don’t shoot from the hip

Numbers are becoming more popular for most people, but the more numbers we get the more useless most of these seem to be. Especially if they are drawn out of a hat; why would you take that into consideration in your decision-making process?

Polling your audiences is fine.
But that is not a statistic that adds up exactly to something like 97%, is it?
Or are you keeping tallies of your straw polls and then doing the statistics?

Comment by DrKPI on Adrian Dayton’s Clearview Social blog

[su_custom_gallery source=”media: 2884″ limit=”7″ link=”image” target=”blank” width=”792px” height=”473px” Title=”Adrian Dayton Clearview Social claims 93% of lawyers… based on straw polls – should we trust this, does it make a difference?” alt=”Adrian Dayton Clearview Social claims 93% of lawyers… based on straw polls – should we trust this, does it make a difference?”]

View slide on Flickr – measure-for-impact – DrKPI

Google Flu Trends is an example that illustrates this problem further. For instance, it:

– looks at historical data – descriptive analytics and research, and
– tries to predict what might happen – predictive analytics – with the help of a model that was developed.

The results are supposed to help us better understand how the flu will spread next winter. Unfortunately, in the Google flu trends versus National Institutes of Health (NIH) challenge, the winner is? NIH! Google estimates are simply far off from the actual data the NIH produces for policy makers and health professionals.

2. Bad data result in bad decisions

Publishing rankings or product tests is popular. Since some readers devour such rankings, publishers can sell more copies, which keeps advertisers happy.

A real win-win situation, right? Not so. Wrong decisions can result in outcomes that are not desirable. For instance, attending the wrong college or polluting more than the test results indicate (think Volkswagen and #dieselgate) is not something we want.

Lucy Kellaway felt so incensed about the ever growing acceptance of making errors in corporate circles, that she wrote:

…I would be exceedingly displeased to learn that the bankers to whom I was handing over a king’s ransom were being taught that errors were perfectly acceptable.

This mistake-loving nonsense is an export from Silicon Valley, where “fail fast and fail often” is what passes for wisdom. Errors have been elevated to such a level that to get something wrong is spoken of as more admirable than getting it right.

By collecting data and using flawed methods we produce rankings or test results that will can seriously hurt people. For instance, when drug certification tests are done improperly and the regulator has no idea, unknown side effects can kill people.

Using the wrong test results to approve or certify a car can result in dismal effects as well. Volkswagen is accused of manipulating tests, and the public got more pollution than it bargained for. VW is working on fixing the 11 million vehicles affected by the diesel cheat, but this will not un-do the damage to the firm’s reputation and our health.

3. Check before you trust the method used

It is always wise to take 5 minutes to do an acid test with any study report we see, such as:

– what does the methodology tell us (e.g., we asked university deans to rank their competitors); and

– does the measure or measures used make sense (e.g., one question about how university developed / improved study programs – result = ASU is more innovative than Stanford or MIT… who are you kidding?).

The Art Review publishes an annual ranking of the contemporary art world’s most influential figures. In short, it helps if you live in London or New York so the Art Review editors or journalists are aware of who you are.

I asked for an explanation of how these numbers develop:

Dear Sir or Madam
I would like to know more about the methodology you used for the ArtReview’s Power 100 List.
Can you help… this would be great to use with my students in a class.
I could not find anything on the website that I could show my students.
Professor Urs E. Gattiker, Ph.D.

14 days later I got an answer from the makers of the ranking:

Subject: Re: Message from user at

We are not following a grid of criteria per se, and the list emerges from a discussion between a panel of international contributors and editors of the magazine, who each advocate for the people they feel are most influential in their region. The influence of the selected people on the list is based on their accomplishments in the past 12 months. I have attached here the introduction to the Power 100, which might help you in defining our approach.
I hope that helps,
Best, Louise

A grid of criteria, what is that? Of course, the office clerk answering me has no clue about research methodology used, as the answer indicates. One could start believing that this Top Art list came from a discussion or using a straw poll. Totally chaotic approach.

You can view the attachment that explains this sloppy method below.

[embeddoc url=”” download=”all” viewer=”google”]

Download the ArtReview criteria with this link.

A friend of mine smiled, and said:

For me this is a great list, Urs. Those on the list rarely if ever represent value for money for serious art collectors. Instead you get buzz and have to pay for their image. The list tells me who we do not need to work with. We use other experts. These give us more value for money. They help us to complement our award-winning collection.

[su_custom_gallery source=”media: 2881″ limit=”7″ link=”image” target=”blank” width=”780px” height=”479″ Title=”Sound research takes plenty of resources” alt=”Sound research takes plenty of resources”]

Bottom line

We all know that data quality is important and frequently discussed. In fact, the trustworthiness of data directly relates to the value it can add to an organisation.

As the image above suggests, doing quality research takes a decent method that results in data that permits careful analysis. Sloppy data are cheap to get, but dangerous if used in decision-making. Such findings are neither replicable nor likely valid.

However, we are increasingly required to present findings in order to attract more readers. Some master this very well like Inc. Another example of theirs I came across was:

Though truly quantifying “best” is impossible, the approach Appelo’s team used makes sense, especially when you read the books that made the list.

The 100 Best Business Books of 2015 by Jeff Haden

And here’s the methodology:
The purpose of our work was to find out which people are globally the most popular management and leadership writers, in the English language.
Step 1: Top lists
With Google, we performed a lot of searches for “most popular management gurus”, “best leadership books”, “top management blogs”, “top leadership experts”, etc. This resulted in a collection of 36 different lists, containing gurus, books, and blogs. We aggregated the authors’ names into one big list of almost 800 people.
Step 2: Author profiles
Owing to time constraints, we limited ourselves to all authors who were mentioned more than once on the 36 lists (about 270 people), though we added a few dozen additional people that we really wanted to include in our exploration. For all 330 authors, we tried to find their personal websites, blogs, Twitter accounts, Wikipedia pages, Goodreads profiles, and Amazon author pages.

So you defer to 36 people and their lists and include those that are mentioned more than once. Fine, if that does then not include the ones you believe should be on the list because you read these books and liked them, no worries. You add a few dozen people (60) and voilà, you have 330 authors (how they ranked them is totally unclear, but interesting – blog reputation, Twitter followers, etc.).

[su_box title=”3 checks you should undertake before accepting a study’s findings” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]

1. Evidence-based management and policy advice

A sloppy method is like following no method.
Can you find a method section, and does the method make sense to you? For example, did the study use a long-form questionnaire to get employment data? Or was it just based on scans of Internet job boards? If the latter, the problem lies with double counting when relying on websites or job search engines.

If the method section does not instill you with confidence that it was done properly, watch out. And, most importantly, don’t complain about a study before you read it carefully!

Interesting read: CRDCN letter to Minister Clement – Census long-form questionnaire (July 9, 2010) explains why Statistics Canada needs to get the funds to collect data for the census to provide evidence-based policy data.

2. Minestrone: Great soup but wrong research method

So the study has a decent method section that makes sense and explains things accurately. What are the chances that somebody else could follow the methodology and get the same result?

To illustrate, if it was done the same way I put together a Minestrone (Italian vegetable soup), you can forget it. I take whatever vegetables are in season, plus, each family’s soup is seasoned differently, guaranteed. This neatly illustrates the fact that if no systematic method is used, it is not science. For the soup this means it turns out different each time anyone makes it.

Without a recipe or method followed, you cannot repeat the performance or generalise from your findings.

3. Buyer beware: Click biting studies using navel gazing metrics

Usage of Sainsbury’s #ChristmasIsForSharing being higher than John Lewis’ #ManOnTheMoon by just 4% is interesting. However, Social Bro’s verdict is based on 50 votes (26 versus 24) from a Twitter poll. In turn, the analytics company uses this data to decide on 2015’s Most Creative Christmas Campaigns. What? Are they real, is their analytics work also that sloppy?

Apparently, even analytics companies like Social Bro have to defer to such navel gazing metrics to get more traffic. Such samples are neither representative nor big enough to draw any inferences.

Just because something is interesting or suggests it is a bit better based on 3 more votes on Twitter, does not mean you should invest your hard earned cash that way. Investing your marketing dollar based on such nonsense is plain dumb.

What is your take?

– what will you change in your data #analytics and #analysis work in 2016?
– what is your favourite example for 20015, illustrating GREAT analytics work and research?
– how do you deal with this data deluge?
– what would you recommend to a novice (ropes to skip)?

More insights about analytics, analysis and big data.

Where do you want to go, reflection is needed.

[su_highlight background=”#fffe99″]Summary[/su_highlight]: David Cameron knows that public approval of RAF air strikes against ISIS in Syria has dropped.
We explain what this teaches Migros, Lidl and Tesco about new product research.

CLICK - CONFIDENCE in measuring ROI of social media and display ads is LOW

Some weeks ago I came across a report (see image) that stated just 29 percent of people feel confident in measuring the ROI (return on investment) of display ads and this drops to just 22 percent for social media marketing.

Accordingly, management is interested in improving its understanding with analyses and analytics when it comes to social media activities. But do managers or politicians understand what we are trying to communicate or convey to them?

If managers read blog entries like this one about how to do surveys, it’s no surprise that they believe it is all easy and cheap to do.

This is the fifth post in a series of entries about big data. Others so far are:

Data analytics: Lessons learned from Ebola
Scottish referendum: A false sense of precision?
– Facebook mood study: Why we should be worried!
– Secrets of analytics 1: UPS or Apple?

Confusion abounds

How are management or politicians supposed to understand the difference between analytics, data and analysis? Can we trust polls or should we learn from the Scottish disaster?

For instance, when we go to a dictionary of statistics and methodology from 1993 (Paul Vogt), neither analytics nor business analytics has an entry, never mind data analysis.

Kuhn: Unless we share a vocabulary, we are not a discipline

However, these days, some would claim data analytics is a science (e.g., Margaret Rouse). Still, if something can be called a science (e.g., physics or neuropsychology), its members share a certain set of beliefs, techniques and values (Gattiker 1990, p. 258).

Do people in data analytics or data analysis share a vocabulary and agree to the meaning of basic terms? Not that I am aware of. Therefore, Thomas Kuhn’s (1970) verdict would be: Not a science (yet).

In web analytics, data analytics or data science as well as social media marketing we agree to disagree. But maybe I can clarify some things.

Sign up for our newsletter; this post is the first in a series of entries on business analysis and analytics.

[su_box title=”2 things business, data, financial and web analytics have in common ” box_color=”#86bac5″ title_color=”#ffffff”]

1. All analytics is art that involves the methodical exploration of a set of data with emphasis on statistical analysis.

2. All analytics include the examination of qualitative and quantitative data.


Analytics gives you the numbers, but fails to provide you with insights. For that, we must move from analytics to analysis, and we only gain the necessary insights if we do the analysis correctly.

[su_custom_gallery source=”media: 2649″ limit=”7″ link=”image” target=”blank” width=”508px” height=”552px” Title=”Diagram: Analysis versus Analytics versus Data – why the difference matters” alt=”Diagram: Analysis versus Analytics versus Data – why the difference matters”]

The graphic above illustrates that proper data is the foundation for doing analytics that permit a thorough analysis. Accordingly, using a sample that is not representative of our potential clients or voters is risky.

Nobody would draw any conclusions about attendance at next season’s football matches by asking a sample of baseball afficionados. So, go ahead and ask your social media platform users to vote for this season’s favourite flavoured drink syrup. But such a poll won’t give you an answer that is representative of your customer base.

Nevertheless, this is exactly what Migros did in 2015 (see Migipedia – few very young users participated in the poll, less than 10 wrote a comment during January 2015). It then published a one-page ad (among many more, see below) in its weekly newspaper (e.g., November 30, 2015), claiming that the chai flavour was the winner.

Making such a decision based on this type of unrepresentative poll is a risky choice. You may actually choose to increase production of the wrong flavour!

[su_custom_gallery source=”media: 2781″ limit=”7″ link=”image” target=”blank” width=”520px” height=”293px” Title=”Polling online community members gives you data from a non-representative sample of your customers – is that good enough to launch a new product?” alt=”Polling online community members gives you data from a non-representative sample of your customers – is that good enough to launch a new product?”]

Collecting data that is based on a representative sample of your customers is a costly exercise.

So why not use your online ‘community’ to do a ‘quick and dirty’ poll?

Surely a Twitter, Facebook or website / corporate blog poll is economical. You do it fast and easy and voilà, you got what you need, right? NOT.

Okay agreed, doing the above will strengthen your hand with a CEO. They might not grasp basic methodology issues of sampling or survey research. Plus, you got data from your online community, which is another reason to invest more money there.

In the Migros example above, having an online poll on your Migipedia platform achieves 3 things:

1. it allows your marketing folks and community managers to show the platform is useful for something;

2. regardless of which flavour wins and gets produced, you can always push it in your company newspaper. This way you reach 3 million readers in Switzerland – a country that has 7.8 million inhabitants,

3. even if the new product turns out to be a flop, thanks to other marketing channels, you sell 150,000 to 300,000 (or more) 1-liter bottles of chai tea syrup during the Christmas Season.

With its many resources and varied marketing channels (e.g., weekly Migros Magazin), Migros can ‘afford’ to use shabby research. It is in the enviable position to succeed, in spite of ‘spending’ so much.

The company might never learn that its analysis actually led the team to choose the second or even third best choice. Nonetheless, your marketing clout ensures that you can show it to management as an example of having done the right thing. Of course, we know it was done for the wrong reasons, but since management probably won’t find out, who cares – right?

[su_custom_gallery source=”media: 2793″ limit=”7″ link=”image” target=”blank” width=”530px” height=”308px” Title=”Polling: Opinion on RAF air strikes against ISIS in Syria – up and down each week” alt=”Polling: Opinion on RAF air strikes against ISIS in Syria – up and down each week”]

One poll is worse than none

As the above image from last week regarding air strikes in Syria shows, poll results can change quite a bit within a week.

For starters, no pollster wanting to stay in business will use a non-representative sample to get opinions. Using such data is unlikely to give you the insights you need for Hillary Clinton or any other candidate to succeed during next year’s US election.

[su_custom_gallery source=”media: 2801″ limit=”7″ link=”image” target=”blank” width=”485px” height=”445px” Title=”Polling: YouGov’s Will Dahlgreen never answered this question – so can you trust these results?” alt=”Polling: YouGov’s Will Dahlgreen never answered this question – so can you trust these results?”]

I left the above comment at the end of the blog post (it has not been published by YouGov so far). I asked about things that a good pollster will always publish with the poll results.

For instance, I asked how data were collected, whether the sample is representative, and what the margin of error was. I could not find any information about any of that. Of course, trust is not improved when one fails to publish a reader comment that raises method issues about your poll.

“YouGov draws a sub-sample of the panel that is representative of British adults in terms of age, gender, social class and type of newspaper (upmarket, mid-market, red-top, no newspaper), and invites this sub-sample to complete a survey.”

How exactly this happens with YouGov we do not know, since the methodology outlined on its website is not very detailed.

But David Cameron knows that while 5 million people have joined the ranks of those opposed to airstrikes in Syria in the past seven days, that could change next week. Polls are more interesting when they show a trend, so Mr Cameron still has hope that the opposition even more.
[su_box title=”5 key pointers for explaining the analyst’s work to your management: The case of survey research or polling” box_color=”#86bac5″ title_color=”#ffffff”]

Collecting quality data is followed by analytics, which subsequently require analysis to draw the proper insights. Analysis requires words in addition to looking at the numbers.

To tackle this challenge successfully, we need to do some preparation, as outlined below.

1. Do you have a strategy or a plan?

What is it you want to collect data for and why? This must be explained in a few sentences.

How will these data help you win the election, get the contract or sell more product?

2. How will data help you execute the plan?

You must know what data you need or the rationale for wanting them (see point 1).

What three steps will you take in the next quarter or six months to execute your strategy?

3. Are the numbers complete?

Most monitoring services can tell you everything about Facebook or Twitter.

But what about smaller websites from climate change activist groups, ISIS sympathesizers or peace activitists’ blogs?

Make sure you get the data you need. Is your sample representative of those whose opinion you must know?

4. Do you need social media monitoring?

Knowing what people say about your brand or company is a good thing. The Volkswagen emission scandal (remember #dieselgate) teaches us that in a crisis, simply monitoring the flood of tweets and status updates on Facebook or LinkedIn is of little use.

Like Volkswagen, you can decide to ignore the social media noise. Change your behaviour and communicate openly and directly (click for German-language radio report).

Unless you use social media monitoring to take action after the data are in, why collect it?

5. Do you have data from your customers?

If you have less than 1,000 employees, don’t make a big fuss about social media monitoring.

Focus on things that matter, such as what your clients report regarding warranty service, and the quality of phone support or user manuals. A tweet matters little.

Feedback can be collected in many ways, including customer surveys, discussions with clients or comments on your corporate blog.

Analysing these data provides insights that help improve product, service and so forth.

What it means

Focus on collecting data that help you serve your customers better. Getting a daily digest about the most important key words regarding your brand (e.g., we use DrKPI, #DrKPI, DrKPI BlogRank, #metrics #socbiz) is probably all you need. Instant data may not be needed unless you are a FT Global 500 company.

Restrict yourself to collecting only those data you absolutely and definitely must have.

Make sure that they meet some minimum quality standards. Only this will enable you to trust the analytics and analysis resulting from that work.

Actionable metrics are what matters

Unreliable or invalid data from clients, social media monitoring and opinion polls is a waste of resources.

Please keep in mind, just collecting data without taking action is a navel-gazing exercise.


Bottom line

Always ensure that analytics leads to analysis that goes beyond navel-gazing metrics. Answer these questions truthfully:

A. What will be done with the findings: Unless you take action based on your data, why measure and collect information at all?

B. What kind of data was collected: Make sure you understand how data were collected. Can this polling data be trusted to be representative of the population (e.g., consumers in my country)?

How was something like influence (e.g., Klout) measured (what kind of proxy measure was used)?

If it is not transparent to you, move on and do not waste your time with such a measure or index.

Keep points A and B in mind before you collect data and / or use somebody else’s findings.

‘Total X’ combines xyz Labs’ proprietary Rambo social media measurement tool, and WalkBack®, the leading measurement source of WOM marketing from the Sambo Group, a Laughing Stock company.

Okay, what does the above mean? Who would want to trust this gobbledygook? If marketers or pollsters cannot explain things clearly and precisely, they tend to cover it up in jargon that tells you nothing.

Regardless, 2016 will mark the year where Lidl, Migros and Tesco will do more of these utterly useless polls, to find another ‘winner’ for a new flavour of drink syrup, mustard or soft drink.

Even though social media, community and marketing managers will claim a victory this year, with so much additional marketing around, who is surprised? Put differently, regardless which syrup the company – Migros – would have produced, I dare to claim it would have flown off the shelf anyway.

Combine all the ads and marketing push, and if it tastes okay, success is in the bag. Unfortunately, those that hate research will attribute part of this success to a useless online poll.

Next time you read something like the above, claiming to rank something, check the methodology. Cannot find anything? Just move on because it is probably hogwash.

Interesting reading

Vogt, Paul W. (1993). Dictionary of statistics and methodology. Newbury Park, CA: Sage Publications. For information see (5th edition 2016).

2 great reading lists for additional resources about research, polls, survey data and much more:


Join the conversation

  1. Do you have an example of a great poll / study?
  2. What is your favourite marketing measure?
  3. What research methodology would you recommend?
  4. Other ideas or concerns you have about marketing research, please state it here.

Of course, I will answer you in the comments. Guaranteed.

As one of the curators of the and corporate blog databases, I see plenty of corporate blogs.

I probably study 20 or more corporate blogs closely each week; maybe another 30 a little less closely.

Recently an advertising CEO from New York asked me the following:

“What are the 5 things that irk you the most when you read ad agency blogs?”

Tough question, but I promised him I would take a stab at it.

Here is my summary of the many social media audits we have done this year on agency blogs for our clients.

What large ad agencies can teach us bloggers

[su_box title=”1. Navel gazing content is out: Your content must focus on your client, NOT you.” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 700″ ]
Nobody wants to listen if all you do is talk about yourself (e.g., products).

Address my concerns or my problems and point out a possible solution (e.g., Checklist: 5 ways to increase sales).

How to do it right:

[su_custom_gallery source=”media: 2682″ limit=”7″ link=”image” target=”blank” width=”520px” class=”alignleft” height=”387px” Title=”Burson-Marsteller – We preach dialogue, but reserve the right to do different on our own corporate blog.” alt=”Burson-Marsteller – We preach dialogue, but reserve the right to do different on our own corporate blog.”]

Burson-Marsteller does many things right, including having content that adds value.

[su_box title=”2. Don’t waste my time: Content that adds value is the only content that matters.” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 520″ ]

After reading your blog entry, how does your reader feel?

Was it worth spending time on?

Did they learn something new?

As this blog below shows, it takes effort to present content that is relevant for your clients.

[su_custom_gallery source=”media: 2668″ limit=”7″ link=”image” target=”blank” width=”520px” class=”alignleft” height=”269px” Title=”Bernet Blog – established it rarely if ever talks about itself – no navel gazing here ” alt=”Bernet Blog – established it rarely if ever talks about itself – no navel gazing here”]

It is hard but the Bernet blog illustrates that ad agencies can do it very well (see above).

Burson-Marsteller also violates point 3 below, by turning off reader comments. That puzzles me. Anybody know why Burson-Marsteller does not want to hear from its blog readers?

[su_box title=”3. Engage and dialogue with me: Do not add insult to injury.” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”520px” ]What we tell our clients and charge for dearly is one thing. What we do on our website is a different story.

If you preach engagement, why would you turn off commenting?

Do you not want to hear from your readers, clients or future customers?

How Landor – part of WPP – does it

Landor talks about engaging, debating and so forth on its blog (see above screenshot).

But all it does is boradcasting, so Web 1.0 – where does it listen, share, discuss and learn? Talk is cheap…

[su_custom_gallery source=”media: 2680″ limit=”7″ link=”image” target=”blank” width=”520px” class=”alignleft” height=”446px” Title=”Landor – Do you want dialogue when you turn off commenting on your blog?” alt=”Landor – Do you want dialogue when you turn off commenting on your blog?”]

We all know that fostering dialogue takes time, of course.

[su_custom_gallery source=”media: 2670″ limit=”7″ link=”image” target=”blank” width=”520px” class=”alignleft” height=”305px” Title=”DrKPI Agency tries to maintain dialogue with its readers, but it’s not easy.” alt=”DrKPI Agency tries to maintain dialogue with its readers, but it’s not easy.”]

Naturally, the above illustrates that not every blog entry gets as much dialogue and social sharing as the next.

Large agencies that span the globe have another challenge to master.

Harnessing economies of scale is great, but boring the local audience with “soft” news does not communicate professionalism.

[su_box title=”4. Do not bore me: Being global is nice, but act locally please.” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”520px” ]

Some believe all you need is a local website.

Once it is up, just report on the last conference you attended, such as the Cannes Lions Festival (a yearly event for self-agrandizing of the ad industry).

Also maybe mention the last charity you helped and show a foto of the last group of interns you took on.

All news that puts you in the right light.

Nevertheless, does such inward looking help your client solve their own challenges?

It seems it is not easy to write content relevant to your local audience when you are part of a large worldwide ad-agency. A case in point is Ogilvy South Africa.

[su_custom_gallery source=”media: 2687″ limit=”7″ link=”image” target=”blank” width=”520px” class=”alignleft” height=”514px” Title=”Ogilvy South Africa – which client cares about a Cannes Lions blog entry – does that add value?” alt=”Ogilvy South Africa – which client cares about a Cannes Lions blog entry – does that add value?”]

Blog entries like those about the Cannes Lions conference cannot be of great interest to a South African client, can they?

Maybe it justifies the author’s junk trip to France from Johannesburg. Anything else?

[su_box title=”5. Read my lips, dialogue is in, broadcasting is out: Houston, we have a problem.” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”520px” ]

Ad agencies advise their clients about which Web 2.0 strategies could result in success.

But why do so many ad agency blogs (i.e. more than 86.9 percent) fail to get their readers to leave comments and join the dialogue?

How can you advise your client and charge for it when you yourself fail to do it right?


Not so easy, as the Deep Edition Digital PR’s blog illustrates. Nevertheless, if you want to be considered an expert and get paid for your advice, should you not do better than those paying you because you supposedly know?

[su_custom_gallery source=”media: 2669″ limit=”7″ link=”image” target=”blank” width=”520px” class=”alignleft” height=”395px” Title=”Deep Edition Digital PR Sweden” alt=”Deep Edition Digital PR Sweden”]

Discuss these issues with us!

Have you come across these issues as well?

Share your viewpoint. I would love to start a dialogue with you.

How do you decide whether or not your ad agency is competent in social media? What criteria do you use?

Bottom line

Next time you look for an ad agency that can help you with social media, check out these three things before giving them the job.

[su_box title=”3 things that tell you if your ad agency is up to social media.” box_color=”#86bac5″ title_color=”#ffffff” radius=”5″ width=”px 520″ ]
1. Navel gazing content is out: Your ad agency’s blog content must focus on your needs, not theirs, in order to provide you with value.

2. Engage and dialogue: Does your agency preach or act accordingly by allowing and encouraging readers to leave comments?

3. Houston, does my ad agency foster dialogue? If their blog gets reader comments, do they write thoughtful replies, or…?


CLICK - DrKPI for improving college marketingIt is again the time of year when parents and prospective students pore over recently published university rankings.

“…US News asked top college officials to identify institutions in their Best Colleges ranking category that are making the most innovative improvements in terms of curriculum, faculty, students, campus life, technology or facilities.”

But should we recommend such rankings, like the one above from the US News & World Report?  Are college rankings good, bad or ugly as Yale’s former Dean of Admissions suggests?

Or could it be the single worst advice we could possibly give a high school student?

Fact 1: College rankings generate revenue for publishers

Media houses know very well that university rankings are of great interest to prospective students and parents. So, newspapers like the Financial Times (FT) feature a weekly special section on education. In addition, the paper publishes numerous rankings throughout the year.

Then there are the various feature reports (see 2015-11-03 FT Special Report on Innovations in Education). Of course, they carry also advertising like the one below from Thunderbird.

[su_custom_gallery source=”media: 2517″ limit=”7″ link=”image” target=”blank” width=”519px” height=”380px” alt=”FT Special Report Innovations in Education – Thunderbird promotes itself as innovative leader – half page – front page color ad $120,000″]

Looking at the FT Special Reports Ad rates shows that getting involved with educational institutions pays well for media houses. Universities are forced to get increasingly famous to attract more resources and qualified students. In turn, advertising in special editions about education is a sure way to reach more of your target audience.

Marketing 101

Publishing college rankings makes sense from a publisher’s perspective. Advertising brings in the revenue needed (FT Special Reports Ad rates), and people read the stuff because as Langville and Meyer (2012, p. 1) suggest:

In America, especially, we are evaluation-obsessed, which thereby makes us ranking-obsessed given the close relationship between ranking and evaluation.

Fact 2: Schools love to use rankings

Everyone certainly loves rankings when they place in the top 10. And regardless of whether we agree with the findings, if we are the top dog, we let the whole world know about it.

The great thing is such rankings are based on a third party’s opinion, which lends it all credibility when we advertise our achievement.  Arizona State University (ASU) continues to tout their top rankings in the US News & World Report list of most innovative schools.

[su_custom_gallery source=”media: 2514″ limit=”7″ link=”image” target=”blank” width=”519px” height=”381px” alt=”US News & World Report – Most innovative school ranking 2015 – an advertising bonanza”]

Marketing 101

You don’t have to be brilliant or innovative, you just have to convince others that you are. Of course, if you have an external reference point that ranks you highly, such as a well-known publication, so much the better for your recruiters.

Can prospective students trust these school rankings?
Are they useful when choosing a university/college or program of study?

Fact 3: This stuff is less useful than you think

It is best to look at the methodology used in a ranking. What measures were used to conclude that ASU should be considered more innovative than Stanford and MIT? Fair question – let’s see.

US News & World Report asked deans and presidents to rank their peers. The magazine wants to compare apples with apples. Hence, national schools such as ASU, Stanford and MIT are ranked with their peers.

By the way, did you know that the US News & World Report puts the United States Naval Academy into the category of national liberal arts colleges?

So how does one measure the innovativeness of a university? We are told:

“…2015 survey that received the most nominations by top college officials for being the most innovative institutions. They are ranked in descending order based on the number of nominations they received. A school had to receive seven or more nominations to be listed.”

In plain English, this means you need to get as many high level university administrators as possible to nominate your university for innovation.

Accordingly, if you manage to make everyone perceive you as innovative, you are. That is all there is to it. Isn’t that wonderful?

Of course, we have no idea if whether a product innovation or a process innovation helped you rank highly. In either case, to claim to have made an invention, and thereby become an innovative university, you should answer things like: Why is this curriculum change an invention? They can be evaluated according to:

  • novelty (new),
  • inventiveness (i.e. must involve a non-obvious inventive step), and
  • industrial applicability (can be used).

Of course, in this case we have no clue what makes a curriculum change a simple change and what makes it an invention.

Marketing 101

US News & World Report rankings illustrate very well that how you measure things matters little. It just has to come across as making sense because 90 percent of readers do not bother to read about your methods or the fine print.

However, if you invest several years of your life in attending a school, while paying through the nose for tuition, fees and so on, you are well-advised to ensure the ranking makes sense to you.

Of course, even if the measure is bad, this does not necessarily mean ASU and Stanford are bad schools. They’re great, but

the US News & World Report’s attempts to measure innovativeness is a useless vanity exercise, to put it politely.

Fact 4: Using just one ranking is the worst

You basically have to do the homework. The five points spelled out in the table below will help you make better sense out of any ranking.

Please keep in mind – the perfect ranking does not exist. Each one has strengths and weaknesses, but you can only learn what those are by following these steps.

[su_box title=”5 critical things to do before trusting a college ranking.” box_color=”#86bac5″ title_color=”#ffffff”]

1. Take the time and make the effort to learn about the methodology. Where is the description, and how thorough is the ranking we are looking at? An example of a good method section is PEW Research‘s study on multiracial Americans, which explains how data were collected, weaknesses of the study, etc. This is also easy for the uninitiated to understand.

If you have done this homework, you know better how much weight you should give the rankings in front of you. That is a great start.

2. Does the study measure what it is supposed to (also called validity)? What criteria were used to make up a component in the ranking? Do these make sense to you?

3. Are there components of the ranking that particularly interest you? We may look at costs as an important factor. It could be interesting to understand how, for instance, a university degree (e.g., undergraduate or graduate) affects one’s career prospects and / or income 10 years after we graduate.

3 very good examples of included interesting factors:

4. Come up with a set of criteria that are important to you (see also image below), such as:

4.1 – location (e.g., which country and what area of the country/city), and
4.2 – costs (e.g., tuition, fees, health insurance, accommodation).

5. Write down a set of criteria that are not that critical to you, such as:

5.1 – GPA of incoming class,
5.2 – number and value of student scholarships, and
5.3 – diversity of faculty (e.g., gender, race, country and language)

The above makes it clear that using just one ranking is plain stupid. Using two is risky and using three or more allows you to pick and choose, thereby empowering you to make the decision that best suits you.

If the ranking uses those criteria that are of limited importantce to you (see point 5), you know what to do – ignore it.


[su_custom_gallery source=”media: 2536″ limit=”7″ link=”image” target=”blank” width=”520px” height=”414px” alt=”Balancing the worth of education with outcomes.”]

One must balance the resources put in and the outcomes we hope for. This also indicates that we need to look at several rankings to choose the right university.

Who is number 1?  Create the best ranking

Of course, in addition to US News & World Report and the Financial Times, others do not want to be left out of this lucrative business. For instance, The Economist (a weekly magazine) also produces a ranking of MBA programs. So does the Wall Street Journal. Of course, even more rankings exist, such as the best 100 Employers to Work For or the Best Consulting Companies (German-language Handelsblatt).

In the case of the Best Consulting Companies, participants are asked three questions about the firm and voilà, we have the 2015 rankings. This may indicate more about how much you advertise (helps increase brand recognition) than how satisfied your clients are with your work.

These examples illustrate, everybody and anybody can create a college ranking. However, to avoid becoming a laughingstock, I urge you to follow the nine steps outlined below.

Join the 3,000+ organizations using the DrKPI Blog Benchmark to double reader comments in a few months while increasing social shares by 50 percent - register now!

How exact and thorough we are when addressing each step will, in turn, affect the overall quality of our rankings.

[su_box title=”9 steps to develop your favorite ranking system for just about anything.” box_color=”#86bac5″ title_color=”#ffffff”]

1. Write a one-page summary of why this ranking is needed and explain its purpose (to help readers… lose weight, pass the certification exam, purchase the best car, etc.).

2. What can readers do with these data? For example, does studying these data help improve performance? Does it show one’s weaknesses? Does it outline how one can improve (see DrKPI BlogRank)?

3. Come up with some indicators or measures that allow the collection of data from individuals (e.g., salary three years after graduation), the institution (e.g., faculty with doctorate), and possibly other indicators (e.g., inflation rate, purchasing power parity (PPP) data from the International Monetary Fund (IMF) to adjust salaries).

4. Use the indicators to make up components that make sense to the uninitiated (e.g., career progress, quality of faculty).

5. Add up the indicators to attain the overall score for each component the school, firm or student achieved.

FT uses three indicators to make up the “idea generation” component of its MBA rankings.

  • percentage of faculty with doctorates,
  • number of doctoral students that graduated last three years, and
  • research output created using a set of 45 journals (no Chinese or Spanish research journals need apply).

6. Convert the component scale to a common one such as 0 to 100, whereby the best gets the top score and average performers hover around 50.

7. Determine the importance of each component.

In many cases, some components are weighted higher than others. That is a value judgment that warrants an explanation. The same goes if you weigh each component the same! Explain your decision to the uninitiated reader.

8. Compute the aggregate score as the weighted sum of the previously calculated scaled component scores.

9. Present the aggregate score from the desired scale, such as 0 to 100.

Thanks to Fung (2013, p. 22-23) for inspiring me to write up this list.

Whenever looking at a university or any other ranking, keep the above in mind. Is the methodology spelled out, explaining the issues raised above? If these things are not made transparent, caution is called for.


Don’t forget: Subscribe to our newsletter!

What is your take?

What’s your favorite ranking (e.g., sports) AND why do you like it?

– Which university ranking did you use when you applied for college?

What do you like the most about rankings?

What advice would you give a high school student regarding college rankings?

FT Global MBA Ranking

As I pointed out above, each ranking has something we might be able to use for our own purposes. The one below shows which business school provides you with the best value (i.e. current income minus tuition, books, lost wages while attending the program, etc.).

Surprising, is it not? The best known schools rank low. But maybe you want to use different criteria to rank… Check it out yourself.

[su_custom_gallery source=”media: 2552″ limit=”7″ link=”image” target=”blank” width=”497px” height=”650px” alt=”Financial Times Global MBA Ranking – Value for money”]

FT Global MBA Ranking – the winner based on value is the University of Cape Town – Graduate School of Business.

Things worth reading

1. Fung, Kaiser (2013). Number Sense. How to use big data to your advantage. New York: McGraw-Hill. Available on

2. Kenrick, Douglas, T. (September 30, 2014). When statistics are seriously sexy. Sex, lies and big data. Psychology Today online. Retrieved November 2, 2015 from

3. Kenrick, Douglas, T. (June 20, 2012). Sexy statistics: What’s the one best question to predict casual sex? The science of sex, beer and enduring love. Psychology Today online. Retrieved November 3, 2015 from

4. Langeville, Amy N. & Meyer, Carl D. (2012). Who’s #1? The science of rating and ranking. Princeton, NJ: Princeton University Press. Available from

5. Rudder, Christian (September 2015). Dataclysm: Love, sex, race, and identity – what our online lives tell us about our offline selves. New York: Broadway Books. Available on

6. Stake, Jeffrey Evans and Alexeev, Michael (October 30, 2014). Who Responds to U.S. News & World Report’s Law School Rankings? Indiana University School of Law-Bloomington Legal Studies Research Paper No. 55. Available at SSRN:

Single worst advice – the answer

After reading this blog entry, it is obvious that using a single ranking is not smart.
Use a few and be aware of each one’s weaknesses and strengths.

Choose the component that helps you the most. If by any chance two rankings use the same component (e.g., salary), compare the numbers and smile.

Nothing is perfect. And since you read all the way to the end, why not write a comment and subscribe to our newsletter?

Download PDF file: How to save advertising dollars on Facebook and YouTube.
2015-09-28 Update thanks to Rubén Cuevas,

Fake views of ads by "bots" cost advertisers more than $6.3 billion US globally during 2015.

Data show, video fraud-detection on DailyMotion, vimeo, YouTube and others fails to filter out invalid traffic properly.
Here I distill our knowledge into 3 takeaways.

Check out what Sir Martin Sorrell WPP has to say about the matter.
According to Media Rating Council (MRC) and IAB (Interactive Advertising Bureau) standards, a viewable impression of a digital ad occurs when 50 percent of an ad’s pixels are on screen for one second.

In December 2014 Google published data regarding display ads in browsers (desktop and mobile). The study revealed that 56 percent of the display ads it served on its own and others’ sites never appeared within view on someone’s screen.

Nobody really knows for sure how Google or any other video platform or ad server come up with these numbers. For instance, Google provides explanations of what one should look for in these numbers it serves advertisers about their ads. How it collects them is, however, not explained.

1. What is the challenge?

The US Association of National Advertisers (ANA) released a report in December 2014, which estimated that

  • 23 percent of video ads, and
  • 11 percent of display ads

are viewed by “bots”. These are computer programs that mimic the behaviour of an Internet user.

The ANA estimated that this would cost advertisers about $6.3 billion US globally in 2015. This is a concern for two reasons.

1. Adertisers are spending ever larger amounts of money across both display and video advertising (see graphic below), and

[su_custom_gallery source=”media: 2336″ limit=”7″ link=”image” target=”blank” width=”530px” height=”310px”]

2. Spending for video ads is estimated to grow 21.9 percent compound annually from 2015 to 2020 (US data) (see also online video celebrities – chart below).

[su_custom_gallery source=”media: 2335″ limit=”7″ link=”image” target=”blank” width=”530px” height=”284px”]

2. Google and Facebook want a larger slice

Google and its YouTube platform want to garner the largest share possible of this growth in video advertising. Nonetheless, the competition will surely want to prevent this.

In April 2015, Facebook boasted it had over 4 billion video views each day. This number continues to grow.

For now, YouTube data suggest many more videos are viewed daily on its video platform than on Facebook.

For Google, display and video ads create tons of cash for the company, but things are changing. For instance, the rate for pay-per-click ads has been dropping (view chart as shown below). Google explains this was lower rates on YouTube than its other platforms.

[su_custom_gallery source=”media: 2334″ limit=”7″ link=”image” target=”blank” width=”530px” height=”288px”]

Join the 3,000+ organizations using the DrKPI Blog Benchmark to double reader comments in a few months while increasing social shares by 50 percent - register now!

Some suggest that in the US, millenials spend nearly 60 percent of their time watching movies on either a smartphone, tablet or desktop/laptop.

To keep advertisers pouring more money into video ads, however, Google and Facebook have to up their game. Accordingly, both must provide strong evidence that their fraud detection systems work. Until fraud detection works, three things must be addressed as outlined below.

3. Focus on not getting charged for invalid video views

Each video platform wants to charge advertisers for video ads according to whatever the market will bear. In turn, advertisers want to keep costs for ads down, but this is becoming a challenge.

Apparently, some companies offer tens of thousands of YouTube views for as little as $5 US. Such data could in part help explain why 23 percent of video ads are viewed by fake consumers.

Of course, no advertiser wants to pay for these “views”.

How does one avoid paying for fraudulent views?

That is difficult to say, because…

Filtering invalid traffic before advertisers are ever charged is not getting easier.

Recent research sheds light on this important issue. Researchers uploaded two videos to each of five video platforms (YouTube, DailyMotion,, TV UOL and Vimeo).

They bought ads on these platforms, which targeted the videos they had previously uploaded. Then, they directed their “bots” to these videos.

What are bots?

[su_box title=”EXPLANATION: What are bots?” box_color=”#86bac5″ title_color=”#ffffff”]

Bots are used by DrKPI, Google and Qwant Search to crawl the web.

They are little programs that allow DrKPI  to collect data about blog entries (e.g., text, data of blog entry, etc.).

Google uses bots to index webpages. Bots can also be computer programs that mimic the behaviour of internet users viewing, e.g., a video ad.

About 60% of internet traffic is due to bots.[/su_box]

Each platform’s two videos were visited by the bots about 150 times. The researchers explain in their paper that the bots used were far from sophisticated tools as cyber criminals might use. Nevertheless, the results are worrisome for advertisers.

[su_custom_gallery source=”media: 2324″ limit=”7″ link=”image” target=”blank” width=”530px” height=”319px”]

If detection mechanisms work properly, marketers do not have to pay for ads on YouTube viewed by robots.

Data show that YouTube seems to have the best fraud-detection mechanism of the five platforms tested. It was followed by DailyMotion.

YouTube’s fraud detection tool identified 25 of the 150 bot visits to a video as real users viewing the video.

This means in 16.67 percent of cases, YouTube wrongfully identified a bot or robot to be a human watching a video.

♥ Curious? Join 1,500+ other subscribers to this blog’s newsletter and read on!

What is most unsettling, however, is that Google charged the researchers for 90 of the registered fake views. This is a 60.67 percent error rate!

So what is the bigger problem:

1. That YouTube wrongfully identified 25 robot “views” to be humans out of 150 times the video ads were “seen” = 16.67 percent error rate, or

2. Google AdWords, instead of charging for the 16.67 percent of views wrongly identified as humans by YouTube, deciding to charge for 90 views done by robots =  a 60.67 percent error or false positive rate?

How could YouTube’s false positive-rate be so inflated? The process of counting views (i.e. public view counter and number of counted and monetized views) is opaque on YouTube.

Thanks to Rubén Cuevas for pointing out: “YouTube has two different mechanisms in place to discount views for the:

public view counter, and also the
monetised view counter”

Important is here to understand as Rubén pointed out to me, the public view counter seems to be more strict in the detection of fake views.

This is to say YouTube increases the count, and therefore, charges the advertiser for even more fake views than the public view counter would suggest.

[su_custom_gallery source=”media: 2376″ limit=”7″ link=”image” target=”blank” width=”443px” height=”242px”]

Read the research findings in detail:

Marciel, Miriam; Cuevas, Ruben; Banchs, Albert; Gonzales, Roberto; Traverso, Stefano; Ahmed, Mohamed and Azcorra, Arturo (July 2015). Understanding the detection of fake view fraud in Video Content Portals. Retrieved September 23, 2015 from

Check out the FT article for non geeks, including comments by the researchers left here:

Cookson, Robert (September 23, 2015). Google charges marketers for ads on YouTube even when viewed by robots. Financial Times, p. 1. Retrieved, September 23, 2015 from

[su_box title=”3 takeaways: Focus on verifiability of video views to fight off deception.” box_color=”#86bac5″ title_color=”#ffffff”]

1. Better process transparency for fraud detection
Having over 15 percent of bot views identified as “real” is a high error rate. While this is bad, YouTube is better than the rest.

YouTube seems to use a sufficiently discriminative fake view detection mechanism, but this applies only to the public view counter.

For the monetized view counter (i.e. those for which advertisers get billed), YouTube seems to ignore this mechanism for discounting fake views (see section 3 above – verifiability).

This is, of course, unacceptable for advertisers. Moreover, it makes the process of how YouTube detects these deceptors totally intransparent for advertisers.

Bottom line: With the help of third party verification, this challenge should be resolved quickly.

2. Improve measurement and use a set of standardized metrics
Even with third parties verifying numbers for advertisers, if our KPIs (key performance indicators) are not comparable we are stuck. For instance, Facebook defines a “view” as someone watching a video for three seconds or more. Others like YouTube talk about around 30 seconds before counting.

These different standards make it difficult for advertisers to get a clear feel and comparable numbers across platforms. Thus, even focusing on numbers, as Google suggests, is of limited value.

Bottom line: Define and agree upon the metrics used by the advertising industry. Make them comparable across social networks and video platforms.

3. Establish third party collecting, verifying and auditing of numbers
Facebook has followed the practice of self-reporting on viewability of ads, pages, reader engagement, and so forth. But as Volskwagen’s #dieselgate shows, self-reporting is always vulnerable to misuse, sloppiness and abuse of the system.

Bottom line: We need third party collecting and verification of numbers. Such efforts must in part focus on minimising charges for advertisers when ads are viewed by robots.

Eliminating fraud in online advertising is key

You are supposed to count the actual number of measured views of a video ad. Ergo, filter out invalid traffic from bots.

In December 2014, the ANA/White Ops study identified 23 percent of video ad impressions as bot fraud. Combine that number with the results from data reported here, and this means:

Google AdWords takes at least 60.67 percent of the 23 percent bot fraud views on YouTube and charges advertisers for them.

Thus, it follows that advertisers pay for at least 14 percent of video ads not viewed by humans!

The lack of transparency, standardized metrics and a regular audit of how video platforms handle fake ad views costs advertisers dearly.

Accurate metrics matter. For the first time ever worldwide mobile advertising will overtake print in 2016 ($71 billion US versus print shrinking to $68 billon US).

As well, social media advertising will top $25 billion US this year. Facebook is expected to take the biggest slice, more than $16 billion US. Instagram will account for “just” $600 million US.

Advertisers are justifiably wary and suspicious. Based on the above predictions, we better make sure that we pay only for those imprints, views, etc., that were executed by humans and not robots. Will #GoogleAW2015 tell us more about how YouTube plans to address this issue? Not really.

Download the checklist as a PDF (320KB file).

Interesting read

a) More content about advertising and viral content
b) Google: hidden ad costs
c) IAB’s efforts to establish a more trustworthy supply chain
d) YouTube frozen views
e) YouTube search for counted views – zero information provided
f) Facebook partners with Moat to verify video ad metrics

Benchmark and test your blog – for free – right now

What is your opinion?

Now that you have read “Epic fail: Video view fraud detection“, I would like to ask you a question or two.

– As an advertiser, how do you deal with this issue? Please share!
– What type of video advertising works best for your business?
– What do you know about Facebook’s handling of this challenge?

More about advertising fraud

Receive email updates on the latest
Strategy / Marketing / Trend research.