Data and Statistics Without Source Information Are Useless

As I was formulating my 2011 business goals for Dow Brook Advisory Services earlier this week, I wondered how many people actually bother to create, record, and monitor goals. Not vague New Years resolutions – real, specific goals. So I did some quick, non-exhaustive research, which consisted of a Google Search query. Here is what I found:

  • 80 percent of people never set goals for themselves
  • Of the 20% of the population that does set goals, roughly 70 percent fail to achieve the goals they have set

SOURCE: http://www.myarticlearchive.com/articles/7/189.htm

  • Of the 20% that do write down goals, only 20% regularly review them

SOURCE: http://smallbusiness.yahoo.com.au/Article/Setting_goals

  • 3 percent of the American population set goals consistently, and are among the wealthiest people in the country

SOURCE: http://www.ehow.com/how_4582821_set-worthwhile-attainable-goals.html

This last item led me to another, more detailed collection of goal setting statistics generated by Harvard University:

“There was a study done at Harvard between 1979 and 1989. Graduates of the MBA program were asked “Have you set clear written goals for your future and made plans to accomplish them?” The results of that question were:

1. Only 3% had written goals and plans
2. 13% had goals but not in writing
3. 84% had no specific goals at all

10 years later Harvard interviewed the members of that class again and found:

1. The 13% who had goals but not in writing were earning on average twice as much as the 84% of those who had no goals at all
2. The 3% who had clear, written goals were earning on average 10 times as much as the other 97% of graduates all together. The only difference between the groups is the clarity of the goals they had for themselves”

SOURCE: http://www.betternetworker.com/forums/viewtopic.php?f=2&t=23456

While the author of the article, Keith Aul, does not provide a source citation for the Harvard study, further quick research revealed that is was discussed in the widely-read book written by Mark McCormack, titled What They Don’t Teach You at Harvard Business School.

I checked a copy of McCormack’s book out of my local library to see for myself how he presents the survey results and cites its source. McCormack provides neither an index nor a bibliography. I manually scanned the entire book (256 pages) to find the passage. I was unsuccessful on the first scan, so I repeated the effort, but at a slower pace and with greater care. Same result.

I had been looking at the original edition of McCormack’s book, published in 1984. I thought that perhaps there might be a later edition that contained the reference to the Harvard study. Some additional research showed that there were indeed subsequent editions. The book was reissued in 1986, 1989, and 1994.

I went back and reread Aul’s quotation of the Harvard study results and noted that the second set of statistics are from a follow-up study supposedly done in 1989. So there is no way that the study could have been quoted in the 1984 or 1986 editions of McCormack’s book, despite the legion of attributions made on the Web. It is possible (but unlikely) that the Harvard study was cited in the 1989 edition of What They Don’t Teach You at Harvard Business School. There is a higher probability that it was included in the 1994 edition. I was unable to find a copy of either edition to peruse, but apparently they were both small printing runs, as they are not available on Amazon.com. Therefore, I do not believe that the Harvard study information is contained in the 1989 or 1994 edition of McCormack’s bo0k either. (Please let me know if I am wrong.)

There is a good blog post, The Harvard Written Goal Study: Fact or Fiction, that chronicles the unsuccessful effort the post author, Sid Savara, made to find a copy of the study on the Web. Yes, unsuccessful. Savara could not find a copy of any such study via Web research. Nor could I. Not even Google Scholar had a clear reference to the research.

Things get even worse. Savara also discusses a similar, apparently fictitious study conducted at Yale University in 1953, the existence of which was denied by Yale in the following FAQ on its Web site.

“It has been determined that no “goals study” of the Class of 1953 actually occurred. In recent years, we have received a number of requests for information on a reported study based on a survey administered to the Class of 1953 in their senior year and a follow-up study conducted ten years later. This study has been described as how one’s goals at graduation related to success and annual incomes achieved during the period.

The secretary of the Class of 1953, who had served in that capacity for many years, did not know of [the study], nor did any of the fellow class members he questioned. In addition, a number of Yale administrators were consulted and the records of various offices were examined in an effort to document the reported study. There was no relevant record, nor did anyone recall the purported study of the Class of 1953, or any other class.”

It seems that both the Harvard and Yale studies, which are frequently trotted out as evidence for the power of setting and writing down goals, are in fact non-existent. Urban legends.

As if that is not shocking enough, note that all of the other goal setting statistics that I cited at the beginning of this post had no source attribution in the blog post from which I pulled them! They were prefaced by meaningless phrases such as “studies show” or statistics prove”.

I originally intended this to be a post about the power of goal setting, but, in light of the discoveries made during my research on the topic, I have decided to make a different point:

It is very difficult to use statistics to support an argument when the data is of unknown origin and its validity cannot be proven.

The amount of data which is generated and made available for analysis is growing exponentially. Yet, that data and the related statistical insight derived from it is utterly useless if the source is not properly documented and cited. Keep that in mind the next time you hear a business colleague, partner, or supplier present a statistic to justify their position.

I would be remiss to close this post without presenting a solution to the problem to which I’ve drawn attention. There are three things that each of us can do to ensure the credibility of the data we make available and the results of its analysis:

  1. Always document the source of any data when it is generated, so those analyzing it can provide a proper source citation
  2. Never generate statistics from a data set without knowing its source
  3. Never cite statistics that are not accompanied by a definitive source citation

If we do those three things, we will be able to have confidence in the data and statistics we are using to make business decisions. If we do not do them, then we are only fooling ourselves when presenting or consuming raw data or “polished” analysis.

About these ads

7 responses to “Data and Statistics Without Source Information Are Useless

  1. Esteban Kolsky

    very nicely done. well said. we can talk alter about the difficulty of obtaining primary research source data, and quoting it. but i agree with you on any secondary research need.

  2. Thanks, Esteban! You’re right about the dearth of primary research source data.

  3. Pingback: Tweets that mention Data and Statistics Without Source Information Are Useless | Together, We Can! -- Topsy.com

  4. Sébastien Wiertz

    Good Point.
    Sources for data are of course important to be mentioned and validated before making any further stats and researches.
    However, it’s also true for blog posts where pictures are taken from Flickr without mentioning the sources ( and therefore work attributions) not to mention people who copy entirely a paragraph from a book without indicating again the source.
    It seems that in the 2.0 world a lot of people were not educated to mention sources like we used to do it in the past.

  5. Larry, excellent point, and a great example of the search being more profitable than the find. It is truly shocking that the goal-setting statistics, attached to the names of prestigious universities, can’t be verified AT ALL. Nice detective work and a great reminder that in this age of so much information, we need to be careful about the data we’re citing and publicizing.

  6. @Sébastien: Thanks for your comment, with which I completely agree. Back in the day, we would have been accused of plagarism in school had we taken the liberties that many do now with other people’s content. Internet publishing has made it too easy to copy and use someone else’s work without proper attribution.

    Another result of internet publishing: We talk quite a lot about information overload. What many do not realize is that the amount of misinformation out there has risen with the volume of published content. That is a bigger problem than information overload, IMO.

  7. @Sadalit: Thanks for reading and commenting. It’s nice to hear from you. Hope you’re well!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s