Potatoes are Good For You,
No We Mean Bad For You,
No We Mean Good For You
Our parents were well-meaning, and our doctors are usually well-meaning, but both have sometimes held beliefs that are unfounded. For instance, for 50 years ulcer patients were prescribed a bland diet originated by a Dr. Sippy and based only on the seemingly logical belief that strong seasoning of almost any kind would inflame ulcers. When a study was finally done, the belief was found to be false.
Even when a study is done, it can be misleading:
(1) The study may be sponsored by an organization that will benefit from the conclusion (e.g. "Chocolate is good for you", the conclusion in a paper on the effects of chocolate. There was no mention of the fact that the study and the paper were sponsored by ----- a candy company);
(2) The study may involve only a small number of subjects, or it may fail to consider an important characteristic (e.g. elements of lifestyle);
(3) The study may be poorly controlled, or it may not be ““double-blind”, or the population studied may be quite specialized;
(4) The conclusion may not be strongly supported by the facts, or a medication may be touted as “more effective” because one, rare, condition responds better to it than to other medications;
(5) A claim may not be based on a study at all, but rather on assumptions (e.g. the Sippy Diet) or by specious “logic” (e.g. a proposed approach is rejected out of hand as necessarily invalid because it contradicts currently-accepted theory);
(6) The conclusion trumpeted in news media may be only preliminary, that is, based on a study that is still underway or that used a statistically insignificant sample;
(7) Some of the data may be erroneous or may have been analyzed incorrectly;
(8) Results may actually have been falsified (e.g. the first “cold fusion” experiments and the first claimed human cloning);
(9) The writer of a newspaper article that summarizes the study results may have misunderstood them (reporters are rarely experts in the fields on which they report);
(10) You may not have understood the conclusion, for instance because the article was not written clearly. And finally:
(11) The headline in a newspaper or even in a medical journal may not accurately represent the conclusions of those who conducted the study (for example, a headline in a 2006 issue of a prestigious medical journal actually stated a conclusion opposite to that of the study).
As long ago as 1986, The Journal of the American Medical Association devoted much of one issue entirely to discussions of the shortcomings in reports of medical studies. Among the weaknesses they listed were:
¨ Failure to disclose that the conclusion does not represent the opinion of all the people who conducted the study;
¨ Failure to disclose limitations in the methods used in the study, which might call the conclusions into question;
¨ Failure to disclose that some data contradicted the report's conclusions;
¨ Failure to disclose that the author has a financial connection to a company that will benefit from the conclusions being reported (this is most frequently found in studies of the effectiveness of new medications).
According to one of the authors:
"... There seems to be no study too fragmented, no hypothesis too trivial, no literature citation too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print."
Drummond Rennie, MD in JAMA, 1986
“… an unbiased reader, roaming at random through a medical library, would find in abundance all the problems I described in 1986”.
Drummond Rennie, MD in JAMA, June 5, 2002
What to do
☼ Read past the headline. Sometimes the article actually says the opposite of what the headline states.
☼ Don’t assume that a TV news item or a newspaper article is accurate.
☼ Ask who sponsored a study that seems to indicate that a particular food or medicine or practice is beneficial or harmful.
☼ If the conclusion is important, find details; the Internet is an excellent source of such information, but be sure you go to the actual source.
☼ Don’t let one study cause you to do something that contradicts your common sense. Before making a significant change in your diet or lifestyle, wait for confirming studies.
The next time you read a newspaper article, or see a television news item, that tells of a new study showing something or other about what we should eat, how our ailments should be treated, or how we should live in general, don't take it as the last word, and don't take it as even the latest reliable word.
W. A. Shapiro