In a typical term paper assignment, faculty ask students to review the literature, synthesize their findings and write a cohesive narrative about a particular topic. They expect students to find the most important research on the subject and determine what the general scientific consensus is, taking into account any disagreements. By the time most students get to their senior year in college, most appear to do an okay job of this.
But do the faculty follow their own guidelines when writing up their own research? A recent study in the journal Ecosphere suggests that researchers aren’t always finding, reading or critically analyzing the original and rebuttal papers.
Banobi, Branch and Hilborn (2011) selected 7 high profile papers originally published in Science or Nature, all of which had at least one rebuttal published. The authors identified papers that cited the original article or the rebuttal and then analyzed:
- Number of citations to the original paper vs. citations to the rebuttal,
- How well the citing paper agreed with the original paper or the rebuttal (and whether this changed after the publication of the rebuttal)
- Whether citations to the original paper decreased over time
After correcting for the effects of self-citation, their results are remarkable:
- Original papers were cited 17 times more than the rebuttals.
- They found a lot of papers that cited only the original paper, and 95% of these accepted the original at face value
- Only about 5% of the citations to the original papers were critical (at all) of the original article.
- Some papers cited the original and the rebuttals as though they both supported the same position!
Why is this happening?
Benobi, et al. suggest that:
This confirms our intuitive sense that most authors, except the relative few that are writing and citing rebuttals, tend to accept a paper’s conclusions uncritically.
Additionally, we can wonder if the authors have really read all of the papers they cite (something suggested by Simkin and Roychowdhury 2003) or found all of the relevant research (suggested by Robinson and Goodman (2010), my discussion here)
The authors suggest that original articles and rebuttals need to be better linked in our information retrieval systems, something that I’ve touched on earlier. But a lack of such system tools does not absolve the authors of their responsibility to find relevant earlier work. Good keyword searches will often easily turn up the rebuttal papers, and citation searching (available for free on Google Scholar if you don’t have Web of Science or Scopus) should be required!
We may also need to examine the possibility that some researchers are just as guilty as their students of not finding and reading the relevant literature.
Cited:
Banobi, J., Branch, T., & Hilborn, R. (2011). Do rebuttals affect future science? Ecosphere, 2 (3) DOI: 10.1890/ES10-00142.1
Robinson, K. A., & Goodman, S. N. (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Annals of internal medicine, 154(1), 50-5. DOI: 10.1059/0003-4819-154-1-201101040-00007.
Simkin, M. V., & Roychowdhury, V. P. (2002). Read before you cite! Complex Systems, 14, 269-272. Retrieved July 15, 2011, from http://arxiv.org/abs/cond-mat/0212043.
Note: Hat tip to Richard P. Grant who posted a link to the Banobi et al. article on Google+.
the Undergraduate Science Librarian 