In a typical term paper assignment, faculty ask students to review the literature, synthesize their findings and write a cohesive narrative about a particular topic. They expect students to find the most important research on the subject and determine what the general scientific consensus is, taking into account any disagreements. By the time most students get to their senior year in college, most appear to do an okay job of this.
But do the faculty follow their own guidelines when writing up their own research? A recent study in the journal Ecosphere suggests that researchers aren’t always finding, reading or critically analyzing the original and rebuttal papers.
Banobi, Branch and Hilborn (2011) selected 7 high profile papers originally published in Science or Nature, all of which had at least one rebuttal published. The authors identified papers that cited the original article or the rebuttal and then analyzed:
- Number of citations to the original paper vs. citations to the rebuttal,
- How well the citing paper agreed with the original paper or the rebuttal (and whether this changed after the publication of the rebuttal)
- Whether citations to the original paper decreased over time
After correcting for the effects of self-citation, their results are remarkable:
- Original papers were cited 17 times more than the rebuttals.
- They found a lot of papers that cited only the original paper, and 95% of these accepted the original at face value
- Only about 5% of the citations to the original papers were critical (at all) of the original article.
- Some papers cited the original and the rebuttals as though they both supported the same position!
Why is this happening?
Benobi, et al. suggest that:
This confirms our intuitive sense that most authors, except the relative few that are writing and citing rebuttals, tend to accept a paper’s conclusions uncritically.
Additionally, we can wonder if the authors have really read all of the papers they cite (something suggested by Simkin and Roychowdhury 2003) or found all of the relevant research (suggested by Robinson and Goodman (2010), my discussion here)
The authors suggest that original articles and rebuttals need to be better linked in our information retrieval systems, something that I’ve touched on earlier. But a lack of such system tools does not absolve the authors of their responsibility to find relevant earlier work. Good keyword searches will often easily turn up the rebuttal papers, and citation searching (available for free on Google Scholar if you don’t have Web of Science or Scopus) should be required!
We may also need to examine the possibility that some researchers are just as guilty as their students of not finding and reading the relevant literature.
Cited:
Banobi, J., Branch, T., & Hilborn, R. (2011). Do rebuttals affect future science? Ecosphere, 2 (3) DOI: 10.1890/ES10-00142.1
Robinson, K. A., & Goodman, S. N. (2011). A systematic examination of the citation of prior research in reports of randomized, controlled trials. Annals of internal medicine, 154(1), 50-5. DOI: 10.1059/0003-4819-154-1-201101040-00007.
Simkin, M. V., & Roychowdhury, V. P. (2002). Read before you cite! Complex Systems, 14, 269-272. Retrieved July 15, 2011, from http://arxiv.org/abs/cond-mat/0212043.
Note: Hat tip to Richard P. Grant who posted a link to the Banobi et al. article on Google+.
the Undergraduate Science Librarian 
If you read both but don’t agree with the rebuttal, do you need to mention / cite it?
That’s a really good question. Do researchers have a responsibility to acknowledge the controversy?
I quite agreed with the position posted. Most faculty behave like that.
I think you should take a more critical eye towards the article you’re referencing before quoting it like fact. While its conclusions may coincide with your own opinion, the paper itself is of questionable quality. Allow me to (ironically) rebut a bit of what that paper claims.
Some of the statements in the Banobi, Branch & Hillborn paper are contentious, and the stark absence of discussion of the article’s authors’ own biases does throw the credibility of the work into question.
I do not see them drawing from all of science for this study. Rather, they are drawing from a rather narrow field that the authors themselves are a part of (marine biology, particularly related to the sustainability of fisheries and magnitude of ecological damage from humans to marine populations). In fact, it appears the authors of this article have a particular axe to grind, as they take one side of a major issue in their field which apparently goes against the grain. I gather this from a perusal of the literature they used for their analysis and their references section.
Take a look at their Table 1. Seven original articles and 24 rebuttals. Banobi, Branch & Hillborn themselves account for 7/24 of the rebuttals, and a number of those rebuttals are actually in the same articles. This article appears to me as an outsider to be an attempt to justify their own work in their field by saying they aren’t getting the attention they deserve in their rebuttals.
This study would have had a lot more meaning if they had done something more comprehensive. It would have been less questionable if they had defined in the text that they were looking at papers from their field, expressed their own potential for biases and controlled with analyses of papers from other fields.
In short, I don’t think this says much about the science community at large. It says much more about themselves and their field to me.
Oh, let me add that the discussion does touch on the fact that they’re talking about ecology fisheries policy especially, but I feel it did a disservice by not being explicit about the nature of the articles they were assessing, their personal relationships to them and by making sweeping generalizations based solely on a look at their highly contentious field.
While this paper was definately limited in the field that it chose to explore, other studies looking at why folks cite (or don’t cite) seemingly relevant papers have discovered similar oversights in other fields (see the Robinson and Goodman paper mentioned above as an example). I don’t think we can say that citation problems are limited to just one field.
We would like to believe that we cite papers based purely on their relevance to the topic at hand, and that we have done our homework in finding everything that is relevant. But papers are cited based on a whole host of factors, including personal relationships and accessibility of the article.