1. Evaluating a Paper (Part 3): Community Response
Last week we talked about an aspect of quality checks on paper publication that occur before the paper is published and available to the public. We focused on peer review, which is a system that enables other people with expertise in the particular field of the research article to offer feedback and suggest ways to make the conclusions of the article more compelling. One of the major goals of the peer review system is to help ensure that published research papers have a basic level of scientific quality, that there aren’t major flaws with the methods that were used or the way the data was interpreted or the logic that ties the data interpretation to the conclusion. However, as we will see, there are certainly papers that turn out to have major problems after they’ve already been published. Today, we’re going to talk about ways in which the scientific community as a whole, as opposed to a small group of hand-picked experts, has the opportunity to offer feedback on the quality of published articles.

2. “We need to get past the antiquated idea that the singular act of publication – or publication in a particular journal – should signal for all eternity that a paper is valid, let alone important.”
Michael Eisen
(Professor of Molecular
and Cellular Biology,
UC Berkeley)
The point of the discussion today, is really that the scientific debate or conversation about particular research results shouldn’t end when the paper is published. It’s really the community's responsibility to think critically about the results that have been published and not to accept the published word as simple fact. A quote that really gets this concept across, is this quote from Dr. Michael Eisen at UC Berkely, which says. . .So what he’s saying is that, simply being published, even in a fancy journal, doesn’t mean that something is true, and we need to continue to evaluate the evidence, even after publication.

3. Revisions/new experiments
Community response
Publish
Write paper
More
experiments
This ties directly into our figure that we’ve been using to represent the expanded scientific process. And here, we’re talking about this part of the diagram, where the community responds to the published results and, based on that response, corrections may be made to the article, more experiments may be done to confirm the results, or in extreme cases, the paper may actually be retracted.
Retract
Corrections
Peer review

4. Community response Science blogs Technical comments Follow-up papers
We’re going to talk about three different ways in which the scientific community can respond to a publication, and we’re going to take the time to look in depth at an example of each of these with respect to the arsenic bacterium paper that we’ve been using as an example for the past couple of weeks.

5. Science blogs Varying levels of technicality. Not peer-reviewed.
Or is it? (comments)
Rapid response to published articles.
The first type of community response that we’re going to look at is the least technical of these three – the science blog. The science blog has many of the same advantages and disadvantages that a secondary non-technical article has in comparison with a research article. They tend to be lower in technicality than technical comments or follow up full-scale research papers, but they can still be a bit technical, as we’ll see in the example we’re going to look at today. Because blog entries are not peer-reviewed, it is important to take their claims with a grain of salt (although as we’ve seen, this is important even with published articles) and although they’re not formally peer reviewed, often the comments left on a blog post can help a non-specialist to judge whether their claims are sound. Often, science blogs are run by active researchers in the field that they are commenting on, making them a good source of information about the topic. Another advantage to using blogging as a community response mechanism is that it is a much faster method for communication than formal submission of scientific critiques to a journal.
Today, we’re going to look at a blog post that was posted only two days after the arsenic bacterium paper was published in Science. Over the course of just a few days, it generated hundreds of comments, both agreeing with and critiquing its analysis. Your task is to read the post (not including the comments unless you have extra time), and answer the questions on the handout. Dr. Facciotti and I will be circulating to respond to questions, and we’ll have a short class discussion afterwards. The answers to the handout are due at the end of class today.

6. Technical comments More technical/formal Focus on specific problem
Short
Don’t present new data
Ok, going into deeper levels of technicality for community response, we have the technical comment, which is a much more formal level of criticism than a blog post and is submitted as a “note” to the published article. Technical comments generally focus on a specific problem with methodology or data interpretation. They tend to be quite short, and generally don’t present novel data, so would not qualify as an independent research article.
With your group, you’re going to look at a technical comment for the arsenic bacterium paper, and answer a series of questions about it, as you just did with the blog entry

7. Follow-up papers Longer than technical comments Present new data
May support or refute original paper
The last example of community response that we’re going to discuss is the follow-up paper, which is a full research article which presents new data, either in support of or refutation of the original article. These may or may not be published in the original journal, whereas technical comments tend to be published in the same journal.
The example follow-up paper for the arsenic bacterium article that you’re going to look at is concerned with the same problem that the technical comment was, but proposes a different solution and offers several pieces of experimental evidence as support for their proposed solution
[After group analysis of follow-up paper] Your homework this week will be to locate and analyze one additional blog post and one additional technical comment about the arsenic bacterium paper and answer the questions of the handout provided on SmartSite.
Now that we’ve had a chance to look at some mechanisms by which the scientific community as a whole can comment on and critique published research articles, we can appreciate that its not always the case that a published article represents sound science. The example we’ve been looking at is a particularly good example of what the community as a whole considers to be bad science, and I’ve used it because its very easy to find a large amount of criticism about this particular paper and a small number of other high publicity papers. It’s important to remember, though, that the quality of a paper isn’t binary, it’s not either good or bad. There may be small problems that only affect the interpretation of one piece of data, and don’t disqualify the overall results, and so a paper may not get as much community response, but it may still be important for a researcher working in that particular field to be able to notice these questionable pieces of data. So you should always be using these critical assessment skills that we’ve been talking about the last few weeks.

8. Journal rankings, what do they really mean?
Impact factor = # citations / time
A common misconception that people often have is that the quality of an individual research article correlates highly with the flashiness or popularity of the journal its published in, meaning that the big name journals like Science, Nature, and Cell are going to have, on the average, much higher quality articles than less high impact journals. This isn’t necessarily true, and we’re going to talk about why.
When we talked about the process an article goes through before it can be published, we mentioned that sometimes an editor will reject a paper without even sending it out for peer review. We said that the two main reasons this could happen are that the science is so bad, the editor (who is usually not an expert in the particular field) can tell, or that the editor doesn’t think the results are “flashy” enough or important enough to be published in their journal. This rejection without review, usually for the second reason, because the results aren’t flashy enough, is more common the more popular or high-impact the journal is. The reason for this, is that what a journal’s ranking is based on, called its impact factor, is the number of times, on average, each of its articles are cited in another paper over a specific time frame. More “flashy” or broad-appeal articles are generally referenced much more often than more specialized papers that may only be read by researchers in a very limited field. So by rejecting papers that they don’t think will be cited very often, an editor of a big-name journal keeps their impact factor high.
A downside of this, of course, is that if research is flashy enough, it can sometimes be published in a high-impact factor journal with major flaws in its analysis, as we’ve seen with the arsenic bacterium paper.

9. Different publishing models
High impact – based on perceived importance of results (eg. Science, Nature).
Specific sub-field – based on fit with specific scope of journal (eg. Extremophiles).
Scientific soundness – based solely on quality of research (eg. PLoS One).
There are, of course, other models for publishing, other than only accepting papers that seem like they would be cited a large number of times. For example, many journals are much more limited in scope to a specific research area than are journals like Science and Nature, and these journals, which still considering the potential impact of a paper, may place more emphasis on the subject of the article fitting their journal’s scope.
Alternatively, there are publishing models which focus solely on the scientific quality, rather than the potential impact of a piece of research, meaning that articles are accepted any time they are judged to be scientifically sound, regardless of how important their conclusions may be. Supporters of this publishing mode emphasize that preventing scientific results from being published keep them from being accessible by others in the scientific community. Even if there are only a small number of other researchers in the field, they need to be able to communicate their findings to each other.

10. Questions?

11. Class discussion of vulpinic acid paper
I’d like to use the last few minutes of class to discuss your thoughts on the paper you looked at last week, that claimed to have found a link between a particular molecule derived from a lichen in killing cancer cells.
First, let’s take a vote, based on your peer review comments, how many groups decided to accept the paper as is? Request minor revisions? Major revisions? Reject outright? (Hopefully, most asked for major revisions or rejected). [ Class discussion about what types of revisions they would need before paper was publishable.]

12. This paper was actually part of an experiment designed to test the peer review process at a number of different journals. It was specifically written so as to be a terrible paper, and should have been rejected outright by any careful reviewer, or likely even by the editor of the journal in many cases. Part of your homework for this week is to read the article “Who’s afraid of peer review” that discusses the results from this study along with a follow-up blog post which criticizes some of the methods used in the study.

13. Homework
Read one technical comment and one blog post associated with the Wolfe-Simon paper. Use these comments to answer questions on handout.
Read “Who’s afraid of peer review?” and associated blog post (