Reviewer number two

Everyone has stories about reviewer number three, but what about the thoughtful and thorough reviewers?

Go to the profile of Jon Marles-Wright
Jan 21, 2016
Upvote 13 Comment

Almost everything we write and publish as scientists is peer reviewed at some point, whether by a colleague or friend at the early stages of writing, or by reviewer number three who thinks your ideas are fatally flawed and that you have the writing style of a small child. At its best, peer review keeps us honest by helping to point out flaws in our ideas, it addresses methodological weaknesses in our results and curbs over interpretation and wild conclusions. Peer review usually fails when it gets personal, when the reviewer is no longer dispassionate and attacks the scientist, or the work, because of vested interests. Whatever the content of the reviews, I always feel a sense of dread when an email comes into my inbox saying that I have comments to respond to.

Every scientist you speak to has a view on the merits of peer review and has a story about some outrageous thing reviewer three once said about their paper or grant. This kind of review is easy to deal with, we get cross, make a joke and then get over it. Easy reviews, where the reviewer requests a few typographical changes are simple to deal with too. Although, sometimes there’s a sense of being short-changed if you’ve waited months for someone to come back and say that your paper is essentially fine, when you have been working in the interim trying to pre-empt gaps that a thorough reviewer might spot in your data.

I recently received a couple of sets of reviews fall into a third category: the thoughtful and helpful review. The first set was for a paper and each review clearly showed that the reviewer had taken the time to read my paper and had written thoughtful comments to make my arguments better and my paper stronger. As well as being great reviews, each one was signed, and they were written by people I have a huge amount of respect for. With this level of engagement from my reviewers I really wanted to address each of their points and try to fit their ideas into my paper. This is impossible when constrained by page limits and the short time given to return the finished paper; I also wanted to keep some of my own ideas in the paper!

The second set of reviews effectively killed a grant proposal with ‘very good’ scores. These scores would usually leave most scientists in anguish; having accepted the fact that the grant would likely be passed over by the panel and reading the actual comments, I was pleased by the fact that the reviewers had both read the proposal and saw potential in it. It was a pretty left-field proposal and the reviewers wanted to see more preliminary data to support our methodology. Whichever way you look at a set of reviews like that, the reviewers were clearly performing their duty to make sure the research funds go to the best proposals. Rather than being deterred by these reviews, my collaborator and I are going to get those results and pitch the proposal again because we’re excited about its potential.

Reading and responding to review comments doesn’t seem to get easier, I think we just get better at dealing with the sense that someone else is picking our work to pieces. The supportive but tough reviews are probably hardest to deal with though, because fair criticism is the hardest type to take. I’ve shared a couple of my recent experiences, what about your experiences of fair and thoughtful reviewers?

Go to the profile of Jon Marles-Wright

Jon Marles-Wright

Chancellor's Fellow, The University of Edinburgh

I'm a structural biochemist using synthetic biology tools to understand protein complexes and metabolism in bacteria and archaea. I work in the Centre for Synthetic and Systems Biology and Institute for Quantitative Biology, Biochemistry, and Biotechnology at The University of Edinburgh.


Go to the profile of Nonia Pariente
Nonia Pariente over 1 year ago

As an editor, fair and thoughtful reviewers are my favorite (and they are on my "use again" list!).; it is nice to see papers grow through peer-review. As an ex-author, I understand your angst perfectly. Thanks for posting!

Go to the profile of Mark Mandel
Mark Mandel over 1 year ago

You are right that the outrageous reviews stand out, but I have been fortunate to have received many thoughtful reviews (including a fresh rejection!). One stands out. On my first submission in grad school, the manuscript was rejected and a reviewer suggested a particular variation on a key experiment. We did it and the result changed the conclusions of the paper. I was grateful to have not published something that was not correct. (We thanked the reviewer in the acknowledgments once the paper was accepted.) I try to keep this in mind, both as an author and as a reviewer.

Go to the profile of Andrew Jermy
Andrew Jermy over 1 year ago

Nice post Jon! We've all been there and I can certainly agree that a thoughtful and thorough referee report is the most helpful but often quite challenging to respond to.

Something to bear in mind that I have learned from seeing under the bonnet of the peer review process over the past 8 years is that for most manuscripts undergoing peer review, the referees will have different expertise which may in part explain why there can seem to be quite left-field comments from one ref. For instance, for the sorts of multi-disciplinary work that we increasingly see coming across our desks, there may be one or two researchers in the world that could happily cover all aspects of a paper, and they will likely be either on the author list or a direct competitor and excluded. So we use a panel of refs with complementary expertise to assess the technical aspects of the work as well as the biological story, depth of insight and novelty. We do weigh this into the decision as well - so if a technical ref says that a study is technically correct but not novel, and the biology ref says it is novel, then we would be more likely to overrule the technical ref.

Go to the profile of Jon Marles-Wright
Jon Marles-Wright over 1 year ago

Thanks for the comments! Your story is great Mark. It's interesting to hear the editors' perspective on this too.

What's the view on balancing out submission versus reviewing? Do people try to balance reviews received against reviews submitted?

Go to the profile of Andrew Jermy
Andrew Jermy over 1 year ago

The systems works through payment in kind (although some are now experimenting with just payment for refereeing - maybe a topic for a future post), as such yes I do think it is reasonable to consider trying to achieve something close to a balance in reviews given vs received.

One thing that I would encourage people to consider is if they have specific and quite rare technical or biological expertise then trying to make sure that the referee assignments that that accept use that knowledge. I have had several researchers over the past couple of years saying that they have become overwhelmed by requests and so limit their activities to journals on which they are editorial board members or are from societies of which they are members. This is of course completely within their prerogative, however having discussed with them the papers that they end up refereeing, the vast majority do not use their specific technical or biological expertise. This is a pain for editors, trying to make sure that we have all of our bases covered with referee expertise, but I also wonder whether it best serves the field in the long run.

Go to the profile of Claudio Nunes-Alves
Claudio Nunes-Alves over 1 year ago

I echo the comments from Andrew and Nonia, I also find the fair and thoughtful reviews the most useful. However, I must admit that for some of the articles that I handle (mostly Reviews and some Perspectives), those detailed referees' comments can be somewhat tricky to manage, because as you so nicely put it, I understand that the authors are willing to address referees' comments but they also want to keep their "own ideas in the paper!" And even though this can be challenging, I do think that most articles that I handle improve with the peer review process, which is one of the things that I like the most about my role, working with authors and referees trying to achieve a nice balance between allowing the authors to express their views and making sure that our articles provide an accurate and balanced representation of the field.

Go to the profile of Michael Chao
Michael Chao over 1 year ago

Late reply, but I agree that the thoughtful reviewers are the ones that really drive research forward. I also had a paper in grad school where a suggested experiment changed one of our conclusions, but rather than put a big, fat reject on the work when we resubmitted, the reviewer looked at the whole of the work, appreciated that we took the time to do the experiment and make the science more robust and still supported publication because it didn't impact the central finding.

I think that's an important lesson to take away, especially in how editors approach referee reports: we have to be open minded enough to accept some limitations and flaws in otherwise impactful work, and also to be fair to the community and request that the science be up to scratch to support the major conclusions.