The Commons Science and Technology Committee of the UK Parliament has announced it is to conduct a review of Peer Review. We take a look at what they have promised to review and wonder what it might mean for communicating science.
Here’s the opening paragraph of the announcement:
“The Committee has today launched an inquiry into peer review. The committee invites evidence on the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication.” Dated 27th January 2010.
As they say in their announcement they are looking for submissions on all aspects of the process including its strengths and weaknesses, its value, how reviewers are selected, if the process differs across the globe, the role of IT in the process and whether there are alternatives to peer review. Wide ranging stuff.
They also say that they will examine “the value and use of peer review in informing public debate.” Now that does sound like an interesting can of worms to have look into. Will they find a rosey picture of loveliness? Or will they find an ugly secret war of scientific words and stifled debate?
It’s not as if we haven’t been here before. New Scientist debated the issues surrounding peer review in an editorial last year (subscription required – sorry). They highlighted the apparent disparities between US stem cell researchers getting published quicker and in higher impact journals than non-US stem cell researchers.
Whilst they admitted that this pattern could be related to all manner of things, including the point that the American’s might simply be doing better research with stem cells, there are still legitimate concerns.
The obvious point is that there is a risk only one side of the story is getting heard – that of the “in-crowd” – the well funded, top level science rock stars that genuinely ‘lead’ a scientific area. They will have a consensus of opinion based on years of research and publications. They won’t agree on everything but they will agree on most things.
Now, what happens when a manuscript comes along that goes against the consensus? Do we get to hear about it?
Well, imagine this situation. A new well funded group in say, China tries to publish a paper that challenges the scientific consensus of the day. It is well written, high quality work and deserves publication in a high impact journal. It goes out to review, only to land on the desk of a member of that elite club, the in-crowd. The reviewer sees the power of their arguments and realises that they might actually be right. He also realises that if this gets out and gains momentum, there would be a risk to his future funding and career prospects. He will make all efforts possible to pick holes in it, delay the review and come up with powerful arguments to get the journal to reject it. The paper will eventually get out, but in a much lower impact journal and much later than it would have done.
Obviously this would be an exceptional case but the risk of bias and the conflicts of interest are clear to see. Do we ever get the chance to see this in action? Not really, because the power of peer review is in its anonymity. Scientists are at liberty to critically review the work without fear of retribution from peers. It is a powerful system that has been around for years and that everyone knows is open to abuse. The checks and balances in the system are lacking because it is very rare that we ever get to see the reviewers comments… except when we are on the receiving end of them.
As the New Scientist pointed out, the solution could be complete disclosure of the review process. Tricky as this might put off some scientists from participating in peer review. Meanwhile, a compromise has been suggested by a group of peeved European stem cell biologists – partial disclosure. Publish the referees’ comments and communications as supplementary material but redact the names. It all sounds rather familiar but I’m guessing Wikileaks have a bit too much on their plate to get involved right now.
There is also the question of whether post publication peer review will make a difference. We’ve seen what happens when peer review gets an airing in public – ‘alien’ bacteria and a massive communications balls up.
If the MPs make this proposal in their review it will add to the growing voices demanding change in the peer review system. It also means that science will become an uglier place to navigate for scientists.
For those that make a living out of talking about science, the journalists and bloggers, it will be a field day.
Alternative Science News Service
