Reports are appearing in a number of main stream media outlets of an apparent breakthrough in the search for a ‘flu vaccine that is universal i.e. that is active against all strains of influenza. That’s important news and could eventually have widespread implications for public health (and probably make someone rather rich).

Some examples of the reports appearing are from The Guardian, Press Association, The Daily Telegraph and MSNBC. The story was originally broken by Alok Jha at the Guardian yesterday (Sunday). It is currently spreading across the net. It is a good piece and Alok should be congratulated on the amount of detail he has documented in the article.

There is one problem. There is no link to the original academic paper describing the data that appears in the article (i.e. the results of a Phase II study about effects). The links that are present refer us to the results of a new Phase I study (about safety) and a commentary. This lack of link to the original research referred to in the article is not unusual. We have discussed why this is a problem before and why it is probably a function of the embargo system.

What is unusual in this case is the length of time we will apparently have to wait before we can actually read the paper. According to Alok himself, the academic paper is, as yet, unpublished. We also contacted Dr Sarah Gilbert, who is leading the research, and she confirmed that no date was set for publication of the study.

The issues here are numerous. We can’t make a judgment about the quality of the science until is it published for one. Next, the paper might not be under peer review or still be under peer review (see this, this and this for other commentators’ opinion) with the risk that it has not been accepted yet. For all we know, there could be multiple issues with the work. We also can’t judge potential conflicts of interest. Who funded it? Who stands to benefit financially? We should not accuse the authors or their institutions of any dodgy practices (far from it, this is The University of Oxford) but, as it stands, we are simply left in the dark.

So, to summarise, we can get a glimpse of the ongoing research by inspecting the Phase I study which is good. But then we have to read the newspapers to find out about the (more important) study on the effects of the treatment. There has to be something odd with that.  It is one thing getting widespread media coverage of such apparently important research (that’s good), but if we can’t actually judge the science on its merits, it is a problem. Sorry, but a major newspaper is not a peer reviewed scientific journal and it is not the place to try and publish original research.

It is now time for whatever journal is involved to get this paper out quickly.

Here’s the opening paragraph of the announcement:

“The Committee has today launched an inquiry into peer review. The committee invites evidence on the operation and effectiveness of the peer review process used to examine and validate scientific results and papers prior to publication.” Dated 27th January 2010.

As they say in their announcement they are looking for submissions on all aspects of the process including its strengths and weaknesses, its value, how reviewers are selected, if the process differs across the globe, the role of IT in the process and whether there are alternatives to peer review. Wide ranging stuff.

They also say that they will examine “the value and use of peer review in informing public debate.” Now that does sound like an interesting can of worms to have look into. Will they find a rosey picture of loveliness? Or will they find an ugly secret war of scientific words and stifled debate?

It’s not as if we haven’t been here before. New Scientist debated the issues surrounding peer review in an editorial last year (subscription required – sorry).  They highlighted the apparent disparities between US stem cell researchers getting published quicker and in higher impact journals than non-US stem cell researchers.

Whilst they admitted that this pattern could be related to all manner of things, including the point that the American’s might simply be doing better research with stem cells, there are still legitimate concerns.

The obvious point is that there is a risk only one side of the story is getting heard – that of the “in-crowd” – the well funded, top level science rock stars that genuinely ‘lead’ a scientific area. They will have a consensus of opinion based on years of research and publications. They won’t agree on everything but they will agree on most things.

Now, what happens when a manuscript comes along that goes against the consensus? Do we get to hear about it?

Well, imagine this situation. A new well funded group in say, China tries to publish a paper that challenges the scientific consensus of the day. It is well written, high quality work and deserves publication in a high impact journal.  It goes out to review, only to land on the desk of a member of that elite club, the in-crowd. The reviewer sees the power of their arguments and realises that they might actually be right. He also realises that if this gets out and gains momentum, there would be a risk to his future funding and career prospects. He will make all efforts possible to pick holes in it, delay the review and come up with powerful arguments to get the journal to reject it. The paper will eventually get out, but in a much lower impact journal and much later than it would have done.

Obviously this would be an exceptional case but the risk of bias and the conflicts of interest are clear to see. Do we ever get the chance to see this in action? Not really, because the power of peer review is in its anonymity. Scientists are at liberty to critically review the work without fear of retribution from peers. It is a powerful system that has been around for years and that everyone knows is open to abuse. The checks and balances in the system are lacking because it is very rare that we ever get to see the reviewers comments… except when we are on the receiving end of them.

As the New Scientist pointed out, the solution could be complete disclosure of the review process. Tricky as this might put off some scientists from participating in peer review. Meanwhile, a compromise has been suggested by a group of peeved European stem cell biologists – partial disclosure. Publish the referees’ comments and communications as supplementary material but redact the names. It all sounds rather familiar but I’m guessing Wikileaks have a bit too much on their plate to get involved right now.

There is also the question of whether post publication peer review will make a difference. We’ve seen what happens when peer review gets an airing in public – ‘alien’ bacteria and a massive communications balls up.

If the MPs make this proposal in their review it will add to the growing voices demanding change in the peer review system. It also means that science will become an uglier place to navigate for scientists.

For those that make a living out of talking about science, the journalists and bloggers, it will be a field day.