How science overtook peer review during the pandemic
When articles from China started flooding the bioRxiv and medRxiv websites in the first few months of 2020, it was a strange and noticeable change. Founded as places where scientists could publish draft research papers before those papers went through traditional peer review, these sites had never really advertised in China or received many submissions. scientists from this country before, said Richard Sever, co-founder of bioRxiv and medRxiv. The sudden change turned out to be a glimpse of the coming pandemic. “We received a wave of submissions from China, then a wave of submissions from Italy. And I remember being with a coworker, looking at the submission numbers, and the graphic was so oddly familiar, ”Sever said. “It looks like the progression of pandemic cases. ”
“Preprint” or “prepress” servers have been around for decades, but during the COVID-19 pandemic, they gained new notoriety and a new level of importance. Two of those sites alone, bioRxiv and medRxiv, hosted 25% of all COVID-19-related scientific research published in the first 10 months of the pandemic – more than 10,000 articles. In contrast, only 78 preprints were uploaded to bioRxiv during the entire Zika 2015-16 outbreak. (The medRxiv site did not yet exist.) COVID-19 has proven to scientists that preprint servers are critical resources in a crisis. At the same time, however, experts said the pandemic had also highlighted gaps in scientific publication. Preprints share much of the same limitations as peer-reviewed research, especially in how the media and the public use research once it is available to everyone.
Websites that publish scientific research before it is peer reviewed and published in a scientific journal have been active since at least 1991, when the classic arXiv site, originally used primarily by physicists, was put online. Even before that, scientists shared drafts and notes with each other through personal correspondence ever since “science” as a field became a thing. These lines of communication exist alongside the traditional peer review process, which places gatekeepers between research and publication. First, an article must be accepted by the editors of a scientific journal. Then it is passed on to a (usually anonymous) panel of other scientists for review. Their ratings will lead to edits, which usually lead to more experiments, and then (hopefully) to eventual release.
But the peer review process does not happen quickly. Editors sort through thousands of dense submissions, and reviewers are volunteers who read in their spare time. On average, it takes about six months for a life science research paper to go from acceptance to publication. “During the outbreak, we maintained the same peer review and editing process, just on an accelerated schedule… but I don’t think this is a sustainable model,” said the Dr Eric Rubin, editor of the New England Journal of Medicine.
He and other scientists said the issue of speed was particularly important for clinical research during the pandemic, as doctors needed the most recent information to make vital decisions for critically ill patients. For example, the large trial of the steroid dexamethasone – which first showed that this cheap and widely available drug could reduce the likelihood of death in COVID-19 patients – was originally published as a preprint. It would appear, traditionally published, in the New England Journal of Medicine eight months later, but preprinting put this knowledge in the hands of physicians more quickly. “I had a doctor contact me and he said, ‘You know, there are probably people who are alive today who would have died without the preprints,’” Sever said.
Such successes helped make preprints more widely accepted by biological and medical scientists. It also helps that research in preprints isn’t statistically significantly different from how those same papers end up being published in a peer-reviewed scientific journal. A 2020 pre-COVID-19 bioscience study that compared preprinted papers and their subsequent, peer-reviewed versions, for example, found the biggest differences were in the details, like how clearly the title reflected conclusions or ease of finding relevant information. information in the article.
But that should be understood as a critique of the peer review process rather than a glowing endorsement of the accuracy of the preprinted paper, said Alice Fleerackers, graduate student and researcher at the Scholarly Communications Lab, a joint project of the Simon Fraser University and the University of Ottawa. “There really is a perception that peer review is a trustworthy quality control mechanism,” Fleerackers said. But research has not been able to support this idea. All the jokes and paranoia about the spread of COVID-19 by 5G cellphone technology, for example, originate in an article published in – and then retracted by – a scientific journal that claims to peer review its content.
“We’ve seen a lot of misinformation and a lot of retractions that came out during COVID … ‘high quality’ – and I’m using aerial quotes here – research in peer-reviewed publications can be just as flawed as research. in preprint, ”Fleerackers said.
The real problem with preprints – which is, oddly enough, also the real problem with peer-reviewed research – is the way these studies are promoted and written on social media and in the press, experts told me. .
“Before the pandemic, I had less worry that poor-quality scientific data was preprinted and then released widely,” said Maia Majumder, computer epidemiologist at Harvard Medical School. “But now ordinary people read them too – and the media is covering them at a rate that far exceeds before 2020. That means when bad science is preprinted and is sensational enough to grab attention, it can. quickly shape the speech around a given topic.
One of the most tweeted and reported preprints in the first 10 months of the pandemic was a study that attempted to estimate the share of the population in Santa Clara County, California that had previously been infected with COVID -19. The results, which found evidence of a higher infection rate in the general population than anyone had previously calculated, were used to support the idea that COVID-19 would cause few deaths and that no lockdowns or measures distancing would be necessary. It was heavily criticized and estimates had to be revised down before BuzzFeed News revealed the research was funded in part by an airline mogul who was critical of social restrictions linked to COVID-19.
It is not known to what extent the media coverage of the study actually reported the results as opposed to the criticism and controversy it sparked, but Fleerackers’ research suggests that digital media does not consistently provide readers with the context. they need to understand the limitations of preprinting. studies. In a study published in January, Fleerackers found that about half the time digital publications did not mention that a COVID-19 preprint study was even discussed. has been a preprint.
Reports on preprints without explaining their limitations risk misleading the public in the same way that reports on peer-reviewed research are inherently accurate, Fleerackers said. While this reality was certainly known before the pandemic, the importance of preprint servers over the past year and a half has really delivered the lesson. Servers like bioRxiv and medRxiv have added important disclaimers to their pages. University press offices have started instituting new rules that control how preprinted articles are promoted to the media. There’s even a new journal specifically dedicated to examining trendy COVID-19 preprints – and debunking them, if necessary.
But the pandemic has also shown how much information in a preprint is beyond scientists’ control when it is posted online. “I hope they develop an awareness,” Fleerackers said, “that if you put something in public, it’s a fair game for journalists and for the public… and for conspiracy theorists.”