The scientific peer review system needs improvement. The goals of the system – to ensure that scientific work is technically accurate and contextualized – are important in building public trust in the work we do do. They also help to improve the quality and presentation of our work.
However, peer review is slow and inefficient. It takes time for journal editors to find scientists willing to review manuscripts, time for those scientists to carefully carry out their work, and more time for editors to synthesize this feedback and come to decisions on acceptance for publication, revision, or rejection. Because the whole system functions on volunteered time and good will, daily obligations can take predence, and delays are inevitable. Some journals like Ecology Letters promise turnarounds in under eight weeks, and others like Science in cases of transformative or urgent work (e.g. on the origin of the recent Ebola outbreak) can be even faster. But in most cases, the process drags on for months, slowing down scientific progress. The problem gets worse if a manuscript is rejected. The authors must then take their feedback and begin the process all over again at another journal, with a new editor, and a new set of reviewers. I myself have a manuscript that has bounced from three journals over two years and is still not acceptable for publication. In other cases I have waited half of a year to even receive a preliminary decision.
Imaginably the process would be much more efficient if the new target journal could draw on the feedback and comments of the previous reviewers and editors. This would reduce unnecessary duplication of effort for all parties and let everyoone spend more time (and public money) doing science rather than reviewing it. The process would also be much more efficient if authors could have their work simultaneously considered by multiple journals in a marketplce, with the peer review process decoupled from the editorial and publication process. Historically, neither of these things have happened, and we muddle along with our sequential volunteer-based system.
Recently a company called Axios Review has tried to change this model. They offer a service to scientists in which they independently obtain peer reviews of manuscripts, then negotiate with journals to find one that wants the manuscript for publication.
Axios looks like a great idea, but I think their implementation causes more problems than solutions. The reason is that they charge a fee scientists to use the service. The fee is currently $250. Reviewers and journal editors do not receive any of this; the company (now a non-profit) retains the money for their management efforts. It looks like money well spent – papers sent through the service have a very high acceptance rate at target journals, according to the Axios website. However this process pushes us towards a two-tier model, building a divide between researchers who have the resources available to pay for this extra level of treatment, and those who do not. It puts those with money at the front of the line, and those without money at the back – even if their work may be of equal quality. A pay-to-publish system builds inequality and does not reduce it. If we are dedicated to improving access to science and scientific careers in our own country and especially in developing countries, I do not think we can support this model. $250 may not seem like a lot of money to the head of a well-funded American research group, but it is a large sum to an American graduate student with no other project funding, or indeed to many researchers outside of the developed world. Do we leave them to slow peer review, low manuscript acceptance rates, and a self-reinforcing system that disadvantages their work? I do not think we can.
I had a constructive conversation with Tim Vines, Axios managing editor, about their system. He made several good arguments. First, he pointed out that their solution clearly does streamline an inefficient process. I agree. Second, he argued that the process of managing peer review does incur some costs, which typically have been averaged across all scientists at the level of journal subscription fees. In this model, they are being passed on to the author instead. This is true, but I think it is fundamentally the equivalent of a regressive taxation system. It is similar to what has happened with open-access publishing, where still in many cases the author must take on the burden of paying to publish. Third, he suggested that if the service gains enough traction with authors it may eventually be possible to flip the revenue model back on to publishers. But until now, publisher-pays models (e.g. Peerage of Science) have not gained much traction. This is fair, too, but the road may be slow.
What would be better? At minimum, I think Axios needs to offer a sliding price scale or fee waivers to reflect the range of financial resources available to different scientists – or an option for well-funded groups to pay-forward these fees for others. More broadly, I think it would be reasonable for journals to request a copy of the peer reviews received by authors during an earlier stage of the process – and to consider making a decision on manuscripts without further peer review. This is difficult as editors often want to know reviewers’ identities, and may not be able to trust that authors have not edited these reviews. A between-journal transfer system (for example, built into the EditorialManager system) could solve this if the journal community wanted it. I also think a community-supported system in which journals could compete to accept manuscripts would also be useful (e.g. see Peerage of Science, Expresso). Both these solutions would not pass along financial costs directly to the authors, as the Axios system currently does. If they could modify their revenue model, then I would be much more supportive of their work. Indeed, I admire their innovative efforts to make change – they have received excellent feedback from several ecologists (1) (2) – but their change is not the change we should ultimately be looking for. Perhaps it will take this sort of innovation to catalyze future change, as is happening slowly with open-access publication. However I am not sure I want to be part of the first wave, and think it is important to remember our values, however they may align with market forces.
I normally review fifteen to twenty manuscripts per year, to match the total number of scientists required for reviewing my own work. This is only what is fair for all of us; no more or less. But when I was asked a few weeks ago to review for Axios, I declined. And I will continue to do so until we find a more equitable way to improve the current system.
Comments
12 responses to “Improving peer review: why Axios is not a solution”
Speaking as a former journal editor and current Axios editor, yes, editors do indeed want and need to know referees’ identities. And as you note, you can’t ask authors to just forward all the reviews their ms may have received previously because you’d have to trust authors not to be selective about it and not to edit the reviews. So just having journals ask authors to pass on the reviews their mss previously received is a total non-stater.
Worth noting that a number of author-pays open access journals deduct the Axios fee from their publication fee, so authors who want to publish in those journals don’t pay any extra to use Axios.
Re: journals possibly bidding for mss, I’ve suggested this idea myself in the past, but the more I think about it the more it seems either unworkable, or worse than the current system, or basically just like the current system, depending on the details of how it’s implemented.
I think the question you need to ask is how to get to the endpoint you consider ideal. Axios is an incremental, narrowly-focused innovation that only aims to solve one problem (the burden placed on the peer review system by rejection and resubmission elsewhere). If it takes off, then there’s an opportunity to modify it with further incremental improvements. Are you saying you’d rather see some other incremental change instead? If so, what? (As Tim noted, Peerage of Science hasn’t taken off…) Or are you saying you’d rather that the entire system was somehow replaced at a stroke with what you’d consider an ideal system, thereby solving many problems at once?
Thanks for engaging, Jeremy – it’s great to get another inside perspective.
I do think that having the review fee subtracted from the publication fee at open access journals is a good start. However this still doesn’t solve the problem of author-pays for open access either, which is admittedly a bigger issue to work through.
Regarding the transfer of reviews between journals – agreed that this would have be handled not by the authors. However many publishers using Editorial Manager already have software functionality to transfers manuscripts and reviews between journals they run. I don’t think it would be that hard to allow similar transfers across different unrelated journals that all use that platform – if enough people wanted this, would there be any technical obstacles to making it possible?
I also don’t see any problems around having journals bid for manuscripts. If author identity is blinded then I do not see the possibility for bias, or why it would unworkable. Can you elaborate?
You do make a good point about incremental vs systemic change. Obviously the latter is very hard. But we need to think through the cascading implications of incremental changes as we make them. And I think I have proposed some alternatives for other incremental changes instead – or modifications to this approach that would make it better.
“However many publishers using Editorial Manager already have software functionality to transfers manuscripts and reviews between journals they run.”
That’s why Axios Review now uses Editorial Manager–ease of transferring mss and reviews between journals.
Re: having journals bid for mss, how would they decide which ones to bid on? Would editors of every journal need to read every ms?
I was imagining a simple system, where authors write a cover letter as with any submission and could target it to say three journals. Then any of these journals could bid for the right to send the paper for the review & have first refusal over it. No need to read the whole MS necessarily; only the cover letter and abstract as usual, with an option to delve deeper if interested. Do you think this would be realistic?
I did not know the Axios system was EM-based – great to know some of this infrastructure is already in place.
Ah, ok. That’s not actually much different from the system we have now, honestly. Instead of submitting to one journal at a time, you’re submitting to three simultaneously. Well, you’re making presubmission inquiries to three journals simultaneously. Which you’re free to do already at leading journals; a presubmission inquiry isn’t a commitment to submit.
So sorry, but this sounds to me like at most a minor tweak on the existing system, and I’m not clear what problem it solves. It certainly isn’t likely to cut down much on the burden on the peer review system. Selective journals already do an increasing number of rejections without external review. And getting invited to submit on the basis of a presubmission inquiry doesn’t prejudge the outcome of the review process. Again, given that selective journals already do lots of desk rejects, I doubt that mss submitted following a presubmission inquiry (or a “bid”) are much more likely to get accepted than those that manage to get sent out for review under the current system.
This is an excellent critique of the most concerning part of Axios Review, the lack of equity in the fee structure. I think most people would agree that fee waivers for authors that have no access to funds is ideal.
I think the biggest issue right now with most of these systems that are trying to improve the efficiency of peer-review (e.g. Axios, Peer Age of Science) is they have very little power until either one of two things happen (1) These systems start getting a lot of good papers the top journals want, so much so that the quality and/or quantity of papers submitted directly to the journals noticeably drops or (2) Reviewers start refusing to review for journals individually. Imagine if 25% of requests to reviewers came back with
“I’m sorry I only review papers for centralized peer-review systems such as Peer Age of Science, Axios Review, or for journals with an extensive and heavily utilized between journal transfer system. In other words, my volunteered time to review manuscripts for free is very valuable, and should not be wasted, and therefore I only review for institutions that are actively fixing the problem of inefficient peer-review.”
I agree that the Axios model, as it stands, is not the ideal final solution. However, I think that unlike Peer Age of Science, the fact that Axios is officially used by a some of the most prestigious ecology journals gives them an opportunity to increase their power. Once they increase their power they can turn the table on the journals and modify their fee structure. Think of it like Tesla, at first they developed a $150,000 sports car, to establish their brand as cool and desirable, and much later started focusing on scaling their products to the masses. Or how about facebook, who initially limited their network to only a few select college campuses. I think that Axios is doing something similar. They are establishing a relationship with journals and gaining some trust and power. Journals are less likely to start buying into the system if they have to pay to do so. However, once they start relying heavily on the system, and see it’s value then they will have no choice but to start paying. I think Axios is being smart here.
Hopefully this process won’t be slow, because as you point out if it is slow, it can create a system where wealthy authors are at a huge advantage.
Note that some journals subtract the $250 Axios fee from their publication cost right now. Even if every journal did this however, it wouldn’t completely address equity issues because the people in greatest need of fee waivers often don’t pay any publishing cost.
Thanks for this comment, Matthew. I agree with most of your points here. Regarding the approach of building relationships and then changing the underlying financial model – this is reasonable to me, but I would want to see an explicit theory of change and long-term plan for making this happen. How can we be sure that this company is indeed interested in making that shift eventually? And more philosophically – should we be happy with this market-driven model in which we trust companies to build market share, then leverage it to produce social good? This line of thinking dominates current philanthropy and business – the Carnegie and Gates and Goldman approach – but also comes under heavy criticism for creating significant structural inequality along the way. What other models can we imagine that don’t require this approach? Market-based solutions are not the only way to solve a problem.
I definitely agree with you. I find that paying for peer-reviewing simply destructs a fundamental peer-reviewing value, which is the will to make science progress on a voluntary basis. Personally, I find that dedicating time to reviewing someone else’s work is one of the most beautiful things about our profession. It shows that even if scientists constantly compete with each other for funding, positions, etc. there is something larger than their personal/academic gains, which is making science progress.
I agree with you Ceres. However, I also get frustrated when I take the time to write a detailed review for a paper, the paper then gets rejected, and then when the paper gets resubmitted elsewhere, my review is not sent to the new journal. It wastes everyone’s time, it prevents authors being held accountable for mistakes, and it is the opposite of open science. That said I really hope Axios can move to a “Journal pays for the service” system as quickly as possible, because authors paying is not ideal for both the reasons you and Benjamin pointed out.
Thanks Matthew for these further thoughts. I wonder if there are alternative ways for authors to be able to transmit journal reviews to other journals with the promise that they have been unmodified – maybe some sort of checksum approach is used in software security?
Thanks for this, Ceres – I agree with this as well, and think it is one of the unique things about science that builds trust in what we do.
Hi Ceres – thanks for your comment. I should clarify that Axios doesn’t charge for peer review. As you say, peer review is a quid pro quo service that academics provide each other. Peer review works best when it’s voluntary and motivated solely by professional pride. The Axios fee covers peer review *management*, whereby professional editorial office staff ensure that the review process is anonymous, fair, and as quick as possible.
Almost all journals have paid editorial office staff, and their salaries are paid from revenue generated by open access fees or subscriptions. Since Axios doesn’t publish anything, we have to charge for the service of peer review management at the ‘point of use’. That aside, reviewing for Axios is the same as reviewing for a journal.