Even within the conversation about open peer review, the policies and practices of what it actually looks like can vary widely — it could mean reviewers signing their names but not including comments, including reviewers’ comments but keeping their names anonymous, journals providing reviews for rejected papers, or something else. How would you define open peer review?
DM: Open peer review is currently not defined as a definitive practice, at present, short of implying that there is at least some openness at some point in the process. One could maybe limit it to two broad themes: 1) that review comments are published alongside an article, but these comments occurred previously and in private, with identities sometimes open, and 2) that the reviewing process itself is performed in public, with all identities open. Examples of 1) are Collabra: Psychology and PeerJ. An examples of 2) is F1000 Research.
What problems do you think an open review process can solve? And conversely, are there any problems you think an open review process can create?
DM: Before you get to open review solving actual problems, I think you can already say that open review is going to be interesting and informative for scientists and scholars. It gives you great background regarding the article’s development and improvement. It could help authors learn about how best to write and craft articles, in the first place, by being able to see any common themes or regular issues uncovered or tackled during the review process. A problem it can solve is any perception of conflict of interest or questionable practices during peer review. For example, if you’ve ever read a paper and thought “how did this get through peer review”, you would be able to actually see how it did, by seeing what the comments were during the process. If you’ve ever thought “oh, this article was only accepted because the author is friends with the editor,”, you would be able to check on the process to ensure that it occurred legitimately, and that a genuine review occurred. So, speaking in very general terms, it helps people’s trust in the communication of science.
As for problems that open review can create, I touch on this in my blog post linked above: most of the problems open review causes are not problems inherent to openness and transparency, they are inherent to “the weird, wonderful, hierarchical world of ambitious, sensitive humans in the academic and research infrastructure.” So, open review could mean that if you wrote a negative review of a more senior academic, and it is made public and you signed your review, you might be afraid that this colleague might hold it against you in future. These problems are important, but in my opinion should be solved by transparently addressing the social elements of the academic system, rather than saying open peer review is bad and should be avoided. I think, in the service of science, we should tackle the problems open review might create in order to better focus on the great problems it solves.
In your opinion, should there be certain parameters around openness during the peer review process? In other words, are there instances or types of information that should not be shared?
DM: Certainly, and these are easier to recognize than define. I think any comment on crucial yet confidential information about an author, or another person, by a reviewer, should be able to be made in confidence. So, therefore, it is important to always have a way to write confidential comments to an editor, or, in the case of public, post-publication review, send confidential comments directly to the author or editor.
At Collabra: Psychology, authors always have the option to choose open review or closed review. Has it surprised you that the majority (77%) of Collabra: Psychology authors have chosen to publish their articles with open reviews?
DM: Yes, I have genuinely been surprised at the number of authors that have opted for open review. I think there are a few drivers at work. Firstly, Collabra: Psychology is clearly presenting itself as a journal embracing open science and transparent practices, so even though open review is an option, early-adopting authors are likely predisposed to open review. Secondly, because it is an option, and many people forget to state in their cover letter whether they are choosing it or not (!) in many instances we get to email directly with the author to check whether or not they are choosing open review. On several occasions, an author has said “well, I wasn’t going to, but tell me more”. Since our first few articles had open review, we were able to not only tell them more, but show them real examples of how it had worked, so perhaps people who wouldn’t ordinarily have chosen it as an A/B decision, got to have a conversation about it which made them more familiar and less fearful of it.
Where do you see the world of peer review headed next? How do you think peer review practices should continue to evolve to facilitate more objective, fair, and transparent science?
DM: Wherever the world of peer review systems and processes is headed, I would like the culture and behavior of scientists reviewing each other’s work to become more objective and helpful, and less interwoven with social and personal sensitivities and the “publish or perish” mentality. While publishing articles should always be important, perhaps if it were a few notches less important, people would be more focused on the transparency and credibility of their science rather than the impact of (and rewards from) their claims, more open to being challenged and questioned, and less likely to be over-defensive about reviewers finding concerns or suggesting improvement to their work.
I am not saying I know exactly how to get there, or that I experience this directly myself (I am not a scholar or researcher). But, I think it is everyone’s duty in scholarly publishing to try and make things better and more efficient, and to keep an eye on what is not working and what is inefficient. I am hopeful the focus on transparency as something embedded in the practice of science, rather than the writing-up and publishing of an article, will improve science, and improve the trust in science. As for specific things, I am particularly interested in:
- Registered Reports, pre-registration, and, in general, the efforts made in stating up front what experiments you are going to be doing, so people understand that you didn’t change your analyses or hypotheses post hoc, or allow undue flexibility in your data analysis.
- Technology solutions like the ones being developed by the Coko Foundation, https://coko.foundation/, which are focused on building person-led (rather than publisher-led, or legacy process-led) systems to manage various editorial processes for journals, books, preprints, etc.
Publishers tend to shy away from change because they do not want to rock the boat or change a situation that is beneficial to their business practices and value-proposition. I myself believe that publishers should be working alongside researchers and scholars to make things better and, in doing so, ensuring publishers’ ongoing value by participating in, rather than blocking, change.