Thanks to The Scholarly Kitchen for allowing us to re-blog the following Peer Review Week 2016 (#PeerRevWk16) -focused post.
About Peer Review Week: Peer Review Week is a global event celebrating the essential role that peer review plays in maintaining scientific quality. The event brings together individuals, institutions, and organizations committed to sharing the central message that good peer review, whatever shape or form it might take, is critical to scholarly communications.
While much of the contemporary debate around peer review focuses on both journal articles and STEM fields, here we are going to focus on humanities and social science (HSS) fields where both longer articles and book-length projects are more common. Does HSS peer review have the same functions, goals, and challenges as STEM peer review? My conversation below with fellow Scholarly Kitchen Chef Karin Wulf and Mary Francis, Editorial Director at the University of Michigan Press, addresses these questions and looks forward to the ways in which HSS peer review might evolve.
Alison Mudditt: Karin, as both a working scholar and someone who oversees a substantial publication program, what are the key ways in which HSS peer review differs from STEM journals models?
Karin Wulf: A key point to make right off the bat is that many more, if not most, HSS fields are “book fields”, disciplines in which scholars typically publish their most significant work in book form rather than in journal articles. Journals still play an important role for HSS; HSS articles tend to be longer, and HSS journals tend to publish a small percentage of submissions. Peer reviewers for HSS journals are thus doing pretty extensive work on lengthy submissions, and mostly for authors whose work won’t be published (at least not in the first journal to which it’s submitted). Generally, though, peer review is valued on both sides of the experience. It’s simply part of the culture for authors and reviewers to participate in the intellectual exchange that takes place in the context of peer reviewing. It is also the case that a lot of articles are developed in the context of a book in progress and that peer reviewers for journals thus have always played an important role in moving a book to eventual publication.
Alison: Beyond the common gatekeeping role, does peer review play a different role in these fields?
Karin: Another important point is one we make in Scholarly Kitchen posts a lot — there is tremendous diversity in the culture and practice among and within disciplines, not only between HSS and STEM. It would be hard to generalize about the differential role of peer review within the cultures of each. In many fields different versions or pieces of a project are often shared to solicit feedback from experts. The difference with formal peer reviewing is that the expertise is solicited (by the journal editor) and usually that it is anonymous to each party (double-blind). The value of double-blind review has been debated quite a bit, but I still come down on the side of anonymity as the best hedge against bias.
Alison: Are there distinctions between HSS book and journal peer review?
Karin: Again, there is significant diversity among and within HSS fields that’s important. But in the aggregate, peer review in journals versus books can be quite distinct. Journal editors tend to commission more reviews for each article than do book editors; reader reports for journal articles also tend to be proportionally longer. Journal editors mostly deal with submissions that come in over the transom and are fully responsible for the substantive responses to reader reports, often writing lengthy letters to authors to interpret the readers’ reports and deliver their decision.
But acquisition editors for university presses are both doing more active recruitment and generally less substantive editorial work with their authors. The role of the review itself is therefore different. Books tend to get fewer reader reports (usually two), though often at two different stages, the original and final manuscript submissions.
What peer reviews of journal article and book manuscripts share is basic assessment: how fresh is the research? How persuasive is the argument? What does this work contribute to the field and/or fields? Peer reviewers are asking these same questions of the two different genres.
Alison: Mary, as a seasoned university press editor, what do you think is distinctive about university press peer review?
Mary Francis: As Karin notes, different peer review practices tend to attach to disciplinary norms and product type, more than type of publisher. University presses, even relatively small ones, produce a wide range of publications: scholarly monographs, journals, textbooks, critical editions of primary sources, guidebooks, regional titles, poetry, and more. While all of these are peer reviewed, one of the distinctive features of university press peer review is how it accommodates such a range of publications. When I was working with the Association of American University Presses (AAUP) acquisitions committee to create their new guidelines, there were many discussions of how to zero in on the most rigorous practices that all book peer review ought to share, and the most salient variations had to do with disciplines being considered.
Alison: Peer review has always been at the heart of university press publishing: why did AAUP decide to create a “best practices” document at this point in time?
Mary: The biggest reason is to call attention to the value that peer review provides. The bulk of the document is a detailed ‘how-to,’ which should make clear how many carefully considered steps and decisions are involved as well as how much expertise is required. Editors play an essential role in framing peer review for their constituents within their overarching project of cultivating intellectual endeavors in a given field, matching the needs of a project to the expertise of peer reviewers, and making sure the final product has the greatest possible reach and influence. At a time when higher education and the core values of scholarly research are under intense public scrutiny, the document was designed to advocate for the value of peer review to those whose support is essential: deans and provosts, foundations and funders, regents, elected representatives.
Alison: What processes shaped its development?
Mary: The creation of the document was an immensely rewarding two-year odyssey of discussion and debate, informed, naturally, by rigorous peer review. The AAUP Acquisitions Committee wrote – and debated — the document jointly. The first draft was reviewed by the AAUP board, then revised and submitted to twenty-five peer reviewers, including fellow university press editors, colleagues from museum publishing, commercial scholarly publishing, and university press publishers in the UK. The revised version was the focus of a very lively AAUP Collaboration Lab at the 2015 AAUP annual conference, and generated more refinements leading up to the release of the complete document this past spring.
Alison: The voices demanding alternatives to blind and double-blind review have risen alongside greater experimentation with new models of long-form content, especially those in digital form. I’m interested as to why AAUP chose not to discuss those new models here – was there a reason for this?
Mary: Alternative forms of peer review, and peer review of multi-modal scholarly outputs were very much part of the committee’s discussions: we talked about how multi-modal scholarly outputs challenge publishers to think carefully about parameters that are easier to take for granted in a world based on print books: version control, preservation, citability, discoverability, etc. In the end, the committee felt that there isn’t enough consensus around peer review of alternative forms (or alternative forms of peer review) for there to be a clear set of best practices that AAUP could endorse at this point in time.
Alison: What about reviewer selection? The guidelines highlight the reviewer’s potential to judge the scholarship and argument presented, but I wonder how both of you think about diversity – race, gender, geographies, and other identities – as you select reviewers? Given the challenges the academy faces in increasing diversity, how do we ensure that peer review doesn’t contribute to privileging certain established voices over others?
Karin: A lot comes down to the press director and their expectations of editors. We don’t want conferences to have “Hasslehoff panels”; similarly, we want review panels to reflect diversity of perspectives. Diversity includes race and gender as well as junior and senior scholars, and scholars from different types of institutions; reviewers shouldn’t all come from R1 universities, for example. But here’s another thing — at least in HSS, editors should want as reviewers not only the scholars whose work is exactly spot on the topic, but what our journal editor might describe as a 360 degree view from different field specialties that have a bearing on the work in question.
Mary: We did talk about how “established voices” manifest themselves in different disciplines. Editors’ desire to diversify beyond “established voices” is about exciting, innovative work that goes beyond familiar methodologies and paradigms, and for this editors are often seeking younger scholars who are thought leaders in their field — a pool more likely to manifest a greater degree of diversity. But as more academics around the world hold contingent positions that make it difficult for them to establish credentials as peer reviewers, the already formidable problem of finding peer reviewers who represent diverse backgrounds and diverse scholarly expertise will only become more challenging.
Alison: Within HSS, there are growing questions about the shortfalls of the (double) blind process and more experimentation with changing the ways in which anonymity functions and trying to remove subjective judgments from the review process, mirroring STM developments (the Open Library of the Humanities [OLH] is but one example*). Listening to the critics of peer review, it would be easy to assume that this is a system of need of fundamental change. Yet from our experience at University of California Press, there seems to be little demand for change. No one seems ready to give up the deep level of review monographs receive, both in terms of overall assessment and deeper commentary, and I don’t see any appetite for expanding that to include post-publication commentary, for example. How do you see the system evolving and do you think more substantial change is likely any time soon?
Karin: Alison, my experience echoes yours. Not much appetite for post-publication review and mostly looking at improvements to double-blind which seems to be working. I sometimes hear about issues with a specific review/er, but usually our editors are pretty good at intervening and interpreting for authors if there is a particularly harsh review. They also let a good intellectual exchange take place — that’s the point of review, to raise the level of argument and scholarly knowledge. The AAUP guidelines address this in the sections on reviewer expectations and how to handle a problematic report; both are important for journal editors, too.
An important thing to keep in mind is that double-blind review for publication usually takes place at the end of a long train of fairly public critical exchange about a piece of work, at conferences and seminars, so there has often been opportunity for “open” review. I don’t really see “a road to reform” in that while double-blind review may not be perfect I hear often that reviewing isn’t seen as a broken system. The challenges of mutual anonymity, namely a lack of diversity among reviewers and biased reviewing, might be just as challenging in open review forums. I almost always sign my reviews, but I’m glad to have the option. I also wonder whether calls for open review are more appealing to some disciplines than others. The OLH for example, and the approach to open reviewing advocated by Kathleen Fitzpatrick, originate in literature and may best reflect movements in that field.
Mary: Like many of Alison’s fellow Scholarly Kitchen Chefs, I believe that the current system of peer review for long-form scholarly work (in whatever format it is published) is so essential to our systems of research and scholarly communication that it will only change incrementally. I suppose my work with the AAUP on this document manifests what I would most like to change: I want more participants in higher education and research to have a nuanced the understanding and appreciation of the virtues of peer review, the hard work that goes into it, the immense value it adds to scholarly communication.
*Editor’s Note: As a point of clarification, OLH has engaged in the debate around preferred peer review methodologies and questioned the status quo, in practice they offer a rigorous policy of double-blind peer-review and a clear signaling the mode of review that has been conducted on the work.
Alison Mudditt has been Director of University of California Press since January 2011, where she has focused on reshaping the Press’s strategy and structure to meet the needs of its diverse audiences in the digital age. Alison more than twenty-five years experience in academic publishing which began at Blackwell in Oxford, and then at Taylor & Francis Inc. in Philadelphia as Publishing Director of the Behavioral Sciences Division. Alison joined SAGE in 2001 as Vice President and Editorial Director, and was appointed Executive Vice President in 2004 where she led the SAGE’s publishing programs across books, journals and digital during a period of tremendous growth. Alison is a regular speaker at industry meetings and is currently Vice Chair of the Scientific Publications Committee and member of the Open Science Committee of the American Heart Association, and member of the Board of Directors of K|N Consultants. She has also served on the Executive Council of the Professional and Scholarly Publishing Division of the American Association of Publishers, and was Co-Chair of the Dean’s Leadership Council at California State University, Channel Islands.