Peer selection vs. “peer review” – why papers in edited volumes should not be “reviewed” externally

I am often asked to review a paper for an edited volume (or special issue of a journal), but I am generally reluctant to do this, for reasons that take a little more space to explain. So here’s why. (This blog post is of course of more general relevance than the general theme of diversity linguistics, but I couldn’t think of a better venue for it.)

We are all familiar with the notion of “peer review”, but I think that the term is not precise enough. What’s most important about the process is peer selection – in other words, the fact that you don’t decide where your papers is published, but one of your peers decides it (in the default case the journal editor). Let us divide the process into its five component stages:

(1) journal selection (by the author)
(2) reviewer selection (by the editor)
(3) providing of peer comments (by the reviewer)
(4) publication recommendation (by the reviewer)
(5) publication decision (by the editor = peer selection)

I would argue that in this process, there are only two steps that are absolutely crucial: journal selection (1) by the author, and publication decision (5) by the editor. To have a basis for her decision, the editor generally needs the advice (i.e. (4)) of colleagues who find time for a more thorough reading of the paper and who may have more expertise in the specialization of the article. But sometimes an editor overrides the recommendation of a reviewer, and not uncommonly the reviewers offer different opinions, so (4) is not absolutely crucial for the process.

Usually reviewers not only give a recommendation, but also provide fairly detailed comments on papers (i.e. (3)). I have often been very impressed by the level of detail in reviews that I get, and I try to be conscientious and thorough in my own reviews. So the peer comments are generally perceived as valuable, but again they are not absolutely crucial. The best papers require few comments, and I think that many papers don’t profit a lot from the comments provided by the peers. Reviewers are rarely as knowledgeable about the paper’s topic as the author, and often they have different agendas.

What’s really important for the system to function is the editor’s decision to accept or reject a paper (i.e. (5), peer selection). Since there’s no guarantee that the paper will be accepted, authors make an effort to submit excellent work, and they will only submit their best work to the journals that they perceive as the best journals (i.e. (1), journal selection).

Now what about edited volumes? They work rather differently from ordinary journal issues, because there is a thematic coherence to the volume. If a paper on a particular topic is missing, this might be perceived as a defect of the volume (and this is clearly the case in handbooks or overview books that need to cover a range of topics). With edited volumes, stage (1) is nomally absent, because the papers are invited (e.g. when a workshop organizer invites the participants to submit a paper afterwards). So what is the purpose of outside reviewers in such cases? Occasionally the editor may be unsure about whether to accept a paper, but in most cases, they know that they want to accept it, but they feel that they have to get a reviewer’s comments so that the volume counts as “peer-reviewed”. Sometimes they know that they don’t want to accept it, and then the reviewer’s task is to provide a justification for rejection, to save the editor’s face. It is my impression that in the great majority of cases, the primary role of the reviewer is to fulfill the general expectation of “peer review”, and perhaps to give some advice to the author for revising the paper. But the important role of reviewers in journal submissions, namely advice to the editor, plays little role in edited volumes.

Hence, I think that a better model would not have publication recommendations for edited volumes, only peer comments. And when there are no recommendations, the reviews do not have to be anonymous – the peer comments could come from the most direct peers, namely the other authors of the same volume. This is something that I’ve seen practiced in a number of cases (e.g. Haspelmath (ed.) 2004, Penke & Rosenbach), and I’ve found that it works quite well. People are generally more motivated to write comments on a paper that’s supposed to come out in a volume where they have a paper themselves, than to write comments on a paper in a volume unconnected to their work. After all, writing reviews for journals is more prestigious – you can list the journals for which you have written reviews in your CV, but you can hardly list the edited volumes for which you’ve done this.

So edited volumes would have a peer-review process that works as follows:

(1) invitation to submit (by the editor = peer selection)
(2) reviewer assignment (from among other volume authors, by the editor)
(3) providing of peer comments (by the reviewer)
(4) publication decision (by the editor) (almost not distinct from (1))

One might argue that this process is not as strict as the corresponding process with journals. I wouldn’t disagree, but I would say that this is unavoidable. Since edited volumes always start out by invitation, it’s simply not possible for the editor to be as objective in taking her publication decision – rejection of an invited paper is potentially face-threatening in the way that rejection of an unsolicited paper isn’t.

Edited volumes give prestige to the author in different way than journal publications. With a journal paper, you show that you’ve made it through the bottleneck of an editor’s selection of the best papers. But with a paper in an edited volume, you show that you’ve been invited to publish together with other colleagues, which also carries a certain prestige. The peer selection happens primarily at stage (1), and in a different way. It’s simply a different kind of bottleneck. Combining the two kinds of bottlenecks, and the two kinds of prestige, simply doesn’t make much sense, it seems to me. This should be recognized by publishers – they shouldn't require external review on the model of journals for edited volumes.

In general, linguists should probably concentrate more on journal publications. These generally have more prestige, especially outside of linguistics, and the more we publish in journals and behave like the natural sciences, the more prestige our field will have. But I’ve learned so many important things from papers that were published in edited volumes (even in festschrifts) that I wouldn’t want to say we shouldn’t have edited volumes at all. These give the authors certain freedoms that journal papers don’t have, and we don’t want to lose those freedoms.

References

Haspelmath, Martin. 2004. Coordinating constructions. Amsterdam: Benjamins.

Penke, Martina & Anette Rosenbach (eds.) 2007. What counts as evidence in linguistics? Amsterdam: Benjamins.


Print This Post Print This Post
This entry was posted in Opinion. Bookmark the permalink.

4 Responses to Peer selection vs. “peer review” – why papers in edited volumes should not be “reviewed” externally

  1. Na'ama Pat-El says:

    I think you are conflating several types of edited volumes: some are proceedings of sorts where the selection of initial presentations is strict and some are indeed “by invitation only”, where selection is often fairly limited. After all, scholars tend to choose from a group of individuals whose work they are familiar with (colleagues, friends etc.). Since the editors in both types are highly invested in the selected group, they certainly need an external reviewer to assess the quality of the work regardless of the volume's theme (which is itself sometimes rather loosely defined). The editor may wish to ignore the review, as does an editor of a journal, but he/she needs to hear conflicting views, if such exist. Authors in the same volume are unlikely to recommend rejection, even if they offer many useful comments. An invitation to present or submit a paper is not blank and without prerequisites. Perhaps if authors remember that fact they will submit their best work to books as they do to journals.

    • Martin Haspelmath says:

      Yes, sometimes editors use the external reviewers as a way of forcing the authors to make certain revisions that they otherwise wouldn't want to make, and occasionally a critical review by an external reviewer gives the editors a face-saving way of rejecting a paper that they don't want. But as a potential reviewer of such a paper, I have little trust that the editor hasn't already made up their mind, so that my review remains unused or is used in a way that I might not be so happy with. Moreover, I'd have to know the other papers in the volume to be able to judge whether the paper fits into it, and whether it is above or below average quality.

  2. Elaine Francis says:

    I agree with your proposal. Articles in edited volumes don't have the same kind of prestige as journal articles and the function of peer review is different. I think it's a great idea to have non-anonymous comments from other authors in the same volume. I don't think it would hurt the publishers' reputation to implement such a policy. It would mostly affect young scholars who are in search of a tenure-track job or who are going up for tenure and promotion, since they could not claim that their articles in edited volumes were peer-reviewed. The solution to this, in my opinion, is to provide better mentoring for young scholars. Editors should think twice about inviting articles from scholars who do not yet have tenure. Dissertation advisers and promotion committees should strongly encourage journal submissions rather than edited volume submissions. I published in edited volumes early in my career because I did not know any better. We need to make sure young scholars know the difference between the two types of "peer-reviewed" articles. Another point is that there is a kind of compromise in which an editor can initiate a special-themed volume with a call for papers, and then have an open competition for the best articles. This preserves the idea of genuine peer review, and I have seen this done with journal special issues (but never with edited volumes). Thanks for a most enlightening post! Elaine

    • Many thanks for these positive comments. But I think we need a more refined terminology. I don't think that it would be fair to say that "articles in edited volumes without external reviews are not peer-reviewed". Crucially, the peer-selection works different – it's selection by invitation. In journals, peer-selection happens after an unsolicited submission, and in edited volumes, it happens in a different way. Publishing in an edited volume is very different from simply putting your paper on your Academia.edu site – only the latter lacks peer-review. Maybe one could distinguish between "open peer-selection" and "invited peer-selection".

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <embed style="" type="" id="" height="" width="" src="" object="" allowfullscreen="" allowscriptaccess="" cachebusting="" bgcolor="" quality="" flashvars=""> <iframe width="" height="" frameborder="" scrolling="" marginheight="" marginwidth="" src=""> <object style="" height="" width="" param="" embed=""> <param name="" value="">