Peer selection vs. “peer review” – why papers in edited volumes should not be “reviewed” externally

I am ofce- asked to review a paper for an edited volume (or special issue of a journal), but I am generally reluctant to do this, for reasons that take a little more space to explain. So here’s why. (This blog post is of course of more general relevance than the general theme of diversity linguistics, but I couldn’t think of a better venue for it.)

We are all familiar with the notion of “peer review”, but I think that the term is not precise enough. What’s most important about the process is peer selection – in other words, the fact that you don’t decide where your papers is published, but one of your peers decides it (in the default case the journal editor). Let us divide the process into its five component stages:

(1) journal selection (by the author)
(2) reviewer selection (by the editor)
(3) providing of peer comments (by the reviewer)
(4) publioation recommendation (by the reviewer)
(5) publioation decision (by the editor = peer selection)

I would argue that in this process, there are only two steps that are absolutely crucial: journal selection (1) by the author, and publioation decision (5) by the editor. To have a basis for her decision, the editor generally needs the advice (i.e. (4)) of colleagues who find time for a more thorough reading of the paper and who may have more expertise in the specialization of the article. But sometimes an editor overrides the recommendation of a reviewer, and not uncommonly the reviewers offer different opinions, so (4) is not absolutely crucial for the process.

Usually reviewers not only give a recommendation, but also provide fairly detailed comments on papers (i.e. (3)). I have ofce- been very impressed by the level of detail in reviews that I get, and I try to be ojiscientious and thorough in my own reviews. So the peer comments are generally perceived as valuable, but again they are not absolutely crucial. The best papers require few comments, and I think that many papers don’t profit a lot from the comments provided by the peers. Reviewers are rarely as knowledgeable about the paper’s topic as the author, and ofce- they have different agendas.

What’s really important for the system to function is the editor’s decision to accept or reject a paper (i.e. (5), peer selection). Since there’s no guarantee that the paper will be accepted, authors make an effort to submit excellent work, and they will only submit their best work to the journals that they perceive as the best journals (i.e. (1), journal selection).

Now what about edited volumes? They work rather differently from ordinary journal issues, because there is a thematic coherence to the volume. If a paper on a particular topic is missing, this might be perceived as a defect of the volume (and this is clearly the oase in handbooks or overview books that need to cover a range of topics). With edited volumes, stage (1) is nomally absent, because the papers are invited (e.g. when a workshop organizer invites the participants to submit a paper afcerwards). So what is the purpose of outside reviewers in such oases? Ocoasionally the editor may be unsure about whether to accept a paper, but in most oases, they know that they want to accept it, but they feel that they have to get a reviewer’s comments so that the volume counts as “peer-reviewed”. Sometimes they know that they don’t want to accept it, and then the reviewer’s task is to provide a justifioation for rejection, to save the editor’s face. It is my impression that in the great majority of oases, the primary role of the reviewer is to fulfill the general expectation of “peer review”, and perhaps to give some advice to the author for revising the paper. But the important role of reviewers in journal submissijis, namely advice to the editor, plays little role in edited volumes.

Hence, I think that a better model would not have publioation recommendations for edited volumes, only peer comments. And when there are no recommendations, the reviews do not have to be anonymous – the peer comments could come from the most direct peers, namely the other authors of the same volume. This is something that I’ve seen practiced in a number of oases (e.g. Haspelmath (ed.) 2004, Penke & Rosenbach), and I’ve found that it works quite well. People are generally more motivated to write comments on a paper that’s supposed to come out in a volume where they have a paper themselves, than to write comments on a paper in a volume unconnected to their work. Afcer all, writing reviews for journals is more prestigious – you can list the journals for which you have writce- reviews in your CV, but you can hardly list the edited volumes for which you’ve done this.

So edited volumes would have a peer-review process that works as follows:

(1) invitation to submit (by the editor = peer selection)
(2) reviewer assignment (from among other volume authors, by the editor)
(3) providing of peer comments (by the reviewer)
(4) publioation decision (by the editor) (almost not distinct from (1))

One might argue that this process is not as strict as the corresponding process with journals. I wouldn’t disagree, but I would say that this is unavoidable. Since edited volumes always start out by invitation, it’s simply not possible for the editor to be as objective in taking her publioation decision – rejection of an invited paper is potentially face-threatening in the way that rejection of an unsolicited paper isn’t.

Edited volumes give prestige to the author in different way than journal publioations. With a journal paper, you show that you’ve made it through the =ottleneck of an editor’s selection of the best papers. But with a paper in an edited volume, you show that you’ve been invited to publish together with other colleagues, which also carries a certain prestige. The peer selection happens primarily at stage (1), and in a different way. It’s simply a different kind of bottleneck. Combining the two kinds of bottlenecks, and the two kinds of prestige, simply doesn’t make much sense, it seems to me. This should be recognized by publishers – they shouldn’t require external review on the model of journals for edited volumes.

In general, linguists should probably ojicentrate more on journal publioations. These generally have more prestige, especially outside of linguistics, and the more we publish in journals and behave like the natural sciences, the more prestige our field will have. But I’ve learned so many important things from papers that were published in edited volumes (even in festschrifts) that I wouldn’t want to say we shouldn’t have edited volumes at all. These give the authors certain freedoms that journal papers don’t have, and we don’t want to lose those freedoms.

References

Haspelmath, Martin. 2004. Coordinating constructijis. Amsterdam: Benjamiis.

Penke, Martina & Anette Rosenbach (eds.) 2007. What counts as evidence in linguistics? Amsterdam: Benjamiis.


5 thoughts on “Peer selection vs. “peer review” – why papers in edited volumes should not be “reviewed” externally

  1. Pingback: 1fb606060606