De-anonymized reviewers in discussion

One big difference between reviewing for conferences like NIPS/ICML and ISIT is that there is a “discussion” period between the reviewers and the Area Chair. These discussions are not anonymized, so you know who the other reviewers are and you can also read their reviews. This leads to a little privacy problem — A and B may be reviewing the same paper P, but A may be an author on a paper Q which is also being reviewed by B. Because A will have access to the text of B’s reviews on P and Q, they can (often) unmask B’s authorship of the review on Q simply by looking at the formatting of the reviews (are bullet points dashes or asterisks, do they give numbered points, are there “sections” to the review, etc). This seems to violate the spirit of anonymous review, which is perhaps why some have suggested that reviewing be unblinded (at least after acceptance).

The extent to which all of this matter is of course a product of the how fast the machine learning literature has grown and the highly competitive nature of the “top tier conferences.” Because the acceptance rate is so low, the reviewing process can appear to be “arbitrary” (read: subjective) and so questions of both review quality and author/review anonymity impact possible biases. However, if aim of double-blind reviewing is to reduce bias, then shouldn’t the discussions also be anonymized?

Advertisements

2 thoughts on “De-anonymized reviewers in discussion

  1. I’d never really considered this before, but why do we have anonymous reviewing at all? Presumably the motivation is to avoid situations whereby the author of a candidate paper pressures the reviewer into accepting a paper. It might also be more difficult to criticize and reject a friend/colleague’s paper without the protection of anonymity. Or perhaps if a reviewer rejects a paper the author can get his revenge later when the tables are turned.

    But there are certainly pro’s too. The quality of reviews are guaranteed to improve when the reviewer has a reputation to preserve. If you reject a paper you better make sure you give a very good reason.

    I think in an ideal world reviews should not be anonymous. Unfortunately, nepotism is a factor in science as it is everywhere else…

    • Certainly if I am non-anonymously reviewing paper by Famous Person A, I would be under pressure as a reviewer to not annoy them in case they were asked to write me a tenure letter, etc.

      Nepotism may be an issue, but I really think it’s ego. All researchers (myself included) have a lot of ego tied up in our work, and to have it criticized causes some emotional hurt.

      The negative side of things is that anonymous reviews can be quite mean in the end — I think this is even more so when the authors are anonymous to the reviewers. I somehow doubt that this is because the negativity is the “true impression” of the reviewer, but rather because (like on the internet) it’s easy to be mean when everyone is faceless. People act out their frustration more.

      I think the real problem is that (most) research communities do not have an agreed-upon notion of what a good review is. People learn from their advisors or from reviews they have gotten, but there aren’t very well (or vocally) articulated principles.

      One exception to this was a PC I served on for which Adam Smith was a chair — he sent out a number of helpful points to keep in mind as a reviewer that I think did help guide the review process/format.

Comments are closed.