Since becoming faculty at TTI, I’ve started to appreciate better the tensions of service commitments and I can see how many people begin to view reviewing as a chore, a burden they must bear to maintain goodwill in the “community.” Since I work in a few different communities now, I end up reviewing papers from a lot of different areas : information theory and signal processing of course, but also machine learning, security, and networks. There’s been a distinct uptick in my reviewing queue, which I find somewhat alarming.
Looking back, I did a quick calculation and in the almost 6 months I’ve been here, I’ve either finished or committed to reviewing 9 journal papers and 16 conference papers. These numbers don’t really mean too much, because some journal papers are shorter (e.g. a correspondence) and some conference papers are long (40+ pages including supplementary material). Page numbers also don’t really help because of formatting differences. I’m hoping my new iPad (ooh, shiny!) will let me pack in some reviewing time during my commute and stop me from killing so many trees.
However, I have no idea if these numbers are typical. I’ve turned down review requests because I felt like I don’t have enough time as it is. So readers : what’s a typical review load like? Should I just suck it up and accept more reviews?
Note that I’m not asking about what’s “fair” in terms of I submit N papers and therefore should review 3N or something like that. Those games are fine and all, but I really wonder what the distribution of review load is across individuals for a given journal. More on that point later…
Update: I should be clear that being on a PC will clearly cause your review load to go up. I am on 2 PCs but for smaller conferences; having 10+ ISIT reviews would add significantly to one’s total load.
An Ergodic Walk 
In the 2010–11 academic year (just FYI), I reviewed 24 papers, slanted towards conferences because that’s where we publish most. It doesn’t appear that I was on any PCs that year that required a big commitment of papers; this year I am on several though, so I expect my load will be higher. In the two-year period 2009–11, I declined 39 reviewer requests, though that was slanted towards lots of declines in 2009–10 when my daughter was born (I turned down everything in that academic year). (These stats from my recent advancement package.)
Especially as an assistant professor, IMHO you’re reviewing *more than enough*. One a week (as you appear to be doing) is a LOT of reviewing. I would advise spreading out your reviews among more venues rather than doing lots of reviews for a few, and accepting reviews from editors/committees whom you’d like to impress.
Also, I try to prioritize reviewing for venues with community-friendly publication policies. Refusing reviews because the venue policy is not a reasonable one sends a strong message to the venue and to the editors who asked for your reviews in my experience.
That usually isn’t an issue for me — I turn down Elsevier maybe once a year, but other than that it’s pretty much IEEE Transactions on Foo and occasionally a Hindawi journal (open access).
Good for you. I have been concerned about IEEE, though, as they are AAP members and did not disclaim the RWA, and in fact signed this letter: http://publishers.org/_attachments/docs/library/aap%20-%20dc%20principles%20frpaa%20letter%20house.pdf
plus their public-domain policies suck: http://cr.yp.to/writing/ieee.html .
There really does need to be an open-access alternative. Maybe the JMLR model would work…
That sounds about the same or a little less than the typical ML reviewing load for people at our stage. But of course things differ by community. And the actual papers you review can make a huge difference!
I think the big difference is the timescale — journal reviews can be delayed by a week, say, and it’s no big deal, so the scheduling problem is easier. By contrast, if you get 5 monstrously large COLT papers to review in a short amount of time, it kind of eats up your life for that month. The review traffic statistics, as it were, affect the subjective perception what “load” is.
Absolutely true.