I know I complain about this all the time, but in my post-job-hunt effort to get back on top of things, I’ve been trying to manage my review stack.
It is unclear to me what the reward for submitting a review on time is. If you submit a review on time, the AE knows that you are a reliable reviewer and will ask you to review more things in the future. So you’ve just increased your reviewing load. This certainly doesn’t help you get your own work done, since you end up spending more time reviewing papers. Furthermore, there’s something disheartening about submitting a review and then a few months later getting BCC-ed on the editorial decision. Of course, reviewing can be its own reward; I’ve learned a lot from some papers. It struck me today that there’s no real incentive to get the review in on time. Parv and Anant may be on to something here (alternate link).
An Ergodic Walk 
What’s the incentive to review anything in the first place? Does it apply to timeliness?
(BTW, the link to the paper is broken.)
The link seems to work for me! Hmmmm. The incentives are primarily “good citizenship” and “reputation” which I will clarify in a followup post.
nothing beats using distortion to guarantee some level of privacy :). I know that wasn’t supposed to be the message but given how paranoid I am about others finding any semblance of my identity in their reviews, the wonderful idea of ‘a knock on the knuckles’ for late reviewing has to be balanced with some privacy guarantees. Which brings me to the question, how exactly are reviewers picked? Is the Parv and Anant model really right in assuming that its picked from a related pool of works?
Yeah there’s a lot of modeling going on there, but I think for the reputation model to be effective it has to be more public and less informal.
Your link to Parv and Anant seems broken to me.
The reward of reviewing on time is the knowledge that you agreed to do something and then kept it, the satisfaction of a job well done. There is nothing wrong with AE’s knowing you are a reliable reviewer.
In general, I guess I’m a believer in the theory that the way to avoid doing work you dont want is to spend your time doing what you’re passionate about and say no to everything else rather than trying to give others the impression that they shouldn’t ask you to do things because even if you say yes you’ll do a cruddy job.
I’ll post an alternate link — it seems to work ok for me.
I agree, there is a value in doing a good job on time, like one said one would. I think that this may not be sustainable (more details in a followup).
I think that a slightly different issue is that some papers are fun to review and some are a real chore. I like reviewing the former, and I don’t mind reviewing the latter because, well, someone has to to do it. But I get the impression that papers in the latter category tend to languish, rather than getting a speedy “thanks but no thanks” or “try again” that they deserve. As a result, the whole thing gets gummed up and reviews come back ridiculously delayed.
Anyway, more thoughts in the next post.
Review on time if you can. Do it because you’re a good person, but professionally, do it because the people on the editorial board will appreciate it and know you’re a reliable fellow. This pays off in the long run. People on editorial boards are good people to have in your plus corner.
In parallel, manage your OWN reviews. Review when you have time to review. Say no when you don’t. “I’d like to help you out but I’m just slammed right now; I just can’t accept another review”. Editors have been cool with this in my experience.
In a few years, there’ll not ever be a moment, 365 days a year, when you don’t have a review over your head.
I think “do it because you’re a good person” ends up being non-sustainable in some communities. The problem with reputations is that they aren’t public enough. I’m not saying that you should get paid or something, but it seems right now that in Information Theory, reviews are (very often) late and (occasionally) cursory.
I think I’m already at the point where I have 2-3 reviews over my head 365 days of the year. It’s been like that since I started my postdoc, really.
Your point about rejecting review requests is a good one, and I’ll take that up in the next post too.
Pingback: clarification on reviewer incentives « An Ergodic Walk
Pingback: clarification on reviewer incentives « An Ergodic Walk