conference selectivity, publication delay, and a proposal

I should preface this post by warning any readers that these thoughts are pretty rambling and tentative. I don’t fully understand all of these issues, but they do interest me, so I find it helpful to write about them. I’m not even certain that my proposed idea here would work, but I think there seems to be some problems with the status quo that need to be addressed.

A recurrent theme in many conversations I had at ISIT was about the nature and purpose of the conference. Part of this concern stems from the stressful nature of the job market, and some from changes in the culture of the field. With regards to the former, the less selective nature of ISIT and electrical engineering conferences in general makes it difficult to evaluate the quality of work of a graduate student without being familiar with the area. Because of the incredibly long delay in journal publication, there may be no objective approval of the work by the time the student graduates. The underlying problem is that information theory is expanding and changing rapidly, and the publication and conference infrastructure hasn’t adapted to deal with the higher pace and volume of research. A lot of different arguments and proposals were thrown out there, but one that seemed to garner some interest was the foundation of a new journal for smaller results — a new Information Processing Letters, if you will.

Several people complained that ISIT was too big and that there were too many “less interesting” results. The trite answers are that the field is pretty big and that what is interesting to specialists may not be interesting to the general information theory community. If we take a comparative view, another model to look at is that of CS theory. There there are two or three very selective conferences — FOCS, STOC, and SODA. The acceptance rate for these conferences is quite low, they are single track, and all of the results presented are therefore “interesting.” Adopting such as system off the shelf misses the point of a conference like ISIT, I think.

ISIT serves several different functions that I know about. The first is to take a snapshot of the field and its progress and capture this in the proceedings. Prior to last year, the proceedings only contained a single-page extended abstract and summary of the results. The original intent that those results which were big enough would get sent to the Transactions. Another function is get sufficient funds via registration fees to pay for the transactions and the operation of the Information Theory Society. Now the proceedings are on CDROM and contain the full (5 page) papers, so many of them are (mostly) fleshed-out short papers. Finally, the conference is supposed to facilitate the contact between different research subcommunities, e.g. to get the coding people to talk to the Shannon theorists.

The first fundamental problem is that information theory has expanded as a research field — there were 4+ sessions on Network Coding alone, and those problems didn’t even exist 10 years ago. As someone told me, the flavor of the conference has shifted slowly over the years, and the incremental changes are hard to detect. The net effect is that the Transactions have become enormous and bulky — many classic papers from years ago were 10 pages, but now it’s rare to see a paper under 15-20 I bet. The conference has also become enormous and bulky, with so many parallel sessions that it’s hard to even get an idea of what is going in the field. Is this because too many papers are accepted? Perhaps. I certainly wouldn’t mind if there were about 25% fewer papers there, just so that I could see a greater percentage of what was going on. Greater selectivity would mean higher quality as well, but there are costs to that. For example, next year the conference is in Nice, and I doubt many grad students in the US would be able to just go there unless they had a paper. A lower acceptance rate would also impact the fundraising aspects.

What about the positive benefits of a low acceptance rate? Some people have argued that a low acceptance rate provides a skewed but somewhat objective measure of the quality of research done by a graduate student. That is, four solid papers at “good conferences” (in the CS sense) mean that the student has developed a solid research program. This prestige factor doesn’t hold in EE and information theory because the prestige is with journal publications, not conferences. But I noted earlier, journals take forever to publish papers, and the published papers tend to be quite long (and not because they have lots of figures). So if the burden of prestige is shifted to conferences, it might be better.

The second fundamental problem is that the pace of research is speeding up, while the venues for providing thoroughly peer-reviewed research are diminished. While all papers at ISIT are reviewed, the reviewing is somewhat uneven (I had very few reviews of my paper, while others had many). Since conferences are more about sharing the results of investigations rather than presenting the most interesting new results of the field, it’s hard to separate those ides which are going somewhere from promising directions and dead-ends, both of which may be interesting in their own right.

One solution that might work for both these problems (increased research pace and more people) would be to have more venues for journal publication. A new journal with thorough peer review and editing but an emphasis on smaller results would possibly alleviate the burden on the Transactions and speed up the pace of journal publication. The conferences could still be big, but then conference presenters with more or less polished end results could journalize those rather than waiting to included in a larger paper sent to the Transactions. It’s worth thinking about in addition to pruning down the size of ISIT a bit to avoid having 12 parallel sessions 5 years from now.

Advertisements

0 thoughts on “conference selectivity, publication delay, and a proposal

  1. I think there are a lot of fascinating issues, and I encourage you to try to take a systems view. What problems are you really trying to solve? What are the incentives for various behaviors?

    Unfortunately, my usual conclusion is that the problems are fairly ingrained in the incentive system. The biggest single problem in machine learning (I don’t know about coding theory) are that there are too many papers published, and that is sufficiently easy to disguise a bad paper as a good one that it is worth one’s while to do so, resulting in a situation where it is very hard for a researcher to figure out what’s worth reading. Of course, how can we blame individuals for acting in their own interest? As long as academics are judged largely on their publications, by people who need shortcuts such as quality and some rough measure of quantity such as publication venue, these problems will persist.

    Also, remember that it is rarely worthwhile to ask chickens and pigs how to run a farm.

  2. It seems to me the dominant problem is not evaluating whether a paper is novel or advances the field. I don’t think people really spend that much time blowing up their own results so that they look more interesting than they are. The paper that most people I met complained about were of the form “we took this ad-hoc method and applied it to this problem and this is how well it did” without providing any insight even into why it worked as well as it did and what this says about the nature of the problem.

  3. From my perspective, a paper like the one you describe is useless. I don’t want to read it, and I’m at least as well off if it isn’t ever written or published. However, if it is easy for me to *rapidly* determine that I don’t need to read it, then the marginal cost to me (and to everyone else who only wants to read the few good papers) of this paper’s existence is low. On the other hand, the marginal benefit to the author might be quite high, especially if the paper is published in the same conference as good papers, and because the conference contains the good papers in the field, it has a reputation for quality, and *especially* if the people evaluating the author, who are probably not in the same field as the author, are unable to tell that the paper is a useless paper.

    So in some sense, a field in which most of the papers published in the good conferences are bad, people who are experts in the field can find the good papers very rapidly, and people who are non-experts cannot is best “for the field” — research is not impeded, and everyone gets tenure.

    In machine learning, I consider myself fairly expert, but I feel I frequently have to read a lot of a paper to tell that it’s bad. I spend a large fraction of my time reading papers that turn out to be bad. So in machine learning, the high acceptance rate seems like a big problem to me. But I don’t really have any good answers.

    It’s even worse in speech, the other community I’m in. There, the top conferences (ICASSP, ICSLP) have enormously high acceptance rates (over 50%), and the papers are so short (4 pages) that I can’t tell if they’re bad even after I’ve read the whole paper. You often have to try (and usually fail) to actually implement a partially described method before you can say anything.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.