Daily Archives: June 11, 2014

NIPS 2014 Review Quality Control Procedure

Posted on by

I got this email yesterday:

Dear Author of a NIPS 2014 Submission,

You are in for a treat! This year we will carry out an experiment that will give us insight to the fairness and consistency of the NIPS reviewing process. 10% of the papers, selected at random, will be duplicated and handled by independent Area Chairs. In cases where the Area Chairs arrive at different recommendations for accept/reject, the papers will be reassessed and a final recommendation will be determined.

I welcome this investigation — as an author and reviewer, I have found the NIPS review process to be highly variable in terms of the thoroughness of reviews, discussion, and the consistency of scores. I hope that the results of this experiment are made more publicly available — what is the variance of the scores? How do score distributions vary by area chair (a proxy for area)? There are a lot of ways to slice the data, and I would encourage the organizing committee to take the opportunity to engage with the “NIPS community” to investigate the correlation between the numerical measures provided by the review process and the outcomes.

Advertisement

Readings

Posted on by

My first semester on a “real job” was sufficiently busy to prevent me from reading as much as I would have wanted. I also blame the driving commute.

Hidden History of New Jersey [Joseph G. Bilby, James M. Madden, and Harry Ziegler]. This was a gift from Erin, who apparently has not written up her analysis of the Twins’ crushing of the Yankees yet. This is a collection of short essays about New Jersey and some of the quirkier characters who provide local color. There’s a heavy focus on military history, which was less interesting to me, but later chapters delve into the immigration history and politics in Jersey City and elsewhere that provide a useful context and analogy for our current situation. There’s a wealth of references, although as the authors point out, no book on the history of the Klan in New Jersey. Apparently they had some sort of summer resort there.

Husband of a Fanatic [Amitava Kumar]. Kumar looks at Hindu-Muslim and India-Pakistan relations after the Babri Masjid riots through the lens of his own marriage to a Pakistani Muslim woman. A fascinating and harrowing book which did not leave me particularly optimistic about the new Modi government.

The Pun Also Rises [John Pollack]. A present from my brother, this book is a delightful (or if you are No Fun, painful) tour through the history and variety of puns and joking wordplay. Pollack waxes poetic a bit, but this is fun read.

Carpe Jugulum [Terry Pratchett]. A Discworld novel with the witches and vampires. Brain candy.

Your Republic Is Calling You [Young-Ha Kim]. This is a novel about a North Korean “sleeper agent” in Seoul who thinks he’s been forgotten but after years of no instructions is given a day to rendezvous with a pickup that will take him back to the North. He’s grown comfortable in his new life though, and things are difficult. The back of the book compares Kim’s writing to Murakami’s (hard to tell because it’s all in translation). I initially felt that poisoned my reading of the book, but in the end I think that they are similar in tone/affect. I rather enjoyed this book, and it made me want to investigate more contemporary Korean literature.