Martin Robillard · Blog

ICSE Reviews: A Free but Limited Resource

15 April 2016 by Martin P. Robillard and Alessandro Orso

The ICSE 2017 Call for Research Papers is out, and there has been some discussion on social media and through email on the fact that submissions to the research track are limited to three papers per author.

One of the main reasons for this policy is that every year more people submit more papers, but the pool of qualified reviewers willing to make the necessary commitment does not grow in proportion. Increasing the size of the program committee is not a panacea, because of coordination overhead and calibration challenges (the ICSE 2017 reviewers already include 34 program board members and 94 program committee members). Allowing the imbalance between the demand for reviewing and reviewing resources to continue to increase is likely to condemn the review process to lose its meaning. The NIPS Experiment is a disconcerting indication that randomness in paper acceptance decisions is not an exaggeration. At the same time, the very perception of a random review process is an encouragement to submit more papers. We are stuck in a vicious cycle.

A key to breaking the cycle is the recognition that the time donated by ICSE reviewers is a valuable and limited resource. We cannot increase it at will. When solicited to serve on the ICSE 2017 program committee, for instance, close to two dozen regular ICSE authors declined to serve, due to lack of time. It is also not uncommon that some who initially volunteer to review find themselves unable to fully deliver on their commitment (leading to, for instance, cancellations or two-line reviews authors occasionally complain about). The fact that other communities deal with larger submission numbers should not mislead us, as each community has different standards and characteristics. Case in point, some communities use multiple program committees for different sub-areas to deal with large submission numbers, a solution that has been discussed several times within the ICSE Steering Committee before being dismissed due to its drawbacks.

In short, with an unlimited access to a limited number of reviewers, the tragedy of the commons is staring us in the face.

In addition to continuing to make intense demands on reviewers' time and scaling up the magnitude of the review process, this year we are also asking authors to make concessions for the long-term sustainability of the peer-reviewing offered by ICSE. With an acceptance rate below 20%, there is room for additional pre-submission reviewing to be done within research groups and institutions, by colleagues, and so on.

We are working hard to make the expectations for acceptance at ICSE 2017 as clear and uniformly applied as possible. Our vision is that if authors submit fewer papers and still get a similar (or possibly even higher) number of submissions accepted, we will maintain or even improve the quality of the ICSE program while using reviewing resources in a more efficient, fair, and sustainable way.