Bellairs workshop On
Software Analysis for
Recommendation Systems

8-15 February 2008, Barbados

Theme and Goals

Recommendation Systems for software development are tools that provide developers with information to guide them in a number of activities (e.g., software navigation, debugging, refactoring), or to alert them of potential issues (e.g., the introduction of bugs, conflicting changes, failure-inducing changes) . Recommendation systems can draw from a wide variety of input data, and benefit from different types of analyses. The goal of this workshop is to explore how different software analyses for recommendation systems can be combined and evaluated to provide maximum value to developers.

Location and Facilities

The workshop will be held at McGill University's
Bellairs research institute, located directly on a beautiful beach in Barbados. The Bellairs institute provides very basic accommodation (in double occupancy only), and partial board, for a minimal fee (in the order of 50 USD per day - exact price subject to change). The workshop activities were held directly at the institute.

About Bellairs Workshops

Numerous computer science workshops are held at the Bellairs station every year. SARS2008 will be the third in a series of Bellairs Workshops on Software Evolution. Information about previous software evolution workshops can be found by following the links below:


The workshop is organized by
Martin Robillard. Due to restricted space availability, attendance at the workshop is by invitation only.


SARS Participants
The SARS 2008 Participants at the Bellairs Station

Workshop Summary

Plenary discussions, talks, small-group work sessions, and informal meetings took place to explore how different recommendation systems for software development could be improved, combined, and evaluated.

The Sunday morning session was dedicated to defining recommendation systems in the context of software development. Of the proposed definitions, participants selected three tentative definitions of recommendation systems for software development but did not reach a consensus on how to merge them:

  1. A recommendation system is a system that recommends relevant information items to assist decision making for achieving a specific task.
  2. A recommendation system is a software tool that can produce recommendations that a user may need to consider to complete a specific sub-goal in the completion of a software engineering task.
  3. Recommendation systems for software engineering collect data from software-relevant artefacts, analyze it for relevance to a specific task and context, and nominate a set of elements and properties relevant to an engineer for their consideration.
A talk on the future of programming environments by Andreas Zeller highlighted the opportunities offered by software development artefacts (version histories, bug histories, source code, execution logs, build logs, email archives, user forums, external documentation, tasks histories (who did what), test cases, user interactions, etc) in making recommendations.

The Sunday afternoon session included a discussion on the ethical issues associated with recommendation systems, such as privacy (e.g., how to prevent the misuse of data collected by the recommendation system), and behaviour shaping (e.g., how recommendation system influence developers' behaviour). A number of strategies to address these issues were discussed.

On Monday, participants presented talks on the application of machine learning (ML, by Tobias Scheffer) and information retrieval (IR, by Andrian Marcus and Giulio Antoniol) to research on recommendation systems. The outcome of these sessions was a discussion of the need for further interaction and collaboration between researchers in the IR, ML and recommendation systems disciplines. In Monday's last session, participants defined the infrastructure of a typical recommendation system as consisting of an extractor, a fact storage module, an analysis module, filtering/cleaning module, a presentation module, and a feedback module.

On Tuesday, Tao Xie and Annie Ying presented talks on recommendation systems for code reuse, and for teams and managers, rep. Informal discussions were continued in the afternoon during a tour of the island.

Wednesday was dedicated to the evaluation of recommendation systems. Participants discussed good and bad practices for evaluating recommendation systems. Because the research on recommendation systems is typically incremental, some participants expressed the need for more repeatable evaluation in order to assess the improvements of new techniques. Ways to improve the evaluation included: (1) specifying the general task and the more specific task that were the target of the recommendation system to avoid overfitting to a particular task, and (2) publishing the raw data used during the evaluation to enable fair comparison. The participants also stressed the importance of triangulation when evaluating a recommendation system.

The workshop was concluded with a "collected wisdom" session on Thursday morning where the networking opportunities, the chance to learn about the systems built by the other participants, and the sessions on the evaluation of recommendation systems were indentified as key aspects of the workshop.

Detailed Information

Continue on to the restricted area (participants only).