The workshop will be held at McGill University's Bellairs research
institute, located directly on a beautiful beach in Barbados.
The Bellairs institute provides very basic accommodation (in
double occupancy only), and partial board, for a minimal fee
(in the order of 50 USD per day - exact price subject to
change). The workshop activities were held directly at the
institute.
About Bellairs Workshops
Numerous computer
science workshops are held at the Bellairs station every
year. SARS2008 will be the third in a series of Bellairs
Workshops on Software Evolution. Information about previous
software evolution workshops can be found by following the
links below:
The SARS 2008 Participants at the Bellairs Station
Workshop Summary
Plenary discussions, talks, small-group work sessions, and
informal meetings took place to explore how different
recommendation systems for software development could be
improved, combined, and evaluated.
The Sunday morning session was dedicated to defining
recommendation systems in the context of software
development. Of the proposed definitions, participants
selected three tentative definitions of recommendation systems
for software development but did not reach a consensus on how
to merge them:
A recommendation system is a system that recommends
relevant information items to assist decision making for
achieving a specific task.
A recommendation system is a software tool that can
produce recommendations that a user may need to consider to
complete a specific sub-goal in the completion of a software
engineering task.
Recommendation systems for software engineering collect
data from software-relevant artefacts, analyze it for
relevance to a specific task and context, and nominate a set
of elements and properties relevant to an engineer for their
consideration.
A talk on the future of programming environments by Andreas
Zeller highlighted the opportunities offered by software
development artefacts (version histories, bug histories,
source code, execution logs, build logs, email archives, user
forums, external documentation, tasks histories (who did
what), test cases, user interactions, etc) in making
recommendations.
The Sunday afternoon session included a discussion on the
ethical issues associated with recommendation systems, such as
privacy (e.g., how to prevent the misuse of data collected by
the recommendation system), and behaviour shaping (e.g., how
recommendation system influence developers' behaviour). A
number of strategies to address these issues were discussed.
On Monday, participants presented talks on the application of
machine learning (ML, by Tobias Scheffer) and information
retrieval (IR, by Andrian Marcus and Giulio Antoniol) to
research on recommendation systems. The outcome of these
sessions was a discussion of the need for further interaction
and collaboration between researchers in the IR, ML and
recommendation systems disciplines. In Monday's last session,
participants defined the infrastructure of a typical
recommendation system as consisting of an extractor, a fact
storage module, an analysis module, filtering/cleaning module,
a presentation module, and a feedback module.
On Tuesday, Tao Xie and Annie Ying presented talks on
recommendation systems for code reuse, and for teams and
managers, rep. Informal discussions were continued in the
afternoon during a tour of the island.
Wednesday was dedicated to the evaluation of recommendation
systems. Participants discussed good and bad practices for
evaluating recommendation systems. Because the research on
recommendation systems is typically incremental, some
participants expressed the need for more repeatable evaluation
in order to assess the improvements of new techniques. Ways to
improve the evaluation included: (1) specifying the general
task and the more specific task that were the target of the
recommendation system to avoid overfitting to a particular
task, and (2) publishing the raw data used during the
evaluation to enable fair comparison. The participants also
stressed the importance of triangulation when evaluating a
recommendation system.
The workshop was concluded with a "collected wisdom" session
on Thursday morning where the networking opportunities, the
chance to learn about the systems built by the other
participants, and the sessions on the evaluation of
recommendation systems were indentified as key aspects of the
workshop.
