SIGMOD 2010 Repeatability & Workability Evaluation

SIGMOD 2008 was the first database conference that proposed testing the code associated to conference submissions against the data sets used by the authors, to test the repeatability of the experiments presented in the submitted papers. A detailed report on this initiative has been published in ACM SIGMOD Record, 37(1):39-45, March 2008.

The experience has been continued in a slightly modified form in conjunction with the SIGMOD 2009 conference. A report on this effort will soon appear in the SIGMOD Record.

The repeatability and workability evaluation in conjunction with SIGMOD 2010 will continue along the lines of the 2009 edition, with some improvements related to the procedure.

The goal

On a voluntary basis, authors of accepted SIGMOD 2010 papers can provide their code/binaries, experimental setups and data to be tested for

repeatability of the experiments described in the accepted papers;
workability in the sense of running different/more experiments with different/more parameters than shown in the respective papers;

by a repeatability/workability committee (to be announced), under the responsibility of the repeatability/workability editors-in-chief (RWE in short). The RWE are Ioana Manolescu and Stefan Manegold.

The procedure

Authors of accepted papers will be contacted by e-mail as soon as acceptancy is determined and invited to submit

(download URIs for) code (executables or sources)
(download URIs for) data
instructions on:
- how the experiments described in the paper should be re-run;
- how further experiments could be run, on inputs similar to those used in the paper, but different from those.
the accepted paper

Submissions will be handled using a Repeatability/Workability Conference Management Tool (RWCMT, in short).

Past experience has demonstrated that repeatability and workability can greatly benefit from the availability of authors to interact with the repeatability reviewer, and help solve minor issues related to the installation, configuration, and usage of the code. Thus, the electronic tool will enable repeated interactions between the reviewers and the authors, in the style of a message board. Thus, the 2010 process will involve such interaction, too.

For reference, the instructions sent to the authors in 2009 can be found here.

The RWE will designate for each submission:

A first reviewer, which will do all the verification work as far as he/she is able to (download and install the code, run it, write a repeatability/workability report). This should span at most 2/3rds of the reviewing period.
A second reviewer, which will check the report of the first, help clarify any pending issues. The second reviewer is expected to interfere, if needed, in the last 3rd of the reviewing period.

The first and second reviewer will interact until they are both satisfied with the terms of the report. They will both sign the report. If there is disagreement that the reviewers cannot work out, the RWE have the final say. They may propose alternative wording for the report, more tests, and/or endorse responsibility together with one reviewer, if the other cannot agree with the chosen wording (and thus is unwilling to sign it).

During the evaluation, the first reviewer and the authors interact via the RWCMT. The second reviewer and the RWE may also participate to the discussion. The recommendation is that the first reviewer is left alone with the authors during the first 2/3rds of the reviewing period, to avoid confusion. The RWCMT documents all interaction between the reviewers and the authors.

The identity of the reviewers is hidden during the evaluation process, but obviously will be revealed afterwards by the reviewers signing their report.

The output

The final RW report will include

a summary of the interaction with and fixes by the authors that were required to get the experiments running properly;
a repeatability result (what could be repeated);
a description of what else the first reviewer was able to run and to which extent the results are expected.

The final RW report may be published in the ACM PubZone, assuming the authors agree to this.

Code archiving

Participating in the repeatability/workability evaluation does not entail the code is archived for everyone to use subsequently. Pending authors' agreement, the code could be uploaded in the SIGMOD PubZone.