Congratulations for your recent paper in our journal. I welcome you to our reproducibility section.


A reproducibility paper is a companion article for you original IS paper. The idea of this paper is to give incentives to reproducibility by introducing your experiment as a computational resource. Any reproducible paper must provide a detailed collection of reproducibility resources together with a very detailed reproducibility protocol based on the former ones with the aim of allowing the exact replication of all your methods and results in an as much easy as possible manner. In addition, we expect that any reproducible paper contributes to set a standard experimentation platform for any future work in the same line of research, and thus it being reused in future research.


For the reasons detailed above, your reproducible paper should provide all details on software used, versions, useful links, how to compile/install, examples on how to use and replicate the results and plots of the paper, etc. Likewise, you should also provide all your raw output data files for an easy comparison and inspection, despite they will be generated from scratch by your experiment artifact. All data processing and plots should be automated using R scripts, or other proper scripting language, starting from your raw output data. Workflow diagrams which provide an easy explanation of your reproducibility protocol are welcome.


Once you submit this paper together with the artifact (software and data), reviewers will analyze them and try to reproduce, vary, and validate the results published in your original paper (raw results, tables, plots, ...). Reviewers may suggest ways to improve your code and reproducibility coverage as well. I should note that the review is not blinded -- authors will know who the reviewers are. Here, we are trying to be as transparent as possible.


If your results can be validated, and if your artifact is in good shape, then your paper and artifact are accepted: (1) the companion paper (reproducibility paper) is published by Elsevier, having the reviewers as co-authors as well -- reviewers (and the editor) can add information in the paper w.r.t. the reproducibility aspect; and (2) a reference to the artifact will be created for the reproducibility paper.

PS: when I say code in "good shape", I mean code and data that can be understood and reproduced without too much effort. We certainly understand that the code comes from a research environment, rather than a multi-million dollar company!


I send enclosed the guidelines for authors. We encourage authors to use ReproZip<https://www.reprozip.org/> to make their experiments reproducible, as well as publishing their reproducibility dataset in Mendeley and submitting a companion Data in Brief article to introduce the former one. It is very important that the companion reproducibility dataset being self-contained, it means that all data, software and resources required by your experiments should gathered in a consolidated dataset although your source code repository being GitHub. Likewise, note that our ultimate goal is that all your experiments can be reproduced in the long-term. For this reason, we encourage the use of self-contained lightweight virtual machines based on Reprozip or Docker which provide a full setup of the execution environment of your experiments.


I provide below some reproducibility papers published in the past as examples.

- https://doi.org/10.1016/j.is.2015.12.004

- https://doi.org/10.1016/j.is.2017.02.002

- https://doi.org/10.1016/j.is.2019.03.007

Please, do not hesitate to contact with me if you have any question.

Juan Lastra-Díaz
Reproducibility Editor
Information Systems
jlastra@invi.uned.es