Artifact Evaluation

ATVA’19 will contain a voluntary artifact evaluation phase for accepted papers. An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully replicable. As an example, a typical artifact would consist of the tool (in binary or source code form) and its documentation, the input files (e.g., models analysed or programs verified) used for the tool evaluation in the paper, and a configuration file or document describing the parameters used in the experiments. The ATVA’19 Artifact Evaluation Committee will read the accepted paper and evaluate the submitted artifact w.r.t. the following criteria:

  • consistency with and replicability of results presented in the paper,
  • completeness,
  • documentation, and
  • ease of use.

Papers that succeed in artifact evaluation will receive a badge.

Important Dates

  • June 23, 2019 (AoE): Artifact submission deadline
  • July 24, 2019 (AoE): Notification

Artifact Submission

An artifact submission consists of

  • an abstract that summarizes the artifact and its relation to the paper,
  • a .pdf file of the accepted paper (uploaded via EasyChair), which may be modified from the submitted version to take reviewers’ comments into account (the results obtained during the evaluation will be checked with the results in this .pdf),
  • a link to a public repository (such as Zenodo, Figshare, GitHub, etc.) to a page with the artifact. We require using a public repository to protect anonymity of the reviewers. (We recommend the authors to use Zenodo for its easy way to allow updates of the artifact and providing DOIs, which can be used to refer to the artifact from the paper. If you cannot provide your artifact publicly [e.g. due to the license of included SW or the need of a different OS than Linux], contact the AE chair.) The page needs to have a .zip file (available for download) containing
    • a directory with the artifact itself,
    • a text file LICENSE that contains the license for the artifact (it is required that the license at least allows the AEC to evaluate the artifact w.r.t. the criteria mentioned above), and
    • a text file README that contains detailed, step-by-step instructions on how to use the artifact to replicate the results in the paper.
  • SHA-256 hash of the .zip file.

Artifacts are submitted using the ATVA’19 Artifact Evaluation submission page in Easychair. Please add the link to your artifact and the SHA-256 at the end of your abstract!!! 

Guidelines for Artifacts

We expect artifact submissions to package their artifact and write their instructions such that AEC members can evaluate the artifact using the Artifact Evaluation virtual machine (VM) for VirtualBox available here.

The submitted artifact will be a .zip file containing the software and packages to be installed in the VM.

If the artifact requires additional software or libraries that are not part of the virtual machine, these need to be included in the .zip file (e.g. in the form of Debian packages) and the instructions must include all necessary steps for their installation and setup (see the page with the VM for hints). AEC members will not download software or data from external sources, and the artifact must work without a network connection. In case you feel that this VM will not allow an adequate replication of the results in your paper, please contact the AE chair prior to artifact submission.

It is to the advantage of authors to prepare an artifact that is easy to evaluate by the AEC. Some guidelines:

  • Document in detail how to replicate most, or ideally all, of the (experimental) results of the paper using the artifact.
  • Keep the evaluation process simple through easy-to-use scripts and provide detailed documentation assuming minimum expertise of users.
  • For experiments that require a large amount of resources (hardware or time), it is recommended to provide a way to replicate a subset of the results of the paper with reasonably modest resources (RAM, number of cores), so that the results can be reproduced on various hardware platforms including laptops, and in a reasonable amount of time.
  • State the resource requirements, or the environment in which you successfully tested the artifact, in the instructions file (RAM, number of cores, CPU frequency).

See here for an example of an artifact (in particular the linked .zip file).

Please make sure your submitted artifact can be easily evaluated using your instructions. Before submitting, try to install your artifact into the VM from the scratch (i.e., into a freshly imported VM). Check that the results can be obtained by following your README file. Make sure that the relation of the obtained results and the paper is clear, i.e., say which table in the paper the obtained results correspond to. If reproducing data that are given in a figure, provide gnuplot, R, graphviz, etc. files that can generate the figure.