Magma Experiment for Code/Bug-based Coverage Benchmarking

Version 1.0
Madadi, Ardi, Seyed Ardeshir, 2026, "Magma Experiment for Code/Bug-based Coverage Benchmarking", https://doi.org/10.17617/3.UQGK4A, Edmond, V1
Learn about Data Citation Standards.
    The dataset is too large to download. Please select the files you need from the files table.
Contact Owner Share
Dataset Metrics
1 Download
Description

Magma Experiment for Code- and Bug-based Coverage Benchmarking

Overview

This dataset contains the raw experimental artifacts for the Magma portion of our study on the concordance (split-half reliability) of coverage-based and bug-based fuzzer benchmarking procedures. [oai_citation:0‡FSE2026.pdf](sediment://file_000000006394720c8478a4c3a5cd9f3e)

The artifacts are intentionally released at a low level (queues, logs, and bug information) so that others can reproduce, audit, and recompute outcomes under alternative analysis choices.

Note: This dataset is not linked directly from the paper. The paper links to a companion code repository, and that repository links to this dataset.

Experimental Setup (Summary)

  • Benchmark suite: Magma v1.2.0
  • Benchmarks used: 18 fuzz drivers (benchmarks) across multiple programs
  • Fuzzers: 8 fuzzers
  • Trials: 20 independent trials per (fuzzer, benchmark) combination (subject to supported combinations)
  • Campaign length: 23 hours per trial
  • Bug evaluation: Magma canary-based ground truth; we count triggered bugs
  • Coverage evaluation: branch coverage (LLVM tooling)

What This Dataset Contains

This dataset includes the raw outputs needed to compute bug-based results directly and to compute coverage-based results via offline replay:

  • Execution queues produced during fuzzing campaigns (per fuzzer, benchmark, trial)
  • Raw bug information from Magma’s canary-based bug reporting (including logs/outputs per run)
  • Per-run metadata required to group results and reproduce the analysis workflow

Important: How Coverage Results Are Obtained

Coverage is not precomputed in this dataset. This dataset includes the execution queues and other artifacts, but coverage values must be derived using the companion code repository.

To extract coverage:

  1. Obtain the companion code repository referenced by the paper.
  2. Configure it to point to this dataset on disk.
  3. Run the provided pipeline to replay the execution queues and compute branch coverage via LLVM tooling.

If you download only this dataset without the companion code, you can still analyze bug-based outcomes, but you cannot reproduce the paper’s coverage-based results.

Directory Layout

The dataset is organized hierarchically by fuzzer, benchmark, and trial. The exact naming is intended to match the assumptions of the companion analysis scripts.

Conceptually:

<fuzzer>/<benchmark>/<trial>/...

Each trial directory corresponds to one 23-hour campaign run and contains the artifacts produced by that run (e.g., queues, logs, bug-triggering information, and auxiliary metadata).

File Count

At release time, this dataset contains exactly 2,860 files.

Known Omissions / Unsupported Combinations

  • SymCC × exif is not present because SymCC does not compile on the exif benchmark; therefore, no such run exists.

Data Completeness Note (Minimal)

A small number of runs (seven) were re-executed after a temporary server failure to complete the expected trial matrix, but were excluded from the paper. The released dataset is intended to be complete for the supported fuzzer–benchmark combinations.

Relationship to Paper and Code

  • Paper: defines the research questions, metrics, and statistical analysis. [oai_citation:1‡FSE2026.pdf](sediment://file_000000006394720c8478a4c3a5cd9f3e)
  • Companion code repository: performs coverage extraction (queue replay), aggregation, ranking, and concordance computations.
  • This dataset: provides the raw artifacts consumed by that code.

Intended Use

  • Reproduction of bug-based outcomes and reanalysis of trial variability
  • Recomputation of coverage-based outcomes via replay using the companion code
  • Alternative analyses of bug and code coverage.

This dataset is not a pre-aggregated results table; it is raw experimental output.

License

This dataset is released under CC BY 4.0.

Citation

Please cite this dataset via its DOI (doi:10.17617/3.UQGK4A). When referencing the methodology or results, also cite the accompanying paper, In Bugs We Trust? On Measuring the Randomness of a Fuzzer Benchmarking Outcome

License/Data Use Agreement

CC BY 4.0

Table
Tree
Filter by
1 to 10 of 2,860 Files
ball.tar
ar/afl/libpng/libpng_read_fuzzer/0/TAR Archive - 22.3 MB
Published Jan 12, 2026
0 Downloads
MD5: 792f767f6bd8c39c49eefc8afa04fb44
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/1/TAR Archive - 21.2 MB
Published Jan 12, 2026
0 Downloads
MD5: d7ceb896cd5eda34553ea6ea424b18e2
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/10/TAR Archive - 18.6 MB
Published Jan 12, 2026
0 Downloads
MD5: 9bf8a31275239599ae1948004a6aae19
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/11/TAR Archive - 21.3 MB
Published Jan 12, 2026
0 Downloads
MD5: 53e7e4493df8f63e19ce5ae3c5f87d2d
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/12/TAR Archive - 19.2 MB
Published Jan 12, 2026
0 Downloads
MD5: d270a8380d2a5a04f58d7138ff9821b0
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/13/TAR Archive - 21.2 MB
Published Jan 12, 2026
0 Downloads
MD5: 40e4b94cf1a95740add084d1a3d91b78
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/14/TAR Archive - 20.2 MB
Published Jan 12, 2026
0 Downloads
MD5: c2a98d9ab3f8817ac5ebcb60f8db03b8
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/15/TAR Archive - 20.3 MB
Published Jan 12, 2026
0 Downloads
MD5: 1c0b5d00333587c0cc83ac090beedec2
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/16/TAR Archive - 19.5 MB
Published Jan 12, 2026
0 Downloads
MD5: 6a999fb8485e25cb2f0453a1145a044a
DATA
ball.tar
ar/afl/libpng/libpng_read_fuzzer/17/TAR Archive - 21.5 MB
Published Jan 12, 2026
0 Downloads
MD5: c4a5fd1211013961b03e986a29a2a70f
DATA
Edit File

This file has already been deleted (or replaced) in the current version. It may not be edited.

Restrict Access

Restricting limits access to published files. People who want to use the restricted files can request access by default. If you disable request access, you must add information about access to the Terms of Access field.

Learn about restricting files and dataset access in the User Guide.

Enable access request
You must enable request access or add terms of access to restrict file access.
Save Changes
Edit Embargo

The selected file or files have already been published. Contact an administrator to change the embargo date or reason of the file or files.
Edit Retention Period

The selected file or files have already been published. Contact an administrator to change the retention period date or reason of the file or files.
Delete Files

The file will be deleted after you click on the Delete button.

Files will not be removed from previously published versions of the dataset.

Select File(s)

Please select one or more files.

Share Dataset

Share this dataset on your favorite social media networks.

Continue
Dataset Citations

Citations for this dataset are retrieved from Crossref via DataCite using Make Data Count standards. For more information about dataset metrics, please refer to the User Guide.

Sorry, no citations were found.
Inaccessible Files Selected

The selected file(s) may not be downloaded because you have not been granted access or the file(s) have a retention period that has expired or the files can only be transferred via Globus.

You may request access to any restricted file(s) by clicking the Request Access button.

Ineligible Files Selected

The selected file(s) may not be transferred because you have not been granted access or the file(s) have a retention period that has expired or the files are not Globus accessible.

You may request access to any restricted file(s) by clicking the Request Access button.

Download Options

The files selected are too large to download as a ZIP.

You can select individual files that are below the 4.7 GB download limit from the files table, or use the Data Access API for programmatic access to the files.

Select File(s)

Please select a file or files to be downloaded.

Inaccessible Files Selected

The selected file(s) may not be downloaded because you have not been granted access or the file(s) have a retention period that has expired.

Click Continue to download the files you have access to download.

Ineligible Files Selected

Some file(s) cannot be transferred. (They are restricted, embargoed, with an expired retention period, or not Globus accessible.)

Click Continue to transfer the elligible files.

Delete Dataset

Are you sure you want to delete this dataset and all of its files? You cannot undelete this dataset.

Delete Draft Version

Are you sure you want to delete this draft version? Files will be reverted to the most recently published version. You cannot undelete this draft.

Unpublished Dataset Preview URL

Preview URL can only be used with unpublished versions of datasets.

Unpublished Dataset Preview URL

Are you sure you want to disable the Preview URL? If you have shared the Preview URL with others they will no longer be able to use it to access your unpublished dataset.

Delete Files

The file(s) will be deleted after you click on the Delete button.

Files will not be removed from previously published versions of the dataset.

Compute

This dataset contains restricted files you may not compute on because you have not been granted access.

Deaccession Dataset

Are you sure you want to deaccession? This is permanent and the selected version(s) will no longer be viewable by the public.

Deaccession Dataset

Are you sure you want to deaccession this dataset? This is permanent an it will no longer be viewable by the public.

Version Differences Details

Please select two versions to view the differences.

Version Differences Details
 
Version:
Last Updated:
Version:
Last Updated:
Select File(s)

Please select a file or files for access request.

Select File(s)

Embargoed files cannot be accessed. Please select an unembargoed file or files for your access request.

Edit Tags

Select existing file tags or create new tags to describe your files. Each file can have more than one tag.

Request Access

  You need to Sign Up or Log In to request access.

Dataset Terms

Please confirm and/or complete the information needed below in order to request access to files in this dataset.

This dataset is made available under the following terms. Please confirm and/or complete the information needed below in order to continue.

Our Community Norms as well as good scientific practices expect that proper credit is given via citation. Please use the data citation shown on the dataset page.

CC BY 4.0

Preview Guestbook

Upon downloading files the guestbook asks for the following information.

Account Information

Package File Download

Use the Download URL in a Wget command or a download manager to download this package file. Download via web browser is not recommended. User Guide - Downloading a Dataverse Package via URL

https://edmond.mpg.de/api/access/datafile/

Compute Batch
Clear Batch
Dataset Persistent Identifier Change Compute Batch
Compute Batch
Submit for Review

You will not be able to make changes to this dataset while it is in review.

Publish Dataset

Are you sure you want to republish this dataset?

Make sure you have checked legal specifications for export control for academic publications by the German Federal Office for Economic Affairs and Export Control.

Select if this is a minor or major version update.

Publish Dataset

This dataset cannot be published until Edmond is published by its administrator.

Return to Author

Return this dataset to contributor for modification. The reason for return entered below will be sent by email to the author.

Curation Status History
StatusDateAssigner
No records found.
Add/Edit a Version Note
Styled Citation