Cleverest - Artifact of FSE '26 paper "Evaluating LLM-based Regression Test Generation"

Cleverest is a feedback-directed, zero-shot LLM-based regression test generation technique proposed in the paper "Evaluating LLM-based Regression Test Generation". We evaluate its effectiveness on 72 commits of six subject programs: three already in WAFLGo dataset (Mujs, Libxml2, and Poppler) and five newly added programs (JerryScript, Z3, PHP, JQ and Micropython). This repository contains the implementation of Cleverest along with the dataset and instructions to reproduce the main results in the paper.

Citation

The paper will be published by FSE 2026 with title "Evaluating LLM-based Regression Test Generation", while the first version of the preprint is available on arXiv:2501.11086 with a different title "Can LLM Generate Regression Tests for Software Commits?".

If you want to use Cleverest in your research or refer it, please use the following citation:

@inproceedings{FSE26-llmtesting,
  author       = {Liu, Jing and Lee, Seongmin and Losiouk, Eleonora and B{\"o}hme, Marcel},
  title        = {Evaluating LLM-based Regression Test Generation},
  year         = {2026},
  booktitle    = {Proceedings of the ACM International Conference on the Foundations of Software Engineering},
  series       = {FSE'26},
  numpages     = {21},
}

The supplementary material is available at supp/supplementary.pdf.

Data Replication

In the paper, we run experiments on 8 software (mujs, libxml2, poppler, jerryscript, z3, php, jq, micropython) for 2 scenarios (bug-finding and bug-reproduction) under 10 different ablation settings (default, MSGONLY, DIFFONLY, GENCMD, TEMP1.0, gpt-4o-mini, deepseek-r1, ITER10, NOFEEDBACK, ENHANCED/REDUCED for FIX scenario) and repeat 10 times. Each software has 180 experiments (200 for libxml2, poppler and jq becuase these three have CLI options to test with GENCMD).

The result of all experiments on all commits is aggregated as a CSV file in figure/aggregated_results.csv with 6760 data rows. The tables in the paper are then drawed from this CSV file.

The data of experiments in the paper is available at repdata.tar.xz (uncompressed size 6 GB), which contains 1500 text files prefixed by SUMMARY_ and 1500 folders prefixed by exp_. For each software, there are 180 (200 for libxml2, poppler and jq) SUMMARY_* files and exp_* folders corresponding to different ablation settings. The SUMMARY_* file contains the configuration of a experiment and the result for each commit in a table format. The exp_* folder contains intemidiate data containing full history of interacting with LLM (chat_*.log), all generated test cases (INPUT_*), non-trival test cases that trigger bugs, cause output difference or reach commit-changed code (TRIGGER_*) and other useful intermediate files.

Evaluation Setup

The system is tested on Docker container aflplusplus/aflplusplus:v4.21c running Ubuntu 22.04.

You can launch Docker container with necessary dependencies installed with following commands:

docker build -t cleverest .
docker run --rm -it --name clever cleverest
docker exec -it clever bash

You will need to specify OpenAI API key in environment variable before running experiments, or set it in openai Bash script.

export OPENAI_API_KEY=your_api_key

Before running full experiments, you can run a very basic test for only one bug-introducing commit 8c27b12 of MuJS with only one LLM query by running the following command:

# $conf is the file containing information about program and commit under test
export conf=mujs1.env
MAX_ITER=1 ./run.sh $conf

The script should take less than 20 seconds to execute and print some debug information. If Cleverest successfully found the bug, you should see "Bug triggered after commit 8c27b12" along with some AddressSanitizer output.

You should also get a text file prexied by SUMMARY_ and a folder prefixed by exp_ in current directory. The SUMMARY_* file contains the configuration of experiment and the result for each commit in a table format. The exp_* folder contains full history of interacting with LLM (chat_*.log), all generated test cases (INPUT_*), non-trival test cases that trigger bugs, cause output difference or reach commit-changed code (TRIGGER_*) and other useful intermediate files.

Build Subject Programs

Even though run.sh will automatically build the subject programs if they do not exist, it's recommended to build them before running experiments to accurately measure the execution time needed by Cleverest. You can build all subject programs for commits under test by running the following command:

export conf=mujs.env  # or libxml2.env, poppler.env, jerryscript.env, z3.env, php.env, jq.env, micropython.env
SCENARIO=BIC ./b.sh $conf  # build bug-introducing commits for bug-finding scenario
SCENARIO=FIX ./b.sh $conf  # build bug-fixing commits for bug-reproduction scenario

Reproducing Results

RQ1: Evaluation of Capabilities

Run the following command to run experiment under default setting:

./run.sh $conf
# same as setting the default configuration
SCENARIO=BIC GIT_INFO=FULL MAX_ITER=5 LLM_TEMP=0.5 LLM=gpt-4o ./run.sh $conf

RQ2: Ablation Study

Prompt Synthesizer

GIT_INFO=MSGONLY ./run.sh $conf
GIT_INFO=DIFFONLY ./run.sh $conf
GENCMD=1 ./run.sh $conf

LLM Module

LLM_TEMP=1.0 ./run.sh $conf
LLM=gpt-4o-mini ./run.sh $conf
LLM=deepseek-r1 ./run.sh $conf

Execution Analyzer

NOFEEDBACK=1 ./run.sh $conf
MAX_ITER=10 ./run.sh $conf

RQ3: Effectiveness of Commit Messages

The enhanced/reduced commit messages produced by gemini-2.5-flash are stored in msg/gemini.yaml for reference. To run experiments with enhanced/reduced commit messages, run the following commands:

SCENARIO=FIX GIT_INFO=ENHANCED ./run.sh $conf
SCENARIO=FIX GIT_INFO=REDUCED ./run.sh $conf

RQ4: Comparison to the State-of-the-Art

Follow instructions in fuzz/README.md to run WAFLGo and ClevFuzz experiments.