ThrowBench

This repository contains the data for ThrowBench. Data is given in JSONL format. Records have the following fields:

• bug_id: The RunBugRun bug id
• code: Program code
• exception_type: The name of the exception thrown, or no_exception if none thrown. This is the target label.
• exception_message: The full message of the exception thrown`
• language: The langugue, either c_sharp, java, python or ruby
• input: The program input that triggers the exception
• inputs: Other triggering inputs (input was randomly selected from this list)
• locs: Program length (in lines of code)

Model Outputs

Model outputs can be found in the results directory. Files have the following fields:

• model: Model name
• predicted_exception: Answer given by model
• actual_exception: Ground-truth answer
• bug_id: See above
• output: Full model output
• language: See above

Scripts

The evaluation script can be found in run.py