Paper-facing provenance brief
A concise account of where TrajRepair-Bench v2 comes from, what was inherited from upstream benchmarks, what we constructed, and how every failed trajectory became an execution-validated repair instance.
benchmark_v2/
Dataset snapshot
TrajRepair-Bench combines locally generated failures on Terminal-Bench tasks with public audit failures from Harbor-Index 1.0. Both lanes are normalized into the same repair protocol: revise a complete action list, replay it from a clean environment, and judge it by execution.
INSTANCES.csv contains 270 rows because 15 shipped development-only instances are recorded in the manifests and split file but not in that table.Primary acquisition sources
The two boards differ in acquisition, not in evaluation semantics. The Terminal lane provides controlled model and scaffold diversity. The Harbor lane provides long, heterogeneous public failures over a wide collection of upstream task families.
Task packages come from Terminal-Bench. Failed trajectories were generated locally by running seven model labels through mini-swe-agent and OpenHands, then retaining failures that could be faithfully replayed.
Task packages and public audit trajectories come from Harbor-Index 1.0, with task metadata available through the Harbor Hub. All selected runs use the Terminus-2 harness.
Provenance boundary
This distinction should be explicit in the paper. TrajRepair-Bench does not claim authorship of the upstream tasks. Its contribution is the transformation of real failed runs into reproducible, repairable, execution-validated instances.
Task description, clean environment definition, official verifier semantics, and upstream reference assets used for dataset construction.
Model reasoning, tool actions, terminal observations, the executable action sequence, and the upstream failure verdict.
Normalized action list, reproduced failure, leak-audited failure scene, augmented checks, split metadata, clean-replay judge, and edit metrics.
Upstream coverage
Counts below are regenerated from the two 285-instance manifests and the frozen 200-instance board. “Terminal-Bench” includes 159 direct Terminal-board instances plus 19 Terminal-Bench-family tasks represented through Harbor.
| Upstream family | Primary domain | Full pool instances | Full pool tasks | Main-board instances | Main-board tasks |
|---|---|---|---|---|---|
| Terminal-Bench | Terminal and systems engineering | 178 | 65 | 132 | 62 |
| GSO | Repository-level performance optimization | 34 | 7 | 21 | 7 |
| BixBench | Computational biology and bioinformatics | 20 | 5 | 8 | 5 |
| AlgoTune | Algorithm and numerical optimization | 16 | 4 | 14 | 4 |
| Spider2 | Enterprise data and DBT workflows | 10 | 2 | 8 | 2 |
| CyberGym | Real-world vulnerability analysis | 10 | 2 | 6 | 2 |
| SciCode | Research-level scientific coding | 8 | 2 | 4 | 2 |
| DA-Code | Executable data-science analysis | 4 | 1 | 3 | 1 |
| USACO benchmark | Competitive programming | 3 | 1 | 2 | 1 |
| CodePDE | Numerical PDE solver generation | 2 | 1 | 2 | 1 |
| Total | — | 285 | 90 | 200 | 87 |
Failure producers
The models listed here produced the source failures; they are not the repair models evaluated later. The dataset intentionally contains multiple model and scaffold identities so that repair systems face heterogeneous failure styles.
159 trajectories · 7 model labels · 2 scaffolds
126 trajectories · 9 model labels · Terminus-2
Construction protocol
Admission depends on executable evidence. Every instance must preserve the original task semantics, reproduce the source failure, expose a faithful but leak-free failure scene, and support clean replay of a submitted action list.
Project reasoning, terminal actions, and observations into a normalized R/P/O trajectory and executable action list.
Replay the original action list from the official clean environment. Keep the instance only when the failure is reproduced.
Capture the post-failure container for diagnosis, then remove tests, oracle material, and authoring artifacts.
Add executable goal, enabler, and constraint checks derived from the specification and upstream assets.
Deduplicate by failure signature or repair similarity, apply quality gates, and freeze the development/test split.
Quality and evaluation
A repair system submits a complete revised action list. The evaluator computes the diff automatically, replays the list from a clean environment, and reports both upstream and strengthened correctness verdicts.
All 285 source action lists have corresponding replayed failure scenes. Harbor records preserve a zero reward on all 126 selected failures.
Repair@1 (official) uses the task's inherited official verifier and remains the externally comparable result.
Repair@1 (strict) requires both the official verifier and augmented specification checks to pass.
Replaced, deleted, and inserted actions are derived from the old/new list alignment. Correctness remains primary.
Release structure
The benchmark ships 285 verified instances. The main board is the frozen test set. Development instances are separated according to whether their underlying task also appears on the test board.
118 Terminal-board and 82 Harbor-board failures across 87 tasks. Frozen-config final evaluation only.
Underlying tasks do not occur on the main board. Intended for genuine method development.
Shares an underlying task with the main board. Suitable for analysis, not parameter selection.
Every manifest instance appears in exactly one frozen split; all have workspace and image metadata.
Paper-ready wording
This paragraph is deliberately conservative: it distinguishes upstream material from our transformations and uses the current 10-family counting convention.
Citation checklist
Cite both the immediate acquisition source and the underlying benchmark families. If space is limited, keep the full family-level provenance in an appendix or dataset card.
Release and governance
These controls make the dataset auditable and reduce contamination or accidental answer leakage.