Long-Horizon Evaluation
Research question
How is a system judged when the task does not end in one turn?
Working thesis
Task completion is a weak measure of a long-running system. What matters is behavior across time and under pressure: recovery quality, provenance integrity, evidence sufficiency, handoff quality, and continuity under interruption — measured in harnesses built to apply adversarial pressure rather than to confirm success.
A thesis is the lab’s current position, not a result. What supports it is listed below; what remains unresolved is stated at the end of this page.
Publications
2 attributedApparatus
4 systemsSystems attached to this program are what the lab builds to investigate the question. Maturity, evidence level, and current activity are recorded separately on each system — a system attached here is not thereby claimed to be production-ready.
Boundary
Simulation & Evaluation Engine
A simulation and synthetic-environment engine for agent rollouts, deterministic replay, evaluation scorecards, and training-dataset export.
Cerberus
Multi-Head Reasoning & Evaluation Harness
A three-head reasoning and evaluation harness built on structured disagreement: one head proposes, one head critiques adversarially, and one head arbitrates and verifies.
M-Class Harness
Governed Evaluation Methodology
A governed evaluation methodology, scoring framework, and review standard for coding, reasoning, and multi-step AI workflows.
Long-Horizon Harness
Extended-Task Execution Environment
An execution and evaluation environment for extended tasks involving interruptions, memory degradation, state change, recovery, and handoffs.
Open questions
Published as open. These are the parts of the program the lab does not consider settled.
- 01
What is the minimum evidence that a long-horizon result is reproducible?
- 02
How are recovery and continuity scored without rewarding systems that simply never stop?
- 03
Which adversarial pressures generalize, and which only break one implementation?