ChronicleBenchmark
In progress. The evaluation below is designed and its harness is in the repository, but no results have been run yet. This page will report them, including null or negative results, when they exist.
The question is whether a computer-use agent completes more tasks when it can query a knowledge graph of its own earlier work than the same model without it, and whether the graph matters or just having history does.
Benchmarks
| Benchmark | Why |
|---|---|
| OSWorld 2.0 | 108 long workflows in Ubuntu VMs with self-hosted services. 43 of them are tagged implicit-state inference: the agent has to recover state the instruction doesn't give it, which is the problem Chronicle is built for. |
| OSWorld-Verified | 369 shorter desktop tasks, each from a fresh VM. Chronicle has no user history here, so any gain has to come from experience across tasks. This is the harder test for Chronicle. |
How the graph is built
The protocol follows the online setting of Agent Workflow Memory. Tasks run once, in a fixed shuffled order that every arm shares. After each task, the agent's own observations and actions from that task are pushed into Chronicle through POST /add and its graph is rebuilt. Later tasks can query everything recorded so far.
- Chronicle never sees benchmark evaluator results, reference solutions, or human demonstrations of the test tasks.
- The agent's runs are recorded whether they succeeded or failed, as they would be on a real machine. Its actions and reasoning are not recorded, matching Chronicle, which never records keystrokes.
- All three arms run the same recorder, so per-step latency is equal, and the injected context counts toward each arm's token cost.
Arms
| Arm | Agent gets |
|---|---|
| Raw model | The benchmark's standard observation and action space |
| Plain memory | The recorded frames that best match the task, ranked by BM25 over the same recordings |
| Chronicle | The get_context pack for the task: knowledge graph, timeline, and cited evidence |
Both memory arms get the same preamble and the same 8,000-character budget, added to the instruction at the start of each task. The plain-memory arm separates the value of the graph from the value of having any history at all. Letting the agent call Chronicle's tools itself is a later experiment.
Reporting
Results will give binary success and, for OSWorld 2.0, partial score per arm, with paired bootstrap confidence intervals and an exact McNemar test on the same tasks. They will also report steps, tokens, and dollar cost per task, results split by phenomenon and by position in the task order, and the model version and dates. The first run is a pilot on one model with a subset of tasks, and it will be labeled as a pilot.
The full protocol, including what counts as leakage and how the task subset is chosen, is in bench/PROTOCOL.md.