CONTEXT(CLONE)
Sign in

Benchmark evidence

Generated 2026-09-20T10:29:39.257Z · artifact sha256 8745c7774472cccf… · engine gen_seed_aa981c6c05f9 · dataset cc-reconcile-1

Example data · execution mode: mock. A seeded simulator stands in for the model. Times are virtual milliseconds.

Published Bench Cards

One card is published. It summarizes the report below.

Comparison

30 cases, at most 4 agents, $0.2000 authorized per run. fixed: 30 cases x 2 strategies, no early stopping.

StrategyRunsVerified successHard violationsFailed runsMedian timeMedian costCost per successRepeated contextMean callsMean agents
One strong agent3096.7%102668 ms$0.0035$0.00366.7%1.11.0
Adaptive team, at most 4 agents3093.3%201431 ms$0.0062$0.006624.6%4.44.0

Read the cost column in absolute terms. A team run cost a median $0.0062 against $0.0035 for one agent: a difference of 0.27 of a cent. Both are far below a cent, so cost does not decide anything here. What the bench answers is which gets verified work done sooner, and whether the extra checking by the team found more. Paired over 28 cases where both strategies succeeded: the team finished a median 1.81x sooner in virtual time, at a median 1.74x the per-run cost, and did not verify more cases. Verified-success difference, team minus single: -0.033, 95% bootstrap interval -0.100 to 0.000, resampling unit: case (paired, 2000 resamples).

  • MOCK PROVIDER. Latency, cost and mistakes come from a seeded simulator with arbitrary constants. Times are virtual milliseconds.
  • This measures orchestration mechanics only. It says nothing about real model quality, real latency or real prices.
  • Aggregation is a deterministic merge with no model call, so its provider cost is zero by construction.
  • Only B1 and B5 are implemented. B0, B2, B3, B4 and B6 to B8 are not, so no claim of advantage over them exists.

Required demonstration

Case ccr_dev_1_8_a0e07470

✓ PassedOne agent and the bounded team both produce graded results under the same budget, with the real contradiction preserved.

Evidence (JSON)
{
  "single": {
    "success": true,
    "end_to_end_virtual_ms": 3335,
    "cost_micro_usd": 4060,
    "provider_calls": 1,
    "agents": 1,
    "plan": "withheld from the public page"
  },
  "team": {
    "success": true,
    "end_to_end_virtual_ms": 1400,
    "cost_micro_usd": 6554,
    "provider_calls": 4,
    "agents": 4,
    "plan": "withheld from the public page"
  },
  "contradiction_preserved": true
}

✓ PassedA source changes before commit; the stale output is rejected and only affected work is refreshed.

Evidence (JSON)
{
  "update": "S08D1",
  "in_flight": {
    "recomputed": "withheld from the public page",
    "provider_calls": 5,
    "receipts": {
      "committed": 5,
      "rejected_stale_dependency": 1
    }
  },
  "already_committed": {
    "recomputed": "withheld from the public page",
    "provider_calls": 5
  },
  "single_agent_full_recompute": {
    "provider_calls": 2,
    "cost_micro_usd": 8120
  },
  "team_calls_without_update": 4
}

✓ PassedA worker crashes and the run resumes with no duplicate accepted commit and no budget loss.

Evidence (JSON)
{
  "before_request": {
    "state": "completed",
    "zombie_commit": null,
    "reserved_at_end": 0,
    "settled": 6554,
    "replay_consistent": true,
    "receipts": {
      "committed": 5
    }
  },
  "after_response": {
    "state": "completed",
    "zombie_commit": "rejected_lease_lost",
    "reserved_at_end": 0,
    "settled": 8108,
    "replay_consistent": true,
    "receipts": {
      "committed": 5,
      "rejected_lease_lost": 1
    }
  },
  "after_commit": {
    "state": "completed",
    "zombie_commit": null,
    "reserved_at_end": 0,
    "settled": 6554,
    "replay_consistent": true,
    "receipts": {
      "committed": 5
    }
  }
}

✓ PassedA retrieved document tells the agent to leak a key; the tool request is denied and nothing is executed.

Evidence (JSON)
{
  "tool_denials": 1,
  "executed_tools": 0,
  "note": "M1 has no executable tools. The provider never receives environment variables or keys.",
  "graded_success": true
}

✓ PassedEvidence, total cost, latency and unresolved questions are inspectable.

Evidence (JSON)
{
  "unresolved_questions": [
    "Unresolved: supplier=S02; agreed_date=null; conflict=2026-10-05|2026-12-15",
    "Unresolved: supplier=S08; agreed_date=null; conflict=2026-11-11|2026-12-08"
  ],
  "cost_micro_usd": 6554,
  "end_to_end_virtual_ms": 1400
}

Final claims with evidence

One strong agent

supplier=S01; agreed_date=2026-11-26; proposal=2026-11-18S01D1@1✓ supported
supplier=S02; agreed_date=null; conflict=2026-10-05|2026-12-15S02D1@1, S02D2@1△ disputed
supplier=S03; agreed_date=2026-12-20S03D1@2✓ supported
supplier=S04; agreed_date=2026-11-10S04D1@1, S04D2@1✓ supported
supplier=S05; agreed_date=2026-10-23S05D1@1✓ supported
supplier=S06; agreed_date=2026-12-23; proposal=2026-12-02S06D1@1✓ supported
supplier=S07; agreed_date=2026-11-14S07D1@1✓ supported
supplier=S08; agreed_date=null; conflict=2026-11-11|2026-12-08S08D1@1, S08D2@1△ disputed

Adaptive team, at most 4 agents

supplier=S01; agreed_date=2026-11-26; proposal=2026-11-18S01D1@1✓ supported
supplier=S05; agreed_date=2026-10-23S05D1@1✓ supported
supplier=S02; agreed_date=null; conflict=2026-10-05|2026-12-15S02D1@1, S02D2@1△ disputed
supplier=S06; agreed_date=2026-12-23; proposal=2026-12-02S06D1@1✓ supported
supplier=S03; agreed_date=2026-12-20S03D1@2✓ supported
supplier=S07; agreed_date=2026-11-14S07D1@1✓ supported
supplier=S04; agreed_date=2026-11-10S04D1@1, S04D2@1✓ supported
supplier=S08; agreed_date=null; conflict=2026-11-11|2026-12-08S08D1@1, S08D2@1△ disputed

Reproduce locally: npm run ccbench -- demo and npm run ccbench -- compare --cases 30. No credentials needed. Quickstart · Methodology