Docs

Run the bench locally

ContextClone makes a swarm of agents accountable: what each agent was allowed to see, what it cost, whether each claim held up, and whether the team was worth it. Plain words first here too, then the mechanism.

This page describes the code as it is today: a local engine and CLI that run against a seeded mock provider. A hosted preview exists for internal review only. There is no public hosted service, no published package and no public repository yet. You need a checkout of the source.

Quickstart

When it is released, it will need Node.js 22.5 or newer, because the local store uses node:sqlite. Mock mode needs no credentials and makes no network calls.

1. Install from the repository root

npm install

2. Run the demonstration

npm run ccbench -- demo

You should see five lines, one for each executed check, followed by the report paths:

PASS  graded_comparison  ...
PASS  stale_rejected_selective_refresh  ...
PASS  crash_recovery  ...
PASS  injection_contained  ...
PASS  inspectable  ...

report: .contextclone/report-latest.html
artifact: .contextclone/artifacts/<sha256>.json

3. Run the comparison

npm run ccbench -- compare --cases 30

This runs every generated case under both strategies and adds the comparison to the same report. Open .contextclone/report-latest.html in a browser. The evidence page on this site shows one such report.

4. Run the tests

npm run bench:test

This verifies the local implementation. It does not run a paid provider or establish real-model performance.

Output goes to .contextclone. Override it with --out. Print every option with npm run ccbench -- --help.

CLI commands

Coming soon, with the open-source release. Listed so you can see the whole surface before you commit to it.

• npm run ccbench -- demo

  Runs the required M1 demonstration: five executed checks. Writes an HTML and JSON report. Exits non-zero if a check fails.

• npm run ccbench -- compare --cases 30

  One agent versus a bounded adaptive team on generated cases. Options: --seed, --suppliers, --max-agents, --family dev|heldout, --fault-rate, --error-rate, --hallucination-rate.

• npm run ccbench -- run --strategy team --name my-run

  One durable run stored in a SQLite file under .contextclone/runs. Use --strategy single for the one-agent strategy.

• npm run ccbench -- replay .contextclone/runs/my-run.db

  Rebuilds committed state from the event log and receipts, compares it with the stored state and lists any differences.

• npm run ccbench -- export .contextclone/runs/my-run.db

  Writes a content-addressed JSON bundle of the run: receipts, usage entries and events.

• npm run bench:test

  Runs the test suites of the four engine packages: contracts, providers, core and bench.

Live mode

compare --live sends requests to the OpenAI Responses adapter and spends real money. It refuses to start without --budget-usd (per run) and --max-total-usd (whole comparison), and it stops if cases x 2 strategies x per-run budget exceeds the total. It needs OPENAI_API_KEY, OPENAI_MODEL_DEFAULT, OPENAI_PRICING_VERSION, OPENAI_CONTEXT_WINDOW_TOKENS, OPENAI_INPUT_MICRO_USD_PER_MTOK and OPENAI_OUTPUT_MICRO_USD_PER_MTOK. The adapter is unit tested. No live comparison has been run or published.

Architecture

The dependency path is contracts and engine-contracts, then providers, then core, then bench, then the CLI or the web app. Each layer imports only the ones before it.

packages/contracts

Runtime schemas and content hashing for intent contracts, task contracts, context packages, agent results, receipts and benchmark manifests.

packages/engine-contracts

Interfaces for the replaceable engine modules and the boundary with the protected host.

packages/providers

The seeded fake provider with a virtual clock, an OpenAI Responses adapter, and a decision-model verifier adapter.

packages/core

The protected host, scheduler, budget reservation, leases, the verify-then-commit gate, invalidation, replay and the SQLite store. Also holds the seed engine generation.

packages/bench

Case generator, deterministic grader, case runner, strategy comparison, the required demonstration and report writer.

apps/cli

The ccbench command. Thin wrapper over bench and core.

apps/web

This site and the portal under /app, which calls the same engine server-side.

Protected host and replaceable engine

Engine modules propose. The host authorizes, validates and records. Five modules sit on the replaceable side: context compilation, decomposition, context routing, effort selection and improvement proposal. Tenant and source-scope filtering, budgets, leases, result verification, the commit gate and grading stay in the host and are not part of the mutable surface.

An engine generation is identified by the hash of its module sources. The published sample ran on gen_seed_aa981c6c05f9. Only this seed generation exists. No successor has been proposed, qualified or adopted.

What happens in a run

1. An intent contract fixes the goal, source scope and authorized budget. Sources are ingested as revisioned context atoms.
2. The engine proposes a decomposition. The host compiles a context package for each task from authorized atoms only and records what was omitted.
3. Before each attempt the host reserves the maximum permitted charge. A protected allowance is held back for final synthesis and verification. If the reservation does not fit, the run fails explicitly instead of overspending.
4. A worker holds a lease on the task. Its result is checked without a model: schema, evidence existence, evidence span and entity coverage.
5. The commit gate rechecks ownership and that the result still rests on current sources. A result built on a superseded source is rejected rather than accepted late.
6. Every decision writes a hash-chained receipt and an event. Replay rebuilds the state from those records and reports any difference.

Leases, receipts and the semantic check

A lease has a fixed lifetime of 120 seconds and no heartbeat. That exceeds the 60-second provider timeout, which is enough for four agents and is a known limit before long tool calls exist. When a lease expires the host reclaims the task and issues a newer fencing token, so a late commit from the first worker is rejected.

An optional independent decision model can check each dated claim against its cited documents inside the commit path. A score below 0.5 rejects the attempt, and retries are bounded. If the decision provider is unreachable the check is skipped and the receipt records SKIPPED. The deterministic checks still gate the commit. The check covers agreed-date claims only. It is not general truth verification.

No tool is executable in M1. Every tool request is validated and denied, which is how the injection check in the demonstration is contained.

Locally, state lives in SQLite through node:sqlite. The hosted preview runs the engine on in-memory SQLite, then imports the finished run into Postgres in one transaction that rechecks the receipt hash chain and applies the server-resolved tenant to every row. Runs execute inside the request. There is no background worker yet.

Benchmark methodology

The fixture: CC-Reconcile

A generator, not a fixed demonstration. Each seed produces a set of supplier documents and the instruction to report the agreed delivery date for each supplier, cite source IDs and keep unresolved conflicts. Every case contains a proposal trap, where an unapproved proposal must not be reported as agreed, and a real contradiction between two signed orders that must stay unresolved. Supplier count varies between cases so that both serial and split plans occur. Development and held-out splits never share a paraphrase template. Dataset version: cc-reconcile-1.

Grading

A deterministic grader runs outside the engine with a hidden key that is never passed to a run, a worker or a provider. A case succeeds only if every supplier is recovered and every claim is correct. A dated claim must cite only valid evidence. An unresolved claim must not be marked supported and must cite the conflicting sources. Reporting an unapproved proposal as agreed is counted as a hard violation.

Strategies

Both strategies get the same cases, provider seed, tools and per-run budget. Implemented today:

• B1: one strong agent with the same tools, sources and budget
• B5: deterministic adaptive scheduler, at most 4 agents, no learning

The test plan names nine baselines, B0 to B8. B0, B2, B3, B4 and B6 to B8 are not implemented, so no comparison against them exists.

What is reported

• Verified success rate, hard violations and failed runs for each strategy. Failed cases stay in the table.
• Median end-to-end time, median cost, cost per success, repeated-context fraction, mean provider calls and mean participating agents.
• Paired over cases where both strategies succeeded: median speedup and median cost ratio. A cost ratio is always shown beside the absolute amounts, because a multiple of a fraction of a cent means little on its own.
• Verified-success difference, team minus single, with a 95% interval from a paired bootstrap over cases, 2,000 resamples, seeded.
• A manifest: dataset version, split hash, task IDs, seeds, model profile, policy version, engine generation, resource caps, grader version, environment and stop rule.

The stop rule is fixed in advance: all cases under both strategies, no early stopping. Read quality, time and cost together, and count the time of your people: the cost that matters is the total cost of an answer you can trust, which is model cost plus the hours needed to check it.

The required demonstration

Five executed checks on one case: a graded comparison with the contradiction preserved; a source change before commit that rejects the stale output; a worker crash with no duplicate commit and no budget loss; a retrieved document that attempts an injection and is denied; and an inspectable record of evidence, cost, latency and open questions.

Interpret the result

Current measured results use a seeded mock provider. Real-provider gains, recursive improvement and large-scale coordination remain hypotheses or engineering targets. Keep failures visible, and distinguish virtual time from wall-clock execution.

Known limits

• MOCK PROVIDER. Latency, cost and mistakes come from a seeded simulator with arbitrary constants. Times are virtual milliseconds.
• This measures orchestration mechanics only. It says nothing about real model quality, real latency or real prices.
• Aggregation is a deterministic merge with no model call, so its provider cost is zero by construction.
• Only B1 and B5 are implemented. B0, B2, B3, B4 and B6 to B8 are not, so no claim of advantage over them exists.
• One benchmark family of eight in the test plan is implemented. The sample has 30 cases, which is a pilot, not a confirmatory sample.
• No public hosted service, no production deployment of tenant-isolated persistence, no background worker, no learned policies and no recursive generations exist yet.

Next: inspect the sample report or check the build status.