# Chip-Bench

> Chip-Bench is an open benchmark for AI chip design. It has four tracks. The CPU track
> asks AI coding agents to design a working RISC-V CPU in synthesizable SystemVerilog and
> then optimize it for speed and area (PPA). The hill-climb track hands the agent a working
> design and one number to improve - twenty frozen optimization tasks across six layers of
> the chip stack. The network track asks for interface IP - PCIe controllers that have to
> stay correct and live while an independently written link partner misbehaves. The
> accelerator track asks for datapath blocks of a transformer (exp, rsqrt, GELU, sigmoid,
> vector reduction, SIMD exp). A design
> only scores if it stays correct on every clock cycle and inside its resource budget.
> Yosys, nextpnr, OpenSTA, SymbiYosys and hidden compliance suites produce every number: no
> model grades anything.

Maintained by ARA Labs / Agent Native Research (https://www.agenticresearch.sh/).
Version 0.3, provisional. Seventeen CPU runs, seventy-six graded hill-climb runs, nine network
runs, and eight accelerator tasks (Grok 4.6 on all eight, 18 September 2026).
Canonical page: https://chipbench.ai/

## What the benchmark measures

- **Level 1 - TinyRV-14.** Empty `rtl/` directory. Build a CPU for a 14-instruction
  RISC-V subset. Must fit in 10,000 LUTs. Memory can stall; the request handshake is
  binding; reset must leave every register at zero.
- **Level 2 - RV32IM.** Empty `rtl/` directory. Full RV32IM with exceptions and
  interrupts. Must fit in 20,000 LUTs, retire a divide within 40 cycles, clear two
  throughput gates at one instruction per cycle, and run CoreMark at 0.6 or more
  instructions per cycle. Exactly one architectural outcome per cycle.
- **Level 3 - Optimize.** Every agent is handed the same working reference core, the
  same tests and a frozen harness, and is asked to make the CPU faster without breaking
  it. Importing a CPU built outside the run is banned. Correctness cannot be traded for
  speed: breaking a gate scores zero. This is the chip-optimization / PPA level.

## The hill-climb suite (20 tasks, 6 layers)

Each task ships a working baseline, a frozen expert bar that scores 1.0, and a frozen
grader. Score is `max(0, log(baseline/agent) / log(baseline/bar))` under hc-0.3.0,
uncapped: matching the author's expert solution scores 1.00, beating it scores above
1.00, the shipped baseline scores 0. The bar stays frozen so a better model can keep
pulling away from a good one. The strongest verified design is the record (orange on
the site), not the 1.0 point. Agents may not touch the harness; every submission is
re-graded from a pristine harness on hidden workload variants the agent never saw.

Ten tasks have been graded by five models, seventy-six runs in all once Gemini 3.8 Flash,
Grok 4.6 and the first six GLM 5.3 tasks are counted. Mean over the ten tasks every
completed run attempted: claude-fable-5-1 1.525, claude-opus-5 1.305, gpt-6-astra 1.283,
grok-4.6-build 1.131, gemini-3.8-flash 1.015. GLM 5.3 is mid-suite and does not yet enter
that mean.

### Logic & synthesis
- `c1_approx_area` (tier T2) - Shrink a 16-bit reciprocal circuit, never off by more than 4 LSB on any of its 65,536 inputs.
  Metric lut4_eq; baseline 1140, bar 338. Best run: codex-astra at 1.313.
- `c1_pipeline_fmax` (tier T2) - Cut a slow arithmetic datapath into pipeline stages so the clock can run faster.
  Metric fmax_mhz; baseline 27.61, bar 125.19. Best run: cc-fable at 1.060.
- `c1_recipe_pareto` (tier T2) - Write synthesis recipes that beat the default flow on area and logic depth — on circuits you never see.
  Metric pareto_hv_ratio; baseline n/a, bar 1.0 by construction. Best run: codex-astra at 1.503.

### Microarchitecture
- `c2_branch_pred` (tier T2) - Guess which way each branch goes, using 8192 bits of memory and nothing else.
  Metric geomean_mpkb; baseline 121.995258, bar 75.503499. Best run: cc-fable at 1.980.
- `c2_cache_ctrl` (tier T3) - Design the data-cache controller that spends the fewest cycles waiting for memory.
  Metric geomean_cycles; baseline 771728.2, bar 540193.7. Best run: cc-fable at 1.808.
- `c2_chacha_tput` (tier T2) - Build a ChaCha20 core with the best area x time per block on an ECP5 FPGA.
  Metric area_time; baseline 11235.202409046806, bar 726.08. Best run: codex-astra at 1.395.
- `c2_divider_al` (tier T2) - Build an RV32M divider that is both small and quick on a hidden mix of operands.
  Metric area_x_avg_latency; baseline 26044, bar 7971.744501. Best run: cc-fable at 1.754.

### Memory system
- `c3_mem_sched` (tier T3) - Order the DRAM request queue so more accesses land on a row that is already open.
  Metric cycles; baseline 1393727.5, bar 812916.4. Best run: cc-fable at 2.352.
- `c3_prefetcher` (tier T2) - Fetch the next address before the program asks for it, in 2 KB of C++ state.
  Metric geomean_cpa; baseline 7.10621, bar 5.788255. Best run: cc-fable at 1.410.
- `c3_replacement` (tier T2) - Choose which cache line to evict, with 4 bits of metadata per line.
  Metric geomean_mpka; baseline 283.992858, bar 241.17407. Best run: grok at 1.062.

### Physical design
- `c4_clock_gating` (tier T3) - Gate the clocks to cut power, with a proof the circuit still behaves the same.
  Metric power_mw; baseline 8.822872, bar 8.136056. Best run: grok at 1.843.
- `c4_flow_tune` (tier T2) - Turn the physical-design flow knobs to shrink area x delay on sky130.
  Metric adp_geomean; baseline 14603.618, bar 12379.578. Best run: grok at 1.114.
- `c4_sdc_refine` (tier T2) - Prove which timing paths can never happen, and let the clock go faster.
  Metric effective_period_ns; baseline 3.8304, bar 2.7422. Best run: gemini-flash and grok both at 1.000.

### Verification & formal
- `c5_induction` (tier T2) - Write the invariants a bounded proof needs to close on a credit-based link.
  Metric proven; baseline 2, bar 8 properties / 20 s. Best run: grok at 1.125 (gemini-flash 1.073); both closed all 8, grok was faster.
- `c5_mutation_eff` (tier T2) - Write the testbench that kills the most injected bugs per simulated kilocycle.
  Metric kills_per_kcycle; baseline 7.4732, bar 149.2537. Best run: grok at 1.230.
- `c5_testset_min` (tier T2) - Catch every injected bug, run the fewest cycles: pick the subset of 200 tests.
  Metric total_cycles; baseline 32268, bar 7227. Best run: gemini-flash at 0.680.

### Co-design, reliability, security, analog
- `c6_const_time` (tier T2) - Make modular exponentiation take the same time no matter what the secret is.
  Metric leak; baseline 0.420441, bar 0.002. Best run: gemini-flash and grok both at 1.000.
- `c6_custom_isa` (tier T3) - Add your own RISC-V instructions, then rewrite the kernels to spend fewer cycles.
  Metric kernel_cycles_geomean; baseline 386664.3, bar 47400.6. Best run: glm-5.3 at 1.641.
- `c6_opamp_fom` (tier T2) - Size a two-stage Miller op-amp for the most gain-bandwidth per microamp, in 60 SPICE runs.
  Metric fom_geomean; baseline 389.6273, bar 3189.4607. Both graded runs scored 0 (missed a spec).
- `c6_seu_harden` (tier T2) - Harden the flip-flops that matter against particle strikes, inside an area budget.
  Metric V; baseline 0.406404, bar 0.061391. Best run: gemini-flash at 1.320.

## The network track (2 tasks, PCIe)

Interface and interconnect IP. A CPU that is wrong computes a wrong number; a controller
that is wrong deadlocks, drops a completion, reorders a posted write past a read, or
overruns a flow-control credit - failures that only appear when a second, independently
written agent is talking back to it. Both tasks are graded against a protocol, by a link
partner that lives only on the grader side. These are completion tasks, not optimization tasks:
there is no baseline, no record and no metric to climb. The bar is exact-correct - every case in
the hidden suite - and no run has cleared it on either task.

- `n1-pcie-ep` - A Gen1-class x1 PCI Express endpoint controller: data link control,
  credit-based flow control, 12-bit sequence numbers, LCRC, ACK/NAK and retry, Type 0
  configuration space, BAR decode, completer and requester paths, ordering and error
  reporting. Top module `pcie_ep_ctrl`, 25,000 mapped LUT4 ceiling. 46 public cases,
  95 hidden.
- `n2-pcie-ltssm` - The physical-layer logical sub-block and link training state machine
  for an upstream port over PIPE x4: ordered-set framing, descrambling, per-lane deskew,
  polarity inversion, lane reversal, 17 LTSSM states, 2.5 and 5.0 GT/s speed change, hot
  reset and disable. Top module `pcie_pl_ltssm`, 12,000 mapped LUT4 ceiling. 28 public
  cases, 113 hidden.

The two tasks meet at one boundary, the Link Layer Symbol Interface: n1 sits above it on
one lane, n2 produces it on four.

### Pilot results, 7 September 2026

One run per agent, a 150-minute wall-clock cap against the 6 hours the task specifies,
isolated workspaces, no network. Nothing is self-reported: every figure is recomputed by
the grader from the RTL the agent left on disk.

**All six runs pass every public case.** The public testbench ships in the workspace by
design, so passing it is necessary and not sufficient. The hidden suites are what rank
the field, and they rank it differently on the two tasks.

- `n1-pcie-ep` hidden cases passing, of 95: cc-opus5 (Claude Opus 5) 94,
  codex-sol (GPT-5.6 Sol) 86, grok46 (Grok 4.6) 86.
- `n2-pcie-ltssm` hidden cases passing, of 113: codex-sol 110, cc-opus5 103, grok46 86,
  and the in-house reference design 96.
- Six of the 95 endpoint cases were written from mutants that survived the mutation gate
  rather than from a reading of the specification, and those six are what separated the
  top of the field.
- Codex's endpoint design never retires a tag behind an unsuccessful completion, which is
  the defect reported against LitePCIe: it takes a real endpoint off the bus until the
  machine is power-cycled.
- Both packages are still blocked on reference calibration and tool pins, so these are
  pilot figures, not leaderboard figures.

## Headline results (public scores, v0.3)

- 17 CPU runs recorded across 3 levels; 7 of them are Level 3 optimization designs.
- Of the first three agent tools evaluated (Claude Code, Codex, Grok), only Claude Code
  cleared all three levels (Level 2 before its CoreMark gate was added). A later clean-room Codex run with GPT-6 Astra also cleared all
  three levels in public evaluation.
- Level 2 gained a CoreMark gate and was rerun on 14 September 2026: one build of EEMBC
  CoreMark 1.0 must validate and average at least 0.6 retired instructions per cycle.
  `codex-astra` (GPT-6 Astra in Codex) passed every public check, including three formal
  jobs, at CoreMark IPC 0.771 on 8,911 LUT4-equivalents, and sealed in 2 hours 3 minutes.
  `cc-fable` (Claude Fable 5.1 in Claude Code) also passed every public check, with the
  highest CoreMark IPC (0.837), the smallest design (6,007 LUT4-equivalents) and the fewest
  cycles on every benchmark program except the divide test, and sealed in 1 hour 9 minutes.
  GLM 5.3 (in the pi coding agent) sealed a design that passes every functional and
  performance check, CoreMark included at IPC 0.739 on 11,163 LUT4-equivalents, but is
  proof pending: the depth-24 interface proof used the whole two-hour formal budget and
  reached step 18 of 24, so the cover and liveness jobs never ran. No proof reported a
  counterexample. Z.ai rate-limited its run throughout, ending 111 of its 112 turns.
  Gemini 3.8 Flash passed every test case but failed CoreMark (IPC 0.530) and its interface
  proof, which found a data request withdrawn before memory accepted it.
- `cc-fuzz` holds the highest public Level 3 score, 5.8416, but one of its formal proofs
  never finished.
- `cc-fable` is the fastest design that passed every check: 5.6896, 41.03 MHz, all 97
  directed tests, all 350 random differential programs and the proof. It used 17,543
  logic cells against 13,766 for cc-fuzz.
- `codex-astra` scored 5.1833 on Level 3 and also passed the public checks on Levels 1 and 2.
  On Level 1 it reached 39,040 median cycles, 2,763 routed LUT4 and 41.3 MHz. On the current
  Level 2 it reached both throughput gates at cycle 515 and CoreMark IPC 0.771, used 8,911
  LUT4-equivalents and routed at 27.6 MHz. No sealed hidden result is available for either construction level.
- cc-fable and cc-fuzz run the same model (Claude Fable 5.1). cc-fuzz ran inside a
  fuzzing-style exploration loop (https://arxiv.org/abs/2608.09855) and ended up 2.7%
  faster on 21.5% less chip area - but it had also read notes from earlier runs, so the
  loop is not proven to be the cause.

## How submissions are verified

There is no LLM judge. An independent architectural interpreter, delayed request and
response schedules, directed assembly and compiled C, random differential programs,
protocol assertions, Yosys synthesis, nextpnr place-and-route, OpenSTA/OpenROAD timing
closure, and bounded formal jobs under SymbiYosys with yices. Every published number was
printed by one of those tools.

## Known limits - read before citing

- One run per configuration. No configuration has the independent runs a model-level
  claim would need.
- The original Level 1-2 verifier was written by Claude Code and may share blind spots
  with its own design. Level 3 uses a supplied, frozen harness.
- Every number published is a **public** score. The official Level 3 score comes from
  hidden workload variants only the maintainers can run.
- On the hill-climb tasks the site plots the raw public metric of every evaluation a run
  made, with the public baseline and the public record as reference lines. The task table
  next to the chart carries both the hidden final metric that the score is computed from
  and the best public metric the run reached, so the part of a gain that did not
  generalise is visible as the gap between those two columns.
- Token accounting is not comparable across harnesses. Output tokens are the closest
  thing to a fair cross-agent measure.

## Links

- [Benchmark results and method](https://chipbench.ai/): the full leaderboard, task
  contracts, exploration curves, the network track's hidden-suite results and run
  traces, token efficiency and failure analysis.
- [Run traces and generated RTL](https://huggingface.co/datasets/AgentNativeResearchLab/cpu-design-benchmark):
  every trajectory and every generated design, on Hugging Face.
- [Agent-Native Research Artifact standard](https://github.com/ARA-Labs/Agent-Native-Research-Artifact):
  the artifact format each run ships in.
- [ARA Labs on GitHub](https://github.com/ARA-Labs)
- [Other benchmarks by Agent Native Research](https://www.agenticresearch.sh/research)
- Contact: amber@ara-commons.com

## Topics

AI for chip design, AI chip design benchmark, benchmark for chip design, LLM hardware
design, agentic hardware engineering, RTL generation benchmark, SystemVerilog code
generation, RISC-V CPU design with AI, CPU optimization, chip optimization, PPA
optimization (power, performance, area), AI for EDA, logic synthesis and place-and-route
with AI, FPGA LUT area budget, formal verification of AI-generated RTL, recursive
self-improvement (RSI), self-improving AI systems, AI coding agent evaluation, PCIe
controller design, PCIe LTSSM, link training, protocol compliance testing, interface IP,
interconnect verification, mutation testing.