THE OPEN BENCHMARK FOR AI CHIP DESIGN

Can AI build
a real chip?

Chip-Bench hands an agent a hardware contract and an empty directory. No tests, no reference model, no harness. It has to write synthesizable RTL, prove the design against adversarial memory schedules, and fit inside a real area budget.

EXPLORE THE RESULTS SEE THE TASKS

3 agents evaluated 3 levels shipped 9 runs recorded 1 agent cleared all three

OVERVIEW

Only one agent cleared all three levels.

Coding benchmarks ask whether an agent can finish a software task. Hardware asks something harder. A design has to hold its contract on every clock edge, survive memory that stalls whenever it likes, recover from wrong guesses without committing them, and still fit in the silicon you budgeted for it.

All three agents write arithmetic that computes the right answer. That turned out to be the easy half. On Levels 1 and 2 the failures cluster where two events land on the same edge: a halt while a request is unanswered, an interrupt while an instruction retires, a reset while a response is in flight. Level 3 then holds correctness fixed and makes speed the score, and the ordering barely changes.

LEVELS & TASKS

Each level asks a harder question than the one before it.

Three levels are built and scored. Levels 1 and 2 ask whether an agent can build a CPU that is correct. Level 3 holds correctness fixed and asks how fast it can make one. Pick a level to see the exact contract the agent was handed.

EITHER CHANNEL MAY STALL FOR ANY FINITE NUMBER OF CYCLES INSTR MEM READ ONLY TINYRV‑14 32‑BIT CPU · 14 INSTRUCTIONS LUI ADDI ADD SUB AND OR XOR LW SW BEQ BNE JAL JALR EBREAK ONE RETIREMENT PER CYCLE, IN ORDER DATA MEM READ / WRITE REQ RESP REQ RESP RESET RETIRE HALT / FAULT IN OUT 14 INSTRUCTIONS · ANY INTERNAL SCHEDULE · SYNTHESIZABLE SYSTEMVERILOG
CONTRACT 01

The handshake is binding

Once a request is visible, only a completed handshake or a reset may clear it. Not a branch, and not a halt.

CONTRACT 02

Reset means quiet

Every register reads zero, pending transactions are discarded, and nothing is asserted on the release edge.

CONTRACT 03

Fit in 10,000 LUTs

Synthesis and place-and-route decide whether the submission counts at all. Over the line is a rejection.

EFFICIENCY

The agent that spent the most tokens is the only one that passed.

Wall-clock time separates almost nothing: seven of the nine runs finished between 43 minutes and two hours. What separates them is how much work happened inside that window, and how often they stopped to measure it.

Totals include cached context reads, which dominate an agentic loop and are counted differently by each harness. Compare the output column across agents and treat the totals as within-agent figures. Timestamps are local wall clock: Levels 1 and 2 ran on 1 September 2026, Level 3 on 2 September. Runs overlapped on one host.

OUTPUT TOKENS vs CORRECTNESS

100%75%50% 25%0% 30K100K300K700K OUTPUT TOKENS · LOG SCALE DIRECTED PASS RATE
LEVEL 01 LEVEL 02

Extra work only paid off on the harder level.

Grok cleared the small CPU perfectly on 114,000 output tokens. Claude Code cleared it too, on 530,000. That is four and a half times as many tokens for the identical score. On an easy contract the extra thoroughness bought nothing.

The pipelined CPU inverts that. Claude Code wrote 686,000 output tokens and passed every case. Codex wrote 72,000 and passed 31 of 97. Grok wrote 138,000 and passed 26 of 97. Every run below 200,000 output tokens failed the level.

Those tokens went into test infrastructure. Codex shipped a chip and nothing else. Claude Code spent most of its run building what the benchmark forgot to provide: an interpreter, a fuzzer, a protocol checker and bounded formal proofs. This benchmark set out to measure chip quality and ended up measuring whether an agent will build its own instruments.

LEVEL 03 · THE CLIMB

Most of the speedup arrives in the first twenty minutes.

Level 3 logs a score every time an agent measures its design, so the whole optimization run is on the record. Each dot is one scored build. The dips are experiments that made the design worse and were reverted.

MINUTES INTO THE RUN SCORE
01 / 03

The instruction cache is the whole game

The memory model charges four cycles per word, and the reference core fetches one word at a time. Claude Code added an 8 KB instruction cache first and jumped from 0.97 to roughly 4.0 in one step. Codex never added one, used no block RAM at all, and never got past 3.1.

02 / 03

Grok optimized cycles and lost the clock

Grok reached a 4.08× cycle speedup, second best of the four. Its routed clock fell to 25.1 MHz, so the frequency factor cut the score to 2.56. Cutting cycles while lengthening the critical path moves the score backwards.

03 / 03

Measuring often is what moves the score

Codex measured 7 times, Claude Code 16. Every measurement is a chance to keep a change or throw it away, and Claude Code took more than twice as many of them inside a shorter run.

FAILURE ANALYSIS

Codex and Grok break the same rule while shutting down.

Both of them drop a memory request that memory has not yet accepted, at both levels. They treat halting as a state the CPU can announce. It is a state the CPU has to arrive at, once every outstanding transaction has been answered.

codex

  • Withdrew a live request. On halt it drops imem_req_valid while the request is still unaccepted. The verifier reports dropped without acceptance.
  • Spoke on the reset edge. At Level 1 it asserted a fetch in the first cycle after reset, and reported halted in the same cycle it was still retiring.
  • Let work slip past an interrupt. It takes an interrupt only when the execute slot happens to be empty, so ordinary instructions retire after the interrupt is already eligible.
  • Missed the area budget twice. 10,970 LUTs against 10,000, then 21,845 against 20,000. It resets whole register arrays instead of masking validity.
  • Shipped no tests. Both submissions contain rtl/ and nothing else. No harness, no reference model, no formal jobs.

grok

  • The same halt bug. Its pending-halt path also clears a stalled instruction request under backpressure.
  • Traps that never end. Several exception cases re-enter the handler until the test times out.
  • Read a value that had moved. A CSR re-reads its source from the register file after the real producer retired on the same edge, and gets a stale zero instead of 0x12345678.
  • Decoded the same instruction repeatedly. 71 strict lint warnings from duplicated decode outputs that are then ignored. Modularity without one authoritative control record.
  • Built a harness, then stopped. Its Level 1 formal jobs pass. Its Level 2 formal/ directory is empty, and it passes zero random differential cases.

claude code

  • Treated the request as state. Once a data request is visible, a latch stops any later control event from cancelling it.
  • Made retirement the arbiter. One slot picks exactly one outcome per cycle, and every write, redirect and flush derives from that choice.
  • Masked reset instead of clearing it. Per-register validity bits give the same visible zero state while letting the register file map into distributed RAM. 478 flip-flops against 1,362 and 1,309.
  • Still leaves speed on the table. 38.79 MHz routed against a 50 MHz attempt, three bubbles on a taken redirect, two cycles per memory operation.
  • Graded its own homework. It wrote the verifier that judges all three entrants. That is the largest caveat on this page.

FINDINGS

Six things that would raise the next score.

METHOD

All three went through one verifier.

Each agent received the task documents and an empty rtl/ directory, and could change nothing else. All three submissions then went through one common verifier: an independent architectural interpreter, delayed request and response schedules, directed assembly and compiled C, random differential programs, protocol assertions, Yosys synthesis, nextpnr place-and-route, and bounded formal jobs.

Where an agent shipped its own harness, that harness was left alone and reported separately. Grok's strict lint warnings were made non-fatal so its functional suite could run at all; the strict result is reported above rather than hidden.

READ THIS BEFORE CITING ANY OF IT

KNOWN LIMITS · v0.1 · 01 SEP 2026
  • One run per agentThese are six specific artifacts. The protocol asks for five independent runs before anyone claims anything about an agent family, and we have one.
  • The winner wrote the rulerNo official harness existed, so Claude Code built the verifier that judges all three entrants. It may share blind spots with its own design.
  • No sealed evaluationThe hidden suite described in the task documents does not exist yet. Every number here comes from public tests.
  • Token accounting is not comparableEach harness counts context and cache reads differently. Output tokens are the closest thing to a fair cross-agent measure, and even that is rough.
  • Level 3 scores are public, not officialThe official Level 3 score comes from hidden workload variants that only the maintainers can run. Everything shown here is the public score, which is the hill-climbing signal the agents could see.

WHY THIS EXISTS

We want to know whether a system can build hardware that is correct, efficient, reproducible and real. Hardware that merely looks correct is easy to produce and worth nothing.

ARA Labs · Chip-Bench