Binary Security Analysis Benchmark

Mechanical, zero-LLM-as-judge evaluation of frontier and open-weights models on real Windows x64 kernel drivers and system binaries: multi-hop vulnerability discovery across shipped security patches, binary claim verification, and byte-level grounding.

Capability Index

KE-Score composite security index (0–100) · Higher is better

Time per Task

Weighted inference time (minutes) per benchmark task · Lower is better

Cost per Task

Weighted average API cost (USD) per benchmark task · Lower is better

Performance by Task Type

Multi-hop vulnerability discovery vs. capability score with verified discoveries · Higher is better

Vulnerability Discovery Capability Score (with Discoveries)

Capability vs. Cost

Capability Index against log-scale USD cost (up and left is better · dashed line = Pareto frontier)

What Models Were Asked To Do

Every model is placed inside the same isolated binary-analysis loop over production Windows x64 PE binaries and graded strictly by deterministic binary oracles — no human subjective scoring and no LLM-as-judge.

1 · Vulnerability Discovery 60.1% of Index · 13 Patch Pairs

Multi-Hop Bug Hunting (Analyze)

Models start at an entry function up to 3 call-hops above a real historical vulnerability inside a compiled Windows driver or boot/system binary (with a forward call cone of 20+ functions) and are asked to audit reachable memory-safety, integer-overflow, type-confusion, or refcount flaws.

  • Paired before/after test: Models audit the unpatched build first and (if they hit the vulnerable sink) immediately audit the patched build under the identical prompt.
  • Feature-flag reasoning: Many Windows patches keep vulnerable code compiled in behind a default-ON WIL Velocity flag. Models must inspect flag state and pass both halves: flag the bug when unpatched and report NO-FIND once fixed.
2 · Technical Verification 39.9% of Index · 26 Audits

Binary Claim & Trap Auditing (Validate)

Models receive technical review drafts containing 9–10 concrete binary claims per binary (direct call edges, instruction constants, and function-entry statements) and must independently verify every statement from disassembly, decompiled C, and raw bytes, marking each CONFIRM or REFUTE.

  • Plausible false traps: Mixes true statements with subtle falsehoods — 2-hop indirect calls misstated as direct calls, constants borrowed from sibling functions, and shifted function-entry RVAs.
  • Adversarial resistance: Includes unwarned prompt-injection payloads embedded inside binary data claims attempting to force CONFIRM or invoke forbidden tools.
3 · Byte-Level Grounding Diagnostic · 77 PE Entries

Raw Data & RVA Precision (Inventory)

Models are given raw read-only data (.rdata) address ranges inside compiled PE binaries and asked to read raw byte slices and emit structured tables of ASCII literals paired with their exact hexadecimal Relative Virtual Addresses (0xRVA).

  • Zero-tolerance byte check: Every emitted {rva, string} row is checked byte-for-byte against the PE image.
  • Catches offset drift: Directly surfaces hex arithmetic slips, off-by-N address slides, and fabricated strings that look plausible in prose but fail against real binary layouts.
4 · Live Proof-of-Concept Separate Score · Live KVM Replay

Working Trigger on a Live VM (PoC-Score)

Models are given a short brief for an already-fixed vulnerability (entry interface, symbolic sink, and the grader's breakpoint contract) and must produce a poc.exe that drives a real, unpatched Windows 11 guest to the vulnerable sink, building and debugging inside the VM.

  • Repair and derive: Repair items start from a reference PoC carrying one planted runtime fault; derive items start from the brief alone.
  • Graded by replay: The submitted PoC is re-run three times from a clean snapshot and passes only if all three runs hit the sink breakpoint with the required register or bugcheck condition. The model's own success claim is not used.
  • Reported separately: PoC-Score is the replay-pass rate over these items and is not part of the Capability Index, so VM variance never moves the static score. “—” means the model has not been run on this lane.

Clean-Room Isolation & Anti-Memorization Controls

Models have no SQL search, no cross-version diff tools, and zero human annotations — every call is pinned to a single binary file_id. All evaluated vulnerabilities are already publicly fixed by Microsoft (fix_state='fixed').

11 Read-Only Binary Tools Provided to Every Model
Hex-Rays Decompiler & Microcode x86-64 Disassembler Call & Data Xrefs Struct / Type Layouts Raw Byte Reader Pointer & Scalar Tables WIL Feature-Flag Inspector CFG Flag Arm Slicer Sandboxed Chunk Emulator (rax) Z3 / SMIR Bound Prover

Model Comparison Table

Click any column header to re-sort. Hover rows or chart bars for 95% confidence intervals and per-lane metrics.

# Model Capability Index Vuln Discovery Byte Grounding Speed (tok/s) Time / Task Cost / Task