Recorded run · NIST Juliet · AArch64
Region recall was 0/14. Good and bad ranked the same.
Seven labelled CWE pairs, gcc -O0 -g aarch64 ELFs. Quick ranking never named the sink function. verify found libc-sink callers on both twins, including the good ones. This is a failed retrieval check, not a detector score.
Region recall is 0/14 at k=1, 3, and 5. Every good/bad pair ranked identically. verify caller recall is 8/8 on the libc-sink binaries, including the good twins, and abstains on CWE-190, CWE-416, and CWE-476. An import name is not a hit. Juliet already names the CWE and the sink.
01 · FROZEN SLICE
One baseline flow per pinned CWE. Linux char/system cases.
Source: NIST Juliet C/C++ 1.3 zip 2017-10-01-juliet-test-suite-for-c-cplusplus-v1-3.zip, SHA-256 ada9d7e1…388f38eb. Selection was frozen before scoring. Prefer *_01 / src_char_declare_cpy_01. No Win32, no sockets. Host gcc on Kali Pi 5 produced 14 ELF 64-bit LSB PIE aarch64 binaries with DWARF, not stripped. No compile abstentions.
CWE flow variant labelled sink libc sink
121 src_char_declare_cpy_01 *_bad / goodG2B strcpy
122 c_src_char_cpy_01 *_bad / goodG2B strcpy
190 int_max_add_01 *_bad / goodG2B+goodB2G none (data + 1)
415 malloc_free_char_01 *_bad / goodG2B+goodB2G free
416 malloc_free_char_01 *_bad / goodG2B+goodB2G none (printLine after free)
476 char_01 *_bad / goodG2B+goodB2G none (data[0])
78 char_console_system_01 *_bad / goodG2B system
gcc -O0 -g -DINCLUDEMAIN -DOMITGOOD -I testcasesupport -o BIN_bad CASE.c io.c
gcc -O0 -g -DINCLUDEMAIN -DOMITBAD -I testcasesupport -o BIN_good CASE.c io.c
r2b brief BIN --quick --no-save --json
r2b verify BIN --import strcpy --json # libc sink only
Binaries were not executed. No POLLs, POVs, decompiles, or model calls. Source-named functions were used only to score after briefs existed.
02 · RANKING
The labelled sink function never made a region.
A region hit requires the ranked subject to be the labelled Juliet function (*_bad, goodG2B, goodB2G). imports:* listing a libc name is not a hit. entry:main is not a hit: the 12-line snippet is the INCLUDEMAIN prologue (srand / printLine("Calling …")), not the sink.
CWE regions verify region caller pair
121 entry:main, imports:memory strcpy dynamic @ *_bad/goodG2B miss hit identical
122 entry:main, imports:memory strcpy dynamic @ *_bad/goodG2B miss hit identical
190 entry:main abstain (no libc sink) miss — identical
415 entry:main free dynamic @ labelled (noisy) miss hit identical
416 entry:main abstain (printLine) miss — identical
476 entry:main abstain (data[0]) miss — identical
78 imports:process, entry:main system mixed; real site dynamic miss hit identical
imports:memory is the string strcpy. imports:process is the string system. That is the cheap PLT pivot the brief already advertises, not retrieval of the labelled function.
03 · IDENTICAL PAIRS
Good and bad still call the same sink. Ranking cannot split them.
Good-only and bad-only are compiled from the same file with OMITBAD versus OMITGOOD. Both still call strcpy / system / free (or neither, for 190/416/476). --quick therefore ranks the same region ids. verify sees a stack buffer feeding the first argument and reports <dynamic> on both sides.
ranked: entry:main, imports:memory
next_argv: []
verify strcpy: dynamic @ *_bad
ranked: entry:main, imports:memory
next_argv: []
verify strcpy: dynamic @ goodG2B
The good CWE-78 source is the constant suffix *.* on "ls "; that did not resolve to all-constant. This is the expected negative control. The Linux ELFs used plain strcpy at -O0; fortify did not rewrite the import.
04 · VERIFY CALLERS
Ask for a named import and you get a caller. On both twins.
Caller recall is 8/8 of the binaries that have a labelled libc sink, including the good twins. CWE-190, CWE-416, and CWE-476 abstain: their sinks are data + 1, printLine after free, and data[0].
CWE-78 is the only case that emits next_argv: r2b verify … --import system --json. Following it finds a caller. It does not split good from bad. Decompile was not auto-queued.
CWE-78 verify status: mixed
real system() site: <dynamic> in *_bad / goodG2B
extra attributed sites: "fgets() failed", "command execution failed!", imp.exit
CWE-415 verify --import free: labelled function appears
plus extra imp.malloc / imp.exit / imp.free xrefs around the same bytes
free is not a default --quick dangerous import; verify ran because the protocol named it
CWE-415 is a noisy caller hit, not a double-free finding. CWE-78 mixed status is extra string comments and PLT xrefs, not a good/bad split.
05 · TIMING AND HOST
Fourteen sequential briefs. Eight sequential verifies. No Ghidra.
14 sequential brief --quick. Per-binary 24.4–59.6 s. First CWE-121 bad paid a cold-start tax (53.6 s). CWE-78 good is the slowest (59.6 s).
8 sequential verify --import on libc-sink binaries only.
ARM64, 4 CPUs, 8 GiB. r2b 0.1.0, radare2 6.0.5, gcc 15.3.0 aarch64-linux-gnu, Python 3.11.13. Tree ae6310c.
Ghidra 12.1.2 is installed and was not used. No model calls. Targets not executed.
All 14 outputs are ELF 64-bit LSB PIE, ARM aarch64, dynamically linked, DWARF, not stripped. Not Mach-O.
Queued r2b verify --import system on both twins. Empty on the other twelve. Decompile was not auto-queued.
THE HONEST CLAIM
r2b did not detect these CWEs. Ranking is not a CWE detector.
Region recall is zero at every requested cutoff. Identical good/bad ranking is the recorded result on all seven pairs. verify locates a libc-sink caller when asked; it does not classify the labelled good source versus the labelled bad source. Import name alone is not a hit, and was not scored as one.