Repository navigation
LLM PoC gate: real-library drafts, compile feedback, tri-state verdicts - #152
Merged
Merged
Conversation
PoC drafting produced nothing usable on cJSON: every draft copied the target
function body into the PoC and then called it with malformed input, so the
crash happened inside the copy and said nothing about the shipped library.
The root cause was the validator, not the model. The rule "the target function
name must appear in the PoC" cannot be satisfied for a static function, so
copying was the only way through the gate.
- llm_adapter: allowlisted /chat/completions transport (DeepSeek), keeping the
existing guards: no redirects, no retries, request/response byte ceilings,
credential screening in both directions, digest bound to the request
- llm_poc: a draft that includes the project header may reach the target
through the public API, so the "name must appear" rule is waived for it
- llm_poc: reject drafts that define the target function themselves; the
detector ignores nested calls such as `if (print(x)) {`
- llm_poc: one feedback turn on rejection, and one on a failed build, carrying
only the validator note or the compiler output, never new evidence
- llm_poc: drafts that never build end as compile_failed and are not written
to disk; a crash inside the compile hook counts as "not checked", not as a
failed draft
Measured on the cJSON tree: 10/10 drafts copied the target body before,
0/5 after; all 5 include the real header and compile against the real library.
Drafting alone cannot tell whether a PoC compiles, links the real project, or crashes inside the shipped code. This adds the last two steps of the gate and a verdict that separates the cases that used to look identical. - compile and run every draft; the link set is inferred from the linker's own undefined-symbol messages (lld-link, GNU ld and MSVC formats), so no caller has to hand-write --link for a component - classify each draft as real_link / self_contained / copied_target; only a real_link draft that crashes can support a finding - three-state verdict: confirmed (crash inside the library), refuted (linked, ran, did not crash), undetermined (copied body, no crash without a memory checker, build failure) - a "did not crash" observation is not promoted to a refutation when the pattern can corrupt memory silently and no memory checker is present: CVE-2026-87933 returns 0 on the default heap and faults under a quarantining allocator, so the same PoC would otherwise flip verdict with the build - tools/harness: guarded heap for Windows builds. ASan has no mingw runtime and lld rejects --wrap, so malloc/free/realloc/calloc are redirected at the preprocessor level onto per-allocation pages that are quarantined on free. Freed addresses are never reused, otherwise the next allocation lands on the same page and the use-after-free runs to completion Measured on the cJSON tree, same harness and build: CVE-2026-87933 (real UAF) -> confirmed, crash_inside_library five cJSON P6 candidates -> refuted, no crash That pair is the point: the harness discriminates instead of making everything look exploitable.
…rary gate
A crash inside the library turned out to be necessary but not sufficient. On the
UDMF case two drafts from the same model both SIGSEGVed inside
libcj_unified_data_channel_ffi.z.so; the exit code, the address range and the
"real_link" shape were identical. Only the symbolized frame separated them:
draft 1 (guessed CParameters) -> SetDataParameters, offset 0x16e5d
draft 2 (ABI from the header) -> FfiUDMFSetProperties, offset 0x1718a
The first one would have been filed as confirmed under the old tri-state rule.
This commit adds the missing discrimination and the loop that uses it.
poc_verify
- verdict_for() takes expected/observed and returns 'wrong_site' when the
crash frame is not the claimed one; with no frame resolver the old
library-level verdict is unchanged
- frame comparison is by symbol (a namespace-qualified or ".cfi" suffixed
name still matches), never by offset: the same source line moves between
builds, as the IPC case showed (0x1ed9b vs 0x218c4)
- verify(..., frames={...}) / --frames FILE reads an external resolver's map,
so nothing here depends on a debugger being installed
llm_poc
- generate_poc_llm(..., runtime_hook=fn) is the fourth gate: after the text
and compile gates a draft is run, and a crash that misses the claimed frame
is fed back once with the observed frame before being recorded wrong_site
- a draft that never lands is not written to disk, same rule as compile_failed
- a hook that throws counts as "not checked", never as "draft is wrong"
poc_real (new)
- resolve_frame() builds a draft with the OHOS clang, runs it under the musl
loader through WSL, and returns the crash offset plus the addr2line inline
chain; two gdb rounds run with ASLR on and must agree on the offset
- runtime_hook() adapts that to the gate above; soname_alias covers drafts
that dlopen a guessed library name
- tools/harness/ohos/run_real.sh is the WSL-side runner
tools/harness/linux (new)
- LD_PRELOAD guarded heap for the same purpose where gdb and ASan are
available directly
Tests: 20 (poc_real) + 24 (poc_verify) + 28 (llm_poc), and the existing LLM
suites still pass.
…ASLR rounds"
Recomputing the archived evidence does not reproduce either number in the old
record:
gdb-full.txt (hiappevent-evidence-20261003.zip)
one SIGSEGV, $rip = 0x7ffff554dc9a
libcj_hiappevent_ffi.z.so r-xp 0x7ffff5548000, segment file offset 0x8000
-> file_offset 0xdc9a, not 0x11c9a
The package also holds a single gdb session, so "two rounds with different ASLR
layouts" has no record behind it either. Both claims sit in the competition
report's table 3-8, where a reviewer can check them with one command.
Six places carried 0x11c9a: the case file, CHANGELOG, the confirmation note,
VULN_REPORT, the works-report draft and the case ledger. All now say 0xdc9a
with a short correction note stating what the archive actually shows. If a
second gdb round exists outside this package, that record should be added and
this change reverted rather than argued around.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Drafting LLM PoCs produced 10/10 drafts that copied the vulnerable function body into the PoC and called a copy of it, so any crash was a tautology and none of it could enter an evidence pack.
What changed
Drafting (
ohvul/llm_poc.py,ohvul/llm_adapter.py)/chat/completionstransport (DeepSeek); redirects, retries and byte ceilings unchangedif (print(x)) {are not mistaken for definitions)compile_failedand are not written to diskVerification (
ohvul/poc_verify.py,tools/harness/)--linktools/harness/guarded_heap.h: Windows guarded heap (ASan has no mingw runtime, lld has no--wrap), redirecting malloc/free at the preprocessor level onto quarantined pagestools/harness/linux/: the same idea as anLD_PRELOADallocator where gdb and ASan are available directlyA crash is not evidence until the frame says so (
ohvul/poc_real.py,tools/harness/ohos/run_real.sh)Two drafts from the same model, same exit code, same library, same address range — and only one of them is the bug:
verdict_for(..., expected=, observed=)returnswrong_sitewhen the crash frame is not the claimed one; with no frame resolver the previous library-level verdict is unchanged0x1ed9bvs0x218c4)resolve_frame()builds a draft with the OHOS clang, runs it under the musl loader through WSL, and returns the offset plus theaddr2lineinline chain; two gdb rounds run with ASLR on and must agree on the offsetruntime_hook()adapts that into a fourth gate, so a draft that misses the claimed frame is fed the observed frame once and then rejectedEvidence
Same harness and build, CVE-2026-87933 (real UAF) vs the cJSON P6 candidates:
The pair is the point: the harness discriminates instead of making everything look exploitable.
Real library, UDMF
SetProperties(tag=NULL), two drafts of the same finding:Tests
tests/test_llm_poc.py(28),tests/test_poc_verify.py(24),tests/test_poc_real.py(20), plus the existing LLM suites — 112 pass.Also in this branch
The HiAppEvent case file offset was wrong: the archived
gdb-full.txtrecomputes to0xdc9a, not0x11c9a, and holds a single gdb round, not two. Six places carried the old number (case file, CHANGELOG, confirmation note, VULN_REPORT, works-report draft, case ledger); all now say0xdc9awith a note recording what the archive shows.Not covered
The
runtime_hookimplementation still needs WSL with a built OHOS tree; on a host without it the gate simply cannot run, and the verdict falls back to library-level attribution.