fix(core-engine): freeze the security frame's dead func_internal_density read (#2714) - #2746
Merged
Merged
Conversation
…ity read (#2714) `security_auditor._construct_feature_matrix` read `func_internal_density` out of artifact telemetry, but nothing has ever written that key there: signal_processor's `telemetry_payload`, galaxyscope's augmentation block and the ecosystem pass all skip it, and the density is computed only inside `record_keeper.py` (~L496) while the `file_data` row is written. The `.get` default therefore fired for every file on every scan -- the column has been a constant 0.0 in the security training frame since it was added, the same shape as the `prompt_injection`/`agentic_rce` placeholders documented in `analysis_lens.py` (#1020). Freeze it as an explicit, documented 0.0 rather than delete it. `audit_repository` aligns the frame by name via `df.reindex(columns=self.feature_names, fill_value=np.nan)`, so dropping the key would hand a trained model NaN where it has always seen 0.0 -- identical only if no tree splits on the feature, which is unverifiable here (the model artifact has never lived in this repo). Populating it for real is a scored-model input change and needs a retrain; that stays open, not bundled here. Swept the rest of the row for the same defect: `control_flow_ratio`, `func_complexity_gini`, `ownership_entropy`, `author_distribution`, `densities.cog_raw`, `ecosystem_baseline_cluster`, `ecosystem_z_score` and `dist_to_0..10` all have live writers. This was the only dead read. The new test pins the constant against a telemetry payload that *does* carry the key, so the frame can't start moving by accident if some future pass begins emitting it. Verified it fails against the old read. No golden-master bless owed: the feature matrix is only built when a model loads, and no fixture records this column. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2714.
The bug
security_auditor._construct_feature_matrix(L385) readfunc_internal_densityout of artifact telemetry, but nothing has ever written that key there. Telemetry is assembled in three places and none of them emit it:signal_processor.telemetry_payload(~L829)galaxyscope.pyL2311 —control_flow_ratio,popularity, identity keys)signal_processor.pyL1197-1200 —ecosystem_*,dist_to_N)The density is computed only inside
record_keeper.py(~L496) while thefile_datarow is written. So the.getdefault fired for every file on every scan: the column has been a constant 0.0 in the security frame since it was added — the same shape as theprompt_injection/agentic_rceplaceholders documented inanalysis_lens.py(#1020).I swept the rest of the row for the same defect.
control_flow_ratio,func_complexity_gini,ownership_entropy,author_distribution,densities.cog_raw,ecosystem_baseline_cluster,ecosystem_z_scoreanddist_to_0..10all have live writers. This was the only dead read.The fix: freeze, don't delete
The issue offered two options; this takes the conservative one, with a wrinkle worth recording.
Deleting the key is not a no-op.
audit_repositoryaligns the frame by name —df.reindex(columns=self.feature_names, fill_value=np.nan)— so dropping the column would hand a trained modelnp.nanwhere it has always seen0.0, and XGBoost routes NaN down a tree's default branch. That is identical only if no tree splits on the feature, which is near-certain for a constant training column but unverifiable here:gitgalaxy_malware_xgb_multiclass.jsonhas never existed in this repo (no file, no git history, no training script, not shipped bypyproject/MANIFEST.in), so the model is an external artifact users supply.So the key stays, as an explicit
0.0with a comment naming it a permanent placeholder and pointing atrecord_keeperas the metric's real home. Output is byte-identical for any model, and the code no longer implies a live feature.Populating it stays open, deliberately. Making the column vary is a scored-model input change: it needs a retrain and a look at what consumes the frame, and it does not belong in a cleanup PR.
Verification
test_func_internal_density_is_a_frozen_placeholderpins the constant against a telemetry payload that does carry the key — so the frame cannot start moving by accident if some future pass begins emitting it. Confirmed it fails against the oldtel.get(...)read and passes on the fix.tests/security_auditing+tests/core_engine/test_zero_dependency.py: 200 passed.audit_check.py: all clear (the one ruff finding was a pure line-shift of the pre-existingPERF203in this file, baseline regenerated).audit_repositoryreturns early otherwise), and no fixture records this column.Related: #2705 (where this was found), #1020 (placeholder-feature precedent).
🤖 Generated with Claude Code