Skip to content

fix(painless): pass script hashes and resolved plans - #179

Merged
catinspace-au merged 2 commits into
mainfrom
fix/hash-keyed-scripts
Nov 15, 2026
Merged

fix(painless): pass script hashes and resolved plans#179
catinspace-au merged 2 commits into
mainfrom
fix/hash-keyed-scripts

Conversation

@catinspace-au

@catinspace-au catinspace-au commented Nov 14, 2026

Copy link
Copy Markdown
Contributor

Each compiled_painless! site carries its script hash and resolved plan. The generated tree carries pipeline script text only where a runner still reads it. The code that needed the text is in dfe-transform-elastic-dev #19.

  • Bespoke runners are keyed by a hash that drops comments outside string literals. All 284 entries are re-keyed.
  • Tests built on pipeline scripts live in -dev and compile here only under --cfg dfe_script_tests.
  • bespoke_registry, painless_binding and multi_binding read the plans back.
  • RemapKeysThroughTable rejects targets that are not paths. It matched okta's two risk-behaviour scripts with script text as the target. That prevented RiskBehaviors from running.
  • The generated modules compile in 15 crates split by package, re-exported as filebeat from dfe-transforms.
  • never_ran rises from 61 to 63. The multi-binding floor falls from 58 to 56. Grouped by the old 200-character head, never_ran is still 61.

Pipeline script audit for this branch (scripts/script_audit.py from -dev):

scripts 1785 (20 pipeline files did not parse)
text carried: 221 scripts at 351 sites
allowed: crates/dfe-transforms/src/filebeat/aws_guardduty/default.rs
leaked: 0 scripts

The allowed file is a generated remove() of a field path a script also removes.

The largest crate, dfe-transforms-symantec, has one 6 MB module. Clippy peaks at 9.4 GiB on desktop-derek with sccache off. Its test-profile build also peaks at 9.4 GiB and takes 354 s.

The next five largest crates peak under 1.5 GiB. Before the split the whole tree peaked at 15.7 GiB.

On desktop-derek, the workspace passes plain cargo test except tests/broker.rs. Those tests share one metrics recorder per process. Broker passes 9 of 9 under nextest, which CI uses.

Each removed public file matches its -dev copy at #19's head 0982acca. The table shows the sha256 of public main's content:

Public path -dev path sha256 (16) Check
crates/dfe-painless/benches/painless.rs overlay/crates/dfe-painless/benches/painless.rs f301e049169b267f match
crates/dfe-painless/src/action_mapping_tests.rs overlay/crates/dfe-painless/src/action_mapping_tests.rs b02e569fe1ec98f1 match
crates/dfe-painless/src/actor_kind_tests.rs overlay/crates/dfe-painless/src/actor_kind_tests.rs e443c8685974ddbe match
crates/dfe-painless/src/collect_rows_tests.rs overlay/crates/dfe-painless/src/collect_rows_tests.rs 7beaaa63e6a963a1 match
crates/dfe-painless/src/common_tests.rs overlay/crates/dfe-painless/src/common_tests.rs 9778ec7075bd53a1 match
crates/dfe-painless/src/deep_merge_tests.rs overlay/crates/dfe-painless/src/deep_merge_tests.rs 530b8c36562c7a78 match
crates/dfe-painless/src/element_mapping_tests.rs overlay/crates/dfe-painless/src/element_mapping_tests.rs 633ce60ab18fce8f match
crates/dfe-painless/src/member_kv_fold_tests.rs overlay/crates/dfe-painless/src/member_kv_fold_tests.rs 7affe735e6ce5241 match
crates/dfe-painless/src/member_ladder_tests.rs overlay/crates/dfe-painless/src/member_ladder_tests.rs 810e4ae83fadf091 match
crates/dfe-painless/src/member_tags_tests.rs overlay/crates/dfe-painless/src/member_tags_tests.rs cb8dd4ab55064a70 match
crates/dfe-painless/src/member_value_wrap_tests.rs overlay/crates/dfe-painless/src/member_value_wrap_tests.rs 962225aafebafef8 match
crates/dfe-painless/src/params_tests.rs overlay/crates/dfe-painless/src/params_tests.rs 468c23a7352f5843 match
crates/dfe-painless/src/securityhub_ocsf_tests.rs overlay/crates/dfe-painless/src/securityhub_ocsf_tests.rs 7b00c0eca527b26a match
crates/dfe-painless/src/set_ladder_tests.rs overlay/crates/dfe-painless/src/set_ladder_tests.rs ee4188a8c9ae13ea match
crates/dfe-painless/src/stringify_member_tests.rs overlay/crates/dfe-painless/src/stringify_member_tests.rs c49991038333aa5e match
crates/dfe-painless/src/time_order_flag_tests.rs overlay/crates/dfe-painless/src/time_order_flag_tests.rs 82c94f5ed624290a match
crates/dfe-painless/src/url_action_tests.rs overlay/crates/dfe-painless/src/url_action_tests.rs fb5ae7d41adb8b0e match
crates/dfe-painless/tests/singleton_unwrap.rs overlay/crates/dfe-painless/tests/singleton_unwrap.rs 05cbf76d5f760605 match
crates/dfe-painless/tests/which_matcher.rs overlay/crates/dfe-painless/tests/which_matcher.rs ef5a89714d9f65e0 match

The 42 test modules moved from public files to -dev. Each matches the module body at public main with its 4-space indent removed:

Public file whose test module moved -dev path
crates/dfe-painless/src/append_records.rs (test module) overlay/crates/dfe-painless/src/append_records_tests.rs
crates/dfe-painless/src/byte_size.rs (test module) overlay/crates/dfe-painless/src/byte_size_tests.rs
crates/dfe-painless/src/coercion.rs (test module) overlay/crates/dfe-painless/src/coercion_tests.rs
crates/dfe-painless/src/collect_by_literal.rs (test module) overlay/crates/dfe-painless/src/collect_by_literal_tests.rs
crates/dfe-painless/src/collect_present.rs (test module) overlay/crates/dfe-painless/src/collect_present_tests.rs
crates/dfe-painless/src/delimited_table.rs (test module) overlay/crates/dfe-painless/src/delimited_table_tests.rs
crates/dfe-painless/src/dotted_keys.rs (test module) overlay/crates/dfe-painless/src/dotted_keys_tests.rs
crates/dfe-painless/src/dump_into_map.rs (test module) overlay/crates/dfe-painless/src/dump_into_map_tests.rs
crates/dfe-painless/src/entity.rs (test module) overlay/crates/dfe-painless/src/entity_tests.rs
crates/dfe-painless/src/expr.rs (test module) overlay/crates/dfe-painless/src/expr_tests.rs
crates/dfe-painless/src/field_tables.rs (test module) overlay/crates/dfe-painless/src/field_tables_tests.rs
crates/dfe-painless/src/gather_members.rs (test module) overlay/crates/dfe-painless/src/gather_members_tests.rs
crates/dfe-painless/src/group_records.rs (test module) overlay/crates/dfe-painless/src/group_records_tests.rs
crates/dfe-painless/src/guarded_records.rs (test module) overlay/crates/dfe-painless/src/guarded_records_tests.rs
crates/dfe-painless/src/hex.rs (test module) overlay/crates/dfe-painless/src/hex_tests.rs
crates/dfe-painless/src/hoist.rs (test module) overlay/crates/dfe-painless/src/hoist_tests.rs
crates/dfe-painless/src/indicator_expiry.rs (test module) overlay/crates/dfe-painless/src/indicator_expiry_tests.rs
crates/dfe-painless/src/issue_lifecycle.rs (test module) overlay/crates/dfe-painless/src/issue_lifecycle_tests.rs
crates/dfe-painless/src/item_writes.rs (test module) overlay/crates/dfe-painless/src/item_writes_tests.rs
crates/dfe-painless/src/labelled_keys.rs (test module) overlay/crates/dfe-painless/src/labelled_keys_tests.rs
crates/dfe-painless/src/labelled_score.rs (test module) overlay/crates/dfe-painless/src/labelled_score_tests.rs
crates/dfe-painless/src/level_labels.rs (test module) overlay/crates/dfe-painless/src/level_labels_tests.rs
crates/dfe-painless/src/list_records.rs (test module) overlay/crates/dfe-painless/src/list_records_tests.rs
crates/dfe-painless/src/lists.rs (test module) overlay/crates/dfe-painless/src/lists_tests.rs
crates/dfe-painless/src/map_entries.rs (test module) overlay/crates/dfe-painless/src/map_entries_tests.rs
crates/dfe-painless/src/member_fan_out.rs (test module) overlay/crates/dfe-painless/src/member_fan_out_tests.rs
crates/dfe-painless/src/named_arms.rs (test module) overlay/crates/dfe-painless/src/named_arms_tests.rs
crates/dfe-painless/src/needle_table.rs (test module) overlay/crates/dfe-painless/src/needle_table_tests.rs
crates/dfe-painless/src/nth_separator.rs (test module) overlay/crates/dfe-painless/src/nth_separator_tests.rs
crates/dfe-painless/src/pair_table.rs (test module) overlay/crates/dfe-painless/src/pair_table_tests.rs
crates/dfe-painless/src/plan.rs (test module) overlay/crates/dfe-painless/src/plan_tests.rs
crates/dfe-painless/src/record_fold.rs (test module) overlay/crates/dfe-painless/src/record_fold_tests.rs
crates/dfe-painless/src/records.rs (test module) overlay/crates/dfe-painless/src/records_tests.rs
crates/dfe-painless/src/rescale.rs (test module) overlay/crates/dfe-painless/src/rescale_tests.rs
crates/dfe-painless/src/seconds_between.rs (test module) overlay/crates/dfe-painless/src/seconds_between_tests.rs
crates/dfe-painless/src/split_fanout.rs (test module) overlay/crates/dfe-painless/src/split_fanout_tests.rs
crates/dfe-painless/src/threat_artifacts.rs (test module) overlay/crates/dfe-painless/src/threat_artifacts_tests.rs
crates/dfe-painless/src/totals.rs (test module) overlay/crates/dfe-painless/src/totals_tests.rs
crates/dfe-painless/src/typed_member_rename.rs (test module) overlay/crates/dfe-painless/src/typed_member_rename_tests.rs
crates/dfe-painless/src/windows.rs (test module) overlay/crates/dfe-painless/src/windows_tests.rs
crates/dfe-painless/src/trim_delimited.rs (test module) overlay/crates/dfe-painless/src/trim_delimited_tests.rs
crates/dfe-painless/src/last_element.rs (test module) overlay/crates/dfe-painless/src/last_element_tests.rs

@catinspace-au
fix(painless): pass script hashes and resolved plans
80e0f3b 
compiled_painless! passes each generated call site its script's hash and resolved plan. The text ships only where a bound runner still reads it: 351 sites over 221 scripts, down from 1,873 sites.

- The script hash drops comments outside string literals. The 284 bespoke entries are re-keyed to it.
- The stats catalogue uses that hash. Three scripts sharing one 200-character head now count separately. This moves never_ran from 61 to 63.
- RemapKeysThroughTable refuses a target that is not a path. It had claimed okta's two risk-behaviour scripts with script text as its target. RiskBehaviors runs them now.
- EpochToMillis and EpochNanosToDateTime match their helper by length and SHA-256.
- The tests built on pipeline scripts move to the private repository, which compiles them under --cfg dfe_script_tests.
- bespoke_registry, painless_binding and multi_binding stay in this repository and read the plans back.
- The generated modules compile in 15 crates split by package. dfe-transforms re-exports them as filebeat, so no rustc holds the whole tree.
@catinspace-au
chore(merge): merge main for the scalo 2.14.30 rebuild
09f1c65 
Takes main's scalo 2.14.30 pins and its regenerated config schema and deployment contract unchanged.
@catinspace-au
catinspace-au merged commit cfe03dc into main Nov 15, 2026
26 checks passed
@catinspace-au
catinspace-au deleted the fix/hash-keyed-scripts branch November 15, 2026 03:44
@github-actions

github-actions Bot commented Nov 15, 2026

Copy link
Copy Markdown

Released in v1.1.32 -- https://github.com/hyperi-io/dfe-transform-elastic/releases/tag/v1.1.32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@catinspace-au