Fix broken header newlines and list-rule sanitisation - #49
Conversation
- SanitisedFileFirstLine in the -MakeConfig default template used
literal backtick-escape text (`r`n) instead of the {1} newline
placeholder that eval-config-string actually expands, so the
generated header printed as garbage on one line.
- List-type rules ("regex"="replacement","delimiter") never matched
anything because of a typo ($matchesGroups instead of $matchGroups),
an undefined variable that always evaluated to $null.
output-keylist and the final console summary both sorted the found keys with Sort-Object -Property name, but the dictionary enumerator exposes Key/Value, not Name, so the sort silently no-ops and the result gets piped through Out-String, whose default table formatter depends on console width - which is invalid outside an interactive terminal (headless run, Task Scheduler, redirected output). Net effect: KeyList.txt and the "Processed Keys" summary always came out blank even though sanitisation itself was working correctly. Replaced both with an explicit "Key - Value" line join so the output no longer depends on host formatting state.
cavejay
left a comment
There was a problem hiding this comment.
Left inline notes on the three underlying bugs this PR fixes (header placeholder, list-rule typo, and the KeyList/console summary sorting-on-a-nonexistent-property + console-width-dependent formatting). All three were reproduced against the repo's own tests/ fixtures and against a larger 8000-line real-world logset (Linux/Apache/HDFS/OpenStack samples) before and after the fix — see the follow-up comment for a minimal repro any reviewer can run locally.
Generated by Claude Code
| log mkconf trace "We're going to make the config file here: $confloc" | ||
| # Apologies if you're trying to read this next string. | ||
| $defaultConfig = "; Strippy Config file`r`n;Recurse=true`r`n;InPlace=false`r`n;Silent=false`r`n;MaxThreads=5`r`n`r`n[ Config ]`r`nIgnoredStrings=""/0:0:0:0:0:0:0:0"",""0.0.0.0"",""127.0.0.1"",""name"",""applications"","""",""unknown"",""null"","".""`r`n`r`n; These settings can use braces to include dynamic formatting:`r`n; {0} = Date/Time at processing`r`n; {1} = NewLine`r`n; #notimplemented {2} = Depends on context. Name of specific file being processed where relevant otherwise it`s the name of the Folder/File provided to Strippy `r`nSanitisedFileFirstLine=""This file was Sanitised at {0}.``r``n==``r``n``r``n""`r`nKeyListFirstLine=""This keylist was created at {0}.""`r`n;KeyFileName=""Keylist.txt""`r`n;AlternateOutputFolder="".\sanitisedoutput""`r`n`r`n[ Rules ]`r`n;""Some Regex String here""=""Replacement here""`r`n""((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))[^\d]""=""Address""`r`n""\\\\([\w\-.]*?)\\""=""Hostname""`r`n" | ||
| $defaultConfig = "; Strippy Config file`r`n;Recurse=true`r`n;InPlace=false`r`n;Silent=false`r`n;MaxThreads=5`r`n`r`n[ Config ]`r`nIgnoredStrings=""/0:0:0:0:0:0:0:0"",""0.0.0.0"",""127.0.0.1"",""name"",""applications"","""",""unknown"",""null"","".""`r`n`r`n; These settings can use braces to include dynamic formatting:`r`n; {0} = Date/Time at processing`r`n; {1} = NewLine`r`n; #notimplemented {2} = Depends on context. Name of specific file being processed where relevant otherwise it`s the name of the Folder/File provided to Strippy `r`nSanitisedFileFirstLine=""This file was Sanitised at {0}.{1}=={1}{1}""`r`nKeyListFirstLine=""This keylist was created at {0}.""`r`n;KeyFileName=""Keylist.txt""`r`n;AlternateOutputFolder="".\sanitisedoutput""`r`n`r`n[ Rules ]`r`n;""Some Regex String here""=""Replacement here""`r`n""((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))[^\d]""=""Address""`r`n""\\\\([\w\-.]*?)\\""=""Hostname""`r`n" |
There was a problem hiding this comment.
Bug: eval-config-string only expands {0}/{1} placeholders (see its definition around line 431) — it does not interpret backtick escapes read back from a config file on disk. The old default template baked in literal `r`n text here instead of using the {1} newline placeholder the comment two lines above already documents, so every -MakeConfig run shipped a header that printed as one garbled line, e.g.:
This file was Sanitised at 09/10/2026 05:05:25.`r`n==`r`n`r`nAddress2
Switched to {1}, matching the pattern tests/sensitiveLists/list.conf already uses correctly (SanitisedFileFirstLine="...{1}{1}").
Generated by Claude Code
| $_matchGroups = $matchGroups # preserve original list of matches | ||
| $matchGroups = $_matchGroups | ForEach-Object { |
There was a problem hiding this comment.
Bug: $matchesGroups / $_matchesGroups are never assigned anywhere else in this file — the only populated variable is $matchGroups, set two lines above (line 951). So this block always read $null, meaning every list-type rule ("regex"="replacement","delimiter") silently matched nothing, no matter what it was pointed at.
Confirmed with tests/sensitiveLists/list.conf: before this fix, the comma/semicolon-separated fruit and building lists in simpleLists.log passed straight through unredacted. After the fix they get tokenised correctly (Fruit1,Fruit2,...).
Generated by Claude Code
|
|
||
| if (!$quicksave) { log outkey message "`r`nExporting KeyList to $kf" } | ||
| $KeyOutfile = (eval-config-string $script:config.KeyListFirstline) + "`r`n" + $( $finalKeyList.GetEnumerator() | Sort-Object -Property name | Out-String ) | ||
| $KeyOutfile = (eval-config-string $script:config.KeyListFirstline) + "`r`n" + $( ($finalKeyList.GetEnumerator() | Sort-Object -Property Key | ForEach-Object { "$($_.Key) - $($_.Value)" }) -join "`r`n" ) + "`r`n" |
There was a problem hiding this comment.
Bug: Sort-Object -Property name sorts on a property that doesn't exist on Dictionary.GetEnumerator() entries (they expose Key/Value, not Name), so it silently no-ops — and the result is piped into Out-String, whose default table formatting depends on console width. Outside an interactive terminal (headless run, Task Scheduler, redirected output — exactly the -Silent automation case the README documents), that width is invalid (-1) and Out-String renders nothing.
Net effect: KeyList.txt came out blank on every run I tested, even the small fixture files, even though the actual sanitisation was working correctly the whole time (the real key/value pairs were just never written out). Replaced with an explicit "Key - Value" line join so the output no longer depends on host formatting state. Same root cause fixed at line ~1680 below for the console "Processed Keys:" summary.
Generated by Claude Code
Manual verification recipeTwo small self-contained checks — one for the header fix, one for the list-rule + KeyList fixes. I ran both against the base commit ( 1. Header fix (
|
literal backtick-escape text (
rn) instead of the {1} newlineplaceholder that eval-config-string actually expands, so the
generated header printed as garbage on one line.
anything because of a typo ($matchesGroups instead of $matchGroups),
an undefined variable that always evaluated to $null.