Skip to content

Fix broken header newlines and list-rule sanitisation - #49

Open
cavejay wants to merge 2 commits into
masterfrom
claude/strippy-testing-dn8veh
Open

Fix broken header newlines and list-rule sanitisation#49
cavejay wants to merge 2 commits into
masterfrom
claude/strippy-testing-dn8veh

Conversation

@cavejay

@cavejay cavejay commented Sep 10, 2026

Copy link
Copy Markdown
Owner
  • SanitisedFileFirstLine in the -MakeConfig default template used
    literal backtick-escape text (rn) instead of the {1} newline
    placeholder that eval-config-string actually expands, so the
    generated header printed as garbage on one line.
  • List-type rules ("regex"="replacement","delimiter") never matched
    anything because of a typo ($matchesGroups instead of $matchGroups),
    an undefined variable that always evaluated to $null.

- SanitisedFileFirstLine in the -MakeConfig default template used
  literal backtick-escape text (`r`n) instead of the {1} newline
  placeholder that eval-config-string actually expands, so the
  generated header printed as garbage on one line.
- List-type rules ("regex"="replacement","delimiter") never matched
  anything because of a typo ($matchesGroups instead of $matchGroups),
  an undefined variable that always evaluated to $null.
output-keylist and the final console summary both sorted the found
keys with Sort-Object -Property name, but the dictionary enumerator
exposes Key/Value, not Name, so the sort silently no-ops and the
result gets piped through Out-String, whose default table formatter
depends on console width - which is invalid outside an interactive
terminal (headless run, Task Scheduler, redirected output). Net
effect: KeyList.txt and the "Processed Keys" summary always came out
blank even though sanitisation itself was working correctly.

Replaced both with an explicit "Key - Value" line join so the output
no longer depends on host formatting state.

@cavejay cavejay left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left inline notes on the three underlying bugs this PR fixes (header placeholder, list-rule typo, and the KeyList/console summary sorting-on-a-nonexistent-property + console-width-dependent formatting). All three were reproduced against the repo's own tests/ fixtures and against a larger 8000-line real-world logset (Linux/Apache/HDFS/OpenStack samples) before and after the fix — see the follow-up comment for a minimal repro any reviewer can run locally.


Generated by Claude Code

Comment thread strippy.ps1
log mkconf trace "We're going to make the config file here: $confloc"
# Apologies if you're trying to read this next string.
$defaultConfig = "; Strippy Config file`r`n;Recurse=true`r`n;InPlace=false`r`n;Silent=false`r`n;MaxThreads=5`r`n`r`n[ Config ]`r`nIgnoredStrings=""/0:0:0:0:0:0:0:0"",""0.0.0.0"",""127.0.0.1"",""name"",""applications"","""",""unknown"",""null"","".""`r`n`r`n; These settings can use braces to include dynamic formatting:`r`n; {0} = Date/Time at processing`r`n; {1} = NewLine`r`n; #notimplemented {2} = Depends on context. Name of specific file being processed where relevant otherwise it`s the name of the Folder/File provided to Strippy `r`nSanitisedFileFirstLine=""This file was Sanitised at {0}.``r``n==``r``n``r``n""`r`nKeyListFirstLine=""This keylist was created at {0}.""`r`n;KeyFileName=""Keylist.txt""`r`n;AlternateOutputFolder="".\sanitisedoutput""`r`n`r`n[ Rules ]`r`n;""Some Regex String here""=""Replacement here""`r`n""((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))[^\d]""=""Address""`r`n""\\\\([\w\-.]*?)\\""=""Hostname""`r`n"
$defaultConfig = "; Strippy Config file`r`n;Recurse=true`r`n;InPlace=false`r`n;Silent=false`r`n;MaxThreads=5`r`n`r`n[ Config ]`r`nIgnoredStrings=""/0:0:0:0:0:0:0:0"",""0.0.0.0"",""127.0.0.1"",""name"",""applications"","""",""unknown"",""null"","".""`r`n`r`n; These settings can use braces to include dynamic formatting:`r`n; {0} = Date/Time at processing`r`n; {1} = NewLine`r`n; #notimplemented {2} = Depends on context. Name of specific file being processed where relevant otherwise it`s the name of the Folder/File provided to Strippy `r`nSanitisedFileFirstLine=""This file was Sanitised at {0}.{1}=={1}{1}""`r`nKeyListFirstLine=""This keylist was created at {0}.""`r`n;KeyFileName=""Keylist.txt""`r`n;AlternateOutputFolder="".\sanitisedoutput""`r`n`r`n[ Rules ]`r`n;""Some Regex String here""=""Replacement here""`r`n""((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))[^\d]""=""Address""`r`n""\\\\([\w\-.]*?)\\""=""Hostname""`r`n"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: eval-config-string only expands {0}/{1} placeholders (see its definition around line 431) — it does not interpret backtick escapes read back from a config file on disk. The old default template baked in literal `r`n text here instead of using the {1} newline placeholder the comment two lines above already documents, so every -MakeConfig run shipped a header that printed as one garbled line, e.g.:

This file was Sanitised at 09/10/2026 05:05:25.`r`n==`r`n`r`nAddress2

Switched to {1}, matching the pattern tests/sensitiveLists/list.conf already uses correctly (SanitisedFileFirstLine="...{1}{1}").


Generated by Claude Code

Comment thread strippy.ps1
Comment on lines +955 to +956
$_matchGroups = $matchGroups # preserve original list of matches
$matchGroups = $_matchGroups | ForEach-Object {

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: $matchesGroups / $_matchesGroups are never assigned anywhere else in this file — the only populated variable is $matchGroups, set two lines above (line 951). So this block always read $null, meaning every list-type rule ("regex"="replacement","delimiter") silently matched nothing, no matter what it was pointed at.

Confirmed with tests/sensitiveLists/list.conf: before this fix, the comma/semicolon-separated fruit and building lists in simpleLists.log passed straight through unredacted. After the fix they get tokenised correctly (Fruit1,Fruit2,...).


Generated by Claude Code

Comment thread strippy.ps1

if (!$quicksave) { log outkey message "`r`nExporting KeyList to $kf" }
$KeyOutfile = (eval-config-string $script:config.KeyListFirstline) + "`r`n" + $( $finalKeyList.GetEnumerator() | Sort-Object -Property name | Out-String )
$KeyOutfile = (eval-config-string $script:config.KeyListFirstline) + "`r`n" + $( ($finalKeyList.GetEnumerator() | Sort-Object -Property Key | ForEach-Object { "$($_.Key) - $($_.Value)" }) -join "`r`n" ) + "`r`n"

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bug: Sort-Object -Property name sorts on a property that doesn't exist on Dictionary.GetEnumerator() entries (they expose Key/Value, not Name), so it silently no-ops — and the result is piped into Out-String, whose default table formatting depends on console width. Outside an interactive terminal (headless run, Task Scheduler, redirected output — exactly the -Silent automation case the README documents), that width is invalid (-1) and Out-String renders nothing.

Net effect: KeyList.txt came out blank on every run I tested, even the small fixture files, even though the actual sanitisation was working correctly the whole time (the real key/value pairs were just never written out). Replaced with an explicit "Key - Value" line join so the output no longer depends on host formatting state. Same root cause fixed at line ~1680 below for the console "Processed Keys:" summary.


Generated by Claude Code

cavejay commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

Manual verification recipe

Two small self-contained checks — one for the header fix, one for the list-rule + KeyList fixes. I ran both against the base commit (0d3e0804) and against this branch's head to confirm the before/after below is real output, not a guess.

1. Header fix (-MakeConfig default template)

.\strippy.ps1 -MakeConfig
Select-String SanitisedFileFirstLine .\strippy.conf
.\strippy.ps1 .\your.log -o .\out -Silent
Get-Content .\out\your.sanitised.log -TotalCount 4

Before (base commit):

SanitisedFileFirstLine="This file was Sanitised at {0}.`r`n==`r`n`r`n"

This file was Sanitised at 09/10/2026 08:14:24.`r`n==`r`n`r`nConnection from Address1 refused
AllowedUsers: alice,bob,carol

The `r`n never expands — it lands in the file as literal text and the header eats the first content line with it.

After (this branch):

SanitisedFileFirstLine="This file was Sanitised at {0}.{1}=={1}{1}"

This file was Sanitised at 09/10/2026 08:14:29.
==

Connection from Address1 refused

2. List-rule + KeyList fixes

Save these two files next to strippy.ps1:

strippy-review.conf

[ Config ]
SanitisedFileFirstLine="This file was Sanitised at {0}.{1}=={1}{1}"

[ Rules ]
"((([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5]))[^\d]"="Address"
"AllowedUsers: (.+)$"="AllowedUser",","

review-test.log

Connection from 10.0.0.5 refused
AllowedUsers: alice,bob,carol

Run:

.\strippy.ps1 .\review-test.log -ConfigFile .\strippy-review.conf -o .\out -ko .\out\KeyList.txt -Silent
Get-Content .\out\review-test.sanitised.log
Get-Content .\out\KeyList.txt

Before (base commit):

Connection from Address1 refused
AllowedUsers: alice,bob,carol          <- list rule never fired, names left in plain text

--- KeyList.txt ---
(no keys listed — just the boilerplate header/filename lines)

After (this branch):

Connection from Address1 refused
AllowedUsers: AllowedUser3,AllowedUser1,AllowedUser2

--- KeyList.txt ---
Address1 - 10.0.0.5
AllowedUser1 - bob
AllowedUser2 - carol
AllowedUser3 - alice

One aside, not something this PR touches: on the base commit only, that same run occasionally logs a non-fatal Exception calling "Replace" with "2" argument(s): "The value cannot be an empty string." during Sanitising-Stripper, then still completes and produces the output above. Looks like a pre-existing, timing-dependent issue in the multi-threaded key-merge path, unrelated to the three bugs fixed here — flagging so it doesn't look like something this diff introduced if you see it while reproducing.

Also validated all four fixes together against the repo's own tests/ fixtures (no regressions) and against an 8000-line combined real-world logset (Linux/Apache/HDFS/OpenStack samples from loghub) exercising basic, list, and delete rule types — zero leftover sensitive values in the output, correct line deletion counts, and a fully populated KeyList.txt.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants