Skip to content

Add Civil ID S3 recovery audit helper#89

Open
souf92i wants to merge 2 commits into
BAWES-Universe:masterfrom
souf92i:codex-civil-id-s3-recovery-audit
Open

Add Civil ID S3 recovery audit helper#89
souf92i wants to merge 2 commits into
BAWES-Universe:masterfrom
souf92i:codex-civil-id-s3-recovery-audit

Conversation

@souf92i
Copy link
Copy Markdown

@souf92i souf92i commented May 15, 2026

/claim #55

Summary

  • Adds an offline Civil ID S3 recovery audit helper for the Phase 8 missing-data audit path.
  • Classifies exported Civil ID references as present, recoverable from the legacy permanent prefix, recoverable from the temp bucket export, or missing.
  • Emits Markdown or CSV reports and optional copy commands without calling AWS or touching real candidate data.
  • Documents the operator workflow and includes a fixture-based verifier using synthetic data only.

Scope

This is a separate Phase 8 audit/recovery helper slice. It does not overlap with the open Civil ID backend/frontend fixes, S3 credential rotation, IAM policy changes, n8n audit, CloudTrail export, bucket guardrail, or secret-scanning PRs.

No live AWS/IAM/S3 calls were made. No key rotation, deletion, bucket policy change, candidate record export, real S3 inventory, or credential material is included.

Verification

  • node tools/check-civil-id-s3-audit.mjs
  • node --check tools/audit-civil-id-s3-objects.mjs
  • node --check tools/check-civil-id-s3-audit.mjs
  • git diff --check

The repo's documented full suite command currently fails before test bootstrap because run-tests.sh invokes docker-compose without a compose file, while this checkout only includes named compose files such as docker-compose-local.yml, docker-compose-dev.yml, and docker-compose-prod.yml.

Summary by CodeRabbit

  • Documentation

    • Added Civil ID S3 Recovery Audit procedure documentation with offline audit workflows.
  • New Features

    • Added CLI tools for auditing Civil ID S3 object inventories with status classification (present, recoverable, missing) and optional recovery command generation in Markdown or CSV formats.

Review Change Stack

@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 15, 2026

Warning

Rate limit exceeded

@souf92i has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 58 minutes and 16 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 5d085863-190d-45bd-9e7d-44699550888e

📥 Commits

Reviewing files that changed from the base of the PR and between 96dd137 and 2d6af87.

📒 Files selected for processing (2)
  • tools/audit-civil-id-s3-objects.mjs
  • tools/check-civil-id-s3-audit.mjs
📝 Walkthrough

Walkthrough

This pull request adds infrastructure for Phase 8 of Civil ID S3 recovery: documentation and tooling to audit offline Civil ID file status. The audit tool classifies candidates as present, recoverable from legacy storage, recoverable from a temporary bucket, or missing, and outputs findings as Markdown or CSV with optional AWS copy commands.

Changes

Civil ID S3 Recovery Audit

Layer / File(s) Summary
Recovery workflow documentation
docs/civil-id-s3-recovery-audit.md
Offline audit procedure and SQL export query, input artifact requirements (candidates CSV and S3 bucket inventories), CLI usage with format and copy-command options, classification status definitions, and local verification command.
Audit tool core implementation
tools/audit-civil-id-s3-objects.mjs
Node.js CLI for offline Civil ID inventory audit: parses candidates CSV and S3 object-key exports, normalizes filenames, classifies each candidate into present/recover_from_legacy/recover_from_temp/missing status, and outputs as Markdown or CSV with optional AWS S3 copy commands.
Audit verification and fixtures
tools/check-civil-id-s3-audit.mjs
Smoke-test script that generates temporary fixture files, invokes the audit tool with Markdown and CSV output formats, asserts expected status counts and specific S3 copy command strings, and validates per-candidate recovery/missing results.

🎯 2 (Simple) | ⏱️ ~12 minutes

🐰 A recovery audit hops through S3 keys so bright,
Classifying each photo—present, lost, or right,
With copy commands nested in markdown's embrace,
And fixtures that verify each recovery case!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Add Civil ID S3 recovery audit helper' accurately summarizes the main change: addition of helper tooling for Civil ID S3 recovery auditing, matching the core purpose of the changeset.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/audit-civil-id-s3-objects.mjs`:
- Around line 34-43: The CLI currently accepts any --<key> which allows typos to
silently add unexpected options; update the argument parsing block that builds
args (the branch handling arg.startsWith('--') which extracts key and sets
args[key]) to validate the computed key against an explicit allowlist of
permitted flag names (e.g., a Set or array of allowed keys) and throw a clear
Error when an unknown flag is encountered; ensure this validation occurs before
assigning args[key] and incrementing i so unknown flags fail fast.

In `@tools/check-civil-id-s3-audit.mjs`:
- Around line 9-10: Replace usage of URL.pathname with Node's fileURLToPath to
get a proper filesystem path: derive the current module file path from
import.meta.url via fileURLToPath (and dirname if needed) to compute root, then
use path.join to build script; update the symbols root and script (currently
created from new URL('..', import.meta.url).pathname and join(root,
'tools/audit-civil-id-s3-objects.mjs')) so they use
fileURLToPath(import.meta.url) (and path.dirname) before join to ensure Windows
and encoded-character safe paths.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6078ecf3-2fde-4acb-a5a5-fc8b349b3ad3

📥 Commits

Reviewing files that changed from the base of the PR and between 7b023ff and 96dd137.

📒 Files selected for processing (3)
  • docs/civil-id-s3-recovery-audit.md
  • tools/audit-civil-id-s3-objects.mjs
  • tools/check-civil-id-s3-audit.mjs

Comment thread tools/audit-civil-id-s3-objects.mjs
Comment thread tools/check-civil-id-s3-audit.mjs Outdated
Copy link
Copy Markdown
Author

souf92i commented May 15, 2026

Addressed the CodeRabbit feedback in souf92i:codex-civil-id-s3-recovery-audit (2d6af873):

  • added an explicit allowlist for value options so unknown --... flags fail with Unknown argument
  • switched the verifier path setup from URL.pathname to fileURLToPath(...)
  • added a smoke assertion covering the unknown-flag failure path

Verification run locally:

  • node tools/check-civil-id-s3-audit.mjs
  • node --check tools/audit-civil-id-s3-objects.mjs
  • node --check tools/check-civil-id-s3-audit.mjs
  • git diff --check

The repository wrapper sh run-tests.sh is still blocked in this checkout by Docker Compose returning no configuration file provided: not found, so the targeted helper checks above are the relevant passing verification for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant