feat: bring data-platform work product under version control - #1
Merged
Merged
Conversation
Initial commit of the jol-m-data content tree, previously developed outside VCS: - warehouse (dbt models/macros/tests/seeds), quality (expectations, scorecards, anomaly rules), lifecycle (retention jobs, anonymization, legal hold, verification), ingestion (pseudonymizer pipeline), synthetic fixture generators, governance + ADRs - catalog: 19 datasets registered, owned, classified, retention-mapped Verification evidence: - make check green: seed validation, catalog lint, PII tripwire - synthetic + pseudonymizer unit tests present (2 suites) - gitleaks staged scan clean (one documented false positive allowed: empty EXTRACT_STRIPE_KEY placeholder in .envrc.example) Defect fixed in-flight: .pre-commit-config.yaml referenced nonexistent upstream hook check-csv (blocked every commit); replaced with local scripts/check-csv.py structural gate + unittest regression suite (5/5), validated against all 5 fleet CSV registers/seeds. Also fixed: sqlfluff hook demanded the dbt templater with no graceful path for credential-free checkouts (profiles.yml is never committed); converted to a local hook that lints when the environment is provisioned and skips loudly otherwise — CI dbt-ci.yml remains the hard gate.
The template shipped a fabricated SHA (a379cfe...), failing job setup with 'unable to find version'. Resolved via GitHub API: v5.6.0 -> a26af69be951a213d495a4c3e4e4022e16d87065.
…ions The dbt-ci gate ran for the first time after seeding the remote and caught real debt; fixed in-repo, gate kept strict: - codeql.yml: job permissions lacked contents:read (checkout fails on private repos) and actions:read (analyze fetches run metadata) - .sqlfluff: tab_space_size 2->4, aligning the linter with the 4-space model style actually written across warehouse/models (killed 199 LT02) - ST06: reordered select targets (simple before calculations) in int_seller_lifecycle, consent_rates, dim_products - RF04: renamed keyword aliases dim_date.year/month -> year_number/ month_number, dim_sellers.locale -> locale_name (no consumers) - ST02: fct_orders is_revenue CASE collapsed to boolean IN expression - LT02: consent_rates round() continuation indent (sqlfluff fix) Verified locally at CI parity: ephemeral postgres 16 + CI raw schema, sqlfluff 4.3.0 with dbt templater clean, dbt build PASS=67 (3 seeds, 18 models, 46 data tests), ERROR=0.
Org is on the team plan; code scanning is unavailable for private repos (API 403: 'Code Security must be enabled'). A permanently red analyze job would mask real failures. Re-enable by dropping the .disabled suffix once GitHub Advanced Security is procured — tracked as a security-tooling procurement finding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: bring data-platform work product under version control
Initial commit of the jol-m-data content tree, previously developed
outside VCS:
scorecards, anomaly rules), lifecycle (retention jobs, anonymization,
legal hold, verification), ingestion (pseudonymizer pipeline),
synthetic fixture generators, governance + ADRs
Verification evidence:
empty EXTRACT_STRIPE_KEY placeholder in .envrc.example)
Defect fixed in-flight: .pre-commit-config.yaml referenced nonexistent
upstream hook check-csv (blocked every commit); replaced with local
scripts/check-csv.py structural gate + unittest regression suite (5/5),
validated against all 5 fleet CSV registers/seeds.
Also fixed: sqlfluff hook demanded the dbt templater with no graceful
path for credential-free checkouts (profiles.yml is never committed);
converted to a local hook that lints when the environment is provisioned
and skips loudly otherwise — CI dbt-ci.yml remains the hard gate.