From 68c921620867fb36f8f90be4c5306351e636e849 Mon Sep 17 00:00:00 2001 From: Joypal Navaneeth Date: Tue, 1 Sep 2026 14:58:40 +0530 Subject: [PATCH] docs: CI workflow, architecture doc, README status update --- .github/workflows/tests.yml | 23 +++++++++++ README.md | 29 +++++++++++--- docs/ARCHITECTURE.md | 78 +++++++++++++++++++++++++++++++++++++ 3 files changed, 125 insertions(+), 5 deletions(-) create mode 100644 .github/workflows/tests.yml create mode 100644 docs/ARCHITECTURE.md diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..4f213bf --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,23 @@ +name: Tests + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install project (with dev dependencies) + run: pip install -e ".[dev]" + + - name: Run tests + run: pytest -v diff --git a/README.md b/README.md index 44d786d..b4c9263 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,8 @@ # CRM Dedup/Hygiene Toolkit +[![Tests](https://github.com/codeapplied/crm-dedup-hygiene-toolkit/actions/workflows/tests.yml/badge.svg)](https://github.com/codeapplied/crm-dedup-hygiene-toolkit/actions/workflows/tests.yml) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) + Finds and safely merges duplicate organizations and contacts in a CRM, with a full audit trail — rebuilt as a generalized, open-source system. @@ -9,9 +12,11 @@ branding. ## Status -🚧 Early development. DB models, config loading, and CLI skeleton are in -place. Duplicate discovery and merge execution are not built yet — see -[open issues](https://github.com/codeapplied/crm-dedup-hygiene-toolkit/issues). +Core pipeline is built and tested: duplicate discovery, score-based primary +selection, and merge execution (dry-run by default, `--execute` to actually +merge) all work end-to-end against both the bundled sandbox demo and a real +Pipedrive backend. See [open issues](https://github.com/codeapplied/crm-dedup-hygiene-toolkit/issues) +for what's next. ## Architecture @@ -22,9 +27,12 @@ member → score each member by a weighted combination of linked-record counts → the highest-scoring member becomes "primary" → dry run prints the exact merge plan; only an explicit `--execute` performs the merge → call the CRM's native merge endpoint, then post one audit note on the primary summarizing -exactly what was absorbed. Every stage is logged to `MergeLog`, which the ops +exactly what was absorbed. Every run is logged to `MergeLog`, which the ops CLI reads for history. +See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) for the full data-flow +diagram and design rationale. + ## Setup ``` @@ -35,12 +43,23 @@ cp config/rules.example.yaml config/rules.yaml crmdedup init ``` +Leave `.env`'s Pipedrive fields blank to run against the bundled zero-network +sandbox demo — no credentials needed to try it. + ## CLI - `crmdedup init` — create the database - `crmdedup status` — recent merge runs - `crmdedup rules` — show the loaded scoring weights and known-false-positive skip list -- `crmdedup discover` — find duplicate groups (not yet implemented) +- `crmdedup discover` — find duplicate groups (dry run — plans only, writes nothing to the CRM) +- `crmdedup discover --execute` — actually perform each planned merge + +## Development + +``` +uv pip install -e ".[dev]" +pytest -v +``` ## License diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..ecc84f7 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,78 @@ +# Architecture + +## Overview + +The toolkit finds and safely merges duplicate CRM records in three stages: +**discover** (group candidates by a normalized key), **score** (pick which +member of each group survives), and **merge** (actually combine them, with +an audit trail). Every mutating step is dry-run by default — nothing is +written to the CRM unless `--execute` is passed explicitly. + +## Data flow + +```mermaid +flowchart TD + A[CRMClient.list_organizations / list_contacts] --> B[normalize_domain / lowercase email] + B --> C[group by key] + C --> D{group size >= 2?} + D -- no --> Z[discarded, not a duplicate] + D -- yes --> E{in skip_pairs?} + E -- yes --> Y[skipped, known false positive] + E -- no --> F[get_related_counts per member] + F --> G[score = weighted sum of counts] + G --> H[pick_primary: richest wins, lowest-id tiebreak] + H --> I{--execute?} + I -- no --> J[write MergeLog, status=dry_run] + I -- yes --> K[merge_organizations / merge_contacts per absorbed id] + K --> L{merge succeeded?} + L -- yes --> M[add to absorbed_ok] + L -- no --> N[record failure, continue — never abort the group] + M --> O[post one audit note on primary] + N --> P[write MergeLog, status=merged/partial/failed] + O --> P +``` + +## Components + +- **`crm/base.py`** — the `CRMClient` interface every backend implements: + list organizations/contacts, get related-record counts, native merge for + both entity types, post an audit note. +- **`crm/sandbox_client.py`** — zero-network default backend, seeded with + real duplicate groups so `discover`/`--execute` have something genuine to + find and merge without any credentials. +- **`crm/pipedrive_client.py`** — a real Pipedrive API v1 implementation. + Honest about its own limitations (no native `notes_count` field, no live + account tested against in this project) rather than hiding them. +- **`dedup.py`** — pure logic, no I/O: key normalization, grouping, + `skip_pairs` filtering, scoring, primary selection. Fully unit-testable + without a database or network. +- **`pipeline/discover.py`** — orchestrates the CRM client and `dedup.py` + into one run: dry-run by default, `execute=True` actually merges. + +## Design decisions worth explaining + +- **Re-verification is the merge call itself, not a separate pre-check.** + Both CRM backends raise when a source record is already gone (merged by + someone else since the plan was made). Catching that at the merge call is + the re-verification checkpoint — a redundant separate "does it still + exist" check would just be the same information fetched twice. +- **A failure on one group member never aborts the group.** If 3 of 5 + duplicates merge cleanly and 2 fail, that's `status=partial` with 3 + recorded as absorbed, not a rolled-back no-op. +- **Score-based primary selection, one formula for both entity types.** A + weighted sum of linked-record counts (contacts, deals, notes, open + activities) — the richest record survives a merge, not "lowest id" or + "most recently touched." +- **`skip_pairs` is data, not code.** Known false positives (a shared + mailbox, an ambiguous same-key pair) are listed in `config/rules.yaml` + and consulted by the matching logic, not hand-written as special cases + inside `dedup.py`. + +## What this deliberately does NOT do + +- No scheduled/automated merging — `discover --execute` is a command you + run, not a background job. Duplicate merging is high-stakes enough to + warrant a human deciding when to run it. +- No fuzzy/partial-name matching — only exact-key duplicates (same + normalized domain, same email) are found automatically. Partial variants + need a human to catch, same as the reference system this was informed by.