diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml new file mode 100644 index 0000000..4f213bf --- /dev/null +++ b/.github/workflows/tests.yml @@ -0,0 +1,23 @@ +name: Tests + +on: + push: + branches: [main] + pull_request: + branches: [main] + +jobs: + test: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - uses: actions/setup-python@v5 + with: + python-version: "3.12" + + - name: Install project (with dev dependencies) + run: pip install -e ".[dev]" + + - name: Run tests + run: pytest -v diff --git a/README.md b/README.md index 19d127c..ea32478 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,9 @@ # Tender Tracking System +[![Tests](https://github.com/codeapplied/tender-tracking-system/actions/workflows/tests.yml/badge.svg)](https://github.com/codeapplied/tender-tracking-system/actions/workflows/tests.yml) +[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml) +[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE) + A tender/bid tracking tool — rebuilt as a generalized, open-source system. Originally built as an internal work tool; this repo is a from-scratch rebuild using @@ -19,13 +23,47 @@ for the full build history and ## Architecture -Daily scheduled run → per-source scraper/API client → relevance filter → -normalize into a common `Tender` record → dedupe against the DB → export to -Excel → sync to Pipedrive as deals → optionally sync to a calendar. Every -run is logged to `SyncLog`, which the ops CLI reads for pipeline health -(last run per source, error counts). Full breakdown, module responsibilities, -and the reasoning behind the key design decisions: -[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). +```mermaid +flowchart LR + A["Scheduled trigger"] --> B["Scraper.fetch()"] + B --> C{"Relevance filter"} + C -->|dropped| D["counted, not stored"] + C -->|passes| E["normalize"] + E --> F[("DB upsert
dedup + field authority")] + F --> G["Excel export"] + F --> H["Pipedrive sync"] + F --> I["Calendar sync"] + G --> J["Cloud sync"] + F -.read-only.-> K["Reconcile:
DB vs Excel vs Pipedrive"] +``` + +Every run is logged to `SyncLog`, which the ops CLI reads for pipeline +health (last run per source, error counts). Full breakdown, module +responsibilities, a sequence diagram of one daily run, and the reasoning +behind the key design decisions (dry-run-by-default, field-level authority, +local-snapshot diffing): [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). + +## In action + +Real output from the actual CLI (sandbox data, default relevance filter) — +not mockups. Regenerate these with `python scripts/generate_readme_assets.py` +after any CLI change. + +**`tendertracker run --apply`** — fetches, filters (4 of 5 sample records +correctly dropped by the default EV/charging relevance filter), stores, and +re-syncs Excel in one command: + +![tendertracker run --apply output](docs/assets/cli-run.svg) + +**`tendertracker health`** — per-source rollup: last run, last success, +error count: + +![tendertracker health output](docs/assets/cli-health.svg) + +**`tendertracker reconcile`** — catches a status that was edited directly in +Excel and never made it back to the DB, without writing anything itself: + +![tendertracker reconcile output](docs/assets/cli-reconcile.svg) ## Setup diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 0a3d530..66943ae 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -2,33 +2,63 @@ ## Data flow -``` -scheduled trigger (cron / GitHub Actions / manual) - │ - ▼ -per-source Scraper.fetch() ──► RawTender (source-specific raw dict) - │ - ▼ -relevance filter (must_match / boost / exclude, per-portal, optional) - │ non-matching records dropped, counted, never stored - ▼ -normalize() ──► common Tender field set - │ - ▼ -upsert into DB (dedup by source + external_id, field-level authority) - │ - ├──► Excel export (Active/Archived sheets, regenerated from DB) - │ └──► optional cloud sync (OneDrive/SharePoint) - ├──► optional Pipedrive sync (org find-or-create → deal → note) - └──► optional calendar sync (Outlook event per open tender) - -separately, on demand: -reconcile ──► read-only diff: DB vs Excel (status), DB vs Pipedrive (title/value) +```mermaid +flowchart TD + A["Scheduled trigger
cron / GitHub Actions / manual"] --> B["Scraper.fetch()
per source"] + B --> C["RawTender
source-specific raw dict"] + C --> D{"Relevance filter
must_match / boost / exclude
per-portal, optional"} + D -->|dropped| E["counted as filtered,
never stored"] + D -->|passes| F["normalize()"] + F --> G["common Tender field set"] + G --> H[("Upsert into DB
dedup by source+external_id
field-level authority")] + H --> I["Excel export
Active/Archived sheets"] + I --> J["optional cloud sync
OneDrive/SharePoint"] + H --> K["optional Pipedrive sync
org find-or-create then deal then note"] + H --> L["optional calendar sync
Outlook event per open tender"] + H -.on demand, read-only.-> M["reconcile:
DB vs Excel status,
DB vs Pipedrive title/value"] + I -.-> M + K -.-> M ``` Every pipeline run writes a `SyncLog` row (per source, per run) — what `tendertracker status`/`health`/`errors` read. +## Sequence: one daily run + +```mermaid +sequenceDiagram + participant CLI as tendertracker run --apply + participant Scraper + participant Filter as Relevance filter + participant DB + participant Excel + participant Pipedrive + participant Calendar + + CLI->>Scraper: fetch() + Scraper-->>CLI: RawTender records + loop each record + CLI->>Filter: evaluate(title + description) + alt passes + CLI->>CLI: normalize(raw) + CLI->>DB: upsert (dedup, field authority) + else fails + CLI->>CLI: drop, count as filtered + end + end + CLI->>DB: write SyncLog row + CLI->>Excel: export_to_excel() + Excel-->>CLI: Active/Archived sheets written + opt Pipedrive configured + CLI->>Pipedrive: sync (create/update deals) + Pipedrive-->>CLI: deal IDs persisted back to DB + end + opt Calendar configured + CLI->>Calendar: sync (create/update/delete events) + Calendar-->>CLI: event IDs persisted back to DB + end +``` + ## Components | Module | Responsibility | @@ -57,6 +87,48 @@ they're thin API wrappers. Orchestration (what to sync, when, and how to decide something changed) lives in `pipeline/*.py`. This split is what makes each piece independently testable with mocks. +```mermaid +flowchart TB + CLI["cli.py"] + + subgraph Pipeline["pipeline/ (orchestration)"] + relevance["relevance.py"] + normalize["normalize.py"] + run_daily["run_daily.py"] + sync_pipedrive["sync_pipedrive.py"] + sync_calendar["sync_calendar.py"] + reconcile["reconcile.py"] + health["health.py"] + end + + subgraph Scrapers["scrapers/"] + base["base.py"] + sources["sandbox_feed.py, canadabuys_feed.py, etc."] + end + + subgraph Storage["storage/"] + models["models.py"] + db["db.py"] + excel_export["excel_export.py"] + end + + subgraph Integrations["integrations/ (thin API clients)"] + pipedrive_client["pipedrive.py"] + onedrive_client["onedrive_sync.py"] + calendar_client["calendar_sync.py"] + graph_auth["graph_auth.py"] + end + + CLI --> Pipeline + run_daily --> Scrapers + Pipeline --> Storage + sync_pipedrive --> pipedrive_client + sync_calendar --> calendar_client + reconcile --> pipedrive_client + onedrive_client --> graph_auth + calendar_client --> graph_auth +``` + ## Design decisions **Plan-only by default, `--apply` to write for real.** Every mutating diff --git a/docs/assets/cli-health.svg b/docs/assets/cli-health.svg new file mode 100644 index 0000000..37b396e --- /dev/null +++ b/docs/assets/cli-health.svg @@ -0,0 +1,84 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + tendertracker health + + + + + + + + + +                                            Source Health                                             +┌─────────┬──────────────────────────┬─────────────┬─────────────────────────┬────────┬────────────┐ +│Source │Last Run                │Last Status│Last Success           │Errors│Total Runs│ +├─────────┼──────────────────────────┼─────────────┼─────────────────────────┼────────┼────────────┤ +│sandbox│2026-09-01              │success    │2026-09-01             │0     │1         │ +││07:05:03.495655         ││07:05:03.495655        │││ +└─────────┴──────────────────────────┴─────────────┴─────────────────────────┴────────┴────────────┘ + + + + diff --git a/docs/assets/cli-reconcile.svg b/docs/assets/cli-reconcile.svg new file mode 100644 index 0000000..0003873 --- /dev/null +++ b/docs/assets/cli-reconcile.svg @@ -0,0 +1,85 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + tendertracker reconcile + + + + + + + + + + Checked 1 tenders. +                            Discrepancies                             +┌──────────────────┬────────┬──────────┬─────────────┬──────────────┐ +│External ID     │Field │DB value│Other value│Other system│ +├──────────────────┼────────┼──────────┼─────────────┼──────────────┤ +│SAMPLE-2026-0004│status│bidding │open       │excel       │ +└──────────────────┴────────┴──────────┴─────────────┴──────────────┘ + + + + diff --git a/docs/assets/cli-run.svg b/docs/assets/cli-run.svg new file mode 100644 index 0000000..33b2af8 --- /dev/null +++ b/docs/assets/cli-run.svg @@ -0,0 +1,95 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + tendertracker run --apply + + + + + + + + + + APPLIED +                       Run Results                        +┌─────────┬─────────┬─────┬─────────┬──────────┬────────┐ +│Source │Fetched│New│Updated│Filtered│Errors│ +├─────────┼─────────┼─────┼─────────┼──────────┼────────┤ +│sandbox│5      │1  │0      │4       │0     │ +└─────────┴─────────┴─────┴─────────┴──────────┴────────┘ +Excel synced to C:\Users\yaarj\AppData\Local\Temp\tendertracker-assets-sm6oe_tb\demo.xlsx(1 active, +0 archived) + + + + diff --git a/scripts/generate_readme_assets.py b/scripts/generate_readme_assets.py new file mode 100644 index 0000000..43be8d5 --- /dev/null +++ b/scripts/generate_readme_assets.py @@ -0,0 +1,89 @@ +"""Regenerate the CLI-output SVGs embedded in the README. + +Runs the real CLI code paths (not a mockup) against a throwaway temp DB +populated by the sandbox scraper, capturing Rich's actual rendered output +(colors, tables, box-drawing) to SVG. Re-run this after any change to the +CLI's table layouts so the README images stay accurate. + +Usage: python scripts/generate_readme_assets.py +""" + +import os +import shutil +import tempfile +from pathlib import Path + +from rich.console import Console + +REPO_ROOT = Path(__file__).parent.parent +ASSETS_DIR = REPO_ROOT / "docs" / "assets" + + +def main() -> None: + tmp_dir = Path(tempfile.mkdtemp(prefix="tendertracker-assets-")) + original_cwd = Path.cwd() + + try: + (tmp_dir / "config").mkdir() + shutil.copy(REPO_ROOT / "config" / "portals.example.yaml", tmp_dir / "config" / "portals.yaml") + + os.chdir(tmp_dir) + + # Mutate the shared global `settings` singleton in place — every + # module did `from ..config import settings` at import time, binding + # to this one object, so reassigning cli_module.settings alone + # wouldn't be seen by run_daily.py's own separately-bound reference. + import tendertracker.cli as cli_module + from tendertracker.config import settings as demo_settings + + demo_settings.db_path = str(tmp_dir / "demo.db") + demo_settings.excel_export_path = str(tmp_dir / "demo.xlsx") + demo_settings.pipedrive_api_token = None + demo_settings.pipedrive_domain = None + demo_settings.ms_graph_tenant_id = None + demo_settings.ms_graph_client_id = None + demo_settings.ms_graph_client_secret = None + demo_settings.ms_graph_drive_id = None + demo_settings.ms_graph_calendar_user_id = None + + from tendertracker.storage.db import get_engine, get_session_factory, init_db + from tendertracker.storage.models import Tender + + engine = get_engine(demo_settings.db_path) + init_db(engine) + session_factory = get_session_factory(engine) + + ASSETS_DIR.mkdir(parents=True, exist_ok=True) + + # --- tendertracker run --apply ----------------------------------- + console = Console(record=True, width=100) + cli_module.console = console + cli_module.run(apply=True) + console.save_svg(str(ASSETS_DIR / "cli-run.svg"), title="tendertracker run --apply") + + # --- tendertracker health ---------------------------------------- + console = Console(record=True, width=100) + cli_module.console = console + cli_module.health() + console.save_svg(str(ASSETS_DIR / "cli-health.svg"), title="tendertracker health") + + # --- tendertracker reconcile (with a real discrepancy) ----------- + with session_factory() as session: + tender = session.query(Tender).first() + tender.status = "bidding" # DB says bidding; Excel (below) still says the old status + session.commit() + + console = Console(record=True, width=100) + cli_module.console = console + cli_module.reconcile() + console.save_svg(str(ASSETS_DIR / "cli-reconcile.svg"), title="tendertracker reconcile") + + print(f"Wrote SVGs to {ASSETS_DIR}") + + finally: + os.chdir(original_cwd) + shutil.rmtree(tmp_dir, ignore_errors=True) + + +if __name__ == "__main__": + main()