Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: Tests

on:
push:
branches: [main]
pull_request:
branches: [main]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.12"

- name: Install project (with dev dependencies)
run: pip install -e ".[dev]"

- name: Run tests
run: pytest -v
52 changes: 45 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Tender Tracking System

[![Tests](https://github.com/codeapplied/tender-tracking-system/actions/workflows/tests.yml/badge.svg)](https://github.com/codeapplied/tender-tracking-system/actions/workflows/tests.yml)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](pyproject.toml)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)

A tender/bid tracking tool — rebuilt as a generalized, open-source system.

Originally built as an internal work tool; this repo is a from-scratch rebuild using
Expand All @@ -19,13 +23,47 @@ for the full build history and

## Architecture

Daily scheduled run → per-source scraper/API client → relevance filter →
normalize into a common `Tender` record → dedupe against the DB → export to
Excel → sync to Pipedrive as deals → optionally sync to a calendar. Every
run is logged to `SyncLog`, which the ops CLI reads for pipeline health
(last run per source, error counts). Full breakdown, module responsibilities,
and the reasoning behind the key design decisions:
[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).
```mermaid
flowchart LR
A["Scheduled trigger"] --> B["Scraper.fetch()"]
B --> C{"Relevance filter"}
C -->|dropped| D["counted, not stored"]
C -->|passes| E["normalize"]
E --> F[("DB upsert<br/>dedup + field authority")]
F --> G["Excel export"]
F --> H["Pipedrive sync"]
F --> I["Calendar sync"]
G --> J["Cloud sync"]
F -.read-only.-> K["Reconcile:<br/>DB vs Excel vs Pipedrive"]
```

Every run is logged to `SyncLog`, which the ops CLI reads for pipeline
health (last run per source, error counts). Full breakdown, module
responsibilities, a sequence diagram of one daily run, and the reasoning
behind the key design decisions (dry-run-by-default, field-level authority,
local-snapshot diffing): [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md).

## In action

Real output from the actual CLI (sandbox data, default relevance filter) —
not mockups. Regenerate these with `python scripts/generate_readme_assets.py`
after any CLI change.

**`tendertracker run --apply`** — fetches, filters (4 of 5 sample records
correctly dropped by the default EV/charging relevance filter), stores, and
re-syncs Excel in one command:

![tendertracker run --apply output](docs/assets/cli-run.svg)

**`tendertracker health`** — per-source rollup: last run, last success,
error count:

![tendertracker health output](docs/assets/cli-health.svg)

**`tendertracker reconcile`** — catches a status that was edited directly in
Excel and never made it back to the DB, without writing anything itself:

![tendertracker reconcile output](docs/assets/cli-reconcile.svg)

## Setup

Expand Down
116 changes: 94 additions & 22 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,63 @@

## Data flow

```
scheduled trigger (cron / GitHub Actions / manual)
│
▼
per-source Scraper.fetch() ──► RawTender (source-specific raw dict)
│
▼
relevance filter (must_match / boost / exclude, per-portal, optional)
│ non-matching records dropped, counted, never stored
▼
normalize() ──► common Tender field set
│
▼
upsert into DB (dedup by source + external_id, field-level authority)
│
├──► Excel export (Active/Archived sheets, regenerated from DB)
│ └──► optional cloud sync (OneDrive/SharePoint)
├──► optional Pipedrive sync (org find-or-create → deal → note)
└──► optional calendar sync (Outlook event per open tender)

separately, on demand:
reconcile ──► read-only diff: DB vs Excel (status), DB vs Pipedrive (title/value)
```mermaid
flowchart TD
A["Scheduled trigger<br/>cron / GitHub Actions / manual"] --> B["Scraper.fetch()<br/>per source"]
B --> C["RawTender<br/>source-specific raw dict"]
C --> D{"Relevance filter<br/>must_match / boost / exclude<br/>per-portal, optional"}
D -->|dropped| E["counted as filtered,<br/>never stored"]
D -->|passes| F["normalize()"]
F --> G["common Tender field set"]
G --> H[("Upsert into DB<br/>dedup by source+external_id<br/>field-level authority")]
H --> I["Excel export<br/>Active/Archived sheets"]
I --> J["optional cloud sync<br/>OneDrive/SharePoint"]
H --> K["optional Pipedrive sync<br/>org find-or-create then deal then note"]
H --> L["optional calendar sync<br/>Outlook event per open tender"]
H -.on demand, read-only.-> M["reconcile:<br/>DB vs Excel status,<br/>DB vs Pipedrive title/value"]
I -.-> M
K -.-> M
```

Every pipeline run writes a `SyncLog` row (per source, per run) — what
`tendertracker status`/`health`/`errors` read.

## Sequence: one daily run

```mermaid
sequenceDiagram
participant CLI as tendertracker run --apply
participant Scraper
participant Filter as Relevance filter
participant DB
participant Excel
participant Pipedrive
participant Calendar

CLI->>Scraper: fetch()
Scraper-->>CLI: RawTender records
loop each record
CLI->>Filter: evaluate(title + description)
alt passes
CLI->>CLI: normalize(raw)
CLI->>DB: upsert (dedup, field authority)
else fails
CLI->>CLI: drop, count as filtered
end
end
CLI->>DB: write SyncLog row
CLI->>Excel: export_to_excel()
Excel-->>CLI: Active/Archived sheets written
opt Pipedrive configured
CLI->>Pipedrive: sync (create/update deals)
Pipedrive-->>CLI: deal IDs persisted back to DB
end
opt Calendar configured
CLI->>Calendar: sync (create/update/delete events)
Calendar-->>CLI: event IDs persisted back to DB
end
```

## Components

| Module | Responsibility |
Expand Down Expand Up @@ -57,6 +87,48 @@ they're thin API wrappers. Orchestration (what to sync, when, and how to
decide something changed) lives in `pipeline/*.py`. This split is what
makes each piece independently testable with mocks.

```mermaid
flowchart TB
CLI["cli.py"]

subgraph Pipeline["pipeline/ (orchestration)"]
relevance["relevance.py"]
normalize["normalize.py"]
run_daily["run_daily.py"]
sync_pipedrive["sync_pipedrive.py"]
sync_calendar["sync_calendar.py"]
reconcile["reconcile.py"]
health["health.py"]
end

subgraph Scrapers["scrapers/"]
base["base.py"]
sources["sandbox_feed.py, canadabuys_feed.py, etc."]
end

subgraph Storage["storage/"]
models["models.py"]
db["db.py"]
excel_export["excel_export.py"]
end

subgraph Integrations["integrations/ (thin API clients)"]
pipedrive_client["pipedrive.py"]
onedrive_client["onedrive_sync.py"]
calendar_client["calendar_sync.py"]
graph_auth["graph_auth.py"]
end

CLI --> Pipeline
run_daily --> Scrapers
Pipeline --> Storage
sync_pipedrive --> pipedrive_client
sync_calendar --> calendar_client
reconcile --> pipedrive_client
onedrive_client --> graph_auth
calendar_client --> graph_auth
```

## Design decisions

**Plan-only by default, `--apply` to write for real.** Every mutating
Expand Down
84 changes: 84 additions & 0 deletions docs/assets/cli-health.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading