From 92f514df0968e71938641ed99aaec0a24c67ec08 Mon Sep 17 00:00:00 2001 From: 0xaboomar Date: Wed, 9 Sep 2026 01:49:30 +0300 Subject: [PATCH 1/4] docs(config): document server-scoped entries and observation modes --- README.md | 62 +++++++++++++++++++++++++++++++++++++++----- config.json.template | 10 +++---- 2 files changed, 59 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index a824a694..f4de95bd 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,6 @@ Watchdog requires a configuration file (`config.json`) before running. Even plac [ { "server_name": "Test Server 1", - "agency_name": "Test Agency 1", - "agency_id": "agency-1", "oba_base_url": "https://test1.example.com", "oba_api_key": "test-key-1", "gtfs_static_feeds": ["https://gtfs1.example.com"], @@ -37,12 +35,14 @@ Watchdog requires a configuration file (`config.json`) before running. Even plac "vehicle_position_url": "https://vehicle1.example.com", "gtfs_rt_api_key": "api-key-1", "gtfs_rt_api_value": "api-value-1", - "agency_ids": ["agency-1"] + "agency_ids": [] }] } ] ``` +The top-level `agency_id` decides what each entry observes: set it to track only that agency (data for any other agency is ignored); leave it out to track the server and every agency it serves. `agency_ids` on a feed is optional and unused by Watchdog today. See [Two observation modes](#two-observation-modes-agency-vs-server) below for both config shapes. + #### Ways to Provide the Config File #### 1. Local Configuration (recommended for development) @@ -93,13 +93,61 @@ One caveat: v1 and v2 fields can't be mixed in the same entry. If an entry conta Migrating to v2 is recommended whenever convenient — it's the only way to configure multiple static or RT feeds per agency. v1 supports just one of each. -### Server vs. agency scoping +### Two observation modes: agency vs. server + +There are two ways Watchdog can observe an OBA deployment. The choice is made **per entry in `config.json`** by whether you set `agency_id` — every other field (`server_name`, `oba_base_url`, `oba_api_key`, `gtfs_static_feeds`, `gtfs_rt_feeds`) is identical in both modes, so `agency_id` is the only switch: + +- **Agency mode** — `agency_id` is set. Watchdog monitors only that one agency. +- **Server mode (recommended)** — `agency_id` is unset. Watchdog monitors the whole server and every agency it currently serves. + +**Observe a single agency** — set `agency_id` (with its `agency_name`) on the entry. Watchdog tracks only that agency; any data not related to it is ignored: + +```json +{ + "server_name": "Agency Example", + "agency_name": "Test Agency", + "agency_id": "agency-1", + "oba_base_url": "https://test1.example.com", + "oba_api_key": "test-key-1", + "gtfs_static_feeds": ["https://gtfs1.example.com"], + "gtfs_rt_feeds": [{ + "trip_update_url": "https://trip1.example.com", + "vehicle_position_url": "https://vehicle1.example.com", + "gtfs_rt_api_key": "api-key-1", + "gtfs_rt_api_value": "api-value-1", + "agency_ids": ["agency-1"] + }] +} +``` + +**Observe the whole server** — omit `agency_id` from the entry. Watchdog tracks the server and every agency it currently serves: + +```json +{ + "server_name": "Server Example", + "oba_base_url": "https://test2.example.com", + "oba_api_key": "test-key-2", + "gtfs_static_feeds": ["https://gtfs2.example.com"], + "gtfs_rt_feeds": [{ + "trip_update_url": "https://trip2.example.com", + "vehicle_position_url": "https://vehicle2.example.com", + "gtfs_rt_api_key": "api-key-2", + "gtfs_rt_api_value": "api-value-2", + "agency_ids": [] + }] +} +``` -Every entry in `config.json` is server-scoped at the top level: `server_name` is required and identifies the OBA deployment; the operator lists the static feeds each server exposes. `agency_id` is optional and controls the *scope* of the entry: +| | Agency mode | Server mode | +|---|---|---| +| `agency_id` | set (paired with required `agency_name`) | unset | +| Scope | one agency | whole OBA deployment | +| Agency set comes from | the config entry | `/api/where/metrics.json` cross-referenced with each static feed's `agency.txt` | +| Typical source | hand-written config | OBACloud service discovery (one entry per OBA API server) | -- **Agency-mode entry** — `agency_id` is set. Watchdog monitors only that one agency. Today's per-agency pipeline runs once per tick for that agency. `agency_name` is required (paired with `agency_id`). +**Agency mode** — `agency_id` is set. Watchdog monitors only that one agency; today's per-agency pipeline runs once per tick for that agency. `agency_name` is required (paired with `agency_id`). -- **Server-mode entry** — `agency_id` is absent. Watchdog probes `/api/where/metrics.json` every tick to learn which agencies OBA is currently serving, cross-references the live agency IDs against the static feeds' `agency.txt` declarations, and runs the per-agency pipeline for every agency that has BOTH a static bundle AND is reported as currently live. +**Server mode** — `agency_id` is absent. Watchdog probes `/api/where/metrics.json` every tick to learn which agencies OBA is currently serving, cross-references the live agency IDs against the static feeds' `agency.txt` declarations, and runs the per-agency pipeline for every agency that has BOTH a static bundle AND is reported as currently live. Multi-agency static feeds are accepted: a single bundle pointer-shared across the serverKeys of every agency declared in its `agency.txt`. Static feeds whose `agency.txt` is empty, declares multiple agencies ambiguously, or whose declared agency isn't currently reported by OBA are reported to Sentry (`gtfs_static_feed_attribution_status` flips to 0) and the per-agency pipeline is skipped — but the bundle is still downloaded and parsed so the introspection metrics (`gtfs_static_stops_count`, `gtfs_static_routes_count`) keep reporting. diff --git a/config.json.template b/config.json.template index 77ca737d..701bc0e0 100644 --- a/config.json.template +++ b/config.json.template @@ -1,6 +1,6 @@ [ { - "server_name": "Test Server 1", + "server_name": "Agency Example", "agency_name": "Test Agency 1", "agency_id": "agency-1", "oba_base_url": "https://test1.example.com", @@ -15,9 +15,7 @@ }] }, { - "server_name": "Test Server 2", - "agency_name": "Test Agency 2", - "agency_id": "agency-2", + "server_name": "Server Example", "oba_base_url": "https://test2.example.com", "oba_api_key": "test-key-2", "gtfs_static_feeds": ["https://gtfs2.example.com"], @@ -26,7 +24,7 @@ "vehicle_position_url": "https://vehicle2.example.com", "gtfs_rt_api_key": "api-key-2", "gtfs_rt_api_value": "api-value-2", - "agency_ids": ["agency-2"] + "agency_ids": [] }] } -] +] \ No newline at end of file From 7c38b35c70c4930089869cf75c47a1abca518dbc Mon Sep 17 00:00:00 2001 From: 0xaboomar Date: Wed, 9 Sep 2026 01:49:30 +0300 Subject: [PATCH 2/4] docs(readme): reorder config guidance for clarity --- README.md | 32 ++++++++++++++++---------------- 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index f4de95bd..387ac960 100644 --- a/README.md +++ b/README.md @@ -77,22 +77,6 @@ export CONFIG_AUTH_USER="username" export CONFIG_AUTH_PASS="password" ``` -### Backward Compatibility (v1 → v2) - -Watchdog used to accept a flat, single-server config schema. Legacy (v1) configs are still supported: they're **silently converted** to the current array-based schema (v2) at load time, so upgrading doesn't require changing your config or interrupt monitoring. - -When a v1 entry is loaded, Watchdog maps it like this: - -- `name` → `agency_name` -- `gtfs_url` → a single-entry `gtfs_static_feeds` -- `vehicle_position_url` / `trip_update_url` → the matching `gtfs_rt_feeds` entries -- `gtfs_rt_api_key` / `gtfs_rt_api_value` → the per-feed auth fields -- `id` → ignored - -One caveat: v1 and v2 fields can't be mixed in the same entry. If an entry contains both schemas (e.g. a legacy `gtfs_url` alongside `gtfs_static_feeds`), it's rejected and reported to Sentry. Each entry must use one schema or the other. - -Migrating to v2 is recommended whenever convenient — it's the only way to configure multiple static or RT feeds per agency. v1 supports just one of each. - ### Two observation modes: agency vs. server There are two ways Watchdog can observe an OBA deployment. The choice is made **per entry in `config.json`** by whether you set `agency_id` — every other field (`server_name`, `oba_base_url`, `oba_api_key`, `gtfs_static_feeds`, `gtfs_rt_feeds`) is identical in both modes, so `agency_id` is the only switch: @@ -166,6 +150,22 @@ In agency-mode every RT vehicle is labeled with the configured `agency_id`. In s This is a **deliberate breaking change** to the existing v2 format. Existing v2 entries without `server_name` become invalid — operators must add the field. The legacy v1 array-of-flat-objects format continues to work: `id` (int) is still ignored, `name` is repurposed as `server_name`, and a v1 entry with `agency_id` populates `agency_name` from `name` so the entry remains agency-scoped. +### Backward Compatibility (v1 → v2) + +Watchdog used to accept a flat, single-server config schema. Legacy (v1) configs are still supported: they're **silently converted** to the current array-based schema (v2) at load time, so upgrading doesn't require changing your config or interrupt monitoring. + +When a v1 entry is loaded, Watchdog maps it like this: + +- `name` → `agency_name` +- `gtfs_url` → a single-entry `gtfs_static_feeds` +- `vehicle_position_url` / `trip_update_url` → the matching `gtfs_rt_feeds` entries +- `gtfs_rt_api_key` / `gtfs_rt_api_value` → the per-feed auth fields +- `id` → ignored + +One caveat: v1 and v2 fields can't be mixed in the same entry. If an entry contains both schemas (e.g. a legacy `gtfs_url` alongside `gtfs_static_feeds`), it's rejected and reported to Sentry. Each entry must use one schema or the other. + +Migrating to v2 is recommended whenever convenient — it's the only way to configure multiple static or RT feeds per agency. v1 supports just one of each. + ### Application Options - **Fetch Interval** → default `30s` (`--fetch-interval `) From dbbc7d059fbb394f5135226da7197b383e7957f8 Mon Sep 17 00:00:00 2001 From: 0xaboomar Date: Wed, 9 Sep 2026 01:49:30 +0300 Subject: [PATCH 3/4] docs(config): fix scoping references --- README.md | 2 +- cmd/watchdog/main.go | 4 ++-- config.json.template | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 387ac960..1a46c666 100644 --- a/README.md +++ b/README.md @@ -156,7 +156,7 @@ Watchdog used to accept a flat, single-server config schema. Legacy (v1) configs When a v1 entry is loaded, Watchdog maps it like this: -- `name` → `agency_name` +- `name` → `server_name`; when `agency_id` is present, it also populates `agency_name` - `gtfs_url` → a single-entry `gtfs_static_feeds` - `vehicle_position_url` / `trip_update_url` → the matching `gtfs_rt_feeds` entries - `gtfs_rt_api_key` / `gtfs_rt_api_value` → the per-feed auth fields diff --git a/cmd/watchdog/main.go b/cmd/watchdog/main.go index d0059c94..bf0da2d4 100644 --- a/cmd/watchdog/main.go +++ b/cmd/watchdog/main.go @@ -27,8 +27,8 @@ func main() { flag.StringVar(&cfg.Env, "env", "development", "Environment (development|staging|production)") flag.IntVar(&cfg.FetchInterval, "fetch-interval", 30, "Interval (in seconds) at which the application fetches data from realtime APIs and updates Prometheus metrics") - // Server-scope design (deliberate decision, see README "Server vs. agency - // scoping" and config.json.template): + // Server-scope design (deliberate decision, see README "Two observation + // modes: agency vs. server" and config.json.template): // // Every entry in config.json is server-scoped at the top level: server_name // is required, server_url is derived from oba_base_url, and the operator diff --git a/config.json.template b/config.json.template index 701bc0e0..1e172ab2 100644 --- a/config.json.template +++ b/config.json.template @@ -27,4 +27,4 @@ "agency_ids": [] }] } -] \ No newline at end of file +] From f29a312065aa956fb6f59a11835a33ae2ec34c5f Mon Sep 17 00:00:00 2001 From: 0xaboomar Date: Wed, 9 Sep 2026 01:49:30 +0300 Subject: [PATCH 4/4] docs: correct agency-mode isolation claims in README Replace misleading agency isolation wording with accurate descriptions of current behavior: OBA API probes are agency-specific, but static and realtime feed data is not filtered by agency. Also fix 'agency_id is the only switch' to note that agency_name is required, and merge duplicate backward-compatibility sections into one. --- README.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 1a46c666..32ac8ffc 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ Watchdog requires a configuration file (`config.json`) before running. Even plac ] ``` -The top-level `agency_id` decides what each entry observes: set it to track only that agency (data for any other agency is ignored); leave it out to track the server and every agency it serves. `agency_ids` on a feed is optional and unused by Watchdog today. See [Two observation modes](#two-observation-modes-agency-vs-server) below for both config shapes. +The top-level `agency_id` decides what each entry observes: set it to track that agency. OBA API probes query only that agency, though static and realtime feed data is not filtered by agency; leave it out to track the server and every agency it serves. `agency_ids` on a feed is optional and unused by Watchdog today. See [Two observation modes](#two-observation-modes-agency-vs-server) below for both config shapes. #### Ways to Provide the Config File @@ -79,12 +79,12 @@ export CONFIG_AUTH_PASS="password" ### Two observation modes: agency vs. server -There are two ways Watchdog can observe an OBA deployment. The choice is made **per entry in `config.json`** by whether you set `agency_id` — every other field (`server_name`, `oba_base_url`, `oba_api_key`, `gtfs_static_feeds`, `gtfs_rt_feeds`) is identical in both modes, so `agency_id` is the only switch: +There are two ways Watchdog can observe an OBA deployment. The choice is made **per entry in `config.json`** by whether you set `agency_id` — every other field (`server_name`, `oba_base_url`, `oba_api_key`, `gtfs_static_feeds`, `gtfs_rt_feeds`) is identical in both modes. Setting `agency_id` selects agency observation mode; `agency_name` must also be provided. - **Agency mode** — `agency_id` is set. Watchdog monitors only that one agency. - **Server mode (recommended)** — `agency_id` is unset. Watchdog monitors the whole server and every agency it currently serves. -**Observe a single agency** — set `agency_id` (with its `agency_name`) on the entry. Watchdog tracks only that agency; any data not related to it is ignored: +**Observe a single agency** — set `agency_id` (with its `agency_name`) on the entry. Watchdog uses that agency for agency-specific OBA API probes, but static and realtime feed data is not filtered by agency: ```json { @@ -146,12 +146,10 @@ The two metrics below are the operator's view into server-mode health: In agency-mode every RT vehicle is labeled with the configured `agency_id`. In server-mode one RT feed may carry vehicles from multiple agencies, so Watchdog attributes each vehicle by looking up its `TripDescriptor`'s `route_id` in a per-server `route_id → agency_id` index built from `routes.txt` at static-download time. Vehicles whose `route_id` is empty or unknown (and vehicles carrying no vehicle ID at all) are left out of the per-vehicle series and counted in `gtfs_rt_unattributed_vehicles_count{server_name, server_url}` so operators can detect static feeds that don't cover every RT route. The data-quality gauges `gtfs_rt_invalid_vehicle_coordinates` and `gtfs_rt_stopped_out_of_bounds_vehicles` instead file those vehicles under the server-scoped series (empty `agency_id`), so their per-agency series always sum to the server-wide count. See `docs/METRICS.md` for details. -#### Backward compatibility +### Backward Compatibility (v1 → v2) This is a **deliberate breaking change** to the existing v2 format. Existing v2 entries without `server_name` become invalid — operators must add the field. The legacy v1 array-of-flat-objects format continues to work: `id` (int) is still ignored, `name` is repurposed as `server_name`, and a v1 entry with `agency_id` populates `agency_name` from `name` so the entry remains agency-scoped. -### Backward Compatibility (v1 → v2) - Watchdog used to accept a flat, single-server config schema. Legacy (v1) configs are still supported: they're **silently converted** to the current array-based schema (v2) at load time, so upgrading doesn't require changing your config or interrupt monitoring. When a v1 entry is loaded, Watchdog maps it like this: