Skip to content

feat(telemetry): fold telemetry older than a year into quarter-hour rows - #435

Merged
joszz merged 1 commit into
mainfrom
feat/telemetry-retention
Oct 5, 2026
Merged

joszz merged 1 commit into
mainfrom
feat/telemetry-retention

Conversation

@joszz

@joszz joszz commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Pull Request

Summary

Telemetry was the one table that grew with every poll and was never trimmed. A new Worker job, HousekeepingService, runs every six hours:

  • Compaction. Telemetry older than TELEMETRY_FULL_DETAIL_DAYS (default 365) is folded into one row per quarter of an hour. Each window keeps its last row, filled in with the newest earlier value of every field it lacks, so the latest odometer, position and schedules answer the same at the end of every window and never with a later reading. A window is split where a resetting counter drops (today's distance and fuel/energy, the since-charge counters, journey distance), so each day's and each charge's total survives.
  • Safe by construction. Works one UTC day at a time from a new Vehicle.TelemetryCompactedUntil line (migration AddTelemetryCompactedUntil, one nullable column), saving each day and the line in one transaction. It never passes the trip recording line, and skips vehicles that have never recorded a trip. 0 turns it off; values below 90 are raised to 90, the furthest back the stats page and map read.
  • Cleanup. The same job removes place-name and snapped-trip cache entries a day after they expire, and notifications the user deleted once they are 30 days old. Map POI tiles are deliberately kept, since the map shows an expired tile's stations while it refetches.
  • Refactor. The merge accessors move from TelemetryRepository into TelemetryFields, shared by the poll merge, the dashboard's current state and compaction.

What is lost: the order of events within a quarter of an hour, and an old fix carries its window's end time. Old trips can no longer be re-cut from telemetry (saved trips are unaffected). Documented in ARCHITECTURE.md ("Data kept over time"), CONFIGURATION.md, .env.example and the README, including VACUUM FULL to shrink the files after a first large compaction.

Type of Change

  • Bug fix
  • New feature
  • Refactor / cleanup
  • Documentation
  • Dependency update
  • Other

Related Issues

None.

Checklist

  • Tests added or updated (13 new: the fold, the compactor against the trip line and the full-detail window, the purges, the setting)
  • Linting passes (dotnet build, dotnet format --verify-no-changes, no pending model changes)
  • No secrets or personal data committed
  • Responsive on mobile (no UI change)
  • i18n keys added for any new UI strings (none)
  • If unraid/garagestack.xml changed: copied to joszz/unraid-community-apps after merge (adds TELEMETRY_FULL_DETAIL_DAYS; sync follows the merge)

Verification

  • 760 backend tests pass.
  • Throwaway PostgreSQL 18 with 500 synthetic days (120k rows): the first run folded 19,305 rows in 3.8 s (old days from 240 to 97 rows); day totals, end-of-day odometer, each day's last position and the most recent year were unchanged; the second run did nothing. The purges removed exactly the expired and old deleted rows.

🤖 Generated with Claude Code

Telemetry was the one table that grew with every poll and was never trimmed. The Worker's new
HousekeepingService now keeps it in check every six hours.

- Rows older than TELEMETRY_FULL_DETAIL_DAYS (365 by default, 0 keeps everything, anything
  below 90 is raised to 90) are folded into one row per quarter of an hour by
  TelemetryCompaction. Each window keeps its last row, filled in with the newest earlier value
  of every field it lacks, so the latest reading at or before a moment (odometer, position,
  schedules) answers the same at the end of every window and never with a later reading. A
  window is cut where a counter that resets drops, so each day's and each charge's total
  survives.
- TelemetryCompactor works a whole UTC day at a time from Vehicle.TelemetryCompactedUntil
  (migration AddTelemetryCompactedUntil), saving each day and the line in one transaction. It
  never passes TripsRecordedUntil, and skips a vehicle that has never recorded a trip.
- The same job removes place names and snapped trips a day after they expire, and
  notifications the user deleted once they are 30 days old. Map POI tiles are left alone: the
  map shows an expired tile's stations while it fetches the tile again.
- The merge accessors move out of TelemetryRepository into TelemetryFields, so the Worker's
  poll merge, the dashboard's current state and compaction share one implementation.

Checked on a throwaway PostgreSQL 18 with 500 synthetic days: 19,305 rows folded in 3.8 s, with
day totals, end-of-day odometer, each day's last position and the recent year unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@joszz
joszz merged commit 9978310 into main Oct 5, 2026
22 checks passed
@joszz
joszz deleted the feat/telemetry-retention branch October 5, 2026 12:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant