Repository navigation
feat(telemetry): fold telemetry older than a year into quarter-hour rows - #435
Merged
Merged
Conversation
Telemetry was the one table that grew with every poll and was never trimmed. The Worker's new HousekeepingService now keeps it in check every six hours. - Rows older than TELEMETRY_FULL_DETAIL_DAYS (365 by default, 0 keeps everything, anything below 90 is raised to 90) are folded into one row per quarter of an hour by TelemetryCompaction. Each window keeps its last row, filled in with the newest earlier value of every field it lacks, so the latest reading at or before a moment (odometer, position, schedules) answers the same at the end of every window and never with a later reading. A window is cut where a counter that resets drops, so each day's and each charge's total survives. - TelemetryCompactor works a whole UTC day at a time from Vehicle.TelemetryCompactedUntil (migration AddTelemetryCompactedUntil), saving each day and the line in one transaction. It never passes TripsRecordedUntil, and skips a vehicle that has never recorded a trip. - The same job removes place names and snapped trips a day after they expire, and notifications the user deleted once they are 30 days old. Map POI tiles are left alone: the map shows an expired tile's stations while it fetches the tile again. - The merge accessors move out of TelemetryRepository into TelemetryFields, so the Worker's poll merge, the dashboard's current state and compaction share one implementation. Checked on a throwaway PostgreSQL 18 with 500 synthetic days: 19,305 rows folded in 3.8 s, with day totals, end-of-day odometer, each day's last position and the recent year unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
7 of 12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull Request
Summary
Telemetry was the one table that grew with every poll and was never trimmed. A new Worker job,
HousekeepingService, runs every six hours:TELEMETRY_FULL_DETAIL_DAYS(default 365) is folded into one row per quarter of an hour. Each window keeps its last row, filled in with the newest earlier value of every field it lacks, so the latest odometer, position and schedules answer the same at the end of every window and never with a later reading. A window is split where a resetting counter drops (today's distance and fuel/energy, the since-charge counters, journey distance), so each day's and each charge's total survives.Vehicle.TelemetryCompactedUntilline (migrationAddTelemetryCompactedUntil, one nullable column), saving each day and the line in one transaction. It never passes the trip recording line, and skips vehicles that have never recorded a trip.0turns it off; values below 90 are raised to 90, the furthest back the stats page and map read.TelemetryRepositoryintoTelemetryFields, shared by the poll merge, the dashboard's current state and compaction.What is lost: the order of events within a quarter of an hour, and an old fix carries its window's end time. Old trips can no longer be re-cut from telemetry (saved trips are unaffected). Documented in ARCHITECTURE.md ("Data kept over time"), CONFIGURATION.md,
.env.exampleand the README, includingVACUUM FULLto shrink the files after a first large compaction.Type of Change
Related Issues
None.
Checklist
dotnet build,dotnet format --verify-no-changes, no pending model changes)unraid/garagestack.xmlchanged: copied tojoszz/unraid-community-appsafter merge (addsTELEMETRY_FULL_DETAIL_DAYS; sync follows the merge)Verification
🤖 Generated with Claude Code