feat(telemetry): fold telemetry older than a year into quarter-hour rows

.env.example

@@ -236,3 +236,11 @@ OPENCHARGEMAP_API_KEY=
 # figure to correct it. Left unset, a plug-in car keeps the gateway's number and a
 # hybrid shows state of charge as a percentage only.
 # HV_BATTERY_CAPACITY_KWH=1.83
+
+# ── Telemetry detail ───────────────────────────────────────────────────────────
+# Days of telemetry kept exactly as the car reported it. Older telemetry is folded
+# into one row per quarter of an hour, so the database grows far more slowly, while
+# every reading's latest value, each day's totals and your saved trips are kept.
+# Nothing in the app shows finer detail than that past 90 days, so a lower value is
+# raised to 90. Set to 0 to keep every row.
+# TELEMETRY_FULL_DETAIL_DAYS=365

README.md

@@ -298,6 +298,10 @@ docker start garagestack
 
 To restore, stop the container, replace `garagestack-data` with the backup, and start it again.
 
+### Database size
+
+Telemetry older than a year is folded into one row per quarter of an hour, so the database keeps growing, but far more slowly. Saved trips, each day's totals and the latest value of every reading are kept; only the order of events within a quarter of an hour is lost. Compaction cannot be undone, so take a backup first if you want to keep the raw rows. Set `TELEMETRY_FULL_DETAIL_DAYS` to change the period, or to `0` to keep every row. See [Data kept over time](documentation/ARCHITECTURE.md#data-kept-over-time).
+
 ---
 
 ## Push notifications

docker-compose.yml

@@ -117,6 +117,8 @@ services:
       <<: *dotnet-environment
       # Language of push notification texts (en or nl); the Worker has no browser to ask.
       NOTIFICATION_LANGUAGE: "${NOTIFICATION_LANGUAGE:-}"
+      # Days of telemetry kept at full detail before the Worker compacts it. See .env.example.
+      TELEMETRY_FULL_DETAIL_DAYS: "${TELEMETRY_FULL_DETAIL_DAYS:-}"
     volumes:
       - worker_logs:/app/logs
 

documentation/ARCHITECTURE.md

@@ -83,13 +83,24 @@ The trip log (`/trip-log`, `TripEndpoints`) shows only saved trips, because only
 
 The `AddTripLog` migration clears the trips saved before it and resets the recording line, and the Worker saves them again with their new columns. Nothing is lost: until that migration, nothing a driver could enter was kept on a trip.
 
+## Data kept over time
+
+Telemetry is the one table that grows with every poll. The Worker's `HousekeepingService` runs every six hours and keeps it in check.
+
+- **Compaction.** Telemetry older than `TELEMETRY_FULL_DETAIL_DAYS` (a year by default) is folded into one row per quarter of an hour by `TelemetryCompaction` in Core. It is the same last-value-wins merge that folds a poll's messages into one row, over a longer window. Each window keeps its last row, filled in with the newest earlier value of every field that row lacks. "The latest reading at or before a moment" (the odometer, the last position, a schedule the car reported once) therefore answers the same at the end of every window, and never with a reading taken later. A window is cut in two where a counter that starts again from zero drops (the day's distance and energy, the counters since the last charge, the journey distance), so the total of each day and each charge survives. A quarter of an hour divides every UTC offset, so a window never spans the car's local midnight.
+- **The line.** `TelemetryCompactor` works through a whole UTC day at a time from `Vehicle.TelemetryCompactedUntil`, and saves each day together with the line in one transaction, so a run can stop anywhere and the next one carries on. It never passes `TripsRecordedUntil`: the trip recorder still needs every fix it has not cut into trips. A vehicle that has never recorded a trip is not compacted at all. Saved trips keep their own fixes (`Trip.PointsJson`), so compaction changes nothing about them. It does mean old trips can no longer be cut again from telemetry, one more reason never to clear saved trips in a migration.
+- **What it costs.** Nothing in the app reads finer detail than this past 90 days: the statistics page and the map ask for 90 days at most, and the history query thins 90 days to a few points a day anyway. That is why a lower setting is raised to 90. What is lost is the order of events within a quarter of an hour, such as a short charge that started and stopped inside one.
+- **Caches and deleted notifications.** The same job removes place names and snapped trips a day after they expire: nothing reads an expired one, and a lookup may be refreshing it at that moment. It removes notifications the user deleted once they are 30 days old; the notification cooldowns read that history, deleted rows included, at most a week back. Map POI tiles are left alone, because the map shows an expired tile's stations while it fetches the tile again.
+
+PostgreSQL reuses the space of removed rows for new ones rather than handing it back to the disk. To shrink the database files after the first compaction of a long history, run `VACUUM FULL "TelemetrySnapshots";` once. It locks the table for as long as it runs, so pick a quiet moment.
+
 ## Projects under `src/`
 
 | Project | Contains | Depends on |
 | --- | --- | --- |
 | `GarageStack.Core` | Domain models (`Models/`), repository/service interfaces (`Interfaces/`), and pure helpers with no I/O (`Helpers/`) - the shared vocabulary every other project builds on. | nothing (leaf project) |
 | `GarageStack.Data` | EF Core: `AppDbContext`, migrations, concrete repository implementations, and `Demo/` (in-memory fakes used when `DEMO_MODE=true`). | `Core` |
-| `GarageStack.Worker` | The MQTT-ingestion process: `Mqtt/MqttConsumerService` and the message handlers beside it, plus the periodic jobs (maintenance reminders, POI pre-caching, push-notification checks, saving finished trips), each a `PeriodicBackgroundService`. | `Core`, `Data` |
+| `GarageStack.Worker` | The MQTT-ingestion process: `Mqtt/MqttConsumerService` and the message handlers beside it, plus the periodic jobs (maintenance reminders, POI pre-caching, push-notification checks, saving finished trips, housekeeping), each a `PeriodicBackgroundService`. | `Core`, `Data` |
 | `GarageStack.Api` | ASP.NET Core minimal APIs (`Endpoints/`), the SignalR hub (`Hubs/`), and Api-only services (outbound MQTT publishing, POI/charging-station lookups, the Postgres-LISTEN-to-SignalR bridge). | `Core`, `Data` |
 | `GarageStack.Tests` | xUnit tests across all of the above. | all four |
 
@@ -192,9 +203,12 @@ A new value arriving from the car passes through several layers, in this order:
 3. The frontend's `TelemetrySnapshot` interface in `services/vehicleApi.ts`, mirrored by hand.
 4. Wherever it should show up: a card in `frontend/src/cards/registry.ts`, and the demo data in
    `src/GarageStack.Data/Demo/` so the field is visible without a car.
+5. Only for a counter that starts again from zero (per day, per charge, per journey): its entry in
+   `TelemetryCompaction`'s resetting counters, so compaction keeps the total it reached before
+   each reset.
 
-What it does *not* need is a mention in the repository's merge loops or its "does this row carry
-anything" filters, or in `WidgetStatusDto`: the first two are derived from the model itself, and
+What it does *not* need is a mention in the merge loops (`TelemetryFields`) or the repository's
+"does this row carry anything" filters, or in `WidgetStatusDto`: the first two are derived from the model itself, and
 the widget deliberately serves a curated subset. Charts are the same story in reverse: add the
 field to `TelemetryHistoryPoint` and the history query starts returning rows that carry it.
 

documentation/CONFIGURATION.md

@@ -98,6 +98,12 @@ The full guide, with provider examples, is [`AUTHENTICATION.md`](AUTHENTICATION.
 | `TYRE_PRESSURE_LOW_BAR` / `TYRE_PRESSURE_GOOD_BAR` / `TYRE_PRESSURE_HIGH_BAR` | `2.2` / `2.6` / `3.2` | Tyre pressure bands in bar, for the diagram's colours and the tyre notifications. Set them from your car's placard. |
 | `HV_BATTERY_CAPACITY_KWH` | none | Usable size of the traction battery in kWh. The gateway assumes an EV-sized pack, so its kWh figures are far too large on a plain hybrid (an MG HS Hybrid+ carries 1.83 and is reported as 72.5). Empty keeps the gateway's figure on a plug-in car and shows a hybrid's charge as a percentage only. |
 
+## Stored data
+
+| Variable | Default | What it does |
+| --- | --- | --- |
+| `TELEMETRY_FULL_DETAIL_DAYS` | `365` | Days of telemetry kept exactly as the car reported it. Older telemetry is folded into one row per quarter of an hour, keeping the latest value of every reading, each day's totals and the saved trips. Below `90` is raised to `90`, the furthest back the statistics page and the map read. `0` keeps every row. See [Data kept over time](ARCHITECTURE.md#data-kept-over-time). |
+
 ## API, logging and ports
 
 | Variable | Default | What it does |

src/GarageStack.Core/Configuration/EnvironmentAliases.cs

@@ -34,6 +34,9 @@ public static class EnvironmentAliases
         ["TYRE_PRESSURE_HIGH_BAR"] = "TyrePressure:HighBar",
         ["HV_BATTERY_CAPACITY_KWH"] = "HvBattery:CapacityKwh",
 
+        // How much detail old telemetry keeps.
+        ["TELEMETRY_FULL_DETAIL_DAYS"] = "TelemetryRetention:FullDetailDays",
+
         // Who may call the API, and how often.
         ["CORS_ORIGIN"] = "Cors:Origins:0",
         ["RATE_LIMIT_GLOBAL_PER_MINUTE"] = "RateLimits:GlobalPerMinute",

src/GarageStack.Core/Configuration/TelemetryRetentionOptions.cs

@@ -0,0 +1,26 @@
+using Microsoft.Extensions.Configuration;
+
+namespace GarageStack.Core.Configuration;
+
+/// <summary>
+/// How long telemetry keeps every row before the Worker compacts it into quarter-hour rows
+/// (TELEMETRY_FULL_DETAIL_DAYS). 0 or less switches compaction off. A setting below
+/// <see cref="MinimumDays"/> is raised to it, since the statistics page and the map read that far
+/// back at full detail.
+/// </summary>
+public sealed class TelemetryRetentionOptions
+{
+    public const int DefaultDays = 365;
+
+    /// <summary>The longest period the statistics page and the map ask the API for.</summary>
+    public const int MinimumDays = 90;
+
+    /// <summary>Days of telemetry kept at full detail, or null when compaction is off.</summary>
+    public int? FullDetailDays { get; init; } = DefaultDays;
+
+    public static TelemetryRetentionOptions From(IConfiguration configuration)
+    {
+        var days = configuration.IntegerOrDefault("TelemetryRetention:FullDetailDays", DefaultDays);
+        return new() { FullDetailDays = days <= 0 ? null : Math.Max(days, MinimumDays) };
+    }
+}

src/GarageStack.Core/Helpers/TelemetryCompaction.cs

@@ -0,0 +1,93 @@
+using GarageStack.Core.Models;
+
+namespace GarageStack.Core.Helpers;
+
+/// <summary>
+/// Folds old telemetry into fewer rows. The Worker already folds the messages of one poll into a
+/// single row over a 15-second window; compaction does the same over a quarter of an hour, once
+/// nothing reads the finer detail. Each window keeps its last row, filled in with the newest earlier
+/// value of every field that row lacks. "The latest reading at or before a moment" (the odometer,
+/// the last position, the current state) therefore gives the same answer at the end of every window
+/// as before, and never a reading from later on.
+/// </summary>
+public static class TelemetryCompaction
+{
+    /// <summary>
+    /// How much time one compacted row covers. A quarter of an hour divides every UTC offset in
+    /// use, so a window never spans the car's local midnight, wherever it is driven.
+    /// </summary>
+    public static readonly TimeSpan Window = TimeSpan.FromMinutes(15);
+
+    // Counters that start again from zero: at midnight, after a charge, at a new journey. Folding a
+    // reset into the reading before it would lose that reading's total (a day's distance or fuel),
+    // so a window is cut in two where one of them drops.
+    private static readonly Func<TelemetrySnapshot, double?>[] ResettingCounters =
+    [
+        s => s.MileageOfTheDay,
+        s => s.PowerUsageOfDay,
+        s => s.MileageSinceLastCharge,
+        s => s.PowerUsageSinceLastCharge,
+        s => s.CurrentJourneyDistance,
+    ];
+
+    /// <summary>The start of the window <paramref name="at"/> falls in.</summary>
+    public static DateTime WindowStart(DateTime at) => new(at.Ticks - at.Ticks % Window.Ticks, at.Kind);
+
+    /// <summary>
+    /// Compacts one vehicle's <paramref name="rows"/>. The row each window keeps is filled in place,
+    /// so a caller holding tracked entities saves the result as it stands. Rows that are already
+    /// compacted come out unchanged.
+    /// </summary>
+    /// <returns>The rows folded into a kept row, for the caller to remove.</returns>
+    public static IReadOnlyList<TelemetrySnapshot> Fold(IEnumerable<TelemetrySnapshot> rows)
+    {
+        var folded = new List<TelemetrySnapshot>();
+        foreach (var group in Groups(rows.OrderBy(r => r.RecordedAt).ThenBy(r => r.Id)))
+        {
+            var kept = group[^1];
+            for (var i = group.Count - 2; i >= 0; i--)
+            {
+                TelemetryFields.ApplyFirstNonNullFields(kept, group[i]);
+                folded.Add(group[i]);
+            }
+        }
+
+        return folded;
+    }
+
+    // Consecutive rows that share a window and see no counter reset between them, oldest first.
+    private static IEnumerable<List<TelemetrySnapshot>> Groups(IEnumerable<TelemetrySnapshot> ordered)
+    {
+        List<TelemetrySnapshot>? group = null;
+        var windowStart = DateTime.MinValue;
+        var counters = new double?[ResettingCounters.Length];
+
+        foreach (var row in ordered)
+        {
+            var start = WindowStart(row.RecordedAt);
+            if (group is null || start != windowStart || Resets(counters, row))
+            {
+                if (group is not null) yield return group;
+                group = [];
+                windowStart = start;
+                Array.Clear(counters);
+            }
+
+            group.Add(row);
+            for (var i = 0; i < ResettingCounters.Length; i++)
+                counters[i] = ResettingCounters[i](row) ?? counters[i];
+        }
+
+        if (group is not null) yield return group;
+    }
+
+    private static bool Resets(double?[] latest, TelemetrySnapshot row)
+    {
+        for (var i = 0; i < ResettingCounters.Length; i++)
+        {
+            if (ResettingCounters[i](row) < latest[i]) return true;
+        }
+
+        return false;
+    }
+}

src/GarageStack.Core/Helpers/TelemetryFields.cs

@@ -0,0 +1,76 @@
+using System.Linq.Expressions;
+using System.Reflection;
+using GarageStack.Core.Models;
+
+namespace GarageStack.Core.Helpers;
+
+/// <summary>
+/// The fields of a <see cref="TelemetrySnapshot"/> that carry telemetry, and the two ways rows are
+/// merged field by field. Every merge goes through here (the Worker folding a poll into one row,
+/// the dashboard's view of the current state, and compaction of old rows), so a new telemetry field
+/// only needs to be added to the model, not hand-copied into each merge loop.
+/// </summary>
+public static class TelemetryFields
+{
+    // Identity and bookkeeping: never merged.
+    private static readonly HashSet<string> NonMergeable =
+    [
+        nameof(TelemetrySnapshot.Id), nameof(TelemetrySnapshot.VehicleId),
+        nameof(TelemetrySnapshot.Vehicle), nameof(TelemetrySnapshot.RecordedAt),
+        nameof(TelemetrySnapshot.RawTopic),
+    ];
+
+    /// <summary>Every property that carries telemetry, in declaration order.</summary>
+    public static readonly IReadOnlyList<PropertyInfo> Mergeable = typeof(TelemetrySnapshot)
+        .GetProperties(BindingFlags.Public | BindingFlags.Instance)
+        .Where(p => p.CanRead && p.CanWrite && !NonMergeable.Contains(p.Name))
+        .ToArray();
+
+    // Compiled expression-tree accessors instead of live reflection (PropertyInfo.GetValue/
+    // SetValue): merging runs on every MQTT message, multiple times per second per vehicle, and
+    // reflection's per-call overhead is avoidable since the property set is fixed at startup.
+    // Built once here, then invoked like a regular delegate call from then on.
+    private sealed record PropertyAccessor(string Name, Func<TelemetrySnapshot, object?> Get, Action<TelemetrySnapshot, object?> Set);
+
+    private static PropertyAccessor BuildAccessor(PropertyInfo prop)
+    {
+        var instance = Expression.Parameter(typeof(TelemetrySnapshot), "instance");
+        var value = Expression.Parameter(typeof(object), "value");
+
+        var getter = Expression.Lambda<Func<TelemetrySnapshot, object?>>(
+            Expression.Convert(Expression.Property(instance, prop), typeof(object)),
+            instance).Compile();
+
+        var setter = Expression.Lambda<Action<TelemetrySnapshot, object?>>(
+            Expression.Assign(
+                Expression.Property(instance, prop),
+                Expression.Convert(value, prop.PropertyType)),
+            instance, value).Compile();
+
+        return new PropertyAccessor(prop.Name, getter, setter);
+    }
+
+    private static readonly PropertyAccessor[] Accessors = [.. Mergeable.Select(BuildAccessor)];
+
+    /// <summary>Overwrites every field on <paramref name="target"/> with the non-null value from <paramref name="source"/>, if any.</summary>
+    public static void ApplyNonNullFields(TelemetrySnapshot target, TelemetrySnapshot source)
+    {
+        foreach (var prop in Accessors)
+        {
+            var value = prop.Get(source);
+            if (value is not null) prop.Set(target, value);
+        }
+    }
+
+    /// <summary>Fills any still-empty field on <paramref name="target"/> from <paramref name="source"/>, leaving already-set fields untouched.</summary>
+    public static void ApplyFirstNonNullFields(TelemetrySnapshot target, TelemetrySnapshot source, ISet<string>? skip = null)
+    {
+        foreach (var prop in Accessors)
+        {
+            if (skip is not null && skip.Contains(prop.Name)) continue;
+            if (prop.Get(target) is not null) continue;
+            var value = prop.Get(source);
+            if (value is not null) prop.Set(target, value);
+        }
+    }
+}

src/GarageStack.Core/Interfaces/IHousekeepingRepository.cs

@@ -0,0 +1,29 @@
+using GarageStack.Core.Models;
+
+namespace GarageStack.Core.Interfaces;
+
+/// <summary>
+/// Keeps the database from growing without end: folds old telemetry into quarter-hour rows, and
+/// removes what nothing reads any more.
+/// </summary>
+public interface IHousekeepingRepository
+{
+    /// <summary>When the vehicle's oldest telemetry row was recorded, or null when it has none.</summary>
+    Task<DateTime?> GetFirstTelemetryAtAsync(int vehicleId, CancellationToken ct = default);
+
+    /// <summary>
+    /// Compacts the vehicle's telemetry from <paramref name="from"/> up to (not including)
+    /// <paramref name="until"/> and moves <see cref="Vehicle.TelemetryCompactedUntil"/> to
+    /// <paramref name="until"/>, in one transaction.
+    /// </summary>
+    /// <returns>How many rows were folded into others and removed.</returns>
+    Task<int> CompactTelemetryAsync(int vehicleId, DateTime from, DateTime until, CancellationToken ct = default);
+
+    /// <summary>Removes the place names and snapped trips that expired before <paramref name="before"/>.</summary>
+    /// <returns>How many cache entries were removed.</returns>
+    Task<int> RemoveExpiredCacheEntriesAsync(DateTime before, CancellationToken ct = default);
+
+    /// <summary>Removes the notifications the user deleted that were created before <paramref name="createdBefore"/>.</summary>
+    /// <returns>How many notifications were removed.</returns>
+    Task<int> RemoveDeletedNotificationsAsync(DateTime createdBefore, CancellationToken ct = default);
+}

src/GarageStack.Core/Models/Vehicle.cs

@@ -28,6 +28,12 @@ public class Vehicle
     /// </summary>
     public DateTime? TripsRecordedUntil { get; set; }
 
+    /// <summary>
+    /// How far the Worker has compacted this vehicle's telemetry into quarter-hour rows: every row
+    /// before this moment is compacted. Null until the first compaction.
+    /// </summary>
+    public DateTime? TelemetryCompactedUntil { get; set; }
+
     public ICollection<TelemetrySnapshot> TelemetrySnapshots { get; set; } = [];
 }