From 3a3368066f35bfe1141ade2f4bb311869e59ddd2 Mon Sep 17 00:00:00 2001 From: Sean Dolbec Date: Wed, 11 Nov 2026 13:06:53 -0500 Subject: [PATCH] Backups: one "Turn on backups" path, a plain health status and alarms; short breaks are always paid MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Backups (production's encrypted backups were refused for 10+ days with no key and nobody was told): - BackupHealth: one answer (off / starting / working / needs attention) from the active key and the layer reports the scripts already write. After a key is turned on each layer gets one interval of its schedule before an older "refused: no key" report counts as a failure. - Settings → Backups: status first (last good backup, weekly check, key), then a two-step "Turn on backups" path: the kit downloads, the page moves to the check code by itself, and after the code it refreshes when the first backup lands. Plain words on what the kit is and what losing it means. Encryption, the one-time kit, the check code and the reauth rule are unchanged. - While backups are off or failing, super admins see a red alert on the bell, a dashboard banner and a red status bar entry (which used to show the unused Drive backup); BackupHealthAlertJob leaves one notification a day. Time & pay: a punched break of 20 minutes or less is paid time even as a "meal" or where the store marks the kind unpaid (FLSA, 29 CFR 785.18-785.19). One predicate (ClockBreak#unpaid?) feeds the clock, Pay page, timesheet import and so the payroll export. No automatic break deductions exist. Co-Authored-By: Claude Fable 5.2 --- docs/ops/backups.md | 10 +- docs/ops/operator-runbook.md | 2 +- docs/time-and-pay/DECISIONS.md | 12 +- web/app/controllers/backup_keys_controller.rb | 18 ++- web/app/helpers/application_helper.rb | 3 + .../controllers/backup_status_controller.js | 59 +++++++ web/app/jobs/backup_health_alert_job.rb | 25 +++ web/app/models/clock_break.rb | 10 ++ web/app/models/clock_shift.rb | 8 +- web/app/services/backup_health.rb | 151 ++++++++++++++++++ web/app/views/backup_keys/show.html.erb | 128 +++++++++++---- web/app/views/home/index.html.erb | 1 + web/app/views/settings/show.html.erb | 4 +- .../shared/_backup_health_banner.html.erb | 8 + .../views/shared/_statusbar_vitals.html.erb | 5 +- web/app/views/system/_backups.html.erb | 2 +- .../views/time_rules_settings/show.html.erb | 1 + web/config/recurring.yml | 5 + .../integration/backup_health_alerts_test.rb | 110 +++++++++++++ web/test/integration/backup_keys_test.rb | 9 +- .../integration/employee_timesheets_test.rb | 23 +++ .../models/time_and_pay/foundation_test.rb | 52 ++++++ web/test/services/backup_health_test.rb | 136 ++++++++++++++++ 23 files changed, 737 insertions(+), 45 deletions(-) create mode 100644 web/app/javascript/controllers/backup_status_controller.js create mode 100644 web/app/jobs/backup_health_alert_job.rb create mode 100644 web/app/services/backup_health.rb create mode 100644 web/app/views/shared/_backup_health_banner.html.erb create mode 100644 web/test/integration/backup_health_alerts_test.rb create mode 100644 web/test/services/backup_health_test.rb diff --git a/docs/ops/backups.md b/docs/ops/backups.md index 6fa0ed8c3..1dc98b335 100644 --- a/docs/ops/backups.md +++ b/docs/ops/backups.md @@ -11,6 +11,14 @@ file to `storage/backups/.json` after every run (`state` ok/failed, `at`, that has never reported shows **Not reporting**, which means it is not installed on this server. The MCP tool `backup_freshness` reports the same layers to connected assistants. +**Settings → Backups** is the owner's page: one plain status (`BackupHealth`: off, starting, working, needs +attention), the last good backup, the weekly check, and the two-step **Turn on backups** path below. While backups +are off or a layer has failed or gone quiet past its schedule, every super admin sees a red alert on the bell, a +banner on the dashboard and a red entry in the status bar, and `BackupHealthAlertJob` leaves one notification a day. +The rolling snapshot and local set always count; the off-host copy and weekly check count once they have reported; +the mirror and the recovery-kit drill are listed as extra protection and never alarm. After a key is turned on each +layer gets one interval of its schedule to make its first encrypted backup before an older refusal counts. + ## Encryption and the recovery kit (#421) Every backup file is encrypted with [age](https://age-encryption.org) (X25519, ChaCha20-Poly1305: authenticated, so @@ -20,7 +28,7 @@ SER8 (container or host) can decrypt a backup, which is the point: a copy of the mirror is useless without the kit. - **Install age** in the container (and on any computer that will restore): `apt-get install age`. -- **Create the key**: Settings → Backups → *Create backup key* (super admin with `settings.write`, signed in within the +- **Turn backups on**: Settings → Backups → *Turn on backups* (later: *Replace the backup key*) (super admin with `settings.write`, signed in within the last 15 minutes, or a password re-entry for password accounts). The server makes the key pair in memory, stores the public key in `backup_encryption_keys`, and answers with the kit PDF. The private key is never stored, logged or shown again (`test/integration/backup_keys_test.rb` checks every table and the log after creating one). diff --git a/docs/ops/operator-runbook.md b/docs/ops/operator-runbook.md index 29ad00335..df059da9e 100644 --- a/docs/ops/operator-runbook.md +++ b/docs/ops/operator-runbook.md @@ -48,7 +48,7 @@ Shopify sends `customers/data_request`, `customers/redact`, `shop/redact`, `app/ | Chat, payroll, HR and 1099 columns | database | encrypted at rest since #421; run `CONFIRM=1 bin/rails ops:encrypt_sensitive_fields` once after upgrading ([encryption-at-rest.md](encryption-at-rest.md)) | — | | Square access token | Settings → env (`SQUARE_ACCESS_TOKEN`) | create a new token in the Square developer dashboard for the same environment, save, run a Square sync | — | | Google OAuth / Drive | Settings → Google sign-in and Drive sections | regenerate the client secret in Google Cloud, save; reconnect Drive (consent) | Drive refresh token must be re-issued | -| Backup encryption key (age) | public half in `backup_encryption_keys`; private half only in the owner's recovery kit (password manager) | Settings → Backups → Create backup key, save the new kit, type its check code; keep the old kit until its backups age out (~35 days) | — | +| Backup encryption key (age) | public half in `backup_encryption_keys`; private half only in the owner's recovery kit (password manager) | Settings → Backups → Replace the backup key (Make a new recovery kit), save the new kit, type its check code; keep the old kit until its backups age out (~35 days) | — | | `SENTRY_DSN` | `/etc/rupert.env` | replace, restart | — | Every rotation: take a full backup first (`systemctl start rupert-backup.service`), note the date in the Activity page or the operator log. diff --git a/docs/time-and-pay/DECISIONS.md b/docs/time-and-pay/DECISIONS.md index 86fe99ae9..ebbc06435 100644 --- a/docs/time-and-pay/DECISIONS.md +++ b/docs/time-and-pay/DECISIONS.md @@ -39,7 +39,7 @@ to Square. | Rule | Default | Why | | --- | --- | --- | | Rule preset | Federal (FLSA) | The stores' state is not recorded in the repo; the owner picks the state preset (CT, MA, NY, NJ, CA, OR, CO available) and the defaults below update. | -| Meal break | Reminder after 6 h, 30 min, unpaid | Common retail practice and the MA/NY/CT rule; federal law requires none. A meal break under 20 min is treated as paid rest. | +| Meal break | Reminder after 6 h, 30 min, unpaid | Common retail practice and the MA/NY/CT rule; federal law requires none. A punched meal of 20 minutes or less counts as paid time (see "Short breaks are always paid"). | | Rest break | Every 4 h, 10 min, paid | FLSA: short breaks are paid time. | | Break reminders | On | Push/in-app nudge at the threshold; never auto-clocks anyone out. | | Daily overtime | Off | Only CA/AK/NV/CO have it; the CA/CO presets turn it on. | @@ -122,6 +122,16 @@ unpaid breaks (meals by default, per the store's break rules). Before this the i while the Pay page, overtime checks and accrual took paid hours, so the payroll export disagreed with what the worker saw. Sheets approved before the change are not rewritten. +## Short breaks are always paid (2026-11-11) + +A punched break of 20 minutes or less is paid time even when it was punched as a meal or the store marks that +kind of break unpaid, because federal law (FLSA, 29 CFR 785.18 and 785.19) counts short breaks as work and only +lets a real meal period, normally 30 minutes or more, go unpaid. It is a fixed floor, not a setting +(`ClockBreak::SHORT_BREAK_PAID`), and it applies wherever paid hours are counted: the clock, the Pay page, the +timesheet import and so the payroll export. Rupert never deducts a break automatically: only breaks someone +actually punched (or a manager-approved correction recorded) come off. Sheets approved before the change are not +rewritten, and Square timecards keep the paid or unpaid flag Square recorded for each break. + ## Retention, PII, offboarding Punches, allocations and corrections are kept for 4 years (FLSA 3 years, plus a year); nothing is diff --git a/web/app/controllers/backup_keys_controller.rb b/web/app/controllers/backup_keys_controller.rb index 7c396a999..b44aed4c4 100644 --- a/web/app/controllers/backup_keys_controller.rb +++ b/web/app/controllers/backup_keys_controller.rb @@ -2,7 +2,7 @@ # Settings → Backups (#421): the key that encrypts backups, and the owner's recovery kit. # -# "Create backup key" makes an age key pair in memory, stores ONLY the public half (BackupEncryptionKey) and answers +# "Turn on backups" (later "Make a new recovery kit") makes an age key pair in memory, stores ONLY the public half (BackupEncryptionKey) and answers # with the recovery kit PDF holding the private half and the Active Record encryption keys. The private key is not # saved, logged, cached or shown again: losing the kit means losing access to the backups made with it. # @@ -27,6 +27,13 @@ def show @recent_sign_in = recent_sign_in? @password_session = session[:auth_method] == 'password' @workspace = workspace_name + @health = BackupHealth.new(key: @current) + @signature = [@current&.id, @pending&.id, @health.state].join(':') + respond_to do |format| + format.html + # Polled by the page while the kit downloads or the first backup is due: ids and a state word, nothing secret. + format.json { render(json: { signature: @signature }) } + end end # POST /settings/backups/key → the recovery kit PDF (once). @@ -58,7 +65,7 @@ def create def activate key = pending_key if key.nil? - return redirect_to(settings_backups_path, alert: 'No backup key is waiting for its check code. Create a new key.') + return redirect_to(settings_backups_path, alert: 'No recovery kit is waiting for its check code (a code works for one hour). Make a new kit below.') end unless key.code_matches?(params[:check_code]) ActivityLogger.log('backup_key_activation_failed', subject: key, label: key.fingerprint, details: 'Wrong check code') @@ -69,7 +76,12 @@ def activate key.activate!(actor: Current.user) ActivityLogger.log('backup_key_activated', subject: key, label: key.fingerprint, details: ["Backups now encrypt to #{key.fingerprint}", ("#{previous.fingerprint} retired; its kit still opens older backups" if previous)].compact.join('. ')) - redirect_to(settings_backups_path, notice: "Backup key #{key.fingerprint} is active. The next backup is encrypted with it.") + notice = if previous + "Backup key #{key.fingerprint} is active. The next backup is encrypted with it." + else + 'Backups are on. The first encrypted backup is made within five minutes and shows on this page.' + end + redirect_to(settings_backups_path, notice: notice) end private diff --git a/web/app/helpers/application_helper.rb b/web/app/helpers/application_helper.rb index 4888b98f1..379a5a867 100644 --- a/web/app/helpers/application_helper.rb +++ b/web/app/helpers/application_helper.rb @@ -175,6 +175,9 @@ def panel_view? def system_alerts @system_alerts ||= begin alerts = [] + if (health = BackupHealth.alert_for(Current.user)) + alerts << { title: health.title, body: health.summary, path: settings_backups_path, link: health.link_label } + end if (location = card_processing_down_location) alerts << { title: 'Card processing is down', body: "Customers at #{location.name} cannot pay by card until the payment gateway is connected and verified.", path: pos_setting_path(location.id), link: 'Open POS settings' } diff --git a/web/app/javascript/controllers/backup_status_controller.js b/web/app/javascript/controllers/backup_status_controller.js new file mode 100644 index 000000000..5ad99420b --- /dev/null +++ b/web/app/javascript/controllers/backup_status_controller.js @@ -0,0 +1,59 @@ +import { Controller } from "@hotwired/stimulus" + +// Settings → Backups. The recovery kit arrives as a file download, which leaves the page where it was, and the +// first backup lands a few minutes after the key is confirmed. This asks the server for the page's state every few +// seconds while something is expected to change and reloads once it has, so the owner never has to refresh by hand. +export default class extends Controller { + static targets = ["hint"] + static values = { + url: String, + signature: String, + poll: Boolean, + interval: { type: Number, default: 4000 }, + limit: { type: Number, default: 900000 } + } + + connect() { + if (this.pollValue) this.start() + } + + disconnect() { + this.stop() + } + + // The "Turn on backups" form was submitted: the kit downloads and the page must move on to the check code. + watch() { + if (this.hasHintTarget) this.hintTarget.hidden = false + this.start() + } + + start() { + if (this.timer) return + this.startedAt = Date.now() + this.timer = setInterval(() => this.check(), this.intervalValue) + } + + stop() { + clearInterval(this.timer) + this.timer = null + } + + async check() { + if (Date.now() - this.startedAt > this.limitValue) return this.stop() + if (document.hidden || this.inFlight) return + this.inFlight = true + try { + const response = await fetch(this.urlValue, { headers: { Accept: "application/json" }, credentials: "same-origin" }) + if (!response.ok) return + const data = await response.json() + if (data.signature && data.signature !== this.signatureValue) { + this.stop() + window.location.assign(window.location.pathname) + } + } catch (_error) { + // Offline or signed out: the next tick tries again; the owner can always refresh. + } finally { + this.inFlight = false + } + } +} diff --git a/web/app/jobs/backup_health_alert_job.rb b/web/app/jobs/backup_health_alert_job.rb new file mode 100644 index 000000000..7d1a655d9 --- /dev/null +++ b/web/app/jobs/backup_health_alert_job.rb @@ -0,0 +1,25 @@ +# frozen_string_literal: true + +# Hourly: when backups are off or failing (BackupHealth), every active instance owner (super admin) gets one in-app +# notification per day until it is fixed. The bell and the dashboard show the live state on every page; this leaves +# a dated record for someone who has not looked. Backups failed unnoticed from late October to Nov 11, 2026. +class BackupHealthAlertJob < ApplicationJob + queue_as :default + + def perform + return unless BackupHealth.alerts_enabled? + + health = BackupHealth.current + return unless health.alert? + + User.where(role: 'super_admin', active: true).find_each do |owner| + tenant = Tenant.find_by(id: owner.tenant_id) + next unless tenant + + Current.with_tenant(tenant) do + Notification.deliver(owner.id, kind: 'backup_health', title: health.title, body: health.summary, + url: '/settings/backups', dedupe_key: "backup-health:#{Time.current.utc.to_date.iso8601}", actor: nil) + end + end + end +end diff --git a/web/app/models/clock_break.rb b/web/app/models/clock_break.rb index 124f0f4b2..b5d7da6d6 100644 --- a/web/app/models/clock_break.rb +++ b/web/app/models/clock_break.rb @@ -11,6 +11,10 @@ class ClockBreak < ApplicationRecord KINDS = %w[rest meal].freeze LABELS = { 'rest' => 'Break', 'meal' => 'Meal' }.freeze + # Federal floor (FLSA, 29 CFR 785.18-785.19): breaks of about 5 to 20 minutes are hours worked; only a real meal + # period, normally 30 minutes or more, may be unpaid. So a punched "meal" (or any break a store marked unpaid) of 20 + # minutes or less still counts as paid time. Not a setting: a store can be more generous, never less. + SHORT_BREAK_PAID = 20.minutes belongs_to :clock_shift @@ -32,6 +36,12 @@ def seconds(at: Time.current) [(ended_at || at) - started_at, 0].max end + # A short break is paid time whatever it is called and whatever the store's rules say (see SHORT_BREAK_PAID). + def short?(at: Time.current) = seconds(at: at) <= SHORT_BREAK_PAID + + # Does this break come off paid hours? Only when the rules make its kind unpaid AND it ran longer than a short break. + def unpaid?(rules, at: Time.current) = !rules.break_paid?(kind) && !short?(at: at) + private def inside_shift diff --git a/web/app/models/clock_shift.rb b/web/app/models/clock_shift.rb index c59080935..fc41932fb 100644 --- a/web/app/models/clock_shift.rb +++ b/web/app/models/clock_shift.rb @@ -45,9 +45,13 @@ def hours(at: Time.current) end # Seconds of unpaid break inside the shift (the location's rules decide which kinds are unpaid). + # A break of 20 minutes or less is never unpaid (ClockBreak::SHORT_BREAK_PAID). Nothing is deducted automatically: + # only breaks the worker actually punched (or a manager-approved correction recorded) come off. + # A break is judged by its whole length, also when a window clips it (a meal across midnight). def unpaid_break_seconds(at: Time.current, rules: nil) rules ||= TimeRules.for(location_id) - breaks.sum { |b| rules.break_paid?(b.kind) ? 0 : b.seconds(at: [at, ended_at].compact.min) } + limit = [at, ended_at].compact.min + breaks.sum { |pause| pause.unpaid?(rules, at: limit) ? pause.seconds(at: limit) : 0 } end # Hours that count for pay: punches minus unpaid breaks. @@ -67,7 +71,7 @@ def paid_hours_between(from, to, rules: nil) rules ||= TimeRules.for(location_id) unpaid = breaks.sum do |pause| - next 0 if pause.ended_at.nil? || rules.break_paid?(pause.kind) + next 0 if pause.ended_at.nil? || !pause.unpaid?(rules) [[pause.ended_at, end_at].min - [pause.started_at, start_at].max, 0].max end diff --git a/web/app/services/backup_health.rb b/web/app/services/backup_health.rb new file mode 100644 index 000000000..a39b76c08 --- /dev/null +++ b/web/app/services/backup_health.rb @@ -0,0 +1,151 @@ +# frozen_string_literal: true + +# One plain answer to "are backups working?", for the owner: Settings → Backups, the bell, the dashboard banner, the +# status bar and the hourly BackupHealthAlertJob all read it. It only interprets what already exists: the active +# BackupEncryptionKey and the per-layer reports in BackupStatus (written by the backup scripts). +# +# off no active backup key: every scheduled backup is refused (the server never writes an unencrypted one) +# starting the key was just turned on and the first encrypted backup has not landed yet +# ok the layers that run on this server are reporting fresh, encrypted backups +# attention a layer failed, or has not reported inside its schedule +# +# The rolling snapshot and the local set always count. The off-host copy and the weekly check count once they have +# reported at least once (they are installed on the host, outside the app). The off-machine mirror and the +# recovery-kit drill are extra protection: listed on the page, never an alarm on every page. +# +# A layer only reports when its own timer fires, so after a key is turned on each one gets a single interval of its +# schedule to make its first encrypted backup before its older "refused: no key" report counts as a failure. +class BackupHealth + CORE = %w[frequent local].freeze + WHEN_INSTALLED = %w[offhost verify].freeze + EXTRA = %w[offsite restore].freeze + PROBLEM_STATES = %w[failed stale missing].freeze + + Check = Struct.new(:layer, :state, keyword_init: true) do + delegate :key, :label, :every, :report, to: :layer + def ok? = state == 'ok' + def waiting? = state == 'waiting' + def problem? = PROBLEM_STATES.include?(state) + end + + attr_reader :key, :now + + class << self + def current = new + + # Alarms are for real servers. Development and test databases have no backup timers; tests turn this on. + def alerts_enabled? = Rails.env.production? + + def alert_for(user) + return unless alerts_enabled? && user&.super_admin? + + health = current + health if health.alert? + rescue StandardError => e + Rails.logger.warn("[backup-health] #{e.class}") + nil + end + end + + def initialize(layers: BackupStatus.layers, key: BackupEncryptionKey.current, now: Time.current) + @layers = layers.index_by(&:key) + @key = key + @now = now + end + + def state + return :off unless key + return :attention if problems.any? + + last_good_at ? :ok : :starting + end + + def off? = state == :off + def ok? = state == :ok + def starting? = state == :starting + def alert? = %i[off attention].include?(state) + + # Layers whose trouble is worth an alarm. + def checks + @checks ||= (CORE + WHEN_INSTALLED).filter_map do |name| + layer = @layers[name] + next unless layer + next if WHEN_INSTALLED.include?(name) && !layer.reported? + + Check.new(layer: layer, state: effective_state(layer)) + end + end + + def extras + @extras ||= EXTRA.filter_map { |name| (layer = @layers[name]) && Check.new(layer: layer, state: layer.state(now)) } + end + + def problems = key ? checks.select(&:problem?) : [] + def waiting = key ? checks.select(&:waiting?) : [] + + # When the newest encrypted, integrity-checked backup was written on this server. + def last_good_at + return unless key + + checks.select { |check| CORE.include?(check.key) && check.ok? }.filter_map { |check| check.report.at }.max + end + + # When the weekly check last confirmed the copies outside the container against their digests. + def verified_at + check = checks.find { |entry| entry.key == 'verify' } + check.report.at if check&.ok? + end + + def title + case state + when :off then 'Backups are off' + when :attention then 'Backups need attention' + when :starting then 'Backups are on' + else 'Backups are working' + end + end + + def summary + case state + when :off + 'Nothing is being backed up. If this server were lost today, your sales, stock and time records could not be recovered. Turning backups on takes about two minutes.' + when :attention + "#{problems.map { |check| problem_text(check) }.join(' ')} #{last_good_text}".strip + when :starting + 'The first encrypted backup is being made. It appears here within about five minutes.' + else + last_good_text + end + end + + def link_label = off? ? 'Turn on backups' : 'Open Backups' + + def problem_text(check) + report = check.report + return "#{check.label} has never run on this server." unless report + + age = report.at ? "#{ago(report.at)} ago" : 'at an unknown time' + return "#{check.label} has not run since #{age} (expected #{check.every})." if check.state == 'stale' + + reason = check.layer.evidence_issue(now) || report.message.to_s.first(200).presence || 'no reason given' + "#{check.label} failed #{age}: #{reason.chomp('.')}." + end + + private + + def last_good_text + last_good_at ? "Last good backup: #{ago(last_good_at)} ago." : 'There is no good backup yet.' + end + + def ago(time) = ActionController::Base.helpers.distance_of_time_in_words(time, now) + + def effective_state(layer) + raw = layer.state(now) + return raw if raw == 'ok' || key.nil? + + reported_since_key = layer.report&.at && layer.report.at >= key.activated_at + return 'waiting' if !reported_since_key && now < key.activated_at + layer.stale_after + + raw + end +end diff --git a/web/app/views/backup_keys/show.html.erb b/web/app/views/backup_keys/show.html.erb index 429a7d241..e327978e9 100644 --- a/web/app/views/backup_keys/show.html.erb +++ b/web/app/views/backup_keys/show.html.erb @@ -1,54 +1,106 @@ -<%# Settings → Backups: the Settings template (docs/ui-standards.md §4). Backup encryption key and recovery kit (#421). %> -
- <%= page_header 'Backups', subtitle: 'Backups are encrypted to a key whose private half lives only in your recovery kit.' %> +<%# Settings → Backups: the Settings template (docs/ui-standards.md §4). One plain status (BackupHealth), one + "Turn on backups" path that makes the key and its recovery kit (#421), and the key history. %> +<% stamp = ->(time) { time.in_time_zone.strftime('%b %-d, %Y %-l:%M %p %Z') } %> +<% pills = { off: %w[Off pill-rose], attention: ['Needs attention', 'pill-rose'], starting: %w[Starting pill-clay], ok: %w[Working pill-fern] } %> +<% replacing = @current || @pending %> +
+ <%= page_header 'Backups', subtitle: 'Encrypted copies of everything in Rupert, made automatically. Only your recovery kit can open them.' %> -
- <%= section_heading 'Backup encryption', hint: 'The server keeps only the public key; it can lock backups but never open them.' do %> - <% if @current %> - Encrypted - <% else %> - No key - <% end %> +
+ <%= section_heading @health.title do %> + <%= pills.fetch(@health.state)[0] %> <% end %> +

<%= @health.summary %>

<% if @current %>
-
Key fingerprint
<%= @current.fingerprint %>
-
Active since
<%= @current.activated_at.strftime('%b %-d, %Y %-l:%M %p') %>
+
+
Last good backup
+
+ <% if @health.last_good_at %> + <%= stamp.(@health.last_good_at) %> (<%= time_ago_in_words(@health.last_good_at) %> ago) +
Encrypted and integrity-checked + <% else %> + None yet + <% end %> +
+
+
+
Weekly check of the stored copies
+
+ <% verify = @health.checks.find { |check| check.key == 'verify' } %> + <% if @health.verified_at %> + Passed <%= stamp.(@health.verified_at) %> + <% elsif verify&.waiting? %> + First check runs this Sunday + <% elsif verify %> + Not passing + <% else %> + Not set up on this server + <% end %> +
+
+
Backup key
<%= @current.fingerprint %>
+
On since
<%= stamp.(@current.activated_at) %>
-

Every backup file has a .key file beside it naming this fingerprint. Open a backup with the recovery kit that shows the same fingerprint.

- <% else %> -

Backups are refused until a key is active: the server will not write unencrypted backups. Create a backup key below.

+ <% if @health.waiting.any? %> +

Still to make their first run with this key: <%= @health.waiting.map { |check| "#{check.label.downcase} (#{check.every})" }.to_sentence %>.

+ <% end %> + <% if @health.starting? %> +

This page refreshes by itself when the first backup lands.

+ <% end %> +

A backup is made every 5 minutes and a full set every 6 hours. Your recovery kit shows the same key fingerprint as above; it is the only thing that opens these backups.

+ <% end %> + <% if Current.user.can?('system.read') %> +

<%= link_to 'Every backup layer in detail', system_path(anchor: 'backups'), class: 'underline' %>

<% end %>
<% if @pending %>
- <%= section_heading 'Confirm the recovery kit', hint: 'Type the check code printed on the kit you just saved.' %> -

Key <%= @pending.fingerprint %> is waiting. It starts protecting backups only after you enter its check code, so no backup depends on a kit you did not keep. The code expires an hour after the kit was made.

+ <%= section_heading(@current ? 'Confirm the new recovery kit' : 'Step 2 of 2: confirm you saved the recovery kit', hint: 'Type the check code printed on the kit.') %> +
    +
  1. Find the PDF that just downloaded: rupert-backup-recovery-kit-<%= @pending.fingerprint.delete(' ') %>.pdf.
    2. +
  2. Save it in your password manager as a secure note named “Rupert backup recovery kit — <%= @workspace %>”, and print one copy for a safe place.
    3. +
  3. Type the check code from the first page of the kit below.
    4. +
+

Backups <%= @current ? 'switch to this key' : 'start' %> only after the code is entered, so no backup ever depends on a kit you did not keep. The code works for one hour.

<%= form_with(url: activate_settings_backup_key_path, method: :post, local: true, data: { turbo: false }, class: 'space-y-3') do %>
<%= label_tag :check_code, 'Check code', class: 'field-label' %> - <%= text_field_tag :check_code, nil, required: true, autocomplete: 'off', spellcheck: false, placeholder: 'ABCD-EFGH', class: 'input font-mono', data: { lpignore: 'true', '1p-ignore': 'true' } %> + <%= text_field_tag :check_code, nil, required: true, autofocus: true, autocomplete: 'off', spellcheck: false, placeholder: 'ABCD-EFGH', class: 'input font-mono', data: { lpignore: 'true', '1p-ignore': 'true' } %>
- <%= submit_tag 'Activate key', class: 'btn-primary' %> + <%= submit_tag(@current ? 'Activate key' : 'Start backups', class: 'btn-primary') %>
<% end %> +

The PDF did not download, or it is already lost? Nothing depends on it yet: make a new kit below and use its code instead.

<% end %>
- <%= section_heading(@current ? 'Rotate the backup key' : 'Create backup key', hint: 'Makes a new key pair and downloads its recovery kit PDF once.') %> -
- What the recovery kit holds and where to keep it -

The PDF holds the private key that opens the backups, the database encryption keys a restored copy needs, the fingerprint and the restore steps. Rupert does not keep the private key, and the PDF cannot be downloaded again.

-

Save it in your password manager as a separate secure note named “Rupert backup recovery kit — <%= @workspace %>”. It is not a login: sign-in uses Google, so leave the site login entry alone.

+ <%= section_heading((@current ? 'Replace the backup key' : (@pending ? 'Make a new recovery kit' : 'Turn on backups')), + hint: (replacing ? 'Makes a new key and downloads its recovery kit PDF once.' : 'Two steps, about two minutes.')) %> + <% unless replacing %> +
    +
  1. Click “Turn on backups”. A PDF called the recovery kit downloads. It is the only key that opens your backups.
    2. +
  2. Save the kit, then type its check code here. Backups start by themselves and this page shows the first one within five minutes.
    3. +
+ <% end %> +
> + What the recovery kit is, where to keep it, and what happens if it is lost +

Backups are locked so that nobody who copies them, and nothing on this server, can read them. The recovery kit is the key. Rupert does not keep a copy, and the PDF cannot be downloaded again.

+

Keep two copies: one in your password manager as a separate secure note named “Rupert backup recovery kit — <%= @workspace %>”, and one printed in a safe place. It is not a login, so leave the site login entry alone. Do not email it and do not leave it in Downloads.

+

If the kit is lost: day-to-day Rupert keeps working, but backups made with it can never be opened by anyone. Come back here and replace the key straight away; the new kit protects every backup made from then on.

<% if @current %> -

Rotating keeps older backups readable with the old kit; keep both until the old backups have aged out.

+

Replacing the key keeps older backups readable with the old kit; keep both until the old backups have aged out (about 35 days).

<% end %>
<% if @recent_sign_in || @password_session %> - <%= form_with(url: settings_backup_key_path, method: :post, local: true, data: { turbo: false }, class: 'space-y-3') do %> + <%= form_with(url: settings_backup_key_path, method: :post, local: true, class: 'space-y-3', + data: { turbo: false, action: 'submit->backup-status#watch' }) do %> <% unless @recent_sign_in %>
<%= label_tag :password, 'Your Rupert password', class: 'field-label' %> @@ -57,20 +109,36 @@ <% end %>
- <%= submit_tag 'Create backup key', class: 'btn-primary' %> + <%= submit_tag((replacing ? 'Make a new recovery kit' : 'Turn on backups'), class: (replacing ? 'btn-ghost' : 'btn-primary')) %>
+ <% end %> <% else %> -

For safety, creating a key needs a sign-in from the last <%= BackupKeysController::REAUTH_WINDOW.in_minutes.to_i %> minutes. Sign out, sign in again with Google, then come back to Settings → Backups.

+

For safety, this needs a sign-in from the last <%= BackupKeysController::REAUTH_WINDOW.in_minutes.to_i %> minutes. Sign out, sign in again, then come back to Settings → Backups.

<%= button_to 'Sign out', logout_path, method: :delete, class: 'btn-ghost', form: { data: { turbo: false } } %>
<% end %>
+ <% if @current %> +
+ <%= section_heading 'Extra protection', hint: 'Optional layers. They never raise an alarm, but each one closes a real gap.' %> +
    + <% @health.extras.each do |check| %> +
  • + <%= check.label %>: <%= check.key == 'offsite' ? 'a copy on a second computer or cloud bucket. Without it, every backup lives on this one server.' : 'opening a backup with the recovery kit on another computer, to prove the kit works.' %> + <%= check.ok? ? "Done #{time_ago_in_words(check.report.at)} ago" : (check.state == 'missing' ? 'Not set up' : 'Not current') %> +
    • + <% end %> +
+

Ask whoever looks after the server to set these up: <%= link_to 'docs/ops/backups.md', 'https://github.com/techmore/rupert/blob/main/docs/ops/backups.md', class: 'underline' %>.

+
+ <% end %> +
<%= section_heading 'Key history', hint: 'Fingerprints of every key, newest first. Old kits still open the backups made with them.' %> <% if @keys.any? %> @@ -87,7 +155,7 @@
<% else %> -

No backup keys yet.

Create a backup key to start encrypted backups.

+

No backup keys yet.

Turn on backups above to make the first one.

<% end %>

Restore steps: <%= link_to 'docs/ops/backups.md', 'https://github.com/techmore/rupert/blob/main/docs/ops/backups.md', class: 'underline' %>.

diff --git a/web/app/views/home/index.html.erb b/web/app/views/home/index.html.erb index 619c421a5..2c3848d53 100644 --- a/web/app/views/home/index.html.erb +++ b/web/app/views/home/index.html.erb @@ -14,6 +14,7 @@ <% end %>
+ <%= render 'shared/backup_health_banner' %> <%= render 'shared/learning_welcome' %> <% if Current.user.can?("sync.read") %> <%= render "syncs/banner" %> diff --git a/web/app/views/settings/show.html.erb b/web/app/views/settings/show.html.erb index 1ae3b807c..f2cd61346 100644 --- a/web/app/views/settings/show.html.erb +++ b/web/app/views/settings/show.html.erb @@ -40,10 +40,10 @@ <% if Current.user.super_admin? %>
<%= link_to "Download backup", backup_settings_path, class: "btn-primary", data: { turbo: false } %> - <%= link_to "Open backup encryption", settings_backups_path, class: "btn-ghost" %> + <%= link_to "Backups: status and recovery kit", settings_backups_path, class: "btn-ghost" %>
<% unless BackupEncryptionKey.current %> -

Server backups are not encrypted yet and will be refused. Create a backup key in backup encryption.

+

Server backups are off: nothing is being backed up. Turn them on in Backups (about two minutes).

<% end %>

Database restores need offline maintenance with web and jobs stopped and a verified recovery backup. Contact the instance operator.

diff --git a/web/app/views/shared/_backup_health_banner.html.erb b/web/app/views/shared/_backup_health_banner.html.erb new file mode 100644 index 000000000..b575ddbc5 --- /dev/null +++ b/web/app/views/shared/_backup_health_banner.html.erb @@ -0,0 +1,8 @@ +<%# Dashboard: backups off or failing, for the instance owner (BackupHealth). Silent failure is the thing to prevent, + so this stays until backups are healthy; it cannot be dismissed. The bell shows the same alert on every page. %> +<% if (health = BackupHealth.alert_for(Current.user)) %> + +<% end %> diff --git a/web/app/views/shared/_statusbar_vitals.html.erb b/web/app/views/shared/_statusbar_vitals.html.erb index 591f0453e..688ad790b 100644 --- a/web/app/views/shared/_statusbar_vitals.html.erb +++ b/web/app/views/shared/_statusbar_vitals.html.erb @@ -1,12 +1,13 @@ <%# Platform administrators only: server health at a glance (HostVitals, cached 30 s). %> <% vitals = HostVitals.current %> <% level = ->(percent) { percent.nil? ? '' : (percent >= HostVitals::ALERT ? 'vital-alert' : (percent >= HostVitals::WARN ? 'vital-warn' : '')) } %> -<% backup_old = vitals.backup_at.nil? || vitals.backup_at < 26.hours.ago %> +<%# Backups: the encrypted server backups (BackupHealth), the same answer as Settings → Backups and the bell. %> +<% backups = BackupHealth.current %> <%= link_to system_path, class: 'statusbar-item statusbar-vitals', aria: { label: 'Server health' }, title: "Server health, read #{l(vitals.taken_at, format: :short)}. Amber from #{HostVitals::WARN}%, red from #{HostVitals::ALERT}%." do %> CPU <%= vitals.cpu ? "#{vitals.cpu}%" : '—' %> RAM <%= vitals.memory ? "#{vitals.memory}%" : '—' %> Disk <%= vitals.disk ? "#{vitals.disk}%" : '—' %> DB <%= vitals.db_bytes ? number_to_human_size(vitals.db_bytes, precision: 3) : '—' %> - <% if vitals.backup_bytes %>Backup <%= number_to_human_size(vitals.backup_bytes, precision: 3) %>, <%= time_ago_in_words(vitals.backup_at) %> ago<% else %>No backup yet<% end %> + <%= { off: 'Backups off', attention: 'Backups need attention', starting: 'First backup pending' }[backups.state] || "Backup #{time_ago_in_words(backups.last_good_at)} ago" %> <% end %> diff --git a/web/app/views/system/_backups.html.erb b/web/app/views/system/_backups.html.erb index 31d2d8ac6..b7793e6c3 100644 --- a/web/app/views/system/_backups.html.erb +++ b/web/app/views/system/_backups.html.erb @@ -9,7 +9,7 @@ <%= headline[0] %> <% end %> <% unless BackupEncryptionKey.current %> -

No backup key is active, so new backups are refused rather than written unencrypted. <%= link_to 'Create a backup key', settings_backups_path, class: 'underline' %>.

+

No backup key is active, so new backups are refused rather than written unencrypted. <%= link_to 'Turn on backups', settings_backups_path, class: 'underline' %>.

<% end %>
    <% backups.each do |layer| %> diff --git a/web/app/views/time_rules_settings/show.html.erb b/web/app/views/time_rules_settings/show.html.erb index 7035c71ee..e88efb537 100644 --- a/web/app/views/time_rules_settings/show.html.erb +++ b/web/app/views/time_rules_settings/show.html.erb @@ -35,6 +35,7 @@
    > <%= section_heading 'Breaks', hint: 'Reminders and the paid/unpaid rule for meals and rest breaks.' %> +

    A break of <%= ClockBreak::SHORT_BREAK_PAID.in_minutes.to_i %> minutes or less is always paid, whatever it was punched as: federal law counts short breaks as work time. Nothing is ever deducted automatically; only breaks someone actually punched come off.

    <%= number_field_tag field.call('meal_break_after_hours'), rules.meal_break_after_hours&.to_s('F'), id: 'tr_meal_after', min: 0, max: 24, step: 0.5, class: 'input' %>
    <%= number_field_tag field.call('meal_break_minutes'), rules.meal_break_minutes, id: 'tr_meal_minutes', min: 0, max: 120, class: 'input' %>
    diff --git a/web/config/recurring.yml b/web/config/recurring.yml index 5ceab3461..fbe427c51 100644 --- a/web/config/recurring.yml +++ b/web/config/recurring.yml @@ -145,6 +145,11 @@ production: class: BackupJob queue: default schedule: every 6 hours + # Backups off or failing: one notification a day to the instance owner until fixed (BackupHealth). + backup_health_alert: + class: BackupHealthAlertJob + queue: default + schedule: every hour ai_health_check: class: AiHealthCheckJob queue: default diff --git a/web/test/integration/backup_health_alerts_test.rb b/web/test/integration/backup_health_alerts_test.rb new file mode 100644 index 000000000..6d6c87061 --- /dev/null +++ b/web/test/integration/backup_health_alerts_test.rb @@ -0,0 +1,110 @@ +# frozen_string_literal: true + +require 'test_helper' +require_relative '../support/pdf_text' + +# Backups must never fail silently again: the owner sees the state on the dashboard, the bell, the status bar and +# Settings → Backups, and the page walks from "off" to the first good backup. +class BackupHealthAlertsTest < ActionDispatch::IntegrationTest + include PdfText + + CODE = /[A-HJ-NP-Z2-9]{4}-[A-HJ-NP-Z2-9]{4}/ + + setup do + @dir = Pathname(Dir.mktmpdir) + @env = ENV['RUPERT_BACKUP_STATUS_DIR'] + ENV['RUPERT_BACKUP_STATUS_DIR'] = @dir.to_s + BackupHealth.stubs(:alerts_enabled?).returns(true) + @tenant = tenants(:default_tenant) + @owner = users(:regular_user) + @owner.update!(role: 'super_admin', password: 'password123') + Current.tenant = @tenant + post login_path, params: { email: @owner.email, password: 'password123' } + end + + teardown do + ENV['RUPERT_BACKUP_STATUS_DIR'] = @env + FileUtils.rm_rf(@dir) + Current.reset + end + + def write(layer, at:, state: 'ok', message: 'encrypted snapshot ok', fingerprint: BackupEncryptionKey.current&.fingerprint) + @dir.join("#{layer}.json").write({ layer: layer, state: state, at: at.utc.iso8601, message: message, file: '/var/backups/x.age', bytes: 41_000_000, + kept: 30, free_bytes: 90_000_000_000, encrypted: state == 'ok', key_fingerprint: fingerprint }.to_json) + end + + test 'with backups off the owner is told on the dashboard, the bell, Notifications and the status bar' do + get root_path + assert_response :success + assert_select '[data-backup-health-banner=off]', text: /Backups are off.*Nothing is being backed up/m do + assert_select 'a[href=?]', settings_backups_path, 'Turn on backups' + end + assert_select 'a.nav-utility[data-system-alerts="1"][aria-label^="Notifications, 1 alert"] span.bg-rose', '!' + assert_select 'footer.app-statusbar [data-vital=backup][data-backup-health=off].vital-alert', text: 'Backups off' + get notifications_path + assert_select '[data-system-alert]', /Backups are off/ do + assert_select 'a[href=?]', settings_backups_path, 'Turn on backups' + end + end + + test 'people who cannot turn backups on are not shown the alarm' do + @owner.update!(role: 'admin') + get root_path + assert_response :success + assert_select '[data-backup-health-banner]', false + assert_select 'a.nav-utility[data-system-alerts="0"]' + end + + test 'turning backups on: kit, check code, first backup, and the alarm clears' do + write('frequent', at: 1.minute.ago, state: 'failed', message: 'refused: no active backup key', fingerprint: '') + post settings_backup_key_path, params: { acknowledge: '1' } + assert_equal 'application/pdf', response.media_type + code = pdf_text(response.body)[CODE] + pending = BackupEncryptionKey.last + + get settings_backups_path(format: :json) + assert_equal ":#{pending.id}:off", response.parsed_body['signature'], 'the page poller sees the kit was made and reloads' + get settings_backups_path + assert_select '#backup-key[data-backup-health=off]' + assert_select '#backup-key-pending h2', 'Step 2 of 2: confirm you saved the recovery kit' + assert_select '#backup-key-pending', text: /rupert-backup-recovery-kit-#{pending.fingerprint.delete(' ')}\.pdf/ + assert_select '#backup-key-pending input[type=submit][value=?]', 'Start backups' + assert_select '#backup-key-create input[type=submit][value=?]', 'Make a new recovery kit' + + post activate_settings_backup_key_path, params: { check_code: code } + assert_redirected_to settings_backups_path + follow_redirect! + assert_select '.flash-notice', text: /Backups are on\. The first encrypted backup is made within five minutes/ + assert_select '#backup-key[data-backup-health=starting]', text: /The first encrypted backup is being made/ + assert_select '[data-controller=backup-status][data-backup-status-poll-value=true]' + assert_select '[data-last-good-backup]', text: /None yet/ + get root_path + assert_select '[data-backup-health-banner]', false, 'no alarm while the first backup is due' + assert_select 'footer.app-statusbar [data-vital=backup][data-backup-health=starting]', text: 'First backup pending' + + write('frequent', at: 2.minutes.ago) + get settings_backups_path + assert_select '#backup-key[data-backup-health=ok] h2', 'Backups are working' + assert_select '[data-controller=backup-status][data-backup-status-poll-value=false]' + assert_select '[data-last-good-backup]', text: /2 minutes ago.*Encrypted and integrity-checked/m + assert_select '[data-backup-verified]', text: /Not set up on this server/ + assert_select '#backup-key-create h2', 'Replace the backup key' + assert_select '[data-backup-extra=offsite][data-backup-state=missing]', text: /Not set up/ + get root_path + assert_select 'a.nav-utility[data-system-alerts="0"]' + assert_select 'footer.app-statusbar [data-vital=backup][data-backup-health=ok]', text: /Backup 2 minutes ago/ + end + + test 'a backup that stops later raises the alarm again with the reason' do + pair = BackupEncryption::AgeKey.generate + key = BackupEncryptionKey.create!(recipient: pair.recipient, confirmation_digest: BackupEncryptionKey.digest('ABCD-EFGH'), activated_at: 9.days.ago) + write('frequent', at: 3.hours.ago, fingerprint: key.fingerprint) + write('local', at: 2.hours.ago, fingerprint: key.fingerprint) + get root_path + assert_select '[data-backup-health-banner=attention]', text: /Backups need attention.*Rolling snapshot has not run since about 3 hours ago/m do + assert_select 'a[href=?]', settings_backups_path, 'Open Backups' + end + get settings_backups_path + assert_select '#backup-key[data-backup-health=attention] .flash-alert', text: /Last good backup: about 2 hours ago/ + end +end diff --git a/web/test/integration/backup_keys_test.rb b/web/test/integration/backup_keys_test.rb index 4ad28d9a1..a8a956f18 100644 --- a/web/test/integration/backup_keys_test.rb +++ b/web/test/integration/backup_keys_test.rb @@ -46,7 +46,12 @@ def everything_stored assert_response :success assert_equal 'no-store', response.headers['Cache-Control'] assert_select '#backup-key[data-backup-key-state=missing]' - assert_select 'input[type=submit][value=?]', 'Create backup key' + assert_select '#backup-key[data-backup-health=off]', text: /Backups are off.*Nothing is being backed up/m + assert_select '#backup-key-create h2', 'Turn on backups' + assert_select 'form[data-action="submit->backup-status#watch"] input[type=submit][value=?]', 'Turn on backups' + assert_select '#backup-key-pending', false + get settings_backups_path(format: :json) + assert_equal '::off', response.parsed_body['signature'] log = Rails.root.join('log/test.log') log_offset = log.exist? ? log.size : 0 @@ -146,7 +151,7 @@ def everything_stored sign_in get settings_path assert_select 'a[href=?]', settings_backups_path - assert_match(/Server backups are not encrypted yet/, response.body) + assert_match(/Server backups are off/, response.body) get system_path assert_select '[data-backup-key-missing] a[href=?]', settings_backups_path end diff --git a/web/test/integration/employee_timesheets_test.rb b/web/test/integration/employee_timesheets_test.rb index 8925676eb..e5cf505b7 100644 --- a/web/test/integration/employee_timesheets_test.rb +++ b/web/test/integration/employee_timesheets_test.rb @@ -282,6 +282,29 @@ def app_login(user = @user) assert_equal 12000, run.payslips.first.gross_cents end + test 'a short punched meal stays paid through the timesheet import and the pay run; a real meal comes off' do + Current.user = @user + day = Time.zone.parse('2026-07-27 09:00') + quick = ClockShift.create!(employee: @employee, user: @user, employee_key: @employee.id.to_s, employee_name: @employee.name, + employment_type: 'full_time', started_at: day, ended_at: day + 8.hours, request_token: 'short_meal_token_123456') + quick.breaks.create!(kind: 'meal', paid: false, started_at: day + 4.hours, ended_at: day + 4.hours + 15.minutes) + proper = ClockShift.create!(employee: @employee, user: @user, employee_key: @employee.id.to_s, employee_name: @employee.name, + employment_type: 'full_time', started_at: day + 1.day, ended_at: day + 1.day + 8.hours, request_token: 'real_meal_token_1234567') + proper.breaks.create!(kind: 'meal', paid: false, started_at: day + 1.day + 4.hours, ended_at: day + 1.day + 4.5.hours) + workflow = TimesheetWorkflow.new + workflow.change(@sheet, :import_shifts) + hours = @sheet.entries.reload.to_h { |entry| [entry.worked_on.iso8601, entry.hours] } + assert_equal BigDecimal('8'), hours['2026-07-27'], 'a 15-minute meal is paid time' + assert_equal BigDecimal('7.5'), hours['2026-07-28'], 'a 30-minute meal is unpaid' + workflow.change(@sheet, :submit) + @sheet.reload.approve! + People::PayRate.create!(employee: @employee, pay_type: 'hourly', hourly_rate_cents: 2000, effective_on: '2026-07-01') + run = People::PayRun.create!(name: 'Meal week', period_start: '2026-07-27', period_end: '2026-08-02') + People::PayrollCalculator.generate!(run) + assert_equal BigDecimal('15.5'), run.payslips.first.hours + assert_equal 31_000, run.payslips.first.gross_cents + end + test 'clock changes cannot silently leave stale imported hours and removed dates' do Current.user = @user shift = ClockShift.create!(employee: @employee, user: @user, employee_key: @employee.id.to_s, employee_name: @employee.name, diff --git a/web/test/models/time_and_pay/foundation_test.rb b/web/test/models/time_and_pay/foundation_test.rb index 3450f2012..ab9023208 100644 --- a/web/test/models/time_and_pay/foundation_test.rb +++ b/web/test/models/time_and_pay/foundation_test.rb @@ -43,6 +43,58 @@ class TimeAndPayFoundationTest < ActiveSupport::TestCase assert_raises(ActiveRecord::RecordInvalid) { shift.breaks.create!(kind: 'rest', started_at: 9.hours.ago) } end + # FLSA, 29 CFR 785.18-785.19: short breaks are hours worked; only a real meal period may be unpaid. + test 'a punched meal of 20 minutes or less is paid time; a longer one is unpaid' do + at = Time.zone.parse('2026-11-05 09:00') + shift = ClockShift.create!(user: @user, employee: @worker, employee_key: @worker.id.to_s, employee_name: @worker.name, + employment_type: 'part_time', started_at: at, ended_at: at + 8.hours, request_token: SecureRandom.uuid, source: 'pos') + short = shift.breaks.create!(kind: 'meal', paid: false, started_at: at + 2.hours, ended_at: at + 2.hours + 12.minutes) + edge = shift.breaks.create!(kind: 'meal', paid: false, started_at: at + 3.hours, ended_at: at + 3.hours + 20.minutes) + rules = TimeRules.for + assert_not rules.meal_break_paid?, 'meals are unpaid by default' + assert_not short.unpaid?(rules) + assert_not edge.unpaid?(rules), 'exactly 20 minutes is still a short break' + assert_equal 0, shift.reload.unpaid_break_seconds + assert_in_delta 8.0, shift.paid_hours, 0.001 + assert_in_delta 8.0, shift.paid_hours_between(at.beginning_of_day, at.beginning_of_day + 1.day), 0.001 + + long = shift.breaks.create!(kind: 'meal', paid: false, started_at: at + 5.hours, ended_at: at + 5.hours + 20.minutes + 1.second) + assert long.unpaid?(rules) + real = shift.breaks.create!(kind: 'meal', paid: false, started_at: at + 6.hours, ended_at: at + 6.5.hours) + assert real.unpaid?(rules) + assert_equal (20.minutes + 1.second + 30.minutes).to_i, shift.reload.unpaid_break_seconds.to_i + assert_in_delta 8.0 - (50.minutes + 1.second) / 3600.0, shift.paid_hours, 0.001 + assert_in_delta shift.paid_hours, shift.paid_hours_between(at.beginning_of_day, at.beginning_of_day + 1.day), 0.001 + end + + test 'the short-break floor also covers rest breaks a store marked unpaid, and a paid-meal store is unchanged' do + at = Time.zone.parse('2026-11-05 09:00') + shift = ClockShift.create!(user: @user, employee: @worker, employee_key: @worker.id.to_s, employee_name: @worker.name, + employment_type: 'part_time', started_at: at, ended_at: at + 8.hours, request_token: SecureRandom.uuid, source: 'pos') + shift.breaks.create!(kind: 'rest', started_at: at + 1.hour, ended_at: at + 1.hour + 10.minutes) + shift.breaks.create!(kind: 'rest', started_at: at + 4.hours, ended_at: at + 4.hours + 25.minutes) + shift.breaks.create!(kind: 'meal', started_at: at + 6.hours, ended_at: at + 6.hours + 45.minutes) + TimeRules.save!({ 'rest_break_paid' => '0', 'meal_break_paid' => '1' }) + assert_equal 25.minutes.to_i, shift.reload.unpaid_break_seconds.to_i, 'only the 25-minute unpaid rest break comes off' + assert_in_delta 8.0 - 25 / 60.0, shift.paid_hours, 0.001 + end + + test 'a short meal is judged by its whole length when a day boundary clips it, and an open meal turns unpaid after 20 minutes' do + start = Time.zone.parse('2026-11-05 20:00') + midnight = Time.zone.parse('2026-11-06 00:00') + shift = ClockShift.create!(user: @user, employee: @worker, employee_key: @worker.id.to_s, employee_name: @worker.name, + employment_type: 'part_time', started_at: start, ended_at: start + 8.hours, request_token: SecureRandom.uuid, source: 'pos') + shift.breaks.create!(kind: 'meal', paid: false, started_at: midnight - 15.minutes, ended_at: midnight + 15.minutes) + assert_in_delta 3.75, shift.paid_hours_between(midnight - 1.day, midnight), 0.001, '30-minute meal: its 15 minutes before midnight are unpaid' + assert_in_delta 3.75, shift.paid_hours_between(midnight, midnight + 1.day), 0.001 + + open_shift = ClockShift.create!(user: @user, employee: @worker, employee_key: @worker.id.to_s, employee_name: @worker.name, + employment_type: 'part_time', started_at: start + 1.day, request_token: SecureRandom.uuid, source: 'pos') + open_shift.breaks.create!(kind: 'meal', paid: false, started_at: start + 1.day + 2.hours) + assert_equal 0, open_shift.unpaid_break_seconds(at: start + 1.day + 2.hours + 10.minutes) + assert_equal 35.minutes.to_i, open_shift.unpaid_break_seconds(at: start + 1.day + 2.hours + 35.minutes).to_i + end + test 'trusted clock devices authenticate by digest and can be revoked' do device, secret = ClockDevice.register!(name: 'Front register', registered_by: Current.user) assert secret.start_with?(ClockDevice::TOKEN_PREFIX) diff --git a/web/test/services/backup_health_test.rb b/web/test/services/backup_health_test.rb new file mode 100644 index 000000000..0cb5b3012 --- /dev/null +++ b/web/test/services/backup_health_test.rb @@ -0,0 +1,136 @@ +# frozen_string_literal: true + +require 'test_helper' + +# BackupHealth: the single plain answer shown on Settings → Backups, the bell, the dashboard and the status bar. +class BackupHealthTest < ActiveSupport::TestCase + REFUSED = 'refused: no active backup key, so no unencrypted snapshot was written. Create one in Settings → Backups.' + + setup do + @dir = Pathname(Dir.mktmpdir) + @env = ENV['RUPERT_BACKUP_STATUS_DIR'] + ENV['RUPERT_BACKUP_STATUS_DIR'] = @dir.to_s + @now = Time.utc(2026, 11, 11, 18, 0, 0) + end + + teardown do + ENV['RUPERT_BACKUP_STATUS_DIR'] = @env + FileUtils.rm_rf(@dir) + end + + def write(layer, at:, state: 'ok', message: 'ok', encrypted: true, fingerprint: @key&.fingerprint) + @dir.join("#{layer}.json").write({ layer: layer, state: state, at: at.utc.iso8601, message: message, file: '/x', bytes: 1, kept: 2, + free_bytes: 3, encrypted: encrypted, key_fingerprint: fingerprint }.to_json) + end + + def activate_key(at) + pair = BackupEncryption::AgeKey.generate + @key = BackupEncryptionKey.create!(recipient: pair.recipient, confirmation_digest: BackupEncryptionKey.digest('ABCD-EFGH'), activated_at: at) + end + + def health = BackupHealth.new(now: @now) + + test 'without an active key backups are off, whatever the scripts last said' do + write('frequent', at: @now - 2.minutes, state: 'failed', message: REFUSED, encrypted: false) + assert_equal :off, health.state + assert health.alert? + assert_equal 'Backups are off', health.title + assert_equal 'Turn on backups', health.link_label + assert_match(/Nothing is being backed up/, health.summary) + assert_nil health.last_good_at + end + + test 'a key that was just turned on waits for each layer once instead of alarming on the old refusals' do + activate_key(@now - 2.minutes) + write('frequent', at: @now - 3.minutes, state: 'failed', message: REFUSED, encrypted: false) + write('local', at: @now - 4.hours, state: 'failed', message: REFUSED, encrypted: false) + write('verify', at: @now - 3.days, state: 'failed', message: 'no encrypted backup found') + assert_equal :starting, health.state + assert_not health.alert? + assert_equal %w[frequent local verify], health.waiting.map(&:key) + + write('frequent', at: @now - 1.minute) + assert_equal :ok, health.state + assert_equal (@now - 1.minute).to_i, health.last_good_at.to_i + assert_nil health.verified_at + assert_equal %w[local verify], health.waiting.map(&:key) + assert_match(/Last good backup: 1 minute ago/, health.summary) + end + + test 'a layer that still has not run after one interval of its schedule is a problem' do + activate_key(@now - 30.minutes) + write('frequent', at: @now - 31.minutes, state: 'failed', message: REFUSED, encrypted: false) + assert_equal :attention, health.state + assert health.alert? + assert_equal %w[frequent], health.problems.map(&:key) + assert_match(/Rolling snapshot failed 31 minutes ago: refused: no active backup key/, health.summary) + assert_match(/There is no good backup yet/, health.summary) + end + + test 'a failure reported after the key was turned on alarms at once, and a stale layer says when it last ran' do + activate_key(@now - 10.days) + write('frequent', at: @now - 1.minute, state: 'failed', message: 'skipped: 100 MB free on /var/backups, need 6000 MB. Free disk space.') + write('local', at: @now - 9.hours) + assert_equal :attention, health.state + assert_equal %w[frequent local], health.problems.map(&:key) + assert_match(/Rolling snapshot failed 1 minute ago: skipped: 100 MB free/, health.summary) + assert_match(/Local backup set has not run since about 9 hours ago \(expected every 6 hours\)/, health.summary) + assert_equal 'Open Backups', health.link_label + end + + test 'healthy: fresh encrypted backups, the weekly check passed, and extra layers never alarm' do + activate_key(@now - 10.days) + write('frequent', at: @now - 3.minutes) + write('local', at: @now - 2.hours) + write('offhost', at: @now - 2.hours) + write('verify', at: @now - 2.days) + write('offsite', at: @now - 30.days) + state = health + assert_equal :ok, state.state + assert_not state.alert? + assert_equal (@now - 3.minutes).to_i, state.last_good_at.to_i + assert_equal (@now - 2.days).to_i, state.verified_at.to_i + assert_equal({ 'offsite' => 'stale', 'restore' => 'missing' }, state.extras.to_h { |check| [check.key, check.state] }) + end + + test 'the off-host copy and weekly check count only once installed, and a report without encryption evidence fails' do + activate_key(@now - 10.days) + write('frequent', at: @now - 3.minutes) + write('local', at: @now - 2.hours) + assert_equal :ok, health.state, 'never-reported host layers are not installed, not failing' + write('offhost', at: @now - 1.hour, encrypted: false) + assert_equal :attention, health.state + assert_match(/does not confirm an age-encrypted backup/, health.summary) + end + + test 'alerts are for super admins on real servers only' do + owner = users(:regular_user) + owner.update!(role: 'super_admin') + assert_nil BackupHealth.alert_for(owner), 'off in the test environment unless switched on' + BackupHealth.stubs(:alerts_enabled?).returns(true) + assert_equal :off, BackupHealth.alert_for(owner).state + owner.update!(role: 'admin') + assert_nil BackupHealth.alert_for(owner) + assert_nil BackupHealth.alert_for(nil) + end + + test 'the hourly job notifies each instance owner once a day while backups are off' do + Current.tenant = tenants(:default_tenant) + owner = users(:regular_user) + owner.update!(role: 'super_admin') + assert_no_difference('Notification.unscoped.count') { BackupHealthAlertJob.perform_now } + BackupHealth.stubs(:alerts_enabled?).returns(true) + assert_difference(-> { Notification.unscoped.where(user_id: owner.id, kind: 'backup_health').count }, 1) do + 2.times { BackupHealthAlertJob.perform_now } + end + note = Notification.unscoped.find_by!(user_id: owner.id, kind: 'backup_health') + assert_equal 'Backups are off', note.title + assert_equal '/settings/backups', note.url + travel(1.day) { assert_difference(-> { Notification.unscoped.where(kind: 'backup_health', user_id: owner.id).count }, 1) { BackupHealthAlertJob.perform_now } } + + activate_key(2.minutes.ago) + assert_no_difference('Notification.unscoped.count') { BackupHealthAlertJob.perform_now } + ensure + Current.reset + end +end