diff --git a/.gitignore b/.gitignore index 5cd0861747..9c8213d1bc 100644 --- a/.gitignore +++ b/.gitignore @@ -84,3 +84,4 @@ debug-storybook.log .devcontainer/devcontainer-lock.json *.lscache +.gstack/ diff --git a/openspec/changes/add-usage-budget-controls/design.md b/openspec/changes/add-usage-budget-controls/design.md new file mode 100644 index 0000000000..9f066398bf --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/design.md @@ -0,0 +1,887 @@ +# Design: Add Usage Budget Controls + +## Overview + +Add three complementary usage budget controls: + +1. **Organization budget alert emails** — warnings sent when organization accepted event usage crosses configured percentage thresholds. +2. **Automatic smart project throttling** — automatic project-isolated throttling/sampling when a project spikes enough to threaten organization usage. +3. **Optional project event budgets** — user-configured project-level accepted event caps. + +These are event-usage controls, not generic API request throttling controls. + +The implementation should extend existing usage and notification paths: + +- `UsageService` remains responsible for usage totals, budget threshold detection, smart throughput calculations, and limit calculations. +- Existing organization overage notification behavior remains unchanged. +- `OverageMiddleware` remains the event-post gate for coarse request-level enforcement. +- `EventPostsJob` / event post processing is where project-level sampled acceptance should happen after events are parsed. +- Existing organization usage enforcement remains the outer hard limit. +- Existing `ThrottlingMiddleware` remains unchanged for this feature. +- Budget alert and smart throttling emails are asynchronous side effects and must not block event ingestion. + +## Existing code paths + +### Event post enforcement + +`OverageMiddleware` currently: + +1. Skips non-event posts. +2. Gets the default organization id from the authenticated request. +3. Rejects when event submission is globally disabled. +4. Rejects invalid/oversized posts. +5. Calls `UsageService.GetEventsLeftAsync(organizationId)`. +6. Rejects organization overage. +7. Allows the request to continue. + +This change keeps that coarse gate but does not rely on it as the only project-level enforcement layer. + +### Usage tracking + +`UsageService.IncrementTotalAsync(organizationId, projectId, eventCount)` already increments both: + +- organization bucket total +- project bucket total + +Existing cache key helpers accept an optional `projectId`. + +Use these existing counters for organization budget alerts, smart project throttling, and project event budgets. Do not introduce a parallel accepted-event counter. + +### Event post processing + +`EventPostsJob` already: + +- loads project and organization +- parses the post into events +- calculates how many events can be processed +- processes only allowed events +- increments blocked usage for events over plan limit +- increments accepted total usage for processed events +- increments discarded usage for discarded events + +Extend this path for sampled project-level acceptance. + +### Existing organization notice email path + +Existing flow for monthly/hourly overage notices: + +1. `UsageService` publishes `PlanOverage`. +2. `EnqueueOrganizationNotificationOnPlanOverage` subscribes to `PlanOverage`. +3. It enqueues `OrganizationNotificationWorkItem`. +4. `OrganizationNotificationWorkItemHandler` loads organization and users. +5. The handler sends organization emails only to users with verified email addresses and email notifications enabled. +6. `Mailer.SendOrganizationNoticeAsync` renders the organization notice email. + +Budget alerts and project smart throttling notifications should follow this pattern with new message/work item/mailer/template types. + +--- + +## Automatic Smart Project Throttling + +### Overview + +Automatic smart project throttling is the default guardrail for sudden event spikes. + +It addresses historical feedback that: + +- organization-level throttling can punish all projects when one project is noisy +- throttling should notify users when applied +- throttling should still allow a small percentage of events through for troubleshooting +- users should not need to configure many options before this protection works + +This is separate from optional project event budgets: + +- Smart throttling is automatic and adaptive. +- Project event budgets are explicit user-configured caps. + +### Design principle: minimal configuration + +Smart project throttling should not require users to configure project percentages, per-project caps, or per-stack settings. + +The UI may expose status and explanation, but not a large set of tuning options. + +### Throughput calculation + +Smart throttling should use remaining monthly allowance and remaining time in the monthly usage period. + +Preferred formula shape: + +```text +maxThroughput(window) = eventsLeftInMonth / windowsLeftInMonth * burstMultiplier +``` + +Where: + +* `eventsLeftInMonth` is the organization's current effective monthly allowance minus accepted usage. +* `windowsLeftInMonth` is the number of smart-throttling windows remaining in the period. +* `burstMultiplier` preserves existing burst tolerance behavior. A default value such as 10 is acceptable if it matches existing Exceptionless throttle behavior. +* The evaluation window should align with existing usage bucket behavior where possible. + +This is intentionally different from a static `monthlyPlanLimit / timeInMonth` calculation. Customers should not be throttled harshly late in the month when they still have substantial allowance remaining. + +### Project isolation + +When organization-level smart throttling would apply, the system should identify the project or projects contributing to the spike and throttle those projects first. + +Observable behavior: + +* One noisy project should not cause all other projects in the organization to stop ingesting events. +* Other projects should continue under organization-level usage enforcement unless they also exceed smart-throttling criteria. +* If the organization hard monthly limit is exhausted, all projects remain subject to existing organization overage behavior. + +### Sampling while throttled + +When a project is smart-throttled and the organization still has remaining monthly allowance, Exceptionless should accept a small sample of events from that project. + +Recommended v1 behavior: + +```text +sampleRate = 1% to 5% +``` + +Implementation may use a fixed default such as 1% or 5% for v1, as long as behavior is deterministic/testable and does not expose many configuration options. + +Sampling goals: + +* Preserve visibility into whether the problem is still occurring. +* Preserve visibility into whether a deployed fix reduced occurrences. +* Avoid accepting 0 events from a throttled project while the organization still has monthly allowance. +* Avoid allowing a noisy project to consume the entire organization allowance. + +### Event processing location + +Smart throttling and sampled acceptance should be implemented in the event processing path after event posts are parsed, not solely in `OverageMiddleware`. + +Reason: + +* `OverageMiddleware` can reject the entire request before the event count is known. +* It cannot preserve a 1–5% sample of events from a batch. +* `EventPostsJob` already parses the post, calculates `eventsToProcess`, processes only allowed events, and increments blocked usage for the rest. +* Extending this job/path allows project-level partial acceptance while preserving existing usage accounting. + +### Event selection + +When only part of a batch can be accepted, the implementation should avoid always taking only the first events if doing so would bias troubleshooting data. + +Acceptable v1 approaches: + +* deterministic sampling by event id/hash +* random sampling with stable seed per post +* first-N selection only if no safe sampling utility exists, but this should be documented as a known limitation + +### Smart throttling state + +The system may store project throttling state in cache to avoid recalculating expensive decisions on every event. + +Suggested key shape: + +```text +usage-smart-throttle:{yyyyMM}:{organizationId}:{projectId} +``` + +Suggested value: + +```json +{ + "is_throttled": true, + "sample_rate": 0.01, + "effective_until_utc": "..." +} +``` + +TTL: short window, aligned with usage bucket/window duration. + +If cached throttle state expires, the system can re-evaluate based on current usage counters. + +### Smart throttling notification + +When a project enters smart-throttled state, eligible organization users should receive an email notification. + +Notification behavior: + +* Send at most once per project per throttling period or cooldown window. +* Reuse the existing organization notification eligibility rules. +* Do not send repeated emails for every event post while the project remains throttled. +* Include project name, organization name, current accepted usage, current limit context, and a link to project usage. + +Suggested message/work item: + +```csharp +public record ProjectSmartThrottleApplied +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Suggested mailer method: + +```csharp +Task SendProjectThrottledNoticeAsync( + User user, + Organization organization, + Project project, + double sampleRate, + int currentEventCount, + int eventLimit); +``` + +### Relationship to organization overage + +Smart project throttling is not a substitute for organization hard overage enforcement. + +Behavior: + +* If organization monthly allowance remains, smart-throttled projects may still have sampled events accepted. +* If organization monthly allowance is exhausted, existing organization overage behavior applies. +* Project sampling must not allow accepted event totals to exceed the organization hard monthly allowance. + +### Relationship to optional project event budgets + +If a project has an explicit project event budget, that budget participates in the same allowed-event calculation. + +Recommended order: + +```text +organization hard allowance +↓ +explicit project budget, if configured +↓ +automatic smart throttling sample allowance, if active +``` + +The final number of events accepted from a batch should be the minimum allowed by all applicable controls. + +### Relationship to budget alert emails + +Smart throttling email notification is distinct from organization budget alert emails. + +* Budget alerts notify when organization accepted usage crosses configured percentages. +* Smart throttling notifications notify when the system starts sampling/throttling a noisy project. +* A project can be smart-throttled before any budget alert threshold is crossed. +* A budget alert can fire without any project being smart-throttled. + +--- + +## Organization Budget Alert Emails + +### Overview + +Add configurable organization-level event budget alert emails. These alerts warn users before monthly plan overage, based on accepted event usage crossing configured percentage thresholds. + +Budget alerts are informational only. They do not block ingestion and do not replace organization overage enforcement, smart throttling, or project event budgets. + +### Domain model + +Add budget alert settings to Organization. + +```csharp +public class Organization +{ + // existing properties... + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} +``` + +Add a new model: + +```csharp +namespace Exceptionless.Core.Models; + +public class OrganizationBudgetAlertSettings +{ + public bool Enabled { get; set; } + + /// + /// Percentage thresholds of the organization's effective monthly event allowance. + /// Example: [50, 80, 90]. + /// + public SortedSet Thresholds { get; set; } = []; +} +``` + +### Defaults + +Existing organizations: + +```text +budget_alert_settings = null +``` + +Null means disabled. + +New organizations: + +```text +budget_alert_settings = null +``` + +The UI may suggest default thresholds [50, 80], but alerts must not be enabled until a user explicitly saves settings. + +### Validation + +Validation rules: + +* `BudgetAlertSettings == null` is valid. +* `Enabled == false` with empty thresholds is valid. +* If enabled, thresholds must contain at least one value. +* Each threshold must be greater than 0. +* Each threshold must be less than 100. +* Threshold 100 is not allowed because existing monthly overage notification already handles reaching/exceeding the plan limit. +* Duplicate thresholds must be removed. +* Thresholds must be stored sorted ascending. +* Percentage budget alerts must be rejected or disabled for unlimited organizations. + +### API contract + +Add `BudgetAlertSettings` to organization update and view DTOs. + +Preferred cleanup: + +```csharp +public record UpdateOrganization +{ + public string Name { get; set; } = null!; + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +} +``` + +Then update `OrganizationController` generic usage from: + +```csharp +RepositoryApiController +``` + +to: + +```csharp +RepositoryApiController +``` + +If minimizing backend churn is preferred, `BudgetAlertSettings` can be added to `NewOrganization` because the current controller uses `NewOrganization` for both create and update. However, adding `UpdateOrganization` is cleaner because `NewOrganization` currently only contains the required organization name. + +Add to `ViewOrganization`: + +```csharp +public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } +``` + +Serialized JSON uses the existing snake_case policy. + +### Threshold calculation + +Use the same effective organization allowance as existing usage enforcement: + +```csharp +effectiveOrganizationAllowance = organization.GetMaxEventsPerMonthWithBonus(timeProvider) +``` + +If effective allowance is negative/unlimited: + +```text +budget alert thresholds are inactive +``` + +Threshold event count: + +```csharp +thresholdEventCount = ceil(effectiveOrganizationAllowance * thresholdPercent / 100) +``` + +### Triggering alerts + +Budget alert checks should run in the accepted-event usage increment path. + +Recommended location: + +```text +UsageService.IncrementTotalAsync +``` + +Observable behavior must be: + +* Alert fires when usage crosses threshold. +* Alert does not fire before threshold. +* Alert does not fire repeatedly after threshold. +* Multiple crossed thresholds can fire if a large batch jumps over more than one threshold. + +### Deduplication + +Each organization threshold must email at most once per monthly usage period. + +Recommended cache key: + +```text +usage-budget-alert:{yyyyMM}:{organizationId}:{threshold} +``` + +TTL: until end of current monthly usage period + safety buffer. + +The cache key should be set before or atomically with enqueueing the work item to reduce duplicate sends under concurrency. + +### Message and work item + +Add a new message: + +```csharp +namespace Exceptionless.Core.Messaging.Models; + +public record OrganizationBudgetAlert +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Add a new work item: + +```csharp +namespace Exceptionless.Core.Models.WorkItems; + +public record OrganizationBudgetAlertWorkItem +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} +``` + +Add a startup subscriber: + +```text +EnqueueOrganizationBudgetAlertOnUsageThreshold +``` + +### Work item handler + +Add: + +```text +OrganizationBudgetAlertWorkItemHandler +``` + +Behavior: + +* Load organization by id. +* Load users in organization. +* Send only to users with verified email addresses. +* Send only to users with `EmailNotificationsEnabled == true`. +* Use the same eligibility rules as existing organization notices. +* Do not send if organization no longer exists. +* Re-check current organization budget alert settings before sending. +* If budget alerts are disabled or the threshold was removed, skip sending. + +### Mailer + +Add to `IMailer`: + +```csharp +Task SendOrganizationBudgetAlertAsync( + User user, + Organization organization, + int threshold, + int thresholdEventCount, + int currentEventCount, + int eventLimit); +``` + +Add implementation in `Mailer`. + +Add template: + +```text +src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html +``` + +Template must include: + +* organization name +* threshold percentage +* threshold event count +* current accepted event count +* organization event limit +* remaining event count +* link to organization usage +* link to billing/plan management when applicable +* link to notification settings + +Do not include event payload data. + +--- + +## Optional Project Event Budgets + +### Overview + +Add optional project-level event budgets enforced during event processing. This is an event-usage control, not generic API request throttling. + +### Domain model + +Add a nullable event budget configuration to Project. + +```csharp +public class Project +{ + // existing properties... + public ProjectIngestLimit? IngestLimit { get; set; } +} +``` + +Add a new model in `src/Exceptionless.Core/Models/ProjectIngestLimit.cs`: + +```csharp +namespace Exceptionless.Core.Models; + +public class ProjectIngestLimit +{ + public ProjectIngestLimitType Type { get; set; } + public int? FixedLimit { get; set; } + public decimal? PercentOfOrganizationLimit { get; set; } +} + +public enum ProjectIngestLimitType +{ + Fixed = 0, + PercentOfOrganizationLimit = 1 +} +``` + +### Naming + +Use `IngestLimit`, not `RateLimit`, because the feature limits accepted event volume for a project, not generic request rate. + +Use `ProjectIngestLimit`, not `ProjectQuota`, because this is a cap, not a reservation. + +### API contract + +Add `IngestLimit` to: + +```text +src/Exceptionless.Web/Models/Project/UpdateProject.cs +src/Exceptionless.Web/Models/Project/ViewProject.cs +``` + +Recommended DTO shape: + +```csharp +public record UpdateProject +{ + public string Name { get; set; } = null!; + public bool DeleteBotDataEnabled { get; set; } + public ProjectIngestLimit? IngestLimit { get; set; } +} + +public class ViewProject +{ + // existing properties... + public ProjectIngestLimit? IngestLimit { get; set; } + public int? EffectiveIngestLimit { get; set; } + public bool IsSmartThrottled { get; set; } + public double? SmartThrottleSampleRate { get; set; } +} +``` + +### Project update behavior + +Use existing `PATCH /api/v2/projects/{id}`. Do not add a new endpoint. + +Clear project budget: + +```json +{ + "ingest_limit": null +} +``` + +Fixed budget: + +```json +{ + "ingest_limit": { + "type": "fixed", + "fixed_limit": 20000 + } +} +``` + +Percentage budget: + +```json +{ + "ingest_limit": { + "type": "percent_of_organization_limit", + "percent_of_organization_limit": 20 + } +} +``` + +### Validation + +General validation: + +* `ingest_limit == null` is valid and means Off. +* Unknown limit type is invalid. +* Limit values must not be negative. +* Ingest limit validation must not change existing project name validation. + +For Fixed: + +* `fixed_limit` is required. +* `fixed_limit` must be greater than 0. +* `percent_of_organization_limit` should be ignored or cleared server-side. +* Backend should not reject a fixed limit greater than the current organization limit. The effective cap is clamped during evaluation. + +For PercentOfOrganizationLimit: + +* `percent_of_organization_limit` is required. +* Percentage must be greater than 0. +* Percentage must be less than or equal to 100. +* `fixed_limit` should be ignored or cleared server-side. +* If the organization currently has an unlimited event allowance, the API should reject setting a percentage cap because a percentage of unlimited is not meaningful. +* If an existing percentage cap later encounters an unlimited organization due to a plan change, enforcement should treat the percentage cap as inactive until the organization returns to a finite allowance. + +### Effective project limit calculation + +Use the same effective organization max that current usage enforcement uses. + +```csharp +private static int? GetEffectiveProjectIngestLimit(ProjectIngestLimit? limit, int organizationMaxEvents) +{ + if (limit is null) + return null; + + return limit.Type switch + { + ProjectIngestLimitType.Fixed when limit.FixedLimit is > 0 => + organizationMaxEvents < 0 + ? limit.FixedLimit.Value + : Math.Min(limit.FixedLimit.Value, organizationMaxEvents), + ProjectIngestLimitType.PercentOfOrganizationLimit + when organizationMaxEvents > 0 && limit.PercentOfOrganizationLimit is > 0 => + Math.Max(1, (int)Math.Ceiling(organizationMaxEvents * (limit.PercentOfOrganizationLimit.Value / 100m))), + _ => null + }; +} +``` + +### UsageService API + +Add project-aware/budget-aware event allowance calculation. + +The final event count accepted from a parsed batch should be the minimum allowed by: + +1. organization hard allowance +2. explicit project event budget, if configured +3. automatic smart throttling sampled allowance, if active + +Avoid a design that returns only a Boolean. The event processing path needs a count. + +Suggested result: + +```csharp +public record EventIngestAllowanceResult +{ + public int EventsAllowed { get; init; } + public int OrganizationEventsLeft { get; init; } + public int? ProjectEventsLeft { get; init; } + public int? EffectiveProjectLimit { get; init; } + public bool IsSmartThrottled { get; init; } + public double? SmartThrottleSampleRate { get; init; } + public string? Reason { get; init; } +} +``` + +Suggested method: + +```csharp +public Task GetEventIngestAllowanceAsync( + string organizationId, + string projectId, + int submittedEventCount); +``` + +Behavior: + +* Compute organization events left using existing organization usage logic. +* If organization events left is <= 0, allow 0 events. +* Compute explicit project budget events left if configured. +* Compute smart throttling state/sample allowance if project is noisy. +* Return the minimum allowed count. +* Never return an allowed count greater than submitted event count. +* Never return an allowed count greater than organization events left. + +### Refactor current events-left logic + +The current organization-only method should be refactored into a shared helper: + +```csharp +private async Task GetEventsLeftAsync( + string organizationId, + string? projectId, + int maxEventsPerMonth) +``` + +Use existing total and bucket cache keys with optional project id. + +### OverageMiddleware behavior + +`OverageMiddleware` should continue handling request-level event-post gates: + +* non-event posts are skipped +* missing organization context is rejected +* globally disabled event submission is rejected +* missing content length is rejected +* oversized posts are rejected +* organization hard overage can still be rejected early when there are no events left + +Project-level smart throttling and sampled project-budget behavior should not be implemented solely as early middleware rejection because early rejection drops the entire post and cannot preserve a sample. + +If `OverageMiddleware` detects that the organization has no remaining monthly allowance, it may preserve the existing organization overage rejection behavior. + +If the organization has remaining allowance, project-level controls should be evaluated later in `EventPostsJob` / `UsageService` after events are parsed. + +### Status code behavior + +* Organization monthly/bucket hard overage remains `402 PaymentRequired` where existing behavior already returns that status. +* Oversized submissions continue to return `413 RequestEntityTooLarge`. +* Missing content length continues to return `411 LengthRequired`. +* Missing organization context continues to return `401 Unauthorized`. +* Disabled event submission continues to return `503 ServiceUnavailable`. +* Project smart throttling should not require changing the HTTP response for accepted queued posts; the job may accept a sample and block/discard the rest asynchronously. +* Project hard rejection may return `429 TooManyRequests` only in code paths where the server can make an immediate project-level decision without sacrificing sampled acceptance. For the normal queued event-post path, project throttling should be represented by partial processing and blocked/discarded usage accounting rather than a full-request 429. + +### Headers + +Do not repurpose existing generic API throttle headers for project monthly ingest caps or smart project throttling in this change. + +### API throttling middleware + +Do not change `ThrottlingMiddleware` for this feature. + +### Storage and indexing + +Persisted data: + +* Existing organizations have `BudgetAlertSettings = null`. +* Existing projects have `IngestLimit = null`. +* No migration/backfill is required. +* Null/default behavior must preserve current behavior. + +Elasticsearch: + +* Do not add Elasticsearch mappings for `Organization.BudgetAlertSettings` or `Project.IngestLimit` unless filtering/sorting/searching by those fields is required. +* Avoid reindexing. + +### Organization Usage UI + +Add budget alert settings to: + +```text +src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte +``` + +Place above the usage chart. + +Suggested card: + +```text +Budget Alerts +Receive an email when your organization reaches selected percentages of its monthly event allowance. +[ ] Enable budget alerts +Thresholds +[50] [80] [90] +[+ Add threshold] +Current plan allowance: 100,000 events +50% = 50,000 events +80% = 80,000 events +Alerts are sent once per threshold per monthly usage period to organization users who have email notifications enabled. +``` + +### Project Usage UI + +Add project event budget and smart throttling status to: + +```text +src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte +``` + +Copy: + +```text +Project Event Budget +Protect your organization's monthly event allowance by limiting how many accepted events this project can use. +This is a cap, not a reservation. Other projects are not guaranteed unused capacity. +``` + +Smart throttling status copy when active: + +```text +Smart throttling is currently active for this project. Exceptionless is accepting a small sample of events so you can continue troubleshooting while protecting your organization's monthly event allowance. +``` + +Do not expose many tuning knobs. + +### Project usage chart behavior + +If `effective_ingest_limit` is not null, display the dashed limit line as the project limit. + +If `effective_ingest_limit` is null, keep showing the organization limit. + +Show smart throttling status as text or badge if active. + +### Accessibility + +* Slider controls must have accessible labels. +* Numeric inputs must have associated labels. +* Computed threshold/effective limit text should update in screen-reader-friendly text. +* Do not rely on color only. +* Save success/error must use existing toast/error patterns. + +### Security + +* Only users authorized to update the organization may update budget alert settings. +* Only users authorized to update the project may update project event budgets. +* Budget alert and smart throttling emails are sent only to verified users with email notifications enabled. +* Project budgets do not grant additional access. +* Users cannot increase organization allowance through project budgets. +* Smart throttling must not allow accepted usage beyond organization hard allowance. + +### Privacy + +No new personal event data is collected. + +Budget alert and smart throttling emails include organization/project names and aggregate usage numbers only. They must not include event payload data, stack data, user-identifying event details, or project-specific event contents. + +### Failure modes + +**Budget alert email queue failure** + +If mail queue enqueue fails, do not block event ingestion. + +**Smart throttling email queue failure** + +If smart throttling notification enqueue fails, do not block event ingestion. + +**Dedupe cache failure** + +If dedupe cache fails, prefer avoiding ingestion failure. The implementation may skip alert sending rather than risk duplicate emails or ingestion failure. + +**Organization/project settings changed after message publication** + +Re-load settings in the work item handler. If the relevant notification is disabled or no longer applicable, skip sending. + +**Project lookup fails** + +If the project cannot be loaded during project-level evaluation, do not block solely due to project-level settings. Continue organization-level enforcement. + +**Invalid persisted settings** + +Treat invalid persisted settings as inactive for enforcement/notification. Do not crash event ingestion. diff --git a/openspec/changes/add-usage-budget-controls/proposal.md b/openspec/changes/add-usage-budget-controls/proposal.md new file mode 100644 index 0000000000..e4cee16752 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/proposal.md @@ -0,0 +1,224 @@ +# Proposal: Add Usage Budget Controls + +## Summary + +Add usage budget controls that help organizations both **warn before overage** and **prevent noisy projects from consuming the full organization event allowance**. + +This change includes three related capabilities: + +1. **Organization budget alert emails** — configurable percentage thresholds such as 50% and 80% of the organization's monthly event allowance. When accepted event usage crosses a configured threshold, Exceptionless sends a budget alert email to eligible organization users. +2. **Automatic smart project throttling** — when a project/environment starts sending enough events to threaten the organization's budget, Exceptionless should automatically isolate throttling to the noisy project where possible, instead of stopping ingestion for the entire organization. +3. **Optional project event budgets** — project-level budget caps that allow customers to explicitly limit how many accepted events a project can consume in the monthly usage period. + +Budget alerts are warning controls. Smart project throttling is the default automatic guardrail. Project event budgets are optional prevention controls for customers who want explicit per-project limits. + +This change addresses customer feedback asking for budget alerts before plan overage: + +> Hi support team +> +> Is there any budget alerts we can configure in terms of the number of events? For example I want to receive an email when i have reached 50% of my plan, and then when 80% of my plan. +> +> This month we have exceeded twice our plan, and we should have a mechanism to prevent this situation. + +It also addresses historical customer feedback from issue #112 requesting smarter throttling: + +- project-level throttling +- email notification when throttling is applied +- 1–5% of errors delivered even under throttling + +## User-visible behavior + +### Organization budget alerts + +Users managing an organization can configure event budget alert thresholds from Organization Settings → Usage. + +Supported behavior: + +- Alerts are disabled by default for existing organizations. +- Users can enable budget alerts and configure one or more percentage thresholds. +- Example thresholds: 50%, 80%, 90%. +- A threshold represents the organization's current monthly event allowance, including active bonus events if those are included by existing usage enforcement. +- When accepted usage crosses a configured threshold for the first time in a monthly usage period, eligible organization users receive an email. +- Each threshold is emailed at most once per monthly usage period. +- Alerts are not sent for unlimited organizations because a percentage of unlimited usage is undefined. +- Alerts do not block event ingestion. +- Existing monthly overage emails remain unchanged. + +### Automatic smart project throttling + +Exceptionless should automatically protect organization usage when one project starts sending a large event spike. + +Supported behavior: + +- Users should not be required to configure smart throttling. +- The system should avoid exposing a large number of throttling options. +- When throttling is needed, it should be scoped to the noisy project where possible rather than blocking all projects in the organization. +- While a project is throttled, Exceptionless should still accept a small sample of events from that project when the organization has remaining monthly allowance. +- The accepted sample should preserve troubleshooting visibility so users can tell whether the issue is still happening and whether a deployed fix helped. +- When smart throttling is applied, eligible organization users should receive an email notification. +- Smart throttling should use remaining monthly allowance and remaining time in the period, not only the static plan size, to avoid throttling customers who still have substantial monthly allowance left. + +### Optional project event budgets + +Users managing a project can configure a Project Event Budget from Project Settings → Usage. + +Supported modes: + +- **Off**: no project-specific cap; current behavior is preserved. +- **Fixed limit**: cap this project at a configured number of accepted events per monthly usage period. +- **Percentage of organization limit**: cap this project at a percentage of the organization's current monthly event allowance. + +Examples: + +- A project with no configured ingest limit can consume organization events as it does today. +- A project with a fixed limit of `20,000` can accept up to `20,000` events in the current monthly usage period, subject to the organization still having events available. +- A project with a `20%` limit in an organization with `100,000` monthly events has an effective project cap of `20,000`. +- If the organization plan changes, percentage-based project caps are recalculated from the organization's current event allowance. + +When a project limit is reached: + +- Event submissions for that project are limited or sampled. +- Event submissions for other projects in the organization are not limited solely because this project reached its cap. +- Blocked event usage is recorded for the project and organization. +- Existing organization overage behavior remains unchanged. + +## Classification + +- **Type:** Feature +- **Affected areas:** Backend/API, billing/usage enforcement, organization model, project model, Redis/cache usage counters, mail/work items, Svelte UI, generated API types, tests +- **OpenSpec justification:** This change affects event ingestion behavior, organization/project usage limits, smart throttling behavior, email notification behavior, API response contracts, persisted organization/project data, Redis/cache usage state, SDK/client expectations, jobs, and Svelte UI/API contracts. + +## Current implementation context + +Exceptionless currently enforces organization event overage in `OverageMiddleware` for event posts. The middleware checks the authenticated organization id, rejects disabled event submission, rejects oversized submissions, and calls `UsageService.GetEventsLeftAsync(organizationId)` before allowing the event post to continue. + +`UsageService` already tracks both organization and project usage counters. `IncrementTotalAsync` increments organization totals and project totals using existing cache keys that accept an optional `projectId`. + +`EventPostsJob` already parses queued event posts, calculates how many events may be processed, processes only allowed events, increments blocked usage for events over the limit, and increments total/discarded usage for processed events. This is the correct place to implement project-level sampled acceptance. + +Exceptionless already has an organization overage email path. Existing usage overage publishes `PlanOverage`, which enqueues `OrganizationNotificationWorkItem`, and the work item handler sends organization notice emails to verified users with email notifications enabled. Budget alerts and project throttling notifications should reuse that mail/work-item pattern with distinct alert types rather than overloading monthly/hourly overage booleans. + +The existing generic API throttling middleware remains separate. It limits API requests over a short period and is not the same as event-plan usage enforcement. + +## Affected areas + +### Backend/API + +- Add organization usage budget alert settings to organization model/API responses. +- Add project event budget fields to project model/API responses. +- Extend usage tracking to publish budget-alert notifications when configured thresholds are crossed. +- Extend usage/event post processing to compute smart project throttling and sampled acceptance. +- Extend usage/event post processing to compute optional project budget limits. +- Preserve existing organization-level overage behavior and status codes. +- Avoid implementing project sampled throttling solely in middleware. + +### Redis/cache + +- Reuse existing organization/project usage counter keys. +- Add budget alert dedupe state keyed by organization, threshold, and monthly usage period. +- Add smart project throttling state or notification dedupe state keyed by organization/project/window where useful. +- Preserve existing usage counter TTL behavior. +- Do not add new project usage counter families unless required for efficient limit evaluation. + +### Mail/work items + +- Reuse the existing organization notification pattern. +- Add budget-alert-specific message/work item/template. +- Add project-smart-throttling-specific message/work item/template. +- Send emails only to verified users with email notifications enabled, matching existing organization notice behavior. + +### Svelte UI + +- Add organization budget alert controls to the Svelte Organization Settings → Usage page. +- Add project event budget controls to the Svelte Project Settings → Usage page. +- Surface smart throttling status where feasible, especially on Project Usage. +- Show computed thresholds/caps and current usage. +- Update the project usage chart to display the project limit when configured. + +### Legacy Angular UI + +- Legacy Angular parity is out of scope for this change unless the release path requires these settings to be available in the legacy UI. +- This change must not break existing legacy Angular organization or project management behavior. + +### SDK/client compatibility + +- Existing event submission authentication mechanisms must continue to work. +- Existing project API keys and user tokens must continue to work. +- Organization overage behavior must remain compatible. +- Smart project throttling for queued event posts should prefer asynchronous sampled processing over changing the HTTP submission response. +- Budget alert emails must be asynchronous side effects and must not change normal event submission success responses. + +### Tests + +- Add backend unit/integration tests for budget threshold calculation, smart throughput calculation, sampled acceptance, and project budget computation. +- Add usage service tests for threshold crossing, dedupe, organization cap, project cap, and smart throttling behavior. +- Add event post job tests for sampled acceptance and blocked usage accounting. +- Add API tests for organization budget alert and project event budget update/validation. +- Add mail/work-item tests for budget alert and smart throttling email eligibility/dedupe behavior. +- Add Svelte UI tests or targeted manual QA for the Usage page controls. +- Update HTTP samples if request/response contracts are changed. + +## Compatibility risks + +| Risk | Mitigation | +|------|------------| +| Existing event submissions could be rejected unexpectedly | Project event budgets default to null/Off for all existing projects. Smart throttling should preserve sampled acceptance when organization allowance remains. | +| Existing organization overage behavior could change | Keep organization-level monthly/bucket enforcement intact. Organization hard overage remains the outer limit. | +| Middleware-level project rejection would drop 100% of events | Project smart throttling and sampled acceptance must be implemented in the event processing path after events are parsed, not only in `OverageMiddleware`. | +| Too much configuration conflicts with product direction | Smart project throttling must work automatically with minimal or no required user configuration. | +| Existing customers expect some data while throttled | When the organization still has allowance, throttled projects should retain a small accepted sample rather than being fully blocked. | +| Static monthly plan throttling can feel unfair late in the month | Throughput calculations should consider events remaining in the monthly period and time remaining in the monthly period. | +| Project cap could be confused with reserved quota | UI copy must state that this is a cap, not a reservation. Other projects are not guaranteed unused capacity. | +| Percentage caps could behave unexpectedly when organization plans change | Effective percentage caps are recalculated from the current organization event allowance each time limits are evaluated. | +| Unlimited organization plans make percentage caps ambiguous | Percentage mode must be disabled or invalid for unlimited organizations. Fixed limits remain supported for project event budgets. Budget alert percentages are unavailable for unlimited organizations. | +| Budget alerts could send unexpected emails | Default budget alert settings to disabled for existing organizations. Users must opt in. | +| Duplicate threshold emails could annoy users | Each configured threshold is sent at most once per organization per monthly usage period. | +| Existing overage emails could change | Keep existing `PlanOverage` monthly/hourly emails unchanged and add budget alerts and smart throttling emails as separate notification types. | +| Persisted project or organization model changes could require Elasticsearch mapping/reindex | Store new settings source-only unless filtering/searching is added. No Elasticsearch mapping or reindex is required for enforcement. | + +## Non-goals + +- Do not implement API-key/token-level ingest limits in this change. +- Do not implement rate-based billing. +- Do not implement reserved project quotas. +- Do not change plan billing, invoices, Stripe integration, or organization plan limits. +- Do not change generic API request throttling behavior in `ThrottlingMiddleware`. +- Do not add new public event-submission endpoints. +- Do not migrate historical usage data. +- Do not require Elasticsearch reindexing. +- Do not require existing organizations or projects to be backfilled. +- Do not implement per-recipient custom alert thresholds in this change. +- Do not implement Slack/webhook budget alerts in this change. +- Do not implement stack-level adaptive throttling in v1 unless it can be done with low risk; project-level isolation is the v1 scope. +- Do not expose a large set of smart throttling tuning options in the UI. + +## Future considerations + +This design should leave room for future layers: + +1. Organization plan/event allowance. +2. Organization budget alert thresholds. +3. Automatic smart project throttling. +4. Optional project event budget. +5. API key/token ingest cap. +6. Future rate-based or usage-based billing. + +Budget alerts should leave room for future channels and scopes: + +- fixed event-count alert thresholds +- project-specific budget alerts +- token/API-key-specific budget alerts +- Slack/webhook alert delivery +- role-based recipients +- rate-based billing alerts + +The original issue suggested stack-level adaptive throttling with approximate occurrence counts. This change should leave room for that future direction, but v1 should focus on project-level isolation because it is simpler, aligns with project usage counters already tracked by Exceptionless, and directly addresses the customer problem of one product/environment affecting all others. + +## Rollback plan + +- Because project event budgets are opt-in, disabling or removing the UI leaves existing unset project budgets unchanged. +- Because budget alerts are opt-in, disabling or removing the UI leaves existing unset organization alert settings unchanged. +- If smart project throttling causes issues, smart throttling can be disabled while leaving budget alerts and optional project budgets in place. +- If budget alert emails cause issues, budget alert publishing/subscribers can be disabled while leaving stored settings in place. +- Existing organizations and projects default to null/Off, so rolling back model/API exposure has no effect on existing ingestion behavior unless users have already configured settings. +- If stored settings must be disabled quickly, project budget and budget alert evaluation can be short-circuited to treat all settings as unset. diff --git a/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md b/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md new file mode 100644 index 0000000000..49faea95bf --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/api-compatibility/spec.md @@ -0,0 +1,149 @@ +# Spec: API Compatibility + +## MODIFIED Requirements + +### Requirement: Event submission MUST preserve existing organization overage behavior + +Exceptionless event submission MUST preserve existing organization overage and API authentication behavior while adding automatic smart project throttling and optional project event budgets. + +#### Scenario: Organization overage status remains unchanged + +Given an organization has no remaining event allowance +When an event is submitted for a project in that organization +Then the response status code must remain the existing organization overage status code. + +#### Scenario: Project smart throttling preserves queued event post compatibility + +Given an organization has remaining event allowance and a project is under smart throttling +When an event post is submitted through the normal queued event submission path +Then the HTTP response should not be changed solely to report project throttling. + +#### Scenario: Generic API throttle behavior is unchanged + +Given API request throttling is enabled +When non-event API requests are made +Then existing generic API request throttle behavior must remain unchanged. + +#### Scenario: Existing client API keys continue to authenticate + +Given an existing project API key is used for event submission and the project has no configured event budget +When an event is submitted +Then authentication and authorization behavior must match existing behavior. + +### Requirement: Rate-limit headers MUST NOT be repurposed for project monthly controls + +Existing generic API throttle response headers MUST NOT be redefined to mean project monthly event budgets or smart project throttling. + +#### Scenario: Generic API throttle headers remain request-rate headers + +Given generic API throttling emits request-rate headers +When usage budget controls are added +Then those headers must continue to represent generic API request throttling behavior. + +## ADDED Requirements + +### Requirement: Organization API exposes budget alert settings + +The organization API may expose and update budget alert settings as an additive organization field. + +#### Scenario: Organization response includes budget alert settings + +Given an organization has budget alert settings +When the organization is returned from the API +Then the response must include budget_alert_settings. + +#### Scenario: Existing organization response remains compatible + +Given an existing organization has no budget alert settings +When the organization is returned from the API +Then budget_alert_settings may be null and existing fields must remain unchanged. + +#### Scenario: Authorized user updates budget alert settings + +Given a user is authorized to update an organization +When the user updates budget alert settings with valid thresholds +Then the API must persist the settings and include them in the response. + +#### Scenario: Invalid budget alert settings rejected + +Given a user is authorized to update an organization +When the user submits invalid budget alert settings +Then the API must reject the request with a validation error. + +## ADDED Requirements + +### Requirement: Budget alert emails are asynchronous side effects + +Budget alert emails are side effects of accepted usage threshold crossing and must not change event submission success responses. + +#### Scenario: Event crosses budget alert threshold + +Given budget alerts are enabled and an accepted event causes usage to cross a threshold +When the event submission completes +Then the event submission response must remain the normal accepted response. + +#### Scenario: Budget alert failure does not reject event + +Given budget alerts are enabled and budget alert queuing fails +When an event is submitted +Then the event must not be rejected solely because budget alert queuing failed. + +## ADDED Requirements + +### Requirement: Project API accepts event budget configuration + +The project update API may accept an optional project event budget configuration as an additive project field. + +#### Scenario: Clear project event budget + +Given an authorized user can update a project +When the user patches the project with ingest_limit set to null +Then the project event budget must be cleared. + +#### Scenario: Set fixed project event budget + +Given an authorized user can update a project +When the user patches the project with a fixed positive ingest_limit +Then the project must persist the fixed event budget. + +#### Scenario: Set percentage project event budget + +Given an authorized user can update a project and the organization has finite allowance +When the user patches the project with a valid percentage event budget +Then the project must persist the percentage event budget. + +#### Scenario: Invalid fixed limit is rejected + +Given an authorized user can update a project +When the user patches the project with a fixed event budget less than or equal to 0 +Then the API must reject the request with a validation error. + +#### Scenario: Invalid percentage limit is rejected + +Given an authorized user can update a project +When the user patches the project with a percentage event budget less than or equal to 0 or greater than 100 +Then the API must reject the request with a validation error. + +## ADDED Requirements + +### Requirement: Project response MUST include budget and throttling state + +Project responses used by the UI MUST expose the configured project event budget, currently effective cap, and smart throttling state where available. + +#### Scenario: Project has no event budget + +Given a project has no configured event budget +When the project is returned from the API +Then ingest_limit must be null and effective_ingest_limit must be null. + +#### Scenario: Project has fixed event budget + +Given a project has a fixed event budget +When the project is returned from the API +Then effective_ingest_limit must contain the currently effective cap. + +#### Scenario: Project is smart-throttled + +Given a project is currently under smart throttling +When the project is returned from the API +Then the response should indicate smart throttling state where feasible. diff --git a/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md b/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md new file mode 100644 index 0000000000..62a5b3d36e --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/event-ingestion/spec.md @@ -0,0 +1,329 @@ +# Spec: Event Ingestion + +## ADDED Requirements + +### Requirement: Smart throttling works without required user configuration + +Exceptionless must provide automatic smart project throttling without requiring users to configure per-project or per-stack throttling settings. + +#### Scenario: No smart throttling configuration exists + +Given an organization has projects +And no smart throttling settings have been configured +When one project starts sending a large event spike +Then Exceptionless should still be able to apply smart throttling automatically. + +#### Scenario: UI does not require many throttling options + +Given smart throttling is available +When a user views organization or project usage settings +Then the UI must not require the user to configure many low-level throttling parameters before smart throttling can protect the organization. + +### Requirement: Smart throttling uses remaining allowance and remaining time + +Smart throttling must calculate allowed throughput using the organization's remaining monthly event allowance and the time remaining in the monthly usage period. + +#### Scenario: Organization has substantial allowance remaining late in the month + +Given an organization is late in the monthly usage period +And the organization still has substantial event allowance remaining +When smart throttling calculates allowed throughput +Then the calculation must account for remaining event allowance +And must not rely only on the static monthly plan size divided by the original month duration. + +#### Scenario: Organization has little allowance remaining + +Given an organization has little event allowance remaining +When smart throttling calculates allowed throughput +Then the allowed throughput should decrease to protect the remaining monthly allowance. + +### Requirement: Smart throttling isolates noisy projects where possible + +When one project is responsible for a usage spike, smart throttling must apply to that project where possible rather than fully throttling the entire organization. + +#### Scenario: One project spikes + +Given an organization has Project A and Project B +And Project A sends a large event spike +And Project B remains within normal usage +When smart throttling is applied +Then Project A may be throttled or sampled +And Project B must not be blocked solely because Project A spiked. + +#### Scenario: Multiple projects spike + +Given an organization has multiple projects +And more than one project exceeds smart throttling criteria +When smart throttling is applied +Then each noisy project may be evaluated and throttled independently. + +### Requirement: Smart throttling preserves a sample of events + +When a project is smart-throttled and the organization still has remaining monthly allowance, Exceptionless must continue accepting a small sample of events for that project. + +#### Scenario: Project is throttled but organization has allowance + +Given an organization has remaining monthly event allowance +And a project is under smart throttling +When events are submitted for that project +Then Exceptionless must accept a small sample of events +And must block or discard the remainder according to usage accounting rules. + +#### Scenario: Sample rate remains small + +Given a project is under smart throttling +When the accepted sample is calculated +Then the accepted sample should be within the intended 1 to 5 percent range +Unless a lower organization or project hard limit leaves fewer events available. + +#### Scenario: Organization allowance exhausted + +Given an organization has no remaining monthly event allowance +And a project is under smart throttling +When events are submitted for that project +Then existing organization overage behavior must apply +And smart throttling must not allow sampled events beyond the organization hard allowance. + +### Requirement: Smart throttling operates after event posts are parsed + +Smart project throttling must be evaluated in a processing path that knows the number of events in the post so it can accept a sample and block the remainder. + +#### Scenario: Batch contains more events than allowed + +Given a queued event post contains many events +And smart throttling allows only a sample +When the event post is processed +Then the processor must process only the sampled/allowed events +And must record blocked usage for the rest. + +#### Scenario: Full request rejection would prevent sampling + +Given an event post would otherwise be rejected entirely by project-level throttling +When the organization still has remaining allowance +Then the system should prefer sampled processing over full request rejection. + +### Requirement: Smart throttling sends project throttling notification + +When smart throttling is applied to a project, Exceptionless must send an email notification to eligible organization users. + +#### Scenario: Project enters smart-throttled state + +Given a project was not previously smart-throttled +And smart throttling is applied to the project +When notification processing runs +Then eligible organization users must receive a project throttling notification email. + +#### Scenario: Project remains smart-throttled + +Given a project is already smart-throttled +When additional event posts are processed while the project remains throttled +Then Exceptionless must not send duplicate throttling emails for every post. + +#### Scenario: User is not email-eligible + +Given a user belongs to the organization +And the user does not have a verified email address or has email notifications disabled +When a project throttling notification is sent +Then that user must not receive the email. + +### Requirement: Smart throttling usage accounting remains accurate + +Smart throttling must record accepted, blocked, and discarded event counts consistently with existing usage accounting. + +#### Scenario: Sampled events accepted + +Given a project is smart-throttled +When a sample of events is accepted +Then accepted organization and project usage must increase by the number of processed events. + +#### Scenario: Non-sampled events blocked + +Given a project is smart-throttled +When some events in a post are not accepted due to throttling +Then blocked usage must increase by the number of non-accepted events. + +#### Scenario: Budget alerts use accepted events only + +Given budget alerts are enabled and a project is smart-throttled +When non-sampled events are blocked +Then blocked events must not count as accepted usage for organization budget alert thresholds. + +### Requirement: Smart throttling does not replace explicit project budgets + +Automatic smart throttling must coexist with optional project event budgets. + +#### Scenario: Project has no explicit budget + +Given a project has no configured project event budget +When the project spikes +Then automatic smart throttling may still apply. + +#### Scenario: Project has explicit budget + +Given a project has an explicit project event budget and automatic smart throttling also applies +When events are processed +Then the number of events accepted must not exceed the lowest applicable allowance from organization hard limit, project budget, and smart throttling sample. + +## ADDED Requirements + +### Requirement: Projects MUST be able to define optional event budgets + +Exceptionless projects MUST be able to define an optional project-level event budget that caps or limits the number of accepted events for that project within the current monthly usage period. + +#### Scenario: Project has no event budget + +Given a project has no configured event budget and the organization has remaining event allowance +When an event is submitted for the project +Then the explicit project event budget must not block the event +And existing organization-level usage enforcement and automatic smart throttling must apply. + +#### Scenario: Existing projects remain uncapped + +Given a project existed before project event budgets were introduced +When the project is loaded +Then its project event budget must be treated as unset. + +### Requirement: Fixed project event budgets MUST cap accepted project events + +A fixed project event budget MUST cap accepted event volume for a project to a configured positive integer for the current monthly usage period. + +#### Scenario: Fixed project budget allows events below cap + +Given an organization has remaining event allowance and a project has a fixed event budget of 20000 events +And the project has accepted fewer than 20000 events in the current monthly usage period +When an event is submitted for the project +Then the event must not be blocked by the explicit project event budget. + +#### Scenario: Fixed project budget limits events at cap + +Given an organization has remaining event allowance and a project has a fixed event budget of 20000 events +And the project has accepted 20000 events in the current monthly usage period +When events are submitted for the project +Then accepted events for that project must be limited by the project event budget +And blocked usage must be recorded for non-accepted events. + +#### Scenario: Fixed project budget above organization allowance is clamped + +Given an organization has a finite monthly event allowance of 10000 events +And a project has a fixed event budget of 20000 events +When the effective project event budget is evaluated +Then the effective project event budget must be 10000 events. + +#### Scenario: Fixed project budget on unlimited organization + +Given an organization has unlimited event allowance and a project has a fixed event budget of 20000 events +When the effective project event budget is evaluated +Then the effective project event budget must be 20000 events. + +### Requirement: Percentage project event budgets MUST derive from organization allowance + +A percentage project event budget MUST cap accepted event volume for a project to a percentage of the organization's current finite monthly event allowance. + +#### Scenario: Percentage project budget computes effective cap + +Given an organization has a finite monthly event allowance of 100000 events +And a project has a percentage event budget of 20 percent +When the effective project event budget is evaluated +Then the effective project event budget must be 20000 events. + +#### Scenario: Percentage project budget on unlimited organization is inactive + +Given an organization has unlimited event allowance and a project has a percentage event budget +When the effective project event budget is evaluated +Then the percentage project event budget must not produce an effective project cap. + +### Requirement: Project event budgets MUST be caps, not reservations + +Project event budgets MUST prevent a project from exceeding its cap but must not reserve organization event allowance for other projects. + +#### Scenario: Unused project budget does not reserve events + +Given an organization has a monthly event allowance of 100000 events +And Project A has a percentage event budget of 20 percent and uses 0 events +When Project B submits events +Then Project B must not be blocked solely because Project A has unused project budget. + +#### Scenario: Project budget does not increase organization allowance + +Given an organization has no remaining event allowance and a project has remaining project event budget +When an event is submitted for the project +Then the event must be rejected due to organization overage. + +### Requirement: Project event budget exhaustion MUST limit only the capped project + +When one project reaches its project event budget, other projects in the same organization MUST continue to be evaluated independently. + +#### Scenario: One capped project does not block another project + +Given an organization has remaining event allowance +And Project A has reached its project event budget and Project B has no project event budget +When an event is submitted for Project B +Then the event must not be blocked because Project A reached its budget. + +#### Scenario: Multiple projects have independent budgets + +Given an organization has remaining event allowance +And Project A has a fixed event budget of 1000 events and Project B has a fixed event budget of 2000 events +When Project A reaches 1000 accepted events +Then Project A must be limited by its project event budget +And Project B must continue to be evaluated against its own 2000-event budget. + +### Requirement: Organization overage MUST remain the hard outer limit + +Organization-level event allowance MUST remain the hard billing/usage boundary. + +#### Scenario: Organization overage takes precedence + +Given an organization has no remaining event allowance and a project has remaining project event budget +When an event is submitted for the project +Then the event must be rejected due to organization overage. + +### Requirement: Project ingest controls MUST reuse existing usage accounting + +Project budget and smart throttling enforcement MUST use existing project usage counters. + +#### Scenario: Accepted event increments project usage + +Given an organization has remaining event allowance and a project has remaining project event budget +When an event is accepted for the project +Then existing project accepted event usage must be incremented. + +#### Scenario: Blocked event does not count as accepted project usage + +Given a project has reached its project event budget +When an event is submitted for the project +Then accepted project usage must not increase for non-accepted events +And blocked usage must be recorded. + +### Requirement: Project controls MUST support safe defaults on failure + +Project budget and smart throttling evaluation MUST avoid introducing new ingestion failures when project data is missing or invalid. + +#### Scenario: Missing project does not cause project-budget rejection + +Given an event submission has an organization id and the project cannot be loaded +When project controls are evaluated +Then the system must not reject the event solely due to project budget evaluation. + +#### Scenario: Invalid persisted project budget is treated as inactive + +Given a project has an invalid persisted event budget configuration +When event ingestion evaluates project controls +Then the invalid project budget must be treated as inactive. + +### Requirement: Project ingest controls MUST prefer sampled processing over full request rejection + +Project-level ingest controls MUST preserve sampled visibility where possible. + +#### Scenario: Queued event post contains many project events + +Given an event post has been queued and the project is over its smart throttling allowance +And the organization still has remaining allowance +When the event post is processed +Then the job should process an allowed sample and record blocked usage for non-sampled events. + +#### Scenario: Middleware cannot sample project events + +Given a project-level condition would require preserving only 1 to 5 percent of events +When the request is still in middleware before parsing +Then middleware must not be the only enforcement layer. diff --git a/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md b/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md new file mode 100644 index 0000000000..ed43f44397 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/jobs-notifications-and-queues/spec.md @@ -0,0 +1,180 @@ +# Spec: Jobs, Notifications & Queues + +## ADDED Requirements + +### Requirement: Organizations may configure budget alert thresholds + +Organizations may configure event budget alert thresholds as percentages of their effective monthly event allowance. Budget alerts are disabled by default. + +#### Scenario: Existing organization has no budget alerts + +Given an organization existed before budget alerts were introduced +When the organization is loaded +Then budget alerts must be treated as disabled. + +#### Scenario: Enable budget alerts with thresholds + +Given an authorized user can update an organization +When the user enables budget alerts with thresholds 50 and 80 +Then the organization must store budget alert settings as enabled with thresholds 50 and 80. + +#### Scenario: Disable budget alerts + +Given an organization has enabled budget alerts +When an authorized user disables budget alerts +Then budget alert emails must no longer be sent for that organization. + +### Requirement: Budget alert thresholds validate percentage values + +Budget alert threshold percentages must be valid pre-overage warning thresholds. + +#### Scenario: Reject zero threshold + +Given an authorized user can update budget alert settings +When the user configures threshold 0 +Then the API must reject the request with a validation error. + +#### Scenario: Reject threshold at or above one hundred percent + +Given an authorized user can update budget alert settings +When the user configures threshold 100 or above +Then the API must reject the request with a validation error. + +#### Scenario: Enabled alerts require at least one threshold + +Given an authorized user can update budget alert settings +When the user enables budget alerts with no thresholds +Then the API must reject the request with a validation error. + +### Requirement: Budget alerts use effective organization allowance + +Budget alert thresholds must be evaluated against the same effective organization event allowance used by existing usage enforcement. + +#### Scenario: Threshold uses plan allowance + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 +When the threshold event count is calculated +Then the threshold event count must be 50000. + +#### Scenario: Unlimited organization has no percentage budget alert + +Given an organization has unlimited event allowance +When budget alert thresholds are evaluated +Then percentage budget alerts must be inactive and no budget alert email must be sent. + +### Requirement: Budget alerts send when accepted usage crosses thresholds + +Budget alert emails must be triggered when accepted organization event usage crosses a configured threshold for the first time in a monthly usage period. + +#### Scenario: Usage below threshold does not send alert + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 and accepted event usage is 49998 +When one event is accepted +Then no budget alert email must be sent. + +#### Scenario: Usage crossing threshold sends alert + +Given an organization has a monthly event allowance of 100000 +And budget alerts are enabled for threshold 50 and accepted event usage is 49999 +When one event is accepted +Then a 50 percent budget alert email must be queued. + +#### Scenario: Usage already above threshold does not resend alert + +Given the 50 percent alert has already been sent in the current monthly usage period +When another event is accepted +Then another 50 percent budget alert email must not be queued. + +#### Scenario: Large batch crosses multiple thresholds + +Given budget alerts are enabled for thresholds 50 and 80 and accepted event usage is 45000 +When a batch of 40000 events is accepted +Then both the 50 percent and 80 percent budget alert emails must be queued. + +### Requirement: Budget alert emails are sent once per threshold per monthly usage period + +Each configured threshold may generate at most one budget alert email per organization per monthly usage period. + +#### Scenario: Threshold already sent in period + +Given the 80 percent alert was sent earlier in the current monthly usage period +When usage remains above 80 percent +Then no additional 80 percent budget alert email must be sent. + +#### Scenario: New monthly period resets sent threshold state + +Given an organization received an 80 percent alert in the previous monthly usage period +When usage crosses 80 percent in the new period +Then a new 80 percent budget alert email may be sent. + +### Requirement: Budget alert emails use organization email notification eligibility + +Budget alert emails must be sent only to organization users who are eligible for existing organization email notices. + +#### Scenario: Verified user with email notifications enabled receives alert + +Given a user belongs to an organization with a verified email and email notifications enabled +When a budget alert is sent for the organization +Then the user must receive the budget alert email. + +#### Scenario: Unverified user does not receive alert + +Given a user belongs to an organization and does not have a verified email address +When a budget alert is sent for the organization +Then the user must not receive the budget alert email. + +### Requirement: Budget alerts do not block ingestion + +Budget alert processing must not block accepted event ingestion. + +#### Scenario: Alert email enqueue fails + +Given budget alerts are enabled and usage crosses a configured threshold +When the budget alert email cannot be queued +Then the accepted event must not be rejected solely because the alert email failed. + +### Requirement: Budget alerts preserve existing overage notifications + +Budget alerts must not replace or suppress existing monthly/hourly overage notifications. + +#### Scenario: Existing monthly overage still sends + +Given an organization reaches or exceeds its monthly event allowance +When existing monthly overage notification behavior is triggered +Then the existing monthly overage email must still be sent. + +### Requirement: Budget alert emails are separate from smart throttling emails + +Organization budget alerts and project smart throttling notifications must be distinct notifications. + +#### Scenario: Project throttling below budget threshold + +Given organization budget alerts are enabled and accepted usage has not crossed a threshold +When a project enters smart-throttled state +Then no organization budget threshold email must be sent solely because smart throttling was applied. + +## ADDED Requirements + +### Requirement: Smart throttling MUST send project throttling notification email + +When smart throttling is applied to a project, Exceptionless MUST send an email notification to eligible organization users. + +#### Scenario: Project enters smart-throttled state + +Given a project was not previously smart-throttled and smart throttling is applied +When notification processing runs +Then eligible organization users must receive a project throttling notification email. + +#### Scenario: Project remains smart-throttled + +Given a project is already smart-throttled +When additional event posts are processed +Then Exceptionless must not send duplicate throttling emails for every post. + +#### Scenario: User is not email-eligible + +Given a user belongs to the organization and does not have a verified email or has notifications disabled +When a project throttling notification is sent +Then that user must not receive the email. diff --git a/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md b/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md new file mode 100644 index 0000000000..18c006aa47 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/specs/organizations-projects-users-auth/spec.md @@ -0,0 +1,115 @@ +# Spec: Organizations, Projects, Users & Auth + +## ADDED Requirements + +### Requirement: Budget alert settings belong to an organization + +Budget alert settings belong to an organization and may be changed only by users authorized to update that organization. + +#### Scenario: Authorized organization user updates budget alerts + +Given a user is authorized to update an organization +When the user updates budget alert settings +Then the update must be allowed if the payload is valid. + +#### Scenario: Unauthorized user cannot update budget alerts + +Given a user is not authorized to update an organization +When the user attempts to update budget alert settings +Then the API must reject the operation according to existing organization authorization behavior. + +## ADDED Requirements + +### Requirement: Budget alert email recipients respect user preferences + +Budget alert email recipients must respect existing user email verification and email notification preferences. + +#### Scenario: User email notifications disabled + +Given a user belongs to an organization and has email notifications disabled +When a budget alert is sent +Then the user must not receive the budget alert email. + +#### Scenario: User email unverified + +Given a user belongs to an organization and the user's email address is not verified +When a budget alert is sent +Then the user must not receive the budget alert email. + +#### Scenario: Verified user with email notifications enabled + +Given a user belongs to an organization with a verified email and notifications enabled +When a budget alert is sent +Then the user may receive the budget alert email. + +## ADDED Requirements + +### Requirement: Project event budget authorization follows project authorization + +A project event budget belongs to a project and is scoped by the project's owning organization. Only users authorized to update the project may configure its event budget. + +#### Scenario: Authorized project user updates project event budget + +Given a user is authorized to update a project +When the user updates the project's event budget +Then the update must be allowed if the payload is valid. + +#### Scenario: Unauthorized user cannot update project event budget + +Given a user is not authorized to update a project +When the user attempts to update the project's event budget +Then the API must reject the operation according to existing project authorization behavior. + +#### Scenario: Project event budget cannot grant organization capacity + +Given a user configures a project event budget +When events are submitted for the project +Then the project event budget must not allow accepted event usage beyond the organization's effective allowance. + +## ADDED Requirements + +### Requirement: Budget controls preserve existing token and auth behavior + +Budget alert settings, smart project throttling, and project event budget configuration must not change API key authentication, token scopes, or user authorization roles. + +#### Scenario: Existing project token behavior is preserved + +Given an existing project API key has client scope and the project has no configured event budget +When the key is used for event submission +Then token authentication and scope behavior must match existing behavior. + +#### Scenario: Project smart throttling does not disable API key + +Given a project is under smart throttling +When a project API key is used for a non-ingest API route it is authorized to access +Then the API key must not be considered disabled solely because the project is smart-throttled. + +#### Scenario: Project budget does not disable API key + +Given a project reaches its project event budget +When a project API key is used for a non-ingest API route it is authorized to access +Then the API key must not be considered disabled solely because the project reached its event budget. + +#### Scenario: Disabled or suspended token remains rejected + +Given a project has remaining project event budget and the API key is disabled or suspended +When an event is submitted with that API key +Then the event must still be rejected due to token authentication state. + +## ADDED Requirements + +### Requirement: Project event budget MUST NOT preclude future token-level caps + +Project event budget and smart throttling behavior MUST NOT preclude future API-key/token-level ingest caps. + +#### Scenario: Future token cap is absent + +Given token-level ingest caps are not implemented +When a project event budget or smart throttling is evaluated +Then enforcement must consider organization and project scopes only. + +#### Scenario: Project budget remains independent of token count + +Given a project has multiple API keys and a configured project event budget +When events are submitted using different API keys for that project +Then all accepted events for the project must count against the same project event budget. diff --git a/openspec/changes/add-usage-budget-controls/tasks.md b/openspec/changes/add-usage-budget-controls/tasks.md new file mode 100644 index 0000000000..340fea6fc2 --- /dev/null +++ b/openspec/changes/add-usage-budget-controls/tasks.md @@ -0,0 +1,356 @@ +# Tasks: Add Usage Budget Controls + +## OpenSpec + +* Task 1: Create OpenSpec change + * Create proposal, design, tasks, and spec deltas for: + * organization budget alerts + * automatic smart project throttling + * project event budgets + * API compatibility + * organization/project auth behavior + * Verification: + * openspec validate add-usage-budget-controls --strict --no-interactive + +## Backend model and API contract + +* Task 2: Add organization budget alert settings model + * Add OrganizationBudgetAlertSettings. + * Add nullable BudgetAlertSettings to Organization. + * Add validation for enabled settings and percentage thresholds. + * Verification: + * dotnet build + * Add model/validation tests if existing model validation tests are available. +* Task 3: Add budget alert settings to organization API contract + * Add BudgetAlertSettings to ViewOrganization. + * Prefer adding UpdateOrganization with Name and BudgetAlertSettings. + * Update OrganizationController generic update DTO if using UpdateOrganization. + * Regenerate generated API and Svelte schemas. + * Verification: + * dotnet build + * dotnet test -- --filter-class OrganizationControllerTests + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check +* Task 4: Add project event budget domain model + * Add ProjectIngestLimit and ProjectIngestLimitType under src/Exceptionless.Core/Models/. + * Add nullable Project.IngestLimit. + * Add validation for fixed and percentage modes. + * Verification: + * dotnet build + * Add/adjust serializer tests if project model serialization coverage exists. +* Task 5: Add project event budget to project DTOs + * Add ProjectIngestLimit? IngestLimit to UpdateProject. + * Add ProjectIngestLimit? IngestLimit to ViewProject. + * Add int? EffectiveIngestLimit to ViewProject. + * Add smart throttling state fields to ViewProject if feasible: + * bool IsSmartThrottled + * double? SmartThrottleSampleRate + * Ensure Mapperly maps the new fields or add explicit mapping if required. + * Verification: + * dotnet build + * dotnet test -- --filter-class ProjectControllerTests +* Task 6: Regenerate OpenAPI and Svelte generated types + * Regenerate src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts. + * Regenerate src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts. + * Confirm UpdateOrganization, ViewOrganization, UpdateProject, ViewProject, and validation schemas include new fields. + * Verification: + * dotnet build + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + +## Organization budget alert emails + +* Task 7: Add budget alert threshold calculation + * Use the same effective organization allowance as existing usage enforcement. + * Support configured percentage thresholds from 1 through 99. + * Treat thresholds as inactive for unlimited organizations. + * Verification: + * Add tests to UsageServiceTests. + * dotnet test -- --filter-class UsageServiceTests +* Task 8: Publish budget alert messages when thresholds are crossed + * Add OrganizationBudgetAlert message. + * In UsageService.IncrementTotalAsync, detect threshold crossings after accepted event usage increments. + * Publish one message per crossed threshold. + * Do not publish repeatedly after threshold was already sent in the current monthly usage period. + * Verification: + * Tests: + * below threshold does not publish + * crossing threshold publishes + * already-sent threshold does not publish + * batch crossing multiple thresholds publishes each crossed threshold + * dotnet test -- --filter-class UsageServiceTests +* Task 9: Add budget alert dedupe + * Add dedupe key per organization, threshold, and monthly usage period. + * Set TTL through the end of the monthly usage period plus a safety buffer. + * Prefer non-blocking behavior if dedupe cache fails. + * Verification: + * Add tests for one-email-per-threshold-per-period behavior. + * dotnet test -- --filter-class UsageServiceTests +* Task 10: Add budget alert work item and subscriber + * Add OrganizationBudgetAlertWorkItem. + * Add startup subscriber that turns OrganizationBudgetAlert messages into work items. + * Add work item handler that loads organization/users and sends emails. + * Re-check current organization budget alert settings before sending. + * Verification: + * dotnet test -- --filter-class OrganizationBudgetAlertWorkItemHandlerTests + * If no dedicated handler test class exists, add one. +* Task 11: Add budget alert mailer method and template + * Add IMailer.SendOrganizationBudgetAlertAsync. + * Implement the method in Mailer. + * Add organization-budget-alert.html template. + * Include current usage, threshold percentage, threshold event count, event limit, remaining events, and links to organization usage/billing. + * Verification: + * Mailer/template rendering test if existing template tests exist. + * dotnet build + +## Automatic Smart Project Throttling + +* Task 12: Add smart throughput calculation + * Calculate allowed throughput from events left in the monthly period and time/windows left in the period. + * Preserve existing burst tolerance behavior where appropriate. + * Avoid static plan-size-only calculations. + * Verification: + * Tests for early-month, late-month, high-remaining, and low-remaining allowance cases. + * dotnet test -- --filter-class UsageServiceTests +* Task 13: Add project-level smart throttling decision + * Detect projects contributing to usage spikes. + * Apply smart throttling to noisy projects where possible instead of organization-wide full blocking. + * Verification: + * Tests where Project A spikes and Project B remains unaffected. + * Tests where multiple projects are evaluated independently. + * dotnet test -- --filter-class UsageServiceTests +* Task 14: Add sampled acceptance for smart-throttled projects + * Implement 1–5% sampled acceptance when project is smart-throttled and organization has remaining allowance. + * Prefer deterministic/testable sampling. + * Record blocked usage for non-sampled events. + * Verification: + * Tests for sampled acceptance count. + * Tests that blocked usage increments for non-sampled events. + * Tests that accepted usage increments only for sampled events. + * dotnet test -- --filter-class EventPostsJobTests +* Task 15: Move project-level sampled enforcement into event post processing + * Extend EventPostsJob / UsageService after event post parsing to calculate allowed events. + * Avoid using only OverageMiddleware for project-level sampled enforcement. + * Preserve existing organization hard overage middleware behavior. + * Verification: + * Tests for batch event posts where only a sample is processed. + * Tests for organization hard overage still blocking. + * dotnet test -- --filter-class EventPostsJobTests + * dotnet test -- --filter-class OverageMiddlewareTests +* Task 16: Add smart throttling notification message/work item/email + * Add project throttling notification message. + * Add work item and handler. + * Add mailer method and template. + * Send to verified users with email notifications enabled. + * Deduplicate notifications during a throttling cooldown window. + * Verification: + * dotnet test -- --filter-class ProjectSmartThrottleNotificationWorkItemHandlerTests + * dotnet build + +## Optional project event budgets + +* Task 17: Refactor UsageService events-left calculation + * Extract organization-only events-left logic into a helper that accepts optional projectId. + * Reuse existing project-aware cache key helpers. + * Preserve current organization behavior. + * Verification: + * Add tests for organization-only events-left behavior to ensure no regression. + * dotnet test -- --filter-class UsageServiceTests +* Task 18: Add project effective budget calculation + * Implement effective fixed budget calculation. + * Implement effective percentage budget calculation. + * Clamp fixed budgets to finite organization allowance during enforcement. + * Treat percentage budgets as inactive when organization allowance is unlimited. + * Verification: + * dotnet test -- --filter-class UsageServiceTests +* Task 19: Add parsed-event ingest allowance result + * Add EventIngestAllowanceResult. + * Add UsageService.GetEventIngestAllowanceAsync(string organizationId, string projectId, int submittedEventCount) or equivalent. + * Return event count allowed, organization remaining, project remaining, effective project budget, and smart throttling state. + * Verification: + * Tests: + * no project id or missing project falls back safely + * no project budget uses organization + smart throttling + * fixed budget limits allowed event count + * percentage budget computes effective cap from organization allowance + * organization overage takes precedence over project controls + * smart throttling sample and project budget combine by taking the minimum allowed count + * dotnet test -- --filter-class UsageServiceTests + +## Middleware and event processing + +* Task 20: Keep OverageMiddleware as coarse gate + * Preserve existing status codes for organization overage, disabled submission, missing content length, oversized posts, and unauthorized organization context. + * Do not implement sampled project enforcement only in middleware. + * Verification: + * dotnet test -- --filter-class OverageMiddlewareTests +* Task 21: Update EventPostsJob for sampled/project-budget allowance + * After parsing events, ask UsageService for allowed event count. + * Process only allowed events. + * Use deterministic sampling/selection when allowed count is lower than submitted count due to smart throttling. + * Increment blocked usage for non-accepted events. + * Increment accepted usage only for processed events. + * Verification: + * dotnet test -- --filter-class EventPostsJobTests + +## Backend API tests + +* Task 22: Add OrganizationController tests for budget alert settings + * Test enabling budget alerts with thresholds. + * Test disabling budget alerts. + * Test threshold normalization/deduplication. + * Test invalid thresholds reject. + * Test unauthorized organization update remains rejected. + * Verification: + * dotnet test -- --filter-class OrganizationControllerTests +* Task 23: Add ProjectController tests for event budget update behavior + * Test setting fixed event budget. + * Test setting percentage event budget. + * Test clearing event budget. + * Test invalid fixed budget rejects. + * Test invalid percentage rejects. + * Test percentage budget rejects for unlimited organization or is marked inactive according to final implementation decision. + * Verification: + * dotnet test -- --filter-class ProjectControllerTests +* Task 24: Add event submission integration tests + * Verify event submission processing accepts samples under smart throttling. + * Verify organization overage behavior remains unchanged. + * Verify uncapped projects continue under the organization limit. + * Verify crossing budget alert thresholds does not change accepted event response. + * Verification: + * dotnet test -- --filter-class EventControllerTests + * dotnet test -- --filter-class EventPostsJobTests +* Task 25: Update HTTP samples if contracts are changed + * Update tests/http/*.http samples for organization update/get if organization request/response examples exist. + * Update tests/http/*.http samples for project update/get if project request/response examples exist. + * Verification: + * Manual review of tests/http/*.http. + * dotnet build + +## Svelte UI + +* Task 26: Add budget alerts UI to Organization Usage page + * Update src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/usage/+page.svelte. + * Add card for enabling/disabling budget alerts. + * Add threshold editor with default suggestions of 50 and 80. + * Show computed event counts for each threshold. + * Disable percentage alerts for unlimited organizations. + * Use existing organization API query/mutation patterns. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * Manual localhost QA on Organization Settings → Usage. +* Task 27: Add project event budget UI component + * Create src/Exceptionless.Web/ClientApp/src/lib/features/projects/components/project-ingest-limit-card.svelte. + * Support Off, Fixed, and Percentage modes. + * Show computed effective cap. + * Show current project usage against cap when active. + * Show fixed-cap warning when fixed cap exceeds current organization allowance. + * Disable percentage mode when organization allowance is unlimited. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint +* Task 28: Add smart throttling UI status + * Surface project smart-throttled status on Project Usage where feasible. + * Explain that a sample of events is still accepted. + * Link to organization/project usage. + * Avoid exposing many tuning options. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * Manual localhost QA with a smart-throttled project. +* Task 29: Integrate project card into Project Usage page + * Add the card to src/Exceptionless.Web/ClientApp/src/routes/(app)/project/[projectId]/usage/+page.svelte. + * Use existing getProjectQuery, getOrganizationQuery, and updateProject mutation. + * Invalidate/update project query data after saving. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run check + * Manual localhost QA on /project/{projectId}/usage +* Task 30: Update project usage chart limit/status display + * Use effective_ingest_limit as the chart limit when present. + * Otherwise preserve existing organization limit behavior. + * Label the limit line as Project Limit or Organization Limit. + * Show smart throttling status text/badge when active. + * Verification: + * Manual localhost QA with: + * no project budget + * fixed project budget + * percentage project budget + * smart-throttled project + * Confirm chart remains readable and accessible. +* Task 31: Add frontend tests for budget control helpers + * Test organization budget alert threshold normalization. + * Test organization budget alert threshold validation. + * Test budget alert computed event counts. + * Test project budget mode-to-payload conversion. + * Test project percentage cap calculation display. + * Test fixed cap warning. + * Verification: + * cd src/Exceptionless.Web/ClientApp && npm run test:unit + +## Compatibility and validation + +* Task 32: Verify existing request throttling remains unchanged + * Confirm ThrottlingMiddleware behavior and tests are unchanged unless incidental generated code changes require updates. + * Verification: + * dotnet test -- --filter-class ThrottlingMiddlewareTests +* Task 33: Verify existing overage notification remains unchanged + * Confirm existing PlanOverage monthly/hourly paths continue to enqueue and send existing organization notices. + * Confirm budget alerts and smart throttling use separate message/work item/template. + * Verification: + * Existing organization notification tests if present. + * New budget alert and smart throttling work item tests. +* Task 34: Verify no Elasticsearch mapping/reindex is required + * Confirm budget alert settings and project event budget are not used for organization/project search/filter/sort. + * Confirm no organization/project index version bump is required. + * Verification: + * Code review checklist item. +* Task 35: Local dogfood budget alerts + * Run local stack with aspire run or Exceptionless.AppHost. + * Use local QA URL only: http://localhost:7110. + * Enable budget alerts for a low threshold. + * Submit events until threshold is crossed. + * Confirm budget alert work item/email is queued. + * Confirm alert does not send twice for same threshold in same monthly period. + * Confirm disabling alerts prevents future sends. + * Verification: + * Manual QA notes in PR. +* Task 36: Local dogfood project budgets and smart throttling + * Run local stack with aspire run or Exceptionless.AppHost. + * Use local QA URL only: http://localhost:7110. + * Configure a low fixed budget on a project. + * Submit events until budget is reached. + * Confirm: + * project is limited/sampled + * blocked usage increases + * another project can still ingest events + * clearing the budget restores behavior + * Create or simulate a noisy project. + * Confirm smart throttling applies without manual configuration. + * Confirm a sample is still accepted. + * Verification: + * Manual QA notes in PR. + +## Final validation + +* Task 37: Run targeted validation + * Commands: + * dotnet build + * dotnet test -- --filter-class UsageServiceTests + * dotnet test -- --filter-class OverageMiddlewareTests + * dotnet test -- --filter-class OrganizationControllerTests + * dotnet test -- --filter-class ProjectControllerTests + * dotnet test -- --filter-class OrganizationBudgetAlertWorkItemHandlerTests + * dotnet test -- --filter-class ProjectSmartThrottleNotificationWorkItemHandlerTests + * dotnet test -- --filter-class EventControllerTests + * dotnet test -- --filter-class EventPostsJobTests + * dotnet test -- --filter-class ThrottlingMiddlewareTests + * cd src/Exceptionless.Web/ClientApp && npm ci && npm run check + * cd src/Exceptionless.Web/ClientApp && npm run lint + * cd src/Exceptionless.Web/ClientApp && npm run test:unit +* Task 38: Run OpenSpec validation + * Command: + * openspec validate add-usage-budget-controls --strict --no-interactive +* Task 39: Optional full validation before merge + * Commands: + * dotnet test + * cd src/Exceptionless.Web/ClientApp && npm run build + * openspec validate --all --strict --no-interactive diff --git a/src/Exceptionless.Core/Bootstrapper.cs b/src/Exceptionless.Core/Bootstrapper.cs index 334f2b07ae..29f2053809 100644 --- a/src/Exceptionless.Core/Bootstrapper.cs +++ b/src/Exceptionless.Core/Bootstrapper.cs @@ -109,6 +109,8 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); + handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); handlers.Register(s.GetRequiredService); diff --git a/src/Exceptionless.Core/Exceptionless.Core.csproj b/src/Exceptionless.Core/Exceptionless.Core.csproj index 2ee31516e5..a8f38c730c 100644 --- a/src/Exceptionless.Core/Exceptionless.Core.csproj +++ b/src/Exceptionless.Core/Exceptionless.Core.csproj @@ -2,20 +2,24 @@ + + + + diff --git a/src/Exceptionless.Core/Jobs/EventPostsJob.cs b/src/Exceptionless.Core/Jobs/EventPostsJob.cs index 0f771e9b64..0995392cdf 100644 --- a/src/Exceptionless.Core/Jobs/EventPostsJob.cs +++ b/src/Exceptionless.Core/Jobs/EventPostsJob.cs @@ -175,6 +175,38 @@ protected override async Task ProcessQueueEntryAsync(QueueEntryContex return JobResult.Success; } + // Check project-specific ingest limits + var ingestAllowance = await _usageService.GetEventIngestAllowanceAsync(organization.Id, project.Id); + if (ingestAllowance.EventsLeft < 1) + { + if (!isInternalProject) + _logger.LogDebug("Unable to process EventPost {FilePath}: Over project ingest limit", payloadPath); + + await _usageService.IncrementBlockedAsync(organization.Id, project.Id, events.Count); + await CompleteEntryAsync(entry, ep, _timeProvider.GetUtcNow().UtcDateTime); + return JobResult.Success; + } + + // Apply smart throttling: sample events from projects consuming disproportionate resources + var throttleResult = await _usageService.GetSmartThrottleRateAsync(organization.Id, project.Id); + if (throttleResult.IsThrottled) + { + int sampled = (int)Math.Max(1, Math.Ceiling(events.Count * throttleResult.SampleRate)); + if (sampled < events.Count) + { + int discarded = events.Count - sampled; + // Use deterministic sampling to keep a representative sample + events = events.Take(sampled).ToList(); + await _usageService.RecordSmartThrottleAsync(organization.Id, project.Id, discarded, throttleResult); + + if (!isInternalProject) + _logger.LogInformation("Smart throttling applied to EventPost {FilePath}: Accepted {Sampled}/{Total} events (rate={SampleRate:F2})", payloadPath, sampled, sampled + discarded, throttleResult.SampleRate); + } + } + + // Use the more restrictive of org and project limits + eventsToProcess = Math.Min(eventsToProcess, ingestAllowance.EventsLeft); + // Keep track of the original event payload size, we can save some processing for retries in the case it was a massive batch. bool isSingleEvent = events.Count == 1; diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs new file mode 100644 index 0000000000..505da28dc5 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/OrganizationBudgetAlertWorkItemHandler.cs @@ -0,0 +1,99 @@ +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Queues; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class EnqueueOrganizationBudgetAlertOnUsageThreshold : IStartupAction +{ + private readonly IQueue _workItemQueue; + private readonly IMessageSubscriber _subscriber; + private readonly ILogger _logger; + + public EnqueueOrganizationBudgetAlertOnUsageThreshold(IQueue workItemQueue, IMessageSubscriber subscriber, ILoggerFactory loggerFactory) + { + _workItemQueue = workItemQueue; + _subscriber = subscriber; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + return _subscriber.SubscribeAsync(async alert => + { + _logger.LogInformation("Enqueueing budget alert work item for organization: {OrganizationId} Threshold: {Threshold}%", alert.OrganizationId, alert.Threshold); + + await _workItemQueue.EnqueueAsync(new OrganizationBudgetAlertWorkItem + { + OrganizationId = alert.OrganizationId, + Threshold = alert.Threshold, + ThresholdEventCount = alert.ThresholdEventCount, + CurrentEventCount = alert.CurrentEventCount, + EventLimit = alert.EventLimit + }); + }, shutdownToken); + } +} + +public class OrganizationBudgetAlertWorkItemHandler : WorkItemHandlerBase +{ + private readonly IOrganizationRepository _organizationRepository; + private readonly IUserRepository _userRepository; + private readonly IMailer _mailer; + + public OrganizationBudgetAlertWorkItemHandler(IOrganizationRepository organizationRepository, IUserRepository userRepository, IMailer mailer, ILoggerFactory loggerFactory) : base(loggerFactory) + { + _organizationRepository = organizationRepository; + _userRepository = userRepository; + _mailer = mailer; + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var wi = context.GetData()!; + Log.LogInformation("Received budget alert work item for organization: {OrganizationId} Threshold: {Threshold}%", wi.OrganizationId, wi.Threshold); + + var organization = await _organizationRepository.GetByIdAsync(wi.OrganizationId, o => o.Cache()); + if (organization is null) + { + Log.LogWarning("Organization {OrganizationId} not found, skipping budget alert", wi.OrganizationId); + return; + } + + // Re-check that budget alerts are still enabled and the threshold is still configured + if (organization.BudgetAlertSettings is not { Enabled: true } || !organization.BudgetAlertSettings.Thresholds.Contains(wi.Threshold)) + { + Log.LogInformation("Budget alerts disabled or threshold {Threshold}% removed for organization: {OrganizationId}, skipping", wi.Threshold, wi.OrganizationId); + return; + } + + var results = await _userRepository.GetByOrganizationIdAsync(organization.Id); + foreach (var user in results.Documents) + { + if (!user.IsEmailAddressVerified) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has not been verified, skipping budget alert", user.Id, user.EmailAddress); + continue; + } + + if (!user.EmailNotificationsEnabled) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has email notifications disabled, skipping budget alert", user.Id, user.EmailAddress); + continue; + } + + Log.LogTrace("Sending budget alert email to {EmailAddress}...", user.EmailAddress); + await _mailer.SendOrganizationBudgetAlertAsync(user, organization, wi.Threshold, wi.ThresholdEventCount, wi.CurrentEventCount, wi.EventLimit); + } + + Log.LogTrace("Done sending budget alert emails"); + } +} diff --git a/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs b/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs new file mode 100644 index 0000000000..f9f70b0322 --- /dev/null +++ b/src/Exceptionless.Core/Jobs/WorkItemHandlers/ProjectSmartThrottleWorkItemHandler.cs @@ -0,0 +1,101 @@ +using Exceptionless.Core.Mail; +using Exceptionless.Core.Messaging.Models; +using Exceptionless.Core.Models; +using Exceptionless.Core.Models.WorkItems; +using Exceptionless.Core.Repositories; +using Foundatio.Extensions.Hosting.Startup; +using Foundatio.Jobs; +using Foundatio.Messaging; +using Foundatio.Queues; +using Foundatio.Repositories; +using Microsoft.Extensions.Logging; + +namespace Exceptionless.Core.Jobs.WorkItemHandlers; + +public class EnqueueProjectSmartThrottleOnThrottleApplied : IStartupAction +{ + private readonly IQueue _workItemQueue; + private readonly IMessageSubscriber _subscriber; + private readonly ILogger _logger; + + public EnqueueProjectSmartThrottleOnThrottleApplied(IQueue workItemQueue, IMessageSubscriber subscriber, ILoggerFactory loggerFactory) + { + _workItemQueue = workItemQueue; + _subscriber = subscriber; + _logger = loggerFactory.CreateLogger(); + } + + public Task RunAsync(CancellationToken shutdownToken = default) + { + return _subscriber.SubscribeAsync(async throttle => + { + _logger.LogInformation("Enqueueing smart throttle notification for project: {ProjectId} in organization: {OrganizationId}", throttle.ProjectId, throttle.OrganizationId); + + await _workItemQueue.EnqueueAsync(new ProjectSmartThrottleWorkItem + { + OrganizationId = throttle.OrganizationId, + ProjectId = throttle.ProjectId, + SampleRate = throttle.SampleRate, + CurrentEventCount = throttle.CurrentEventCount, + EventLimit = throttle.EventLimit + }); + }, shutdownToken); + } +} + +public class ProjectSmartThrottleWorkItemHandler : WorkItemHandlerBase +{ + private readonly IOrganizationRepository _organizationRepository; + private readonly IProjectRepository _projectRepository; + private readonly IUserRepository _userRepository; + private readonly IMailer _mailer; + + public ProjectSmartThrottleWorkItemHandler(IOrganizationRepository organizationRepository, IProjectRepository projectRepository, IUserRepository userRepository, IMailer mailer, ILoggerFactory loggerFactory) : base(loggerFactory) + { + _organizationRepository = organizationRepository; + _projectRepository = projectRepository; + _userRepository = userRepository; + _mailer = mailer; + } + + public override async Task HandleItemAsync(WorkItemContext context) + { + var wi = context.GetData()!; + Log.LogInformation("Received smart throttle notification for project: {ProjectId} in organization: {OrganizationId}", wi.ProjectId, wi.OrganizationId); + + var organization = await _organizationRepository.GetByIdAsync(wi.OrganizationId, o => o.Cache()); + if (organization is null) + { + Log.LogWarning("Organization {OrganizationId} not found, skipping smart throttle notification", wi.OrganizationId); + return; + } + + var project = await _projectRepository.GetByIdAsync(wi.ProjectId, o => o.Cache()); + if (project is null) + { + Log.LogWarning("Project {ProjectId} not found, skipping smart throttle notification", wi.ProjectId); + return; + } + + var results = await _userRepository.GetByOrganizationIdAsync(organization.Id); + foreach (var user in results.Documents) + { + if (!user.IsEmailAddressVerified) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has not been verified, skipping throttle notification", user.Id, user.EmailAddress); + continue; + } + + if (!user.EmailNotificationsEnabled) + { + Log.LogInformation("User {UserId} with email address {EmailAddress} has email notifications disabled, skipping throttle notification", user.Id, user.EmailAddress); + continue; + } + + Log.LogTrace("Sending smart throttle email to {EmailAddress}...", user.EmailAddress); + await _mailer.SendProjectThrottledNoticeAsync(user, organization, project, wi.SampleRate, wi.CurrentEventCount, wi.EventLimit); + } + + Log.LogTrace("Done sending smart throttle emails"); + } +} diff --git a/src/Exceptionless.Core/Mail/IMailer.cs b/src/Exceptionless.Core/Mail/IMailer.cs index 79b3b7f39e..dc98c759ee 100644 --- a/src/Exceptionless.Core/Mail/IMailer.cs +++ b/src/Exceptionless.Core/Mail/IMailer.cs @@ -10,6 +10,8 @@ public interface IMailer Task SendOrganizationNoticeAsync(User user, Organization organization, bool isOverMonthlyLimit, bool isOverHourlyLimit); Task SendOrganizationPaymentFailedAsync(User owner, Organization organization); Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan); + Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit); + Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit); Task SendUserEmailVerifyAsync(User user); Task SendUserPasswordResetAsync(User user); } diff --git a/src/Exceptionless.Core/Mail/Mailer.cs b/src/Exceptionless.Core/Mail/Mailer.cs index b6498c1d45..3d8fad6963 100644 --- a/src/Exceptionless.Core/Mail/Mailer.cs +++ b/src/Exceptionless.Core/Mail/Mailer.cs @@ -195,6 +195,57 @@ public Task SendOrganizationPaymentFailedAsync(User owner, Organization organiza }, template); } + public Task SendOrganizationBudgetAlertAsync(User user, Organization organization, int threshold, int thresholdEventCount, int currentEventCount, int eventLimit) + { + const string template = "organization-budget-alert"; + string subject = $"[{organization.Name}] Budget Alert: {threshold}% of monthly event allowance used"; + + var data = new Dictionary { + { "Subject", subject }, + { "BaseUrl", _appOptions.BaseURL }, + { "OrganizationId", organization.Id }, + { "OrganizationName", organization.Name }, + { "Threshold", threshold }, + { "ThresholdEventCount", thresholdEventCount }, + { "CurrentEventCount", currentEventCount }, + { "EventLimit", eventLimit }, + { "RemainingEventCount", Math.Max(0, eventLimit - currentEventCount) } + }; + + return QueueMessageAsync(new MailMessage + { + To = user.EmailAddress, + Subject = subject, + Body = RenderTemplate(template, data) + }, template); + } + + public Task SendProjectThrottledNoticeAsync(User user, Organization organization, Project project, double sampleRate, int currentEventCount, int eventLimit) + { + const string template = "project-smart-throttle"; + string subject = $"[{organization.Name}] Smart Throttling Active: {project.Name}"; + + var data = new Dictionary { + { "Subject", subject }, + { "BaseUrl", _appOptions.BaseURL }, + { "OrganizationId", organization.Id }, + { "OrganizationName", organization.Name }, + { "ProjectId", project.Id }, + { "ProjectName", project.Name }, + { "SampleRate", sampleRate }, + { "SamplePercent", (int)(sampleRate * 100) }, + { "CurrentEventCount", currentEventCount }, + { "EventLimit", eventLimit } + }; + + return QueueMessageAsync(new MailMessage + { + To = user.EmailAddress, + Subject = subject, + Body = RenderTemplate(template, data) + }, template); + } + public Task SendProjectDailySummaryAsync(User user, Project project, IEnumerable? mostFrequent, IEnumerable? newest, DateTime startDate, bool hasSubmittedEvents, double count, double uniqueCount, double newCount, double fixedCount, int blockedCount, int tooBigCount, bool isFreePlan) { const string template = "project-daily-summary"; diff --git a/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html b/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html new file mode 100644 index 0000000000..ca434d0f8b --- /dev/null +++ b/src/Exceptionless.Core/Mail/Templates/organization-budget-alert.html @@ -0,0 +1 @@ +{{Subject}}
Exceptionless
 

{{OrganizationName}} has used {{Threshold}}% of its monthly event allowance ({{CurrentEventCount}} of {{EventLimit}} events).

You have {{RemainingEventCount}} events remaining this billing period. Consider upgrading your plan or adjusting your event volume to avoid service interruption.

View Usage
 

You can also view the most frequent events to see which events are counting against your plan limits.

Other Actions
diff --git a/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html b/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html new file mode 100644 index 0000000000..618deb72da --- /dev/null +++ b/src/Exceptionless.Core/Mail/Templates/project-smart-throttle.html @@ -0,0 +1 @@ +{{Subject}}
Exceptionless
 

Smart throttling has been activated for {{ProjectName}} in {{OrganizationName}}.

Your organization has used {{CurrentEventCount}} of {{EventLimit}} events this billing period. To protect your remaining budget, events from this project are now being sampled at {{SamplePercent}}%. This means approximately {{SamplePercent}}% of incoming events will be accepted while the rest are discarded.

Throttling will automatically adjust as your usage changes. To stop throttling, you can upgrade your plan or reduce event volume.

View Usage
 

You can also view the most frequent events to see which events are counting against your plan limits.

Other Actions
diff --git a/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs b/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs new file mode 100644 index 0000000000..1d6bf4219b --- /dev/null +++ b/src/Exceptionless.Core/Models/Messaging/OrganizationBudgetAlert.cs @@ -0,0 +1,10 @@ +namespace Exceptionless.Core.Messaging.Models; + +public record OrganizationBudgetAlert +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} diff --git a/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs b/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs new file mode 100644 index 0000000000..b3e9ca7c2c --- /dev/null +++ b/src/Exceptionless.Core/Models/Messaging/ProjectSmartThrottleApplied.cs @@ -0,0 +1,10 @@ +namespace Exceptionless.Core.Messaging.Models; + +public record ProjectSmartThrottleApplied +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } +} diff --git a/src/Exceptionless.Core/Models/Organization.cs b/src/Exceptionless.Core/Models/Organization.cs index 27b9e3c937..b7cf6e12ef 100644 --- a/src/Exceptionless.Core/Models/Organization.cs +++ b/src/Exceptionless.Core/Models/Organization.cs @@ -171,6 +171,11 @@ public Organization() /// public DataDictionary? Data { get; set; } + /// + /// Budget alert settings for this organization. Null means disabled. + /// + public OrganizationBudgetAlertSettings? BudgetAlertSettings { get; set; } + public DateTime CreatedUtc { get; set; } public DateTime UpdatedUtc { get; set; } public bool IsDeleted { get; set; } @@ -265,6 +270,19 @@ public IEnumerable Validate(ValidationContext validationContex [nameof(SuspendedByUserId)]); } } + + if (BudgetAlertSettings is { Enabled: true }) + { + if (BudgetAlertSettings.Thresholds.Count == 0) + { + yield return new ValidationResult("At least one threshold is required when budget alerts are enabled.", + [nameof(BudgetAlertSettings)]); + } + + if (BudgetAlertSettings.Thresholds.Any(t => t <= 0 || t >= 100)) + yield return new ValidationResult("Budget alert thresholds must be between 1 and 99.", + [nameof(BudgetAlertSettings)]); + } } } diff --git a/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs b/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs new file mode 100644 index 0000000000..e8d55eccb2 --- /dev/null +++ b/src/Exceptionless.Core/Models/OrganizationBudgetAlertSettings.cs @@ -0,0 +1,12 @@ +namespace Exceptionless.Core.Models; + +public class OrganizationBudgetAlertSettings +{ + public bool Enabled { get; set; } + + /// + /// Percentage thresholds of the organization's effective monthly event allowance. + /// Example: [50, 80, 90]. + /// + public SortedSet Thresholds { get; set; } = []; +} diff --git a/src/Exceptionless.Core/Models/Project.cs b/src/Exceptionless.Core/Models/Project.cs index 0e5f223072..7215e2dbd3 100644 --- a/src/Exceptionless.Core/Models/Project.cs +++ b/src/Exceptionless.Core/Models/Project.cs @@ -63,6 +63,11 @@ public Project() public bool DeleteBotDataEnabled { get; set; } + /// + /// Optional project-level event budget configuration. Null means no project-specific cap. + /// + public ProjectIngestLimit? IngestLimit { get; set; } + /// /// The tick count that represents the next time the daily summary job should run. This time is set to midnight of the /// projects local time. diff --git a/src/Exceptionless.Core/Models/ProjectIngestLimit.cs b/src/Exceptionless.Core/Models/ProjectIngestLimit.cs new file mode 100644 index 0000000000..67704e9de6 --- /dev/null +++ b/src/Exceptionless.Core/Models/ProjectIngestLimit.cs @@ -0,0 +1,14 @@ +namespace Exceptionless.Core.Models; + +public class ProjectIngestLimit +{ + public ProjectIngestLimitType Type { get; set; } + public int? FixedLimit { get; set; } + public decimal? PercentOfOrganizationLimit { get; set; } +} + +public enum ProjectIngestLimitType +{ + Fixed = 0, + PercentOfOrganizationLimit = 1 +} diff --git a/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs new file mode 100644 index 0000000000..6a11ee2b45 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/OrganizationBudgetAlertWorkItem.cs @@ -0,0 +1,14 @@ +using Foundatio.Queues; + +namespace Exceptionless.Core.Models.WorkItems; + +public record OrganizationBudgetAlertWorkItem : IHaveUniqueIdentifier +{ + public required string OrganizationId { get; init; } + public required int Threshold { get; init; } + public required int ThresholdEventCount { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + + public string UniqueIdentifier => $"BudgetAlert:{OrganizationId}:{Threshold}"; +} diff --git a/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs b/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs new file mode 100644 index 0000000000..d9a6014603 --- /dev/null +++ b/src/Exceptionless.Core/Models/WorkItems/ProjectSmartThrottleWorkItem.cs @@ -0,0 +1,14 @@ +using Foundatio.Queues; + +namespace Exceptionless.Core.Models.WorkItems; + +public record ProjectSmartThrottleWorkItem : IHaveUniqueIdentifier +{ + public required string OrganizationId { get; init; } + public required string ProjectId { get; init; } + public required double SampleRate { get; init; } + public required int CurrentEventCount { get; init; } + public required int EventLimit { get; init; } + + public string UniqueIdentifier => $"SmartThrottle:{OrganizationId}:{ProjectId}"; +} diff --git a/src/Exceptionless.Core/Services/UsageService.cs b/src/Exceptionless.Core/Services/UsageService.cs index 4a12491675..f42c2ef5b6 100644 --- a/src/Exceptionless.Core/Services/UsageService.cs +++ b/src/Exceptionless.Core/Services/UsageService.cs @@ -444,6 +444,19 @@ public async Task IncrementTotalAsync(string organizationId, string projectId, i long monthTotal = currentTotalCache.Value + bucketTotal; if (monthTotal >= maxEventsPerMonth && monthTotal - maxEventsPerMonth < eventCount) await _messagePublisher.PublishAsync(new PlanOverage { OrganizationId = organizationId }); + + // Check budget alert thresholds — non-fatal: alert failures must not break event ingestion + if (maxEventsPerMonth > 0) + { + try + { + await CheckBudgetAlertThresholdsAsync(organizationId, (int)monthTotal, maxEventsPerMonth); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to check budget alert thresholds for organization {OrganizationId}", organizationId); + } + } } if (bucketTotal >= bucketLimit && bucketTotal - bucketLimit < eventCount) @@ -454,6 +467,179 @@ public async Task IncrementTotalAsync(string organizationId, string projectId, i } } + /// + /// Gets the event ingest allowance for a project, taking into account both organization limits + /// and project-specific ingest limits. Returns how many events can be processed and the sample rate. + /// + public async Task GetEventIngestAllowanceAsync(string organizationId, string projectId) + { + int orgEventsLeft = await GetEventsLeftAsync(organizationId); + if (orgEventsLeft < 1) + return new EventIngestAllowance { EventsLeft = 0, SampleRate = 0, IsOverOrgLimit = true }; + + var project = await _projectRepository.GetByIdAsync(projectId, o => o.Cache()); + if (project?.IngestLimit is null) + return new EventIngestAllowance { EventsLeft = orgEventsLeft, SampleRate = 1.0 }; + + int effectiveLimit = await GetEffectiveProjectLimitAsync(project, organizationId); + if (effectiveLimit < 0) + return new EventIngestAllowance { EventsLeft = orgEventsLeft, SampleRate = 1.0 }; + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + int projectTotal = await GetProjectMonthTotalAsync(utcNow, organizationId, projectId); + + if (projectTotal >= effectiveLimit) + return new EventIngestAllowance { EventsLeft = 0, SampleRate = 0, IsOverProjectLimit = true, EffectiveProjectLimit = effectiveLimit }; + + int projectEventsLeft = effectiveLimit - projectTotal; + int eventsLeft = Math.Min(orgEventsLeft, projectEventsLeft); + + return new EventIngestAllowance { EventsLeft = eventsLeft, SampleRate = 1.0, EffectiveProjectLimit = effectiveLimit }; + } + + /// + /// Calculates the smart throttle sample rate for a project within the organization. + /// Returns 1.0 if no throttling is needed, or a value between 0 and 1 representing + /// the probability of accepting an event from this project. + /// + public async Task GetSmartThrottleRateAsync(string organizationId, string projectId) + { + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + + int maxEventsPerMonth = await GetMaxEventsPerMonthAsync(organizationId); + if (maxEventsPerMonth <= 0) + return SmartThrottleResult.NoThrottle; + + // Calculate fair share: what percentage of events has this project been using? + int orgTotal = await GetOrganizationMonthTotalAsync(utcNow, organizationId); + int projectTotal = await GetProjectMonthTotalAsync(utcNow, organizationId, projectId); + + if (orgTotal <= 0 || projectTotal <= 0) + return SmartThrottleResult.NoThrottle; + + // Only throttle when we're approaching the limit (>80%) + double usageRatio = (double)orgTotal / maxEventsPerMonth; + if (usageRatio < 0.8) + return SmartThrottleResult.NoThrottle; + + // Get total project count for org to determine fair share + var projectCount = await GetOrganizationProjectCountAsync(organizationId); + if (projectCount <= 1) + return SmartThrottleResult.NoThrottle; + + double fairShare = (double)maxEventsPerMonth / projectCount; + double projectShare = (double)projectTotal / orgTotal; + + // If this project is consuming more than 2x its fair share, apply throttling + double fairShareRatio = projectTotal / fairShare; + if (fairShareRatio <= 2.0) + return SmartThrottleResult.NoThrottle; + + // Scale sample rate: heavier consumers get more aggressive throttling + // At 2x fair share: rate = 0.5, at 4x: rate = 0.25, etc. + double sampleRate = Math.Max(0.1, 1.0 / fairShareRatio); + + return new SmartThrottleResult + { + SampleRate = sampleRate, + IsThrottled = true, + ProjectShare = projectShare, + FairShareRatio = fairShareRatio + }; + } + + private async Task CheckBudgetAlertThresholdsAsync(string organizationId, int currentTotal, int maxEventsPerMonth) + { + var organization = await _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + if (organization?.BudgetAlertSettings is not { Enabled: true }) + return; + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + double usagePercent = (double)currentTotal / maxEventsPerMonth * 100; + + foreach (int threshold in organization.BudgetAlertSettings.Thresholds) + { + if (usagePercent < threshold) + break; + + int thresholdEventCount = (int)Math.Ceiling((double)threshold / 100 * maxEventsPerMonth); + + string alertSentKey = GetBudgetAlertSentKey(utcNow, organizationId, threshold); + // Atomically mark as sent for this billing period — only the first writer (increment == 1) sends the alert. + long alertCount = await _cache.IncrementAsync(alertSentKey, 1, TimeSpan.FromDays(32)); + if (alertCount != 1) + continue; + + await _messagePublisher.PublishAsync(new OrganizationBudgetAlert + { + OrganizationId = organizationId, + Threshold = threshold, + ThresholdEventCount = thresholdEventCount, + CurrentEventCount = currentTotal, + EventLimit = maxEventsPerMonth + }); + } + } + + private async Task GetEffectiveProjectLimitAsync(Project project, string organizationId) + { + if (project.IngestLimit is null) + return -1; + + return project.IngestLimit.Type switch + { + ProjectIngestLimitType.Fixed => project.IngestLimit.FixedLimit ?? -1, + ProjectIngestLimitType.PercentOfOrganizationLimit => await CalculatePercentageLimitAsync(project.IngestLimit.PercentOfOrganizationLimit, organizationId), + _ => -1 + }; + } + + private async Task CalculatePercentageLimitAsync(decimal? percent, string organizationId) + { + if (percent is null or <= 0) + return -1; + + int maxEventsPerMonth = await GetMaxEventsPerMonthAsync(organizationId); + if (maxEventsPerMonth < 0) + return -1; + + return (int)Math.Ceiling(maxEventsPerMonth * (double)percent.Value / 100); + } + + private async Task GetProjectMonthTotalAsync(DateTime utcNow, string organizationId, string projectId) + { + var currentTotalCache = await _cache.GetAsync(GetTotalCacheKey(utcNow, organizationId, projectId)); + if (currentTotalCache.HasValue) + return currentTotalCache.Value; + + var project = await _projectRepository.GetByIdAsync(projectId, o => o.Cache()); + return project?.GetCurrentUsage(_timeProvider).Total ?? 0; + } + + private async Task GetOrganizationMonthTotalAsync(DateTime utcNow, string organizationId) + { + var currentTotalCache = await _cache.GetAsync(GetTotalCacheKey(utcNow, organizationId)); + if (currentTotalCache.HasValue) + return currentTotalCache.Value; + + var organization = await _organizationRepository.GetByIdAsync(organizationId, o => o.Cache()); + return organization?.GetCurrentUsage(_timeProvider).Total ?? 0; + } + + private async Task GetOrganizationProjectCountAsync(string organizationId) + { + var cacheKey = $"usage:project-count:{organizationId}"; + var cached = await _cache.GetAsync(cacheKey); + if (cached.HasValue) + return cached.Value; + + var countResult = await _projectRepository.GetCountByOrganizationIdAsync(organizationId); + long count = countResult.Total; + int projectCount = (int)Math.Max(1, count); + await _cache.SetAsync(cacheKey, projectCount, TimeSpan.FromHours(1)); + return projectCount; + } + public async Task IncrementBlockedAsync(string organizationId, string? projectId, int eventCount = 1) { if (eventCount <= 0) @@ -489,6 +675,42 @@ public async Task IncrementDiscardedAsync(string organizationId, string projectI AppDiagnostics.EventsDiscarded.Add(eventCount); } + /// + /// Records smart throttle discard and publishes a deduped notification (once per billing period per project). + /// + public async Task RecordSmartThrottleAsync(string organizationId, string projectId, int discardedCount, SmartThrottleResult throttleResult) + { + await IncrementDiscardedAsync(organizationId, projectId, discardedCount); + + var utcNow = _timeProvider.GetUtcNow().UtcDateTime; + string notifyKey = $"usage:smart-throttle-notified:{utcNow:yyyy-MM}:{organizationId}:{projectId}"; + if (await _cache.GetAsync(notifyKey) is { HasValue: true, Value: true }) + return; + + await _cache.SetAsync(notifyKey, true, TimeSpan.FromDays(32)); + + int projectTotal = await GetProjectMonthTotalAsync(utcNow, organizationId, projectId); + int maxEventsPerMonth = await GetMaxEventsPerMonthAsync(organizationId); + int projectCount = await GetOrganizationProjectCountAsync(organizationId); + int fairShareLimit = maxEventsPerMonth > 0 ? (int)(maxEventsPerMonth / Math.Max(1, projectCount)) : 0; + + try + { + await _messagePublisher.PublishAsync(new ProjectSmartThrottleApplied + { + OrganizationId = organizationId, + ProjectId = projectId, + SampleRate = throttleResult.SampleRate, + CurrentEventCount = projectTotal, + EventLimit = fairShareLimit + }); + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to publish smart throttle notification for project {ProjectId}", projectId); + } + } + public async Task IncrementTooBigAsync(string organizationId, string? projectId) { var utcNow = _timeProvider.GetUtcNow().UtcDateTime; @@ -620,6 +842,31 @@ private string GetProjectSetKey(DateTime utcTime) return $"usage:{bucket}:projects"; } + private string GetBudgetAlertSentKey(DateTime utcTime, string organizationId, int threshold) + { + int monthBucket = GetTotalBucket(utcTime); + return $"usage:budget-alert:{monthBucket}:{organizationId}:{threshold}"; + } + private int GetCurrentBucket(DateTime utcTime) => utcTime.Floor(_bucketSize).ToEpoch(); private int GetTotalBucket(DateTime utcTime) => utcTime.StartOfMonth().ToEpoch(); } + +public class EventIngestAllowance +{ + public int EventsLeft { get; init; } + public double SampleRate { get; init; } = 1.0; + public bool IsOverOrgLimit { get; init; } + public bool IsOverProjectLimit { get; init; } + public int EffectiveProjectLimit { get; init; } = -1; +} + +public class SmartThrottleResult +{ + public static readonly SmartThrottleResult NoThrottle = new() { SampleRate = 1.0 }; + + public double SampleRate { get; init; } = 1.0; + public bool IsThrottled { get; init; } + public double ProjectShare { get; init; } + public double FairShareRatio { get; init; } +} diff --git a/src/Exceptionless.Web/Bootstrapper.cs b/src/Exceptionless.Web/Bootstrapper.cs index f83edd3615..e408f0b1f3 100644 --- a/src/Exceptionless.Web/Bootstrapper.cs +++ b/src/Exceptionless.Web/Bootstrapper.cs @@ -41,5 +41,11 @@ public static void RegisterServices(IServiceCollection services, AppOptions appO services.AddSingleton(); services.AddStartupAction(); + + services.AddSingleton(); + services.AddStartupAction(); + + services.AddSingleton(); + services.AddStartupAction(); } } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts index 96d51972c1..ebfe8e744c 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/api.svelte.ts @@ -6,7 +6,7 @@ import { accessToken } from '$features/auth/index.svelte'; import { type FetchClientResponse, type ProblemDetails, useFetchClient } from '@exceptionless/fetchclient'; import { createMutation, createQuery, useQueryClient } from '@tanstack/svelte-query'; -import type { Invoice, InvoiceGridModel, NewOrganization, SuspensionCode, ViewOrganization } from './models'; +import type { Invoice, InvoiceGridModel, NewOrganization, SuspensionCode, UpdateOrganization, ViewOrganization } from './models'; export async function invalidateOrganizationQueries(queryClient: QueryClient, message: WebSocketMessageValue<'OrganizationChanged'>) { const { id } = message; @@ -389,9 +389,9 @@ export function getPlansQuery(request: GetPlansRequest) { export function patchOrganization(request: PatchOrganizationRequest) { const queryClient = useQueryClient(); - return createMutation(() => ({ + return createMutation(() => ({ enabled: () => !!accessToken.current && !!request.route.id, - mutationFn: async (data: NewOrganization) => { + mutationFn: async (data: UpdateOrganization) => { const client = useFetchClient(); const response = await client.patchJSON(`organizations/${request.route.id}`, data); return response.data!; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts index 85d1fe5de6..d05f925285 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/models.ts @@ -1,4 +1,14 @@ -export type { Invoice, InvoiceGridModel, NewOrganization, ViewOrganization } from '$generated/api'; +export type { Invoice, InvoiceGridModel, NewOrganization, OrganizationBudgetAlertSettings, ViewOrganization } from '$generated/api'; + +export interface UpdateOrganization { + name?: string; + budget_alert_settings?: OrganizationBudgetAlertSettingsUpdate | null; +} + +export interface OrganizationBudgetAlertSettingsUpdate { + enabled: boolean; + thresholds: number[]; +} // TODO: This should be generated from the backend enum - investigate why it wasn't included in the generated API export enum SuspensionCode { diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts index c380dca5e7..99884f2f60 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/organizations/schemas.ts @@ -1,9 +1,21 @@ -import { date, type infer as Infer, number, object, string, enum as zodEnum } from 'zod'; +import { date, type infer as Infer, number, object, string, enum as zodEnum, array, boolean } from 'zod'; import { SuspensionCode } from './models'; export { type NewOrganizationFormData, NewOrganizationSchema } from '$generated/schemas'; +export const BudgetAlertSettingsSchema = object({ + enabled: boolean(), + thresholds: array(number().int().min(1).max(100)) +}); +export type BudgetAlertSettingsFormData = Infer; + +export const UpdateOrganizationSchema = object({ + name: string().min(1, 'Name is required').optional(), + budget_alert_settings: BudgetAlertSettingsSchema.nullable().optional() +}); +export type UpdateOrganizationFormData = Infer; + export const SetBonusOrganizationSchema = object({ bonusEvents: number().int('Bonus events must be a whole number'), expires: date().optional() diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts index a055dfb212..fd0abbc731 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/models.ts @@ -1,6 +1,13 @@ +export { ProjectIngestLimitType } from '$generated/api'; export type { ClientConfiguration, NewProject, NotificationSettings, UpdateProject, ViewProject } from '$generated/api'; export interface ClientConfigurationSetting { key: string; value: string; } + +export interface ProjectIngestLimitUpdate { + type: 'fixed' | 'percent'; + fixed_limit?: number | null; + percent_of_organization_limit?: number | null; +} diff --git a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts index e495e7dd4f..f17b4c158b 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/features/projects/schemas.ts @@ -5,7 +5,7 @@ import { type NotificationSettingsFormData, NotificationSettingsSchema } from '$generated/schemas'; -import { type infer as Infer, object, string } from 'zod'; +import { discriminatedUnion, literal, type infer as Infer, nullable, number, object, optional, string } from 'zod'; export { type NewProjectFormData, NewProjectSchema, type NotificationSettingsFormData, NotificationSettingsSchema }; @@ -15,5 +15,25 @@ export const ClientConfigurationSettingSchema = object({ }); export type ClientConfigurationSettingFormData = Infer; +export const FixedIngestLimitSchema = object({ + type: literal(0), + fixed_limit: number().int().min(1, 'Limit must be at least 1').nullable().optional(), + percent_of_organization_limit: optional(nullable(number())) +}); + +export const PercentIngestLimitSchema = object({ + type: literal(1), + fixed_limit: optional(nullable(number())), + percent_of_organization_limit: number().min(1, 'Percentage must be at least 1').max(999, 'Percentage must be at most 999').nullable().optional() +}); + +export const IngestLimitSchema = discriminatedUnion('type', [FixedIngestLimitSchema, PercentIngestLimitSchema]); +export type IngestLimitFormData = Infer; + export const UpdateProjectSchema = GeneratedUpdateProjectSchema.partial(); export type UpdateProjectFormData = Infer; + +export const UpdateProjectIngestLimitSchema = object({ + ingest_limit: IngestLimitSchema.nullable().optional() +}); +export type UpdateProjectIngestLimitFormData = Infer; diff --git a/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts b/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts index 8ef1330255..5934489784 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/generated/api.ts @@ -7,6 +7,11 @@ export enum StackStatus { Discarded = "discarded", } +export enum ProjectIngestLimitType { + Fixed = 0, + PercentOfOrganizationLimit = 1, +} + export enum BillingStatus { Trialing = 0, Active = 1, @@ -112,6 +117,8 @@ export interface InvoiceLineItem { amount: number; } +export type JsonElement = any; + export interface Login { /** The email address or domain username */ email: string; @@ -128,6 +135,7 @@ export interface NewProject { organization_id: string; name: string; delete_bot_data_enabled: boolean; + ingest_limit?: null | ProjectIngestLimit; } export interface NewSavedView { @@ -137,6 +145,7 @@ export interface NewSavedView { filter?: null | string; time?: null | string; sort?: null | string; + /** @pattern ^[a-z0-9]+(?:-[a-z0-9]+)*$ */ slug?: null | string; view_type: string; filter_definitions?: null | string; @@ -192,6 +201,15 @@ export interface OAuthAccount { extra_data: Record; } +export interface OrganizationBudgetAlertSettings { + enabled: boolean; + /** + * Percentage thresholds of the organization's effective monthly event allowance. + * Example: [50, 80, 90]. + */ + thresholds: number[]; +} + export interface PersistentEvent { /** * Unique id that identifies an event. @@ -256,6 +274,29 @@ export interface PersistentEvent { reference_id?: null | string; } +export interface PredefinedSavedViewDefinition { + key: string; + name: string; + slug: string; + viewType: string; + filter?: null | string; + time?: null | string; + sort?: null | string; + filterDefinitions?: null | JsonElement; + columns?: null | Record; + columnOrder?: string[] | null; + showStats?: null | boolean; + showChart?: null | boolean; +} + +export interface ProjectIngestLimit { + type: ProjectIngestLimitType; + /** @format int32 */ + fixed_limit?: null | number; + /** @format double */ + percent_of_organization_limit?: null | number; +} + export interface ResetPasswordModel { password_reset_token: string; password: string; @@ -364,10 +405,17 @@ export interface UpdateEvent { description?: null | string; } +/** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ +export interface UpdateOrganization { + name: string; + budget_alert_settings: object; +} + /** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ export interface UpdateProject { name: string; delete_bot_data_enabled: boolean; + ingest_limit: object; } /** A class the tracks changes (i.e. the Delta) for a particular TEntityType. */ @@ -538,6 +586,7 @@ export interface ViewOrganization { is_throttled: boolean; is_over_monthly_limit: boolean; is_over_request_limit: boolean; + budget_alert_settings?: null | OrganizationBudgetAlertSettings; } export interface ViewProject { @@ -559,6 +608,12 @@ export interface ViewProject { event_count: number; has_premium_features: boolean; has_slack_integration: boolean; + ingest_limit?: null | ProjectIngestLimit; + /** @format int32 */ + effective_ingest_limit?: null | number; + is_smart_throttled: boolean; + /** @format double */ + smart_throttle_sample_rate?: null | number; usage_hours: UsageHourInfo[]; usage: UsageInfo[]; } diff --git a/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts b/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts index 8642423da4..302631d358 100644 --- a/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts +++ b/src/Exceptionless.Web/ClientApp/src/lib/generated/schemas.ts @@ -27,6 +27,7 @@ export const StackStatusSchema = zodEnum([ "ignored", "discarded", ]); +export const ProjectIngestLimitTypeSchema = union([literal(0), literal(1)]); export const BillingStatusSchema = union([ literal(0), literal(1), @@ -169,6 +170,7 @@ export const NewProjectSchema = object({ .regex(/^[a-fA-F0-9]{24}$/, "Organization id has invalid format"), name: string().min(1, "Name is required"), delete_bot_data_enabled: boolean(), + ingest_limit: lazy(() => ProjectIngestLimitSchema).optional(), }); export type NewProjectFormData = Infer; @@ -277,6 +279,14 @@ export const OAuthAccountSchema = object({ }); export type OAuthAccountFormData = Infer; +export const OrganizationBudgetAlertSettingsSchema = object({ + enabled: boolean(), + thresholds: array(number()), +}); +export type OrganizationBudgetAlertSettingsFormData = Infer< + typeof OrganizationBudgetAlertSettingsSchema +>; + export const PersistentEventSchema = object({ id: string() .length(24, "Id must be exactly 24 characters") @@ -320,6 +330,31 @@ export const PersistentEventSchema = object({ }); export type PersistentEventFormData = Infer; +export const PredefinedSavedViewDefinitionSchema = object({ + key: string().min(1, "Key is required"), + name: string().min(1, "Name is required"), + slug: string().min(1, "Slug is required"), + viewType: string().min(1, "View type is required"), + filter: string().min(1, "Filter is required").nullable().optional(), + time: string().min(1, "Time is required").nullable().optional(), + sort: string().min(1, "Sort is required").nullable().optional(), + filterDefinitions: lazy(() => JsonElementSchema).optional(), + columns: record(string(), boolean()).nullable().optional(), + columnOrder: array(string()).nullable().optional(), + showStats: boolean().nullable().optional(), + showChart: boolean().nullable().optional(), +}); +export type PredefinedSavedViewDefinitionFormData = Infer< + typeof PredefinedSavedViewDefinitionSchema +>; + +export const ProjectIngestLimitSchema = object({ + type: ProjectIngestLimitTypeSchema, + fixed_limit: int32().nullable().optional(), + percent_of_organization_limit: number().nullable().optional(), +}); +export type ProjectIngestLimitFormData = Infer; + export const ResetPasswordModelSchema = object({ password_reset_token: string().length( 40, @@ -414,9 +449,16 @@ export const UpdateEventSchema = object({ }); export type UpdateEventFormData = Infer; +export const UpdateOrganizationSchema = object({ + name: string().min(1, "Name is required").optional(), + budget_alert_settings: record(string(), unknown()).optional(), +}); +export type UpdateOrganizationFormData = Infer; + export const UpdateProjectSchema = object({ name: string().min(1, "Name is required").optional(), delete_bot_data_enabled: boolean().optional(), + ingest_limit: record(string(), unknown()).optional(), }); export type UpdateProjectFormData = Infer; @@ -425,12 +467,7 @@ export const UpdateSavedViewSchema = object({ filter: string().min(1, "Filter is required").nullable().optional(), time: string().min(1, "Time is required").nullable().optional(), sort: string().min(1, "Sort is required").nullable().optional(), - slug: string() - .min(1, "Slug is required") - .max(100, "Slug must be at most 100 characters") - .regex(/^[a-z0-9]+(?:-[a-z0-9]+)*$/, "Slug has invalid format") - .nullable() - .optional(), + slug: string().min(1, "Slug is required").nullable().optional(), filter_definitions: string() .min(1, "Filter definitions is required") .nullable() @@ -577,6 +614,9 @@ export const ViewOrganizationSchema = object({ is_throttled: boolean(), is_over_monthly_limit: boolean(), is_over_request_limit: boolean(), + budget_alert_settings: lazy( + () => OrganizationBudgetAlertSettingsSchema, + ).optional(), }); export type ViewOrganizationFormData = Infer; @@ -598,6 +638,10 @@ export const ViewProjectSchema = object({ event_count: int(), has_premium_features: boolean(), has_slack_integration: boolean(), + ingest_limit: lazy(() => ProjectIngestLimitSchema).optional(), + effective_ingest_limit: int32().nullable().optional(), + is_smart_throttled: boolean(), + smart_throttle_sample_rate: number().nullable().optional(), usage_hours: array(lazy(() => UsageHourInfoSchema)), usage: array(lazy(() => UsageInfoSchema)), }); @@ -633,10 +677,7 @@ export const ViewSavedViewSchema = object({ show_stats: boolean().nullable().optional(), show_chart: boolean().nullable().optional(), name: string().min(1, "Name is required"), - slug: string() - .min(1, "Slug is required") - .max(100, "Slug must be at most 100 characters") - .regex(/^[a-z0-9]+(?:-[a-z0-9]+)*$/, "Slug has invalid format"), + slug: string().min(1, "Slug is required"), time: string().min(1, "Time is required").nullable().optional(), sort: string().min(1, "Sort is required").nullable().optional(), version: int32(), diff --git a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/manage/+page.svelte b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/manage/+page.svelte index 617d3654c8..147738944b 100644 --- a/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/manage/+page.svelte +++ b/src/Exceptionless.Web/ClientApp/src/routes/(app)/organization/[organizationId]/manage/+page.svelte @@ -4,17 +4,20 @@ import { page } from '$app/state'; import ErrorMessage from '$comp/error-message.svelte'; import { Muted } from '$comp/typography'; + import { Badge } from '$comp/ui/badge'; import { Button, buttonVariants } from '$comp/ui/button'; import * as DropdownMenu from '$comp/ui/dropdown-menu'; import * as Field from '$comp/ui/field'; import { Input } from '$comp/ui/input'; import { Spinner } from '$comp/ui/spinner'; + import { Switch } from '$comp/ui/switch'; import { deleteOrganization, getOrganizationQuery, patchOrganization } from '$features/organizations/api.svelte'; import RemoveOrganizationDialog from '$features/organizations/components/dialogs/remove-organization-dialog.svelte'; import { type NewOrganizationFormData, NewOrganizationSchema } from '$features/organizations/schemas'; import { ariaInvalid, getFormErrorMessages, mapFieldErrors, problemDetailsToFormErrors } from '$features/shared/validation'; import { ProblemDetails } from '@exceptionless/fetchclient'; import Stacks from '@lucide/svelte/icons/layers'; + import Plus from '@lucide/svelte/icons/plus'; import X from '@lucide/svelte/icons/x'; import { createForm } from '@tanstack/svelte-form'; import { toast } from 'svelte-sonner'; @@ -70,7 +73,7 @@ onSubmitAsync: async ({ value }) => { toast.dismiss(toastId); try { - await update.mutateAsync(value); + await update.mutateAsync({ name: value.name }); toastId = toast.success('Successfully updated Organization name'); return null; } catch (error: unknown) { @@ -87,10 +90,68 @@ const debouncedFormSubmit = debounce(1000, () => form.handleSubmit()); + // Budget alert settings state + const budgetSettings = $derived(organizationQuery.data?.budget_alert_settings); + let budgetEnabled = $state(false); + let thresholds = $state([]); + let newThreshold = $state(''); + let budgetSaving = $state(false); + let thresholdError = $state(''); + + $effect(() => { + budgetEnabled = budgetSettings?.enabled ?? false; + thresholds = [...(budgetSettings?.thresholds ?? [])].sort((a, b) => a - b); + }); + + async function saveBudgetSettings() { + toast.dismiss(toastId); + budgetSaving = true; + try { + await update.mutateAsync({ + budget_alert_settings: { + enabled: budgetEnabled, + thresholds: [...thresholds].sort((a, b) => a - b) + } + }); + toastId = toast.success('Budget alert settings saved.'); + } catch (error: unknown) { + const message = error instanceof ProblemDetails ? error.title : 'Please try again.'; + toastId = toast.error(`Error saving budget settings: ${message}`); + } finally { + budgetSaving = false; + } + } + + function addThreshold() { + thresholdError = ''; + const val = parseInt(newThreshold, 10); + if (isNaN(val) || val < 1 || val > 99) { + thresholdError = 'Enter a percentage between 1 and 99.'; + return; + } + if (thresholds.includes(val)) { + thresholdError = 'That threshold is already added.'; + return; + } + thresholds = [...thresholds, val].sort((a, b) => a - b); + newThreshold = ''; + } + + function removeThreshold(val: number) { + thresholds = thresholds.filter((t) => t !== val); + } + + function handleThresholdKeydown(e: KeyboardEvent) { + if (e.key === 'Enter') { + e.preventDefault(); + addThreshold(); + } + } + // TODO: Add Skeleton -
+
General organization settings
+
+
+

Budget alerts

+ Send email alerts when monthly event usage crosses percentage thresholds. +
+ +
+
+
+
Enable budget alerts
+ Receive email notifications when usage thresholds are crossed. +
+ (budgetEnabled = checked)} + aria-label="Enable budget alerts" + /> +
+ + {#if budgetEnabled} +
+
Alert thresholds
+
+ {#each thresholds as threshold (threshold)} + + {threshold}% + + + {:else} + No thresholds set. Add at least one below. + {/each} +
+ +
+
+ + New threshold percentage + + {#if thresholdError} +

{thresholdError}

+ {/if} +
+
+ +
+
+ {/if} + +
+ +
+
+
+
+
+
+ +