Skip to content

feat(database): optimize timescaledb queries with continuous aggregat… - #998

Merged
Mosas2000 merged 2 commits into
StellaBridge:mainfrom
DevNetlife:feat/timescaledb-continuous-aggregates
Jul 29, 2026
Merged

feat(database): optimize timescaledb queries with continuous aggregat…#998
Mosas2000 merged 2 commits into
StellaBridge:mainfrom
DevNetlife:feat/timescaledb-continuous-aggregates

Conversation

@DevNetlife

Copy link
Copy Markdown
Contributor

Description
Over 90 days of operation, the prices, health_scores, and liquidity_snapshots hypertables accumulate millions of rows. Aggregating data over long ranges ($\ge 7$ days, 30 days, 90 days) on the Analytics dashboard previously executed expensive raw table queries.

This PR introduces TimescaleDB continuous aggregate materialized views with automated refresh policies for 1-hour and 1-day windows across high-volume hypertables, and refactors backend services to route queries to continuous aggregates when requesting data ranges $\ge 7$ days.

Type of Change
Optimization (performance enhancement)
Database Migration
Refactoring (no logic breaking changes)
Key Changes

  1. Database Migration
    Created migration 048_continuous_aggregates_optimizations.ts defining TimescaleDB continuous aggregate materialized views:
    prices_hourly & prices_daily
    health_scores_hourly & health_scores_daily
    liquidity_hourly & liquidity_daily
    Configured automated refresh policies via add_continuous_aggregate_policy for 1-hour and 1-day bucket intervals.
    Added performance indexes on (symbol, bucket DESC) for each aggregate view.
  2. Service Refactoring
    PriceModel (backend/src/database/models/price.model.ts): Refactored getTimeBucketed() to target prices_daily for lookbacks $\ge 30$ days and prices_hourly for lookbacks $\ge 7$ days, falling back to raw hypertables if unaggregated or in test environments.
    AnalyticsService (backend/src/services/analytics.service.ts): Refactored getHistoricalComparison() for health_score, liquidity, and price metrics to query continuous aggregates when requested range is $\ge 7$ days.
    AggregationService (backend/src/services/aggregation.service.ts): Refactored aggregatePrices(), aggregateHealthScores(), and aggregateVolume() to query continuous aggregate views for long time ranges ($\ge 7$ days) or daily/weekly/monthly intervals.
    LiquidityFragmentationService (backend/src/services/liquidityFragmentation.service.ts): Refactored getFragmentationTrend() for 7d and 30d ranges to query liquidity_hourly and liquidity_daily.
  3. Testing & Verification
    Added test file backend/tests/services/continuousAggregates.service.test.ts to verify query routing logic for $\ge 7$-day ranges.
    Confirmed all service unit tests pass cleanly:
    Test Files 5 passed (5)
    Tests 79 passed (79)
    Impact & Compatibility
    Breaking Change: No
    Short-term Feeds: Unaffected (queries under 7 days continue to read directly from raw hypertables).

Closes #798

…es (StellaBridge#798)

Add migration 048 to configure continuous aggregate views and refresh policies on prices, health_scores, and liquidity_snapshots hypertables. Refactor PriceModel, AnalyticsService, AggregationService, and LiquidityFragmentationService to query continuous aggregates when range >= 7 days.
…llaBridge#798)

Add browser caching step for ~/.cache/ms-playwright and increase job timeout-minutes from 20 to 30 to prevent Playwright installation timeouts.
@Mosas2000

Copy link
Copy Markdown
Contributor

Fix the failing E2E check

@Mosas2000

Copy link
Copy Markdown
Contributor

The implementation of TimescaleDB continuous aggregates is excellently structured, featuring robust fallback mechanisms that will significantly enhance long-range query performance.

@Mosas2000

Copy link
Copy Markdown
Contributor

The thorough test coverage effectively validates the newly optimized routing logic.

Thank you, approving this for merge.

@Mosas2000
Mosas2000 merged commit 94a62d5 into StellaBridge:main Jul 29, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

change: Optimize TimescaleDB Hypertable Queries with Continuous Aggregates and Real-Time Materialized Views

2 participants