feat(dashboard): set Dependency Latency visualization to Heatmap

[anant-shipit]

Fixes #97

Design Decisions Summary

Grouping & Panel Choice:
This metric remains correctly grouped under "System Availability", but the default timeseries panel compressed request duration distributions into a single line, masking bimodal patterns.

Visualization Rationale:
I chose a Heatmap for this panel. Latency is fundamentally a distribution, not a single average. A heatmap visualizes the distribution of request durations over time, making it easy to instantly spot long-tail outliers (like degraded downstream servers) that a simple line chart would hide.

Go Metric Modifications:
No Go metric types were modified. However, the PromQL query was updated to use sum(rate(http_outgoing_request_duration_seconds_bucket[5m])) by (le) to properly feed the heatmap.

Summary by CodeRabbit

• New Features
  • Updated the Dependency Latency dashboard panel to display request latency as a heatmap.
  • Added clearer bucket-based visualization and histogram tooltips for easier latency analysis.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 9e5564e0-ac2a-411f-997a-93ea0b222906

📥 Commits

Reviewing files that changed from the base of the PR and between d93c501 and f457357.

📒 Files selected for processing (1)

• grafana/dashboards/watchdog_metrics_dashboard.json

🚧 Files skipped from review as they are similar to previous changes (1)

• grafana/dashboards/watchdog_metrics_dashboard.json

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

---

📝 Walkthrough

Walkthrough

The Dependency Latency panel changes from a Grafana time series to a heatmap. Its Prometheus target uses outgoing request duration histogram buckets grouped by le, with heatmap rendering and seconds-based axis settings.

Changes

Dependency latency visualization

Layer / File(s)	Summary
Configure dependency latency heatmap grafana/dashboards/watchdog_metrics_dashboard.json	Panel 102 uses heatmap rendering and replaces the duration-sum query with a bucket-based query grouped by le and formatted as a heatmap. The panel adds field, color, tooltip, legend, filtering, row, and axis settings.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to f4573

This localized dashboard change updates the Dependency Latency panel to a heatmap and adjusts its PromQL query; no actionable merge-blocking risk remains beyond normal checks and review.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Linked Issues check	⚠️ Warning	The changes implement the heatmap visualization and histogram-bucket query requested by issue #97. The provided context does not show documentation of the visualization rationale or an assessment of s…	Document why the heatmap is the preferred visualization and state whether server-level filtering provides value. Add the corresponding dashboard filter if the assessment requires it; otherwise document why it is not needed.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the main change: replacing the Dependency Latency timeseries visualization with a heatmap.
Out of Scope Changes check	✅ Passed	The changes are limited to the Dependency Latency Grafana panel and directly support issue #97. No unrelated code or configuration changes are shown.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…

Full details: Linked Issues check

Explanation

The changes implement the heatmap visualization and histogram-bucket query requested by issue #97. The provided context does not show documentation of the visualization rationale or an assessment of server-level filtering, which are also required by the issue.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@grafana/dashboards/watchdog_metrics_dashboard.json`:
- Line 198: Update the PromQL expression in the dashboard’s outgoing request
duration panel to replace the no-op url matcher with server_id=~"$server_id",
preserving the existing rate, sum, and le aggregation so the dashboard respects
server-level filtering.
- Line 161: In the dashboard panel configuration, change the "calculate" setting
from true to false so Grafana uses the pre-calculated Prometheus histogram
buckets and preserves their "le" boundaries.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 15a9f4c4-611e-461f-a1e8-7a8091ffc913

📥 Commits

Reviewing files that changed from the base of the PR and between af77c99 and 5aa9742.

📒 Files selected for processing (1)

• grafana/dashboards/watchdog_metrics_dashboard.json

[coveralls]

coverage: 66.398% (+15.1%) from 51.344% — anant-shipit:feat/issue-97-dependency-latency into OneBusAway:main