Skip to content

Replace distribution health ratios with packet observations - #140

Merged
zhongwencool merged 2 commits into
mainfrom
zw/system-distribution-observations
Sep 28, 2026
Merged

zhongwencool merged 2 commits into
mainfrom
zw/system-distribution-observations

Conversation

@zhongwencool

Copy link
Copy Markdown
Owner

Summary

The System page treated the fourth erlang:dist_get_stat/1 result (pending output packets) as bytes, divided it by dist_buf_busy_limit, and inferred ok/warn health. Those quantities cannot produce a meaningful capacity percentage.

This PR replaces that inference with useful, explicitly scoped observations:

  • Keep connection state, address, and type; remove the distribution Health and Percent columns.
  • Show Rx pkt/s and Tx pkt/s from packet-counter deltas over actual monotonic elapsed time.
  • Show Pending pkts and up to three recent observations, oldest to newest. Oversized histories explicitly elide older values instead of truncating the latest observation.
  • Display Dist busy limit (bytes) independently in runtime configuration.
  • Reset rates/history after reconnection, counter reset, or unavailable sampling; show - while establishing a baseline and N/A for unavailable statistics.
  • Guard internal OTP statistics against missing tables, changed shapes, invalid handles, and disappearing connections.
  • Update the TUI reference and add deterministic and live-peer regression tests.

Scope and semantics

Packet activity is not application-message throughput, capacity utilization, or a health score. Nonzero pending samples do not prove continuous blocking between samples. The CLI's controller-queue context remains unchanged. This PR does not install system monitors, add backpressure diagnostics, or introduce a controller-queue detail view.

Validation

  • mise exec -- rebar3 eunit: 800 tests passed on OTP 28.
  • mise exec erlang@29.0.6 -- rebar3 eunit --module=observer_cli_system_test: 34 tests passed, including real peer traffic and reconnect baseline resets.
  • mise exec -- rebar3 as ci compile: passed.
  • mise exec -- rebar3 fmt --check: passed.
  • git diff --check and staged diff checks: passed.
  • Verified real peer activity and reset-on-reconnection rendering in an OTP 29 terminal session; layout tests cover 139, 180, and 201 columns and ANSI boundaries.

Hosted CI is separate from these local results; the full OTP compatibility matrix was not run locally.

Changes:
- Remove the invalid packet-to-byte percentage and derived distribution health labels.
- Show packet rates using actual elapsed time, pending packet counts, and three-sample history per connection.
- Reset sampling across reconnects, counter resets, and unavailable samples; protect internal OTP probes against failures.
- Label the busy limit as an independent byte-valued configuration and document the observation-only semantics.
- Add sampling, availability, layout, and live peer reconnection regression tests without changing CLI semantics or installing monitors.

Validation:
- mise exec -- rebar3 eunit: 800 tests passed on OTP 28.
- mise exec erlang@29.0.6 -- rebar3 eunit --module=observer_cli_system_test: 34 tests passed.
- mise exec -- rebar3 as ci compile: passed.
- mise exec -- rebar3 fmt --check: passed.
- git diff --check and git diff --cached --check: passed.
- Verified real peer packet activity and reset-on-reconnection output in an OTP 29 terminal session.
- Hosted CI and the full OTP compatibility matrix have not been run locally.
Changes:
- Remove the fallback clause already excluded by net_kernel:nodes_info/0's return type, fixing Dialyzer failures across OTP 26 through 29.
- Preserve the outer exception handler for unavailable distribution information and disappearing resources.

Validation:
- mise exec -- rebar3 as ci dialyzer: passed on OTP 28 with no warnings.
- mise exec -- rebar3 eunit --module=observer_cli_system_test: 34 tests passed.
- mise exec -- rebar3 fmt --check: passed.
- git diff --check: passed.
- The full suite was not rerun locally for this unreachable-clause removal; the hosted matrix will validate the updated commit.
@codecov

codecov Bot commented Sep 28, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 94.54545% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.88%. Comparing base (04aee9f) to head (6304d2f).

Files with missing lines Patch % Lines
src/observer_cli_system.erl 94.54% 3 Missing ⚠️

❌ Your project check has failed because the head coverage (97.88%) is below the target coverage (98.00%). You can increase the head coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #140      +/-   ##
==========================================
- Coverage   97.91%   97.88%   -0.04%     
==========================================
  Files          24       24              
  Lines        7928     7939      +11     
==========================================
+ Hits         7763     7771       +8     
- Misses        165      168       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@zhongwencool
zhongwencool merged commit 30bfc1a into main Sep 28, 2026
7 of 8 checks passed
@zhongwencool
zhongwencool deleted the zw/system-distribution-observations branch September 28, 2026 07:29
@zhongwencool

Copy link
Copy Markdown
Owner Author
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant