Skip to content

feat(python): expose get_stats on IggyClient - #4018

Open
7487 wants to merge 3 commits into
apache:masterfrom
7487:python-sdk-get-stats
Open

feat(python): expose get_stats on IggyClient#4018
7487 wants to merge 3 commits into
apache:masterfrom
7487:python-sdk-get-stats

Conversation

@7487

@7487 7487 commented Sep 1, 2026

Copy link
Copy Markdown

Which issue does this PR address?

Closes #4016

Rationale

get_stats is the server's headline diagnostic call and is exposed by every other SDK; the Python SDK could not reach it.

What changed?

The Python SDK had no binding for get_stats, so server counts, host/version details and cache metrics were unreachable from Python.

A new foreign/python/src/stats.rs wraps Stats, CacheMetrics and CacheMetricsKey following the user.rs pattern. CacheMetricsKey is frozen with __eq__/__hash__, so Stats.cache_metrics converts to dict[CacheMetricsKey, CacheMetrics]. IggyClient.get_stats returns an awaitable resolving to Stats; byte sizes are exposed as integer bytes and times as microseconds, matching the existing getters. Stubs were regenerated with cargo run --bin stub_gen (purely additive diff after ruff).

tests/test_stats.py creates a stream/topic, sends messages, and asserts the stream/topic/partition/message counts moved, the version string is non-empty, and the cache-metrics dict round-trips through key lookup.

Local Execution

  • Passed
  • Pre-commit hooks ran

Ran against a locally built iggy-server from this branch: pytest tests/ gives 323 passed (only test_tls errors locally for lack of a Docker daemon, unrelated). cargo fmt, cargo clippy --all-features --all-targets, ruff check/format and pyrefly are clean.

AI Usage

  1. Claude Code (Fable 5).
  2. Entire implementation and tests, following the pattern proposed in the issue.
  3. Built the extension and ran the new and full Python test suites against a locally built server; regenerated and diffed the stubs; ran clippy/fmt/ruff/pyrefly.
  4. Yes.

The Python SDK had no way to reach the server's headline diagnostic
call, exposed by every other SDK.

Wrap Stats, CacheMetrics and CacheMetricsKey in a new stats module
following the user.rs pattern. CacheMetricsKey is frozen, hashable and
comparable so cache_metrics maps to dict[CacheMetricsKey, CacheMetrics].
Byte sizes are exposed as integer bytes, times as microseconds,
matching the existing getters.

Closes apache#4016
@github-actions

github-actions Bot commented Sep 1, 2026

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer

See CONTRIBUTING.md for details.

@codecov

codecov Bot commented Sep 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 43.75000% with 72 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.97%. Comparing base (328b289) to head (fe8d21d).
⚠️ Report is 29 commits behind head on master.

Files with missing lines Patch % Lines
foreign/python/src/stats.rs 37.39% 72 Missing ⚠️

❌ Your patch check has failed because the patch coverage (43.75%) is below the target coverage (50.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4018      +/-   ##
============================================
- Coverage     85.00%   84.97%   -0.03%     
  Complexity     1402     1402              
============================================
  Files          1225     1226       +1     
  Lines        180283   180411     +128     
  Branches     146587   146587              
============================================
+ Hits         153248   153304      +56     
- Misses        22993    23065      +72     
  Partials       4042     4042              
Components Coverage Δ
Rust Core 85.90% <ø> (ø)
Java SDK 67.29% <ø> (ø)
C# SDK 75.37% <ø> (ø)
Python SDK 87.63% <43.75%> (-2.44%) ⬇️
PHP SDK 85.65% <ø> (ø)
Node SDK 96.24% <ø> (ø)
Go SDK 69.31% <ø> (ø)
Files with missing lines Coverage Δ
foreign/python/src/client.rs 99.85% <100.00%> (+<0.01%) ⬆️
foreign/python/src/lib.rs 100.00% <100.00%> (ø)
foreign/python/src/stats.rs 37.39% <37.39%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@justinmclean

Copy link
Copy Markdown
Member

Nice addition, follows the existing user.rs wrapper pattern and the stub is regenerated. Nothing blocking; a few things worth a look.

  • src/stats.rs:30CacheMetricsKey has no #[new], so Python can't construct one. The dict ends up iterate-only: you can't ask for stream 1 / topic 2 / partition 0 directly. A constructor would fix that cheaply.
  • src/stats.rs:264cache_metrics is a #[getter], so every access re-collects the whole map. Iterating the property and indexing it in the same loop is quadratic, which is what the test does at tests/test_stats.py:66 and :70. Building it once in From<RustStats> would avoid the surprise.
  • tests/test_stats.py:66 — the loop body is the only thing test_get_stats_cache_metrics_dict asserts, and on a server with no cached partitions it never runs. Sending and polling messages first, then asserting the map is non-empty, would make it test the conversion.
  • tests/test_stats.py:70-:73stats.cache_metrics[key] is not None can't fail (a miss raises KeyError), and >= 0 on u32 / u64 is always true.
  • tests/test_stats.py:47-:49 — exact equality on server-global counters. Fine today since the suite runs serially, but pytest-xdist is a declared dev dependency; >= avoids the trap.
  • src/stats.rs:2551.2.3 -> 100200300 should be 1002003. Inherited verbatim from core/common/src/types/stats/mod.rs:73, so not yours. Fixing it means the Rust doc comment plus a stub_gen regen, ideally in core/common too.

This review was drafted by an AI-assisted tool (Apache Magpie), so it may contain mistakes. If you think one of them is misapplied, please reply on the PR, and a maintainer will weigh in.

@ethanlin01x ethanlin01x left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Three small things, mostly about matching the SDK's existing conventions.

Comment thread foreign/python/src/stats.rs Outdated

/// The run time of the server process, in microseconds.
#[getter]
pub fn run_time(&self) -> u64 {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run_time is an IggyDuration, and this SDK already exposes durations as datetime.timedelta. Returning raw microseconds here is inconsistent.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 7a28235run_time now returns datetime.timedelta via the shared iggy_duration_to_py_delta helper, and the stub is regenerated.

Comment thread foreign/python/src/stats.rs Outdated
/// The numeric semantic version of the Iggy server, or `None` when unknown.
/// E.g. 1.2.3 -> 100200300 (major * 1000000 + minor * 1000 + patch).
#[getter]
#[gen_stub(override_return_type(type_repr = "int | None"))]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: the other overrides write builtins.int | None (e.g. config.rs:191). Same meaning here, but this string is copied verbatim into the generated stub, so it stays as the only hand-written bare int across every future regen.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 7a28235 — the override now reads builtins.int | None, and the regenerated stub carries it.

"""

@typing.final
class Stats:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CacheMetrics and CacheMetricsKey both define __repr__, but Stats does not. Since get_stats is a diagnostic call, print(stats) showing an object address is not great. A short repr with a few key fields would help.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in 7a28235Stats.__repr__ now prints hostname, server version and the headline counters (streams/topics/partitions/messages/clients).

@slbotbm

slbotbm commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

/author

@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Sep 3, 2026
7487 and others added 2 commits September 4, 2026 18:19
- CacheMetricsKey gets a #[new] constructor so a key built in Python can
  address a cache_metrics dict entry directly.
- cache_metrics is converted once in From<RustStats> and stored as a
  Py<PyDict>; every access returns the same dict instead of re-collecting
  the whole map.
- run_time is exposed as datetime.timedelta via the shared duration
  helper, matching the SDK's other duration surfaces.
- The semver stub override uses the builtins.int | None convention, and
  Stats gains a __repr__ with the headline fields.
- The numeric semver docstring example is corrected to 1.2.3 -> 1002003
  here and in core/common (get_numeric_version pads minor/patch to three
  digits).
- Tests compare server-global counters with >= (pytest-xdist safe), drop
  assertions that could not fail, and cover key construction, hashing and
  dict addressing without a server. The cache metrics map itself stays
  empty for now: the server replies with a hardcoded empty map.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@7487

7487 commented Sep 4, 2026

Copy link
Copy Markdown
Author

Thanks for the thorough pass — all addressed, point by point:

  • CacheMetricsKey constructor: added #[new], so CacheMetricsKey(stream_id=1, topic_id=2, partition_id=3) now works, and there is a new test that an independently constructed equal key addresses a dict entry (eq/hash round-trip).
  • cache_metrics re-collection: the map is now converted exactly once in From<RustStats> and stored as a Py<PyDict>; the getter hands back the same dict on every access, so iterate-and-index is linear. The test asserts stats.cache_metrics is stats.cache_metrics to pin that down.
  • Non-empty assertion: I could not make that work, and traced why — the server currently hardcodes an empty map in the GetStats reply (core/server/src/responses.rs: cache_metrics: Vec::new()), so the map stays empty no matter how much traffic precedes the call. A non-empty assertion would fail unconditionally. I left the per-entry type checks (with a comment explaining they only run once the server starts populating the field) and covered the key/dict semantics in the new server-free test instead.
  • Vacuous assertions (is not None, >= 0 on unsigned): dropped.
  • Exact equality on server-global counters: switched to >= with a comment naming the pytest-xdist reason.
  • Semver docstring: you're right — get_numeric_version formats {major}{minor:03}{patch:03}, so 1.2.3 -> 1002003. Fixed in the binding, in core/common/src/types/stats/mod.rs, and the stub is regenerated.

Also merged master to catch the branch up.

@7487

7487 commented Sep 4, 2026

Copy link
Copy Markdown
Author

/ready

@github-actions github-actions Bot added S-waiting-on-review PR is waiting on a reviewer and removed S-waiting-on-author PR is waiting on author response labels Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Python SDK: expose get_stats

4 participants