Skip to content

docs(python): build the Python reference from the type stub, for Read the Docs - #161

Closed
JosteinGj wants to merge 4 commits into
mainfrom
docs/python-stubs-sphinx
Closed

JosteinGj wants to merge 4 commits into
mainfrom
docs/python-stubs-sphinx

Conversation

@JosteinGj

Copy link
Copy Markdown
Contributor

What this does

Builds the Python client's API reference from this repository, for Read the Docs. The type stub
datahub_python_bindings/python/intellistream_datahub_sdk/__init__.pyi is now the single source of
the reference: signatures and prose alike. IDEs read the same file on hover, so the site and the
tooltip cannot disagree. The build parses the stub statically and needs no cargo and no compiled
module.

pip install -r docs-python/requirements.txt
sphinx-build -W -b html docs-python docs-python/_build

How the site is built

  • Service pages (timeseries, datasets, …) are written by a small local Sphinx extension,
    docs-python/_ext/service_pages.py. Service classes are never named by a caller, so methods
    read timeseries.by_ids(...) rather than TimeSeriesServiceSync.by_ids(...). Each page opens
    with a sync/async table and groups its methods under Find / Change / … from
    docs-python/structure.toml.
  • Everything else (clients, entities, filters, updates, …) comes from sphinx-autoapi, one page
    per class, grouped by hand in docs-python/*.rst.
  • Drift fails the build. A service method missing from structure.toml, a name in it that
    no service has, and a class in no group all fail -W.

Stub changes

  • Reference prose for every service: 94 methods in numpydoc (Parameters / Returns / Raises /
    See Also / Examples), plus TimeSeries. Behaviour is taken from the backend's endpoint
    contracts, the bindings and the Rust SDK. Every object an example constructs was built against
    the compiled module; the examples were not run against a live server.
  • No type aliases. Each parameter spells out what its Rust type accepts. The shared unions
    claimed more than the methods take; for example, timeseries.by_ids never accepted a
    Resource, and files.delete rejects the IdCollection it was annotated with.
  • Fixes the build and mypy.stubtest needed:
    • builtins.list[...] inside the services: their list method shadowed the builtin, so mypy
      and Pyright treated later annotations in the class as invalid.
    • Constructors are declared as __new__, and non-subclassable classes are @final.
    • The TimeSeries constructor order now matches the runtime.
    • The DatapointsCollectionString constructor is now declared.
    • Page declares its sequence methods.
  • Old prose that contradicted the code was corrected:
    • events.search is not ranked.
    • A duplicate relationship type is a 409.
    • A refused delete is a 409, not a 400.

AGENTS.md gains a short section on where Python reference prose goes and what fails the build.

Not in this PR

  • Read the Docs setup, which happens in the dashboard: import the project, add an automation
    rule to activate SemVer tags, and enable pull-request builds.
  • A mypy.stubtest job in CI to keep the stub true to the module. It currently reports about
    60 findings. Fix/python stub matches runtime #151 overlaps with many of them.
  • The bindings' /// comments are no longer read by the docs build; they still feed help().
  • Bugs found while writing the prose. They are documented as the code behaves today, not
    fixed:
    • datasets.filter and events.filter default to limit=100, while the others use the
      server's 1000.
    • files.search never sends a limit.
    • insert_from_lists silently truncates mismatched lists.
    • units.by_external_ids / by_external_id differ between the sync and async clients.
    • /assets/update and /assets/delete do not check that the target is an asset.
    • datasets.delete resolves ids across all node types.
    • Restoring a file by external id never matches.

Docs pages affected

datahub-sdk-docs should link to this reference instead of hosting its own Python API pages. Its
guides are unaffected.

Conflicts: this rewrites much of the stub. The subscriptions filter rework (removing
SubscriptionFilterForm / DataSort) will conflict in __init__.pyi. After it merges,
docs-python/filters-and-identifiers.rst and the subscriptions.filter docstring need the same
update; the -W build flags the stale group entry.

🤖 Generated with Claude Code

JosteinGj and others added 4 commits September 25, 2026 09:40
… the Docs

The Python client's reference is built by Sphinx from
datahub_python_bindings/python/intellistream_datahub_sdk/__init__.pyi, which is
now the one place its signatures and prose live. The same file is what IDEs show
on hover, so the site and the tooltip cannot disagree. The build reads the stub
statically and needs neither cargo nor the compiled module.

sphinx-autoapi documents the data types, one page per class, grouped by hand in
docs-python/*.rst; a class missing from those groups fails the build. The service
classes are not something a caller names, so a small local extension
(docs-python/_ext/service_pages.py) skips them and writes one page per client
attribute instead: `timeseries.by_ids(...)`, methods grouped by
docs-python/structure.toml, the sync and async clients side by side. A method
missing from that file, or a name it lists that no service has, fails the build.

The stub changes are what the build and stubtest needed:

- Type aliases are gone; every annotation spells out what that parameter
  accepts, taken from the Rust type it binds to. The shared unions claimed more
  than most methods take -- timeseries.by_ids never accepted a Resource.
- `list[...]` inside the services is `builtins.list[...]`: their `list` method
  shadowed the builtin, so mypy and Pyright read every later annotation in the
  class as invalid. DataSort's `property` shadowed `@property` the same way.
- Constructors are `__new__`, as pyo3 builds them, and the classes that cannot
  be subclassed are `@final`.
- TimeSeries' constructor matches the runtime order; DatapointsCollectionString
  declares its real constructor instead of five members it does not have; Page
  declares the sequence methods it implements.
- The timeseries service and TimeSeries carry full numpydoc prose: parameters,
  returns, raises and runnable examples.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: jgjesdal <jostein@intellistream.ai>
…t works

- conf.py: drop the autoapi options and the TOC setting that change nothing in
  the output (checked by diffing the built site with and without each), and
  read only the stub rather than the stub and __init__.py.
- The stub loses two stale comment blocks: one repeated the DatasetUpdate
  docstring behind two lines that belonged elsewhere, the other still claimed
  datasets.search is Latin-letters-only and unranked, both lifted server-side.
- AGENTS.md says where Python reference prose goes, that types are spelled out,
  what fails the build, and how to run it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: jgjesdal <jostein@intellistream.ai>
Every client service now has the numpydoc reference timeseries got first: a
class docstring, and for each of its 94 methods a summary, the behaviour a
caller must know, and Parameters, Returns, Raises, See Also and Examples as
they apply. Each statement was taken from the backend's endpoint contract, the
bindings and the Rust SDK, and every object an example constructs was built
against the compiled module.

Where the old prose disagreed with the code, the code won: events.search is an
unranked substring match returned newest first, not ranked; a duplicate
relationship type is a 409, not a silent 200; datasets.filter, events.filter and
subscriptions.filter send a limit of 100 when none is given, not the server's
1000; a refused delete is a 409 (referenced / would-strand), not a 400.

Two annotations change with it. files.delete and files.restore took the
FileIdentifiable union's definition, which named IdCollection; the binding
accepts int | str | INode | FileUpload and raises TypeError for IdCollection.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: jgjesdal <jostein@intellistream.ai>
…rns it

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: jgjesdal <jostein@intellistream.ai>
@JosteinGj JosteinGj closed this Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant