Skip to content

feat(python): yield the items of a poll, with a resume point - #92

Closed
moetemp wants to merge 31 commits into
moe/up/varargs-unwrap-result-fieldfrom
moe/up/poll-items
Closed

feat(python): yield the items of a poll, with a resume point#92
moetemp wants to merge 31 commits into
moe/up/varargs-unwrap-result-fieldfrom
moe/up/poll-items

Conversation

@moetemp

@moetemp moetemp commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

What changed?

This PR lets a Python poll iterator yield the items of each page, with a point to resume from.

/// @nexus.poll python="next_read_request" items="records"
///   python-resume="resume_read_request" targets="client"
read-records: func(request: read-records-request) -> read-records-response;
  • items="<field>" names a list field of the response, and the iterator yields its items.
  • python-resume= names an optional helper (page request, item) -> request that resumes after an item.
  • The iterator is a PollIterator. Its next_request resumes after the last item it yielded, and it's None after the last page.

The stream sample's read yields each Record.

TypeScript and .NET still yield the responses, and get the items in a follow-up PR.

Why?

The proposal reads Records one by one, and a reader saves its place to resume after a restart. With the page iterator of temporalio#325, each caller looped over the Records and kept the offset itself.

How did you test it?

Link to a test plan if any -

  • Unit Tests
  • Staging
  • End to End Tests

Python tests read the Records of a handle one by one, resume after each, and check that a changed item doesn't change the next request.

This was referenced Oct 28, 2026
Closed
Closed
Closed
Closed
Closed
@moetemp
moetemp force-pushed the moe/up/varargs-unwrap-result-field branch from bb29c8a to 56d99e2 Compare October 29, 2026 04:00
@moetemp
moetemp force-pushed the moe/up/varargs-unwrap-result-field branch from 56d99e2 to e082b61 Compare October 31, 2026 13:08
@moetemp moetemp mentioned this pull request Nov 1, 2026
Closed
moetemp and others added 20 commits November 2, 2026 08:59
@moetemp @tconley1428
feat(python): add a client target
086445d 
`--targets workflow,client` also writes a `client` package, with its own resource classes and a
client extension for each service. Its methods start each operation through `_create_nexus_client`
and return the result. The models are shared, and the Workflow package doesn't change. The
operation renderers take a target and start each line at its indent, so both targets share them.
The stream sample gets the client target.
@tconley1428
Refactor generation around independently selected API targets
ac21a17
@tconley1428
Checkpoint target validation and reviewed documentation
211b63d
@tconley1428
Bind client resources explicitly and add inherited sample configurations
f43c827
@tconley1428
Unify field origins across language selection and API planning
4c878dd
@tconley1428
Keep sample configuration docs in the advanced guide
bc3a6a8
@moetemp @tconley1428
fix: address the review of API targets and the Python client target (t…
b39a797 
…emporalio#346)

* Scoped the Workflow-only sample defaults to the Workflow target.

A default without a target list applies to every target, and these helpers need a Workflow context.

* Used `GenerationTarget` in Python and ran each target check once.

`PythonTarget` copied `GenerationTarget`, and two checks ran twice per run. A
private client method named `_create_nexus_client` replaced the extension hook.

* Rendered each resource method request argument on its own line.

One long line has no trailing comma, so `ruff format` joined the requests in the
samples, and unformatted output got lines of about 200 characters.

* Accepted an empty sourced string again in Python proto decode.

Without a source default in the shared model, decode refused an empty sourced
value. A handler then rejected requests that it accepted before.

* Rendered fields-only resources when the Workflow target is off.

The root classes had stub methods that always raised, and the client package
kept private copies of them. The client now decodes results into the root
classes.

* Generated definitions for resources and sources with no API target.

A handler that implements a service generates definitions only, from the WIT
that its callers share. That WIT can have resources and sources, and the
default build has no `--targets`, so it couldn't generate such WIT at all.

* Gave a returned client handle the Nexus client of its own service.

When another interface returned a resource, the handle kept the client of the
returning service, so its methods started operations on the wrong endpoint.

* Matched the client Nexus protocol to `temporalio.client.NexusClient`.

`Client.create_nexus_client` didn't fit the generated hook, so a real client
failed to type-check and to run. The SDK needs an operation ID, so each call
passes a new random one.

* Added a public factory for each resource on the client extensions.

To rebuild a client handle from stored identity fields, a caller needed the
private `_create_nexus_client` hook and the wire names of the service.

* Rejected unknown settings in examples.toml by name.

A misspelled key was read as a child configuration and the parent then stopped generating. The error now names the key and its section.

* Rejected examples.toml sections that match no input.

A misspelled section name gave definitions-only output with no error, or an error that did not point at examples.toml.

* Rejected child names that share a directory or cannot name a package.

Children whose names map to one directory in any language would merge their files. Names with a leading digit or a Python keyword are not importable packages.

* Built the target-sources sample in TypeScript, Go and .NET.

The target-sources sample now builds for the workflow target in all three languages. Go filled sourced fields only on proto-backed inputs, so operations without a proto input now fill them too.

* Pruned Go imports that the rendered code does not use.

Go output with no Workflow API kept the imports of that API and did not compile. The renderer pruned only aliased imports, and now prunes plain ones too.

* Updated the changelog for the targets change.

The client constructor line describes a new feature, so it moves to added. The breaking entry lists the flag and library changes, and a new fixed entry covers Go output with no Workflow API.

* Renamed the leftover native-api wording to Workflow API.

The flag is gone, so READMEs, specs and comments now name the Workflow API.

* Explained why target-sources borrows Temporal protos.

A reader could copy the sample, but its fields do not match the meaning of the borrowed messages.

* Updated the core-fix changelog and reworded a native API comment.

The sources line still said that resources need a target. The `kw_only`
flips had no entry yet.

* Generated and tested the Python client of the stream sample.

The stream sample is the one System Nexus service with a client, and no sample
built or type-checked it. Only Python has the client target, so Python builds
both APIs in its own variant, and TypeScript and .NET keep the Workflow API.

* Moved the cross-interface resource sample to its own WIT.

The `orders` interface in `target-sources.wit` broke the .NET build of that
sample, which now runs in every language. Two services there declare two
`Operations` classes in one namespace.

* Used a resource field before its source in every language.

Go, TypeScript and .NET replaced a resource field with the source of the same
request field, and Python kept the resource field. Each now sends the
resource's value and fills only the other sources.
@tconley1428
Separate shared models from Python API packages
434533b
@tconley1428
Adapt upstream Stream tests to target sample layout
70ac708
@tconley1428
Allow target-specific repeated source and default directives
8eeb34d
@tconley1428
Use shared generation target lists and trim redundant tests
7f12a25
@tconley1428
Refine Python client APIs and share service contracts
789e0a5
@moetemp
Parsed @nexus.poll on operations, with a required client target.
6e8eee0 
The poll helper names the support function that builds the next request. Only
the client target gets an iterator, so the parser refuses a poll without
`target="client"` and a poll on a resource method.
@moetemp @tconley1428
Generated Python client iterators for @nexus.poll.
434d304 
A WIT stream can't cross a Nexus call, so a read is a request and a response.
The iterator reads every page. The extension creates one Nexus client for the
iteration, and a handle uses its own client.

Co-authored-by: Tim Conley <timothy.conley@temporal.io>
@moetemp
Stop a stream read that doesn't wait at the newest Record
1d8e5fd 
The Stream service sets `next_offset` on an empty page too, so the helper
reads it as the Stream reader does. Each answer's next offset sets the later
reads, also after `start` or `last`.
@moetemp
Build the next poll request before the iterator yields the page
36ad316 
A caller that changes a yielded page then can't change the next read.
`AsyncGenerator` lets a caller close the iterator with `aclose()` or
`contextlib.aclosing`. A handle's iterator also gets the method doc and `Args:`.
@moetemp
Take targets="client" on @nexus.poll, as @nexus.source does
1447fcc 
The poll directive then reads its targets as the other directives do. A
refused poll gives the service, the operation and the input path.
@moetemp
Explain poll pages, timeouts and resumes in the guide
449ceaa 
Also rewrite poll comments that read two ways or use `name` as a verb.
@moetemp
Drop the low-value poll rejection fixtures
221d47b 
A generic directive check is planned for syntax and placement. The kept fixtures cover the client-only rule, an operation shape the iterator can't take, and an iterator name clash.
@moetemp
Shorten the @nexus.poll section of the guide
2e70315 
The full stream helper is in the sample, so the guide keeps a short version and links to it.
moetemp added 11 commits November 3, 2026 02:01
@moetemp
Merged temporalio#345 into the combined base.
7293e5b 
# Conflicts:
#	advanced/samples/typescript/tests/stream-service.test.ts
#	advanced/samples/typescript/wit/stream-service/workflow/models.ts
#	architecture.md
#	src/generator/proto/typescript.rs
#	src/generator/typescript.rs
#	src/planning/resource_resolution.rs
@moetemp
feat(typescript): add a client target with @nexus.poll
5e2ddc6 
The stream sample's TypeScript output gets the client API and the read iterator that Python has.
A client resource takes a service-bound Nexus client, as in Python.
@moetemp
feat(dotnet): add a client target with @nexus.poll
1457f2a 
The stream sample's .NET output gets the client API and the read iterator that Python has.
A client resource takes the `NexusClient` of its service, as in Python.
@moetemp
feat: let a resource declare its type parameters
1180181 
A resource with `@nexus.type-parameter "<alias>"` is generic in Python, TypeScript and .NET. An
operation that returns it, or that its method binds, takes a runtime type for each type parameter
of its output, so a response decodes its generic values.
@moetemp
feat: accept a definition object for a resource's type arguments
5bd1eef 
`@nexus.definition` names an SDK type, such as a Stream definition, that gives the type arguments
of a generic resource. An operation that returns the resource takes it in place of runtime types,
unless a resource method binds the operation.
@moetemp
feat(python): add RPC options to client calls
1aa317f 
Each client call and client resource method takes `rpc_metadata` and `rpc_timeout`, and passes
them to the `start_operation` of its Nexus client.
@moetemp
feat: give a field a source or an api-omit for each target
b020739 
A Stream request fills the producer from a Workflow helper in Workflow code and from an Activity
helper in a client, and the client API leaves the owner unset. A field takes one `@nexus.source`
for each target, and `@nexus.api-omit` takes `targets`.
@moetemp
feat: flatten some fields of one request field
3bbac1d 
`@nexus.flatten-in-api fields="..." fill="<field>=<sibling>"` on a field makes the call take the
named fields of its record, and copies sibling fields into the others. The Stream create takes the
backend of its ref this way, and the ref copies the id.
@moetemp
feat(python): add a Workflow resource factory
c0e998f 
Workflow code builds a handle on an existing Stream with the factory, from an id and a backend.
@moetemp
feat: add @nexus.varargs, @nexus.unwrap and @nexus.result-field
9d48475 
A resource method can take the items of its last list as arguments, take a one-field record as its
field, and return one field of the response. The Stream's publish takes values and returns the
last offset.
@moetemp
feat(python): yield the items of a poll, with a resume point
bfee66d 
`@nexus.poll items=` makes the iterator yield the items of each page, and `resume=` names the helper
that resumes after one item. The iterator's `next_request` lets a caller save its place.
@moetemp
moetemp force-pushed the moe/up/varargs-unwrap-result-field branch 2 times, most recently from 9d48475 to 00292fd Compare November 4, 2026 09:34
@moetemp

moetemp commented Nov 4, 2026

Copy link
Copy Markdown
Owner Author

temporalio#356 replaced it.

@moetemp moetemp closed this Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@moetemp @tconley1428