Skip to content

fix: make asynchronous functions / session functions return uncomputed model output thunks #381

@jakelorocco

Description

@jakelorocco

You can't current stream using mfuncs / session functions because we await the model output thunk before returning it. We should change it to where we pass back uncomputed model output thunks when possible (ie when not doing sampling).

As a part of this, we should make a subclass of model output thunk that is always computed. We can return that from the synchronous functions so that there is a clear delineation of when you are expected to await the computation. We can also do that for asynchronous sampling / sampling strategies.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions