From c5b03503b37413235f45c49d416e10dc7f34e813 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 20 Aug 2026 18:48:58 +0300 Subject: [PATCH 01/49] Rewrite package documentation --- doc.go | 253 ++++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 196 insertions(+), 57 deletions(-) diff --git a/doc.go b/doc.go index fc4e768..21dd3a1 100644 --- a/doc.go +++ b/doc.go @@ -1,80 +1,219 @@ -// Package rill provides composable channel-based concurrency primitives for Go that simplify parallel processing, -// batching, and stream handling. It offers building blocks for constructing concurrent pipelines from -// reusable parts while maintaining precise control over concurrency levels. The package reduces boilerplate, -// abstracts away goroutine orchestration, features centralized error handling, and has zero external dependencies. +// Package rill provides composable, type-safe concurrency primitives for Go: +// functions that transform, filter, batch, reduce, and consume channel-based +// streams, with bounded concurrency and first-class error handling. // -// # Streams and Try Containers +// Rill operates on ordinary Go channels - no custom stream abstraction, +// interfaces, or runtime. Its functions can be used standalone or composed +// into multi-stage pipelines that integrate easily with existing +// channel-based code. // -// In this package, a stream is a channel of [Try] containers. -// Each Try represents either a value or an error. -// When an "empty stream" is referred to, it means a channel of Try containers that has been closed and was never written to. +// # Streams // -// Most functions in this package are concurrent, and the level of concurrency can be controlled by the argument n. -// Some functions share common behaviors and characteristics, which are described below. +// In this package, a stream is a channel of [Try] structs. Such structs hold +// either a value or an error: this simplifies error propagation when multiple +// rill functions are composed together. Instances of [Try] are called items +// in this documentation. // -// # Non-blocking functions +// # Composition and Stages // -// Functions such as [Map], [Filter], and [Batch] take a stream as an input and return a new stream as an output. -// They do not block and return the output stream immediately. All the processing is done in the background by the goroutine pools they spawn. -// These functions forward all errors from the input stream to the output stream. -// Any errors returned by the user-provided functions are also sent to the output stream. -// When such a function reaches the end of the input stream, it closes the output stream, stops processing and cleans up resources. +// Most functions in this package, such as [Map] or [Filter], take a +// stream as input and return a new stream as output. These functions: // -// Such functions are designed to be composed together to build complex processing pipelines: +// - do not block, and return the output stream immediately +// - process input values as they arrive +// - write processing results to the output as they are ready +// - forward input error items to the output as-is +// - write processing errors to the output as they occur +// - close the output stream after the input is fully consumed and processed // -// stage2 := rill.Map(input, ...) -// stage3 := rill.Batch(stage2, ...) -// stage4 := rill.Map(stage3, ...) -// results := rill.Unbatch(stage4, ...) -// // consume the results and handle errors with some blocking function +// Such functions are generic and can be composed into multi-stage pipelines, +// where the output of one stage is the input to the next, and the functions +// themselves are called stages. // -// # Blocking functions +// filtered := rill.Filter(input, ...) +// batches := rill.Batch(filtered, ...) +// results := rill.Map(batches, ...) // -// Functions such as [ForEach], [Reduce] and [MapReduce] are used at the last stage of the pipeline -// to consume the stream and return the final result or error. +// # Sinks // -// Usually, these functions block until one of the following conditions is met: -// - The end of the stream is reached. In this case, the function returns the final result. -// - An error is encountered either in the input stream or in some user-provided function. In this case, the function returns the error. +// A sink is a special type of stage that takes a stream as input, but returns +// a plain value and/or an error instead of a stream. Sinks, such as [ForEach] +// or [MapReduce], are usually the final stage of a pipeline. Sinks can also +// do processing on their input stream, but the lifecycle is different. Sinks: // -// In case of an early termination (before reaching the end of the input stream), such functions return immediately -// but spawn a background goroutine that discards the remaining items from the input channel. This is done to prevent goroutine -// leaks by ensuring that all goroutines feeding the stream are allowed to complete. -// The input stream should not be used anymore after calling such functions. +// - block until the final result (successful or not) is known +// - return the first observed error immediately, regardless of where it +// came from (input or processing) +// - on early return (because of an error or the sink's internal logic), +// keep consuming and discarding the remaining input items (including +// late errors) in the background // -// It's also possible to consume the pipeline results manually, for example using a for-range loop. -// In this case, add a deferred call to [Discard] before the loop to ensure that goroutines are not leaked. +// # Sources // -// defer rill.Discard(results) +// Every pipeline begins with a stream that is created rather than +// transformed. Any channel of [Try] structs can play this role, no matter +// where it comes from - a rill helper such as [FromSlice] or [Generate], a +// third-party library, or hand-written code. This first stream, together +// with the code feeding it, is called the source. // -// for res := range results { -// if res.Error != nil { -// return res.Error -// } -// // process res.Value -// } +// # Extending rill // -// # Unordered functions +// Stages, sinks, and sources are ordinary functions that receive and/or +// return channels. Any user function of a similar shape is fully compatible with +// rill. // -// Functions such as [Map], [Filter], and [FlatMap] write items to the output stream as soon as they become available. -// Due to the concurrent nature of these functions, the order of items in the output stream may not match the order of items in the input stream. -// These functions prioritize performance and concurrency over maintaining the original order. +// For example, it's trivial to create a source that streams rows from a +// database table (just remember to close the channel when the data ends), +// or a sink that collects all observed errors into a slice. Custom reusable +// stages can also be created by composing existing rill functions. // -// # Ordered functions +// # Concurrency // -// Functions such as [OrderedMap] or [OrderedFilter] preserve the order of items from the input stream. -// These functions are still concurrent, but use special synchronization techniques to ensure that -// items are written to the output stream in the same order as they were read from the input stream. -// This additional synchronization has some overhead, but it is negligible for i/o bound workloads. +// Most stages and sinks are concurrent: they process items using a worker +// pool and take the pool size as the argument n. // -// Some other functions, such as [ToSlice], [Batch] or [First] are not concurrent and are ordered by nature. +// # Backpressure // -// # Error handling +// In the context of Go channels, backpressure means that sending to an +// unbuffered channel blocks until the receiver on the other end is ready to +// receive. Rill naturally inherits this property: a slow stage in the +// pipeline blocks the previous stage, and it in turn blocks the stage before that, +// and so on, until the slow stage catches up. +// +// In cases when this is not desirable, use [Buffer] to add slack between stages. +// +// # Ordered stages +// +// By default, results are written to the output stream as they are ready, so +// the order of outputs depends on how the Go runtime schedules the goroutines +// in the worker pool, and how much time each individual item takes to +// process. This is the normal behavior of a worker pool, but sometimes the +// order of outputs matters. // -// Error handling can be non-trivial in concurrent applications. Rill simplifies this by providing a structured error handling approach. -// As described above, all errors are automatically propagated down the pipeline to the final stage, where they can be caught. -// This allows the pipeline to terminate after the first error is encountered and return it to the caller. +// One solution is to disable concurrency within the stage by setting n = 1. +// Another is to use ordered functions, such as [OrderedMap]. These functions +// stay concurrent, but each worker holds its result until all earlier +// results are sent, so the output order matches the input order at the cost +// of some latency. This ordering guarantee holds for both values and errors. +// +// Some stages, such as [Batch] or [Tee], process items sequentially and +// are naturally ordered. +// +// # Error handling // -// In cases where more complex error handling logic is required, the [Catch] function can be used. -// It can catch and handle errors at any point in the pipeline, providing the flexibility to handle not only the first error, but any of them. +// Stages forward errors they encounter downstream: user callbacks never see +// them. As a result, every error, no matter where it originates, eventually +// reaches the sink, which returns the first one it observes to the user code, +// where it can be handled. +// +// When errors need to be handled mid-pipeline, use [Catch]. +// +// # Context and cancellation +// +// Rill itself is context-agnostic: none of its functions take a +// [context.Context]. The stopping mechanism is the user's choice - a +// context, a done channel, or any other signal the source and the callbacks +// understand. +// +// The cancellation model is cooperative and follows from three properties +// of the library: +// +// - pipelines are not first-class objects, but compositions of simpler stages +// - streams are plain channels: stages know nothing about each other; +// data and errors can only travel downstream +// - a source can be infinite, and no stage or sink can know whether it is +// +// The entire model is built around one idea: return control to the user +// code as soon as possible, and let it stop the source from producing new +// items. All other behaviors emerge from this idea: +// +// - stages forward all errors downstream +// - the sink returns the first error it observes, without waiting for +// callbacks already in flight to complete or for its input to end, +// which might not even be possible if the source is infinite +// - the sink keeps draining and discarding its input in the background, +// so that nothing upstream is blocked during the cancellation +// +// A sink can also return early without any error - for example, [Any] does +// so when it finds a match. The model and the responsibilities stay the +// same. +// +// While this may sound complicated, in typical use cases it boils down to at +// most one deferred call, as shown in the examples below. +// +// A pipeline doing I/O. Create a cancellable context before building the +// pipeline, and defer cancel(). Stages doing database or network calls are +// typically context-aware: when the sink returns and the deferred cancel +// fires, the source and all in-flight I/O stop quickly, while the sink's +// background drain disposes of whatever the pipeline still produces, late +// errors included. +// +// ctx, cancel := context.WithCancel(ctx) +// defer cancel() +// +// // context-aware source +// ids := streamUserIDs(ctx) +// +// // context-aware stage +// users := rill.Map(ids, 5, func(id int) (*User, error) { +// return db.GetUser(ctx, id) +// }) +// +// // context-aware sink +// return rill.ForEach(users, 5, func(u *User) error { +// // do something with the user +// return db.Save(ctx, u) +// }) +// +// A sink-only pipeline over a finite source - for example, a standalone +// [ForEach] over a slice. Here even defer cancel() is not strictly +// necessary: after the return, the sink switches into drain mode and +// discards the remaining input items without invoking the user's callback. +// +// err := rill.ForEach(finiteSource, 5, func(x int) error { +// return doSomething(x) +// }) +// +// Manual consumption. Add a deferred [Discard] call before the loop. With no +// sink, there is no one to drain the stream on early exit, so it becomes the +// caller's job - otherwise the goroutines feeding the stream leak: +// +// defer rill.Discard(results) +// for res := range results { +// if res.Error != nil { +// return res.Error +// } +// // process res.Value +// } +// +// [ToSeq2] handles this automatically and does not require a deferred call: +// +// for value, err := range rill.ToSeq2(results) { +// if err != nil { +// return err +// } +// // process value +// } +// +// # Nil handling +// +// Nil channels are valid in Go. They never emit values and are never closed. +// In practice, this means that an attempt to read from a nil channel blocks +// forever. +// +// Rill does not introduce any special semantics for nil channels. If a stage +// receives a channel that blocks forever when read, it returns a channel that +// also blocks forever. If a sink receives such a channel, the sink itself +// hangs. +// +// # Panics +// +// Rill validates its arguments and panics on misuse - a concurrency level +// below one, a nil callback, an invalid batch size. Such panics happen when +// the function is called, before any item is processed, and never depend on +// the data flowing through the pipeline. +// +// Rill does not automatically recover panics in user callbacks: a panicking +// callback can crash the process, as it would in any hand-written concurrent +// code. package rill From 7b1a2c6f021150d3f62b60c7655e7c8cdaec9a44 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 20 Aug 2026 19:23:51 +0300 Subject: [PATCH 02/49] Expand the Concurrency chapter --- doc.go | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/doc.go b/doc.go index 21dd3a1..1ce83c5 100644 --- a/doc.go +++ b/doc.go @@ -69,8 +69,13 @@ // // # Concurrency // -// Most stages and sinks are concurrent: they process items using a worker -// pool and take the pool size as the argument n. +// Most stages and sinks are concurrent, and take the argument n - the +// maximum number of concurrent invocations of the user callback. +// Concurrency is per stage, not per pipeline: each stage enforces its own +// limit, so an I/O-bound stage can use a much larger n than a CPU-bound one. +// +// With n = 1, the callback is never invoked concurrently: items are +// processed one by one, in input order. // // # Backpressure // From 57159e69d566fb8cb5882618befcf81d4e39a07b Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 00:48:58 +0300 Subject: [PATCH 03/49] Refine the cancellation model --- doc.go | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/doc.go b/doc.go index 1ce83c5..638a631 100644 --- a/doc.go +++ b/doc.go @@ -123,19 +123,20 @@ // The cancellation model is cooperative and follows from three properties // of the library: // -// - pipelines are not first-class objects, but compositions of simpler stages -// - streams are plain channels: stages know nothing about each other; -// data and errors can only travel downstream +// - pipelines are not first-class objects, but compositions of simpler +// stages, which know nothing about other stages or their in-flight +// callbacks +// - streams are plain channels: data and errors can only travel downstream // - a source can be infinite, and no stage or sink can know whether it is // // The entire model is built around one idea: return control to the user // code as soon as possible, and let it stop the source from producing new -// items. All other behaviors emerge from this idea: +// items. All other behaviors serve this idea: // // - stages forward all errors downstream // - the sink returns the first error it observes, without waiting for -// callbacks already in flight to complete or for its input to end, -// which might not even be possible if the source is infinite +// in-flight callbacks to complete or for its input to end, which might +// not even be possible if the source is infinite // - the sink keeps draining and discarding its input in the background, // so that nothing upstream is blocked during the cancellation // @@ -148,7 +149,7 @@ // // A pipeline doing I/O. Create a cancellable context before building the // pipeline, and defer cancel(). Stages doing database or network calls are -// typically context-aware: when the sink returns and the deferred cancel +// typically context-aware. When the sink returns and the deferred cancel // fires, the source and all in-flight I/O stop quickly, while the sink's // background drain disposes of whatever the pipeline still produces, late // errors included. @@ -175,7 +176,7 @@ // necessary: after the return, the sink switches into drain mode and // discards the remaining input items without invoking the user's callback. // -// err := rill.ForEach(finiteSource, 5, func(x int) error { +// err := rill.ForEach(rill.FromSlice(finiteSource), 5, func(x int) error { // return doSomething(x) // }) // From 98abf1a231a7de1b2501f7738a6c9c718a40c1e1 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 17:55:05 +0300 Subject: [PATCH 04/49] Refine the Concurrency chapter --- doc.go | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/doc.go b/doc.go index 638a631..6c34aae 100644 --- a/doc.go +++ b/doc.go @@ -69,13 +69,15 @@ // // # Concurrency // -// Most stages and sinks are concurrent, and take the argument n - the -// maximum number of concurrent invocations of the user callback. -// Concurrency is per stage, not per pipeline: each stage enforces its own -// limit, so an I/O-bound stage can use a much larger n than a CPU-bound one. +// Most stages and sinks are concurrent, and take the argument n, which +// acts as both an upper bound and a target for the number of concurrent +// invocations of the user callback. Rill never exceeds this bound, and, +// given enough input, reaches it. With n = 1, the callback is never +// invoked concurrently: items are processed one by one, in input order. // -// With n = 1, the callback is never invoked concurrently: items are -// processed one by one, in input order. +// Concurrency is per stage, not per pipeline: each stage enforces its own +// limit, so an I/O-bound stage can use a much larger n than a CPU-bound +// one. // // # Backpressure // From 285fe7ebec6626b5b2d44ed829706a49b8ec087a Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 17:55:06 +0300 Subject: [PATCH 05/49] Rewrite Map, OrderedMap, ForEach, Any, and All docs --- consume.go | 41 +++++++++++++++++++++++------------------ transform.go | 15 +++++++++------ 2 files changed, 32 insertions(+), 24 deletions(-) diff --git a/consume.go b/consume.go index 6ed8172..56c9593 100644 --- a/consume.go +++ b/consume.go @@ -5,15 +5,17 @@ import ( "sync/atomic" ) -// ForEach applies a function f to each item in an input stream and returns the first error encountered. +// ForEach consumes the stream, calling f on each value. It immediately +// returns the first observed error. Otherwise, it returns nil after the +// input is fully consumed and every call to f has returned. // -// This is a blocking unordered function that processes items concurrently using n goroutines. +// The argument n bounds the number of concurrent calls to f. When n = 1, +// ForEach processes items sequentially in stream order, similar to a +// regular for-range loop: f can safely read and modify shared state without +// synchronization, and all its effects are visible to the caller after +// ForEach returns. // -// When n = 1, ForEach processes items sequentially in stream order, similar to a regular -// for-range loop: f can safely read and modify shared state without synchronization, -// and all its effects are visible to the caller after ForEach returns. -// -// See the package documentation for more information on blocking unordered functions and error handling. +// See the package documentation for the behaviors that all sinks share. func ForEach[A any](in <-chan Try[A], n int, f func(A) error, options ...SinkOption) error { validateN(n) validateNilFunc(f == nil) @@ -91,13 +93,15 @@ func First[A any](in <-chan Try[A], options ...SinkOption) (value A, found bool, // sharing cannot contaminate across calls. var errFound = errors.New("found") -// Any reports whether the input stream contains an item that satisfies the condition f. -// This function returns true as soon as it finds such an item. Otherwise, it returns false. +// Any reports whether the stream contains a value that matches the +// condition f. It consumes the stream, calling f on each value, and +// immediately returns (true, nil) or (false, err) on the first observed +// match or error, respectively. Otherwise, it returns (false, nil) after +// the input is fully consumed and every call to f has returned. // -// Any is a blocking unordered function that processes items concurrently using n goroutines. -// When n = 1, items are processed sequentially in stream order. +// The argument n bounds the number of concurrent calls to f. // -// See the package documentation for more information on blocking unordered functions and error handling. +// See the package documentation for the behaviors that all sinks share. func Any[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...SinkOption) (bool, error) { validateN(n) validateNilFunc(f == nil) @@ -119,14 +123,15 @@ func Any[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...Sin return false, err } -// All reports whether all items in the input stream satisfy the condition f. -// This function returns false as soon as it finds an item that does not satisfy the condition or encounters an error. -// Otherwise, it returns true. +// All reports whether every value in the stream matches the condition f. +// It consumes the stream, calling f on each value, and immediately returns +// (false, nil) or (false, err) on the first observed mismatch or error, +// respectively. Otherwise, it returns (true, nil) after the input is fully +// consumed and every call to f has returned. // -// All is a blocking unordered function that processes items concurrently using n goroutines. -// When n = 1, items are processed sequentially in stream order. +// The argument n bounds the number of concurrent calls to f. // -// See the package documentation for more information on blocking unordered functions and error handling. +// See the package documentation for the behaviors that all sinks share. func All[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...SinkOption) (bool, error) { validateN(n) validateNilFunc(f == nil) diff --git a/transform.go b/transform.go index 68186cc..98c6fff 100644 --- a/transform.go +++ b/transform.go @@ -4,13 +4,15 @@ import ( "github.com/destel/rill/internal/core" ) -// Map takes a stream of items of type A and transforms them into items of type B using a function f. -// Returns a new stream of transformed items. +// Map takes a stream of values of type A and returns a stream of values of +// type B, using f to transform each. When f returns an error, it's written +// to the output instead of a value. // -// This is a non-blocking unordered function that processes items concurrently using n goroutines. -// An ordered version of this function, [OrderedMap], is also available. +// The argument n bounds the number of concurrent calls to f. Results are +// written to the output as they become ready, so their order can differ +// from the input order when n > 1. Use [OrderedMap] to preserve the order. // -// See the package documentation for more information on non-blocking unordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -29,7 +31,8 @@ func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] }) } -// OrderedMap is the ordered version of [Map]. +// OrderedMap is the ordered version of [Map]: the output preserves the +// input order, for values and errors alike. func OrderedMap[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) From 7e7aac5f46fea8ff0d3a04d8bd21c4e4a98c1275 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 18:18:38 +0300 Subject: [PATCH 06/49] Rewrite Filter, FilterMap, and Catch docs --- transform.go | 55 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 32 insertions(+), 23 deletions(-) diff --git a/transform.go b/transform.go index 98c6fff..0e8caee 100644 --- a/transform.go +++ b/transform.go @@ -51,13 +51,16 @@ func OrderedMap[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan }) } -// Filter takes a stream of items of type A and filters them using a predicate function f. -// Returns a new stream of items that passed the filter. +// Filter takes a stream of values and returns a new stream, keeping +// only the values that match the condition f. When f returns an error, +// it's written to the output instead of the value. // -// This is a non-blocking unordered function that processes items concurrently using n goroutines. -// An ordered version of this function, [OrderedFilter], is also available. +// The argument n bounds the number of concurrent calls to f. Results are +// written to the output as they become ready, so their order can differ +// from the input order when n > 1. Use [OrderedFilter] to preserve the +// order. // -// See the package documentation for more information on non-blocking unordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func Filter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -76,7 +79,8 @@ func Filter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[ }) } -// OrderedFilter is the ordered version of [Filter]. +// OrderedFilter is the ordered version of [Filter]: the output preserves +// the input order, for values and errors alike. func OrderedFilter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -95,14 +99,17 @@ func OrderedFilter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-ch }) } -// FilterMap takes a stream of items of type A, applies a function f that can filter and transform them into items of type B. -// Returns a new stream of transformed items that passed the filter. This operation is equivalent to a -// [Filter] followed by a [Map]. +// FilterMap takes a stream of values of type A and returns a stream of +// values of type B, using f to transform each value and decide whether +// to keep the result. When f returns an error, it's written to the +// output instead of a value. // -// This is a non-blocking unordered function that processes items concurrently using n goroutines. -// An ordered version of this function, [OrderedFilterMap], is also available. +// The argument n bounds the number of concurrent calls to f. Results are +// written to the output as they become ready, so their order can differ +// from the input order when n > 1. Use [OrderedFilterMap] to preserve +// the order. // -// See the package documentation for more information on non-blocking unordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func FilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -121,7 +128,8 @@ func FilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <- }) } -// OrderedFilterMap is the ordered version of [FilterMap]. +// OrderedFilterMap is the ordered version of [FilterMap]: the output +// preserves the input order, for values and errors alike. func OrderedFilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -200,17 +208,17 @@ func OrderedFlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) return out } -// Catch allows handling errors in the middle of a stream processing pipeline. -// Every error encountered in the input stream is passed to the function f for handling. -// -// The outcome depends on the return value of f: -// - If f returns nil, the error is considered handled and filtered out from the output stream. -// - If f returns a non-nil error, the original error is replaced with the result of f. +// Catch takes a stream and returns a new stream with the errors +// optionally handled by f. Each error is passed to f, which returns nil +// to drop it from the stream, the same error to keep it, or a different +// one to replace it. Values never reach f and are passed through as-is. // -// This is a non-blocking unordered function that handles errors concurrently using n goroutines. -// An ordered version of this function, [OrderedCatch], is also available. +// The argument n bounds the number of concurrent calls to f. Items are +// written to the output as they become ready, so their order can differ +// from the input order when n > 1. Use [OrderedCatch] to preserve the +// order. // -// See the package documentation for more information on non-blocking unordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func Catch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -229,7 +237,8 @@ func Catch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { }) } -// OrderedCatch is the ordered version of [Catch]. +// OrderedCatch is the ordered version of [Catch]: the output preserves +// the input order, for values and errors alike. func OrderedCatch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) From 076e8227580cdaba8a8eaa95333e79fcb7588e48 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 21:18:13 +0300 Subject: [PATCH 07/49] Rewrite FlatMap and OrderedFlatMap docs --- transform.go | 53 ++++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 47 insertions(+), 6 deletions(-) diff --git a/transform.go b/transform.go index 0e8caee..d16a6cf 100644 --- a/transform.go +++ b/transform.go @@ -148,13 +148,18 @@ func OrderedFilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, err }) } -// FlatMap takes a stream of items of type A and transforms each item into a new sub-stream of items of type B using a function f. -// Those sub-streams are then flattened into a single output stream, which is returned. +// FlatMap takes a stream of values of type A and returns a stream of +// values of type B, using f to expand each value into its own sub-stream. +// The sub-streams are flattened into the output: every item is forwarded, +// values and errors alike. // -// This is a non-blocking unordered function that processes items concurrently using n goroutines. -// An ordered version of this function, [OrderedFlatMap], is also available. +// The argument n bounds the number of sub-streams consumed concurrently: +// each worker consumes one sub-stream to the end before starting the next. +// When n > 1, items from different sub-streams can interleave in the +// output. Use [OrderedFlatMap] to concatenate the sub-streams in the +// input order. // -// See the package documentation for more information on non-blocking unordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -180,7 +185,43 @@ func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan return out } -// OrderedFlatMap is the ordered version of [FlatMap]. +// OrderedFlatMap is the ordered version of [FlatMap]: the output is the +// sub-streams concatenated in the input order. +// +// The argument n bounds the number of concurrent calls to f. The +// sub-streams are prepared concurrently, but - unlike in [FlatMap] - +// consumed one at a time and in order: nothing reads from a sub-stream +// before its turn. In practice, to keep the stage concurrent, a +// sub-stream must do all or part of its expensive work ahead of its +// turn. +// +// Consider a stream of URLs: each file should be downloaded, and its +// lines streamed to the output, all in order. Downloading is the +// expensive work here. +// +// Example 1: f downloads the whole file into memory, then streams the +// lines from there. Up to 5 downloads run concurrently. +// +// rill.OrderedFlatMap(urls, 5, func(u string) <-chan rill.Try[string] { +// lines, err := getFileLines(u) +// return rill.FromSlice(lines, err) +// }) +// +// Example 2: f streams the lines as the file is being downloaded, +// through a [Buffer] that lets the sub-stream run ahead of its turn. +// Again up to 5 concurrent downloads, but each pauses after the first +// 100 lines, until its turn comes. +// +// rill.OrderedFlatMap(urls, 5, func(u string) <-chan rill.Try[string] { +// lines := streamFileLines(u) +// return rill.Buffer(lines, 100) +// }) +// +// The two examples do the same thing: they buffer the lines, with or +// without a bound. Without any buffering, the downloads would run one +// at a time, and the stage would turn sequential. +// +// See the package documentation for the behaviors that all stages share. func OrderedFlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) From d6acc04ad63927361cd44ff18e6269fda0f78410 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 21:36:13 +0300 Subject: [PATCH 08/49] Rewrite Reduce and MapReduce docs --- reduce.go | 48 +++++++++++++++++++++++++++--------------------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/reduce.go b/reduce.go index e13175b..8c301ce 100644 --- a/reduce.go +++ b/reduce.go @@ -8,20 +8,24 @@ import ( "github.com/destel/rill/internal/list" ) -// Reduce combines all items from the input stream into a single value using a binary function f. +// Reduce combines all values of the stream into a single value using +// the binary function f. // -// Treating f as a binary operator "*", Reduce computes in[0] * in[1] * ... * in[N-1]: -// items are combined in stream order, but the parenthesization is unspecified -// and may vary from run to run. This requires f to be associative - -// (a * b) * c == a * (b * c) - so that every parenthesization yields the same -// result. Commutativity is not required. +// Treating f as a binary operator "*", Reduce computes +// in[0] * in[1] * ... * in[N-1]: values are combined in stream order, +// but the parenthesization is unspecified and may vary from run to run. +// This requires f to be associative - (a * b) * c == a * (b * c) - so +// that every parenthesization yields the same result. Commutativity is +// not required. // -// The hasResult return flag is set to true if the stream contained at least one value and no error was encountered, -// otherwise it is set to false. +// Reduce immediately returns (zero, false, err) on the first observed +// error. Otherwise, it returns (zero, false, nil) if the stream is +// empty, or (result, true, nil) after the input is fully consumed and +// every call to f has returned. // -// Reduce is a blocking function that processes items concurrently using n goroutines. +// The argument n bounds the number of concurrent calls to f. // -// See the package documentation for more information on blocking functions and error handling. +// See the package documentation for the behaviors that all sinks share. func Reduce[A any](in <-chan Try[A], n int, f func(A, A) (A, error), options ...SinkOption) (result A, hasResult bool, err error) { validateN(n) validateNilFunc(f == nil) @@ -246,21 +250,23 @@ func Reduce[A any](in <-chan Try[A], n int, f func(A, A) (A, error), options ... return First(out, options...) } -// MapReduce transforms the input stream into a Go map using mapper and reducer functions. -// The transformation is performed in two concurrent phases. +// MapReduce consumes the stream and builds a map: mapper turns each +// value into a key-value pair, and reducer combines the values that +// share a key. // -// - The mapper function transforms each input item into a key-value pair. -// - The reducer function reduces values of the same key into a single value. -// This phase has the same semantics as the [Reduce] function: for each key, -// values are combined in stream order, but the parenthesization is unspecified, -// so the reducer must be associative. +// For each key, the values are combined as in [Reduce]: in stream +// order, with unspecified parenthesization, so reducer must be +// associative. Commutativity is not required. // -// An empty input stream produces an empty map. +// MapReduce immediately returns (nil, err) on the first observed +// error. Otherwise, it returns the map after the input is fully +// consumed and every call to mapper and reducer has returned. An empty +// stream results in an empty map. // -// MapReduce is a blocking function that processes items concurrently using nm and nr goroutines -// for the mapper and reducer functions respectively. +// The arguments nm and nr bound the number of concurrent calls to +// mapper and reducer, respectively. // -// See the package documentation for more information on blocking functions and error handling. +// See the package documentation for the behaviors that all sinks share. func MapReduce[A any, K comparable, V any](in <-chan Try[A], nm int, mapper func(A) (K, V, error), nr int, reducer func(V, V) (V, error), options ...SinkOption) (map[K]V, error) { validateN(nm) validateNilFunc(mapper == nil) From 6a6316f803017e72c0ae8bc722bb29b107cbede0 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 23:06:01 +0300 Subject: [PATCH 09/49] Rewrite Err and First docs --- consume.go | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/consume.go b/consume.go index 56c9593..a8d3975 100644 --- a/consume.go +++ b/consume.go @@ -54,10 +54,11 @@ func ForEach[A any](in <-chan Try[A], n int, f func(A) error, options ...SinkOpt return Err(out, options...) } -// Err returns the first error encountered in the input stream or nil if there were no errors. +// Err consumes the stream and immediately returns the first error it +// contains. Otherwise, it returns nil after the input is fully +// consumed. // -// This is a blocking ordered function that processes items sequentially. -// See the package documentation for more information on blocking ordered functions and error handling. +// See the package documentation for the behaviors that all sinks share. func Err[A any](in <-chan Try[A], options ...SinkOption) error { defer Discard(in, options...) @@ -70,12 +71,12 @@ func Err[A any](in <-chan Try[A], options ...SinkOption) error { return nil } -// First returns the first value or error encountered in the input stream. -// If the stream is empty or its first item is an error, found is false and -// value is the zero value of A. +// First returns the first item of the stream: (value, true, nil) if +// the item is a value, (zero, false, err) if it is an error, or +// (zero, false, nil) if the stream is empty. // -// This is a blocking ordered function that processes items sequentially. -// See the package documentation for more information on blocking ordered functions and error handling. +// The rest of the stream is discarded. See the package documentation +// for the behaviors that all sinks share. func First[A any](in <-chan Try[A], options ...SinkOption) (value A, found bool, err error) { defer Discard(in, options...) From bb2373ad2be130bf32d14236ac6e4e11b9dc2779 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 21 Aug 2026 23:27:55 +0300 Subject: [PATCH 10/49] Drop the consumes-the-stream construct from sink docs --- consume.go | 29 ++++++++++++++--------------- reduce.go | 5 ++--- 2 files changed, 16 insertions(+), 18 deletions(-) diff --git a/consume.go b/consume.go index a8d3975..be3a4ae 100644 --- a/consume.go +++ b/consume.go @@ -5,9 +5,9 @@ import ( "sync/atomic" ) -// ForEach consumes the stream, calling f on each value. It immediately -// returns the first observed error. Otherwise, it returns nil after the -// input is fully consumed and every call to f has returned. +// ForEach calls f on the stream's values, like a concurrent for-range +// loop. It immediately returns the first observed error. Otherwise, it returns nil after f has been +// called on every value and every call has returned. // // The argument n bounds the number of concurrent calls to f. When n = 1, // ForEach processes items sequentially in stream order, similar to a @@ -54,9 +54,8 @@ func ForEach[A any](in <-chan Try[A], n int, f func(A) error, options ...SinkOpt return Err(out, options...) } -// Err consumes the stream and immediately returns the first error it -// contains. Otherwise, it returns nil after the input is fully -// consumed. +// Err immediately returns the first error of the stream. Otherwise, it +// returns nil after the input is fully consumed. // // See the package documentation for the behaviors that all sinks share. func Err[A any](in <-chan Try[A], options ...SinkOption) error { @@ -95,10 +94,10 @@ func First[A any](in <-chan Try[A], options ...SinkOption) (value A, found bool, var errFound = errors.New("found") // Any reports whether the stream contains a value that matches the -// condition f. It consumes the stream, calling f on each value, and -// immediately returns (true, nil) or (false, err) on the first observed -// match or error, respectively. Otherwise, it returns (false, nil) after -// the input is fully consumed and every call to f has returned. +// condition f. It immediately returns (true, nil) or (false, err) on +// the first observed match or error, respectively. Otherwise, it +// returns (false, nil) after f has been called on every value and every +// call has returned. // // The argument n bounds the number of concurrent calls to f. // @@ -124,11 +123,11 @@ func Any[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...Sin return false, err } -// All reports whether every value in the stream matches the condition f. -// It consumes the stream, calling f on each value, and immediately returns -// (false, nil) or (false, err) on the first observed mismatch or error, -// respectively. Otherwise, it returns (true, nil) after the input is fully -// consumed and every call to f has returned. +// All reports whether every value in the stream matches the condition +// f. It immediately returns (false, nil) or (false, err) on the first +// observed mismatch or error, respectively. Otherwise, it returns +// (true, nil) after f has been called on every value and every call has +// returned. // // The argument n bounds the number of concurrent calls to f. // diff --git a/reduce.go b/reduce.go index 8c301ce..97bb18e 100644 --- a/reduce.go +++ b/reduce.go @@ -250,9 +250,8 @@ func Reduce[A any](in <-chan Try[A], n int, f func(A, A) (A, error), options ... return First(out, options...) } -// MapReduce consumes the stream and builds a map: mapper turns each -// value into a key-value pair, and reducer combines the values that -// share a key. +// MapReduce builds a map from the stream: mapper turns each value into +// a key-value pair, and reducer combines the values that share a key. // // For each key, the values are combined as in [Reduce]: in stream // order, with unspecified parenthesization, so reducer must be From 9640991a581eda6340d23d2a579c8b014045b835 Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 22 Aug 2026 01:39:46 +0300 Subject: [PATCH 11/49] Rewrite wrap.go docs --- wrap.go | 78 ++++++++++++++++++++++++++++++++++----------------------- 1 file changed, 46 insertions(+), 32 deletions(-) diff --git a/wrap.go b/wrap.go index 03878f1..580ae11 100644 --- a/wrap.go +++ b/wrap.go @@ -1,14 +1,14 @@ package rill -// Try represents either a value of type A or an error. -// When Error is non-nil, callers must ignore Value. +// Try holds either a value of type A or an error. When Error is +// non-nil, Value is meaningless. type Try[A any] struct { Value A Error error } -// Stream is a type alias for a channel of [Try] containers. -// This alias is optional, but it can make the code more readable. +// Stream is a type alias for a receive-only channel of [Try] structs. +// Using it is optional, but improves readability. // // Before: // @@ -23,11 +23,11 @@ type Try[A any] struct { // } type Stream[T any] = <-chan Try[T] -// Wrap converts a value-error pair into a [Try]. -// If err is non-nil, Wrap returns an error item and ignores value. -// It's a convenience function to avoid creating a [Try] container manually and benefit from type inference. +// Wrap converts a value-error pair into a [Try]. If err is not nil, +// Wrap returns an error item and ignores value. // -// Such function signature also allows concise wrapping of functions that return a value and an error: +// This signature allows concise wrapping of functions that return a +// value and an error: // // item := rill.Wrap(strconv.Atoi("42")) func Wrap[A any](value A, err error) Try[A] { @@ -40,10 +40,12 @@ func Wrap[A any](value A, err error) Try[A] { // FromSlice converts a slice into a stream. // If err is not nil, it is added to the end of the stream. // -// The slice is read in the background until the returned stream -// is fully consumed. Modifying it before is a data race. +// Modifying the slice before the stream is fully consumed is a data +// race. // -// Such function signature allows concise wrapping of functions that return a slice and an error: +// This signature allows concise wrapping of functions that return a +// slice and an error. FromSlice assumes that a non-empty slice along +// with an error is a partial result, and preserves both. // // stream := rill.FromSlice(someFunc()) func FromSlice[A any](slice []A, err error) <-chan Try[A] { @@ -75,12 +77,12 @@ func FromSlice[A any](slice []A, err error) <-chan Try[A] { return out } -// ToSlice converts an input stream into a slice. -// If the stream contains errors, ToSlice returns the values that precede -// the first error, along with that error. +// ToSlice collects the stream's values into a slice. When ToSlice +// encounters an error, it immediately returns that error along with the +// partial slice. Otherwise, it consumes the stream to the end and +// returns a slice of all values. // -// This is a blocking ordered function that processes items sequentially. -// See the package documentation for more information on blocking ordered functions and error handling. +// See the package documentation for the behaviors that all sinks share. func ToSlice[A any](in <-chan Try[A], options ...SinkOption) ([]A, error) { defer Discard(in, options...) @@ -94,10 +96,16 @@ func ToSlice[A any](in <-chan Try[A], options ...SinkOption) ([]A, error) { return res, nil } -// FromChan converts a regular channel into a stream. -// If err is not nil, the function ignores the passed values and returns a stream with a single error. +// FromChan converts a regular channel into a stream. If err is not nil, +// FromChan returns a stream with only that error and ignores values. // -// Such function signature allows concise wrapping of functions that return a channel and an error: +// Otherwise, values are forwarded to the output as they arrive, and the +// output is closed once values is exhausted. A nil input is never +// exhausted, so the output never closes. +// +// This signature allows concise wrapping of functions that return a +// channel and an error. FromChan assumes a non-nil error means +// someFunc() could not construct the channel. // // stream := rill.FromChan(someFunc()) func FromChan[A any](values <-chan A, err error) <-chan Try[A] { @@ -122,13 +130,16 @@ func FromChan[A any](values <-chan A, err error) <-chan Try[A] { return out } -// FromChans creates a stream from independent value and error channels. -// Items from both inputs are added to the output stream as they arrive, and nil -// errors are skipped. -// The output stream is closed only when both input channels are exhausted. -// In particular, if at least one input is nil, the output stream never closes. +// FromChans converts separate value and error channels into a single +// stream. Values and errors are forwarded to the output as they arrive, +// and nil errors are skipped. +// +// The output is closed only when both inputs are exhausted. A nil input +// is never exhausted, so the output never closes. // -// Such function signature allows concise wrapping of functions that return two channels: +// This signature allows concise wrapping of functions that return two +// channels. FromChans assumes someFunc() returns two channels that +// eventually close. // // stream := rill.FromChans(someFunc()) func FromChans[A any](values <-chan A, errs <-chan error) <-chan Try[A] { @@ -168,9 +179,12 @@ func FromChans[A any](values <-chan A, errs <-chan error) <-chan Try[A] { return out } -// ToChans splits an input stream into two channels: one for values and one for errors. -// Both output channels are closed when the input stream is exhausted. -// They must be consumed concurrently to avoid deadlocks. +// ToChans splits the stream into two channels: one for values and one +// for errors. It returns immediately, forwards each item to the +// appropriate channel as it arrives, and closes both channels once the +// input is exhausted. +// +// The channels must be consumed concurrently to avoid a deadlock. func ToChans[A any](in <-chan Try[A]) (<-chan A, <-chan error) { if in == nil { return nil, nil @@ -195,9 +209,9 @@ func ToChans[A any](in <-chan Try[A]) (<-chan A, <-chan error) { return out, errs } -// Generate is a shorthand for creating streams. -// It provides a more ergonomic way of sending both values and errors to a stream, manages goroutine and channel lifecycle. -// Nil errors passed to sendError are skipped. +// Generate is a shorthand for creating streams: it manages the +// goroutine and channel lifecycle. Inside f, send writes a value to the +// stream, and sendError writes an error unless it is nil. // // stream := rill.Generate(func(send func(int), sendError func(error)) { // for i := 0; i < 100; i++ { @@ -206,7 +220,7 @@ func ToChans[A any](in <-chan Try[A]) (<-chan A, <-chan error) { // sendError(someError) // }) // -// Here's how the same code would look without Generate: +// The same stream without Generate: // // stream := make(chan rill.Try[int]) // go func() { From ee06c7f631dc78616b9ffd4dfaba1d266245ea86 Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 22 Aug 2026 12:16:10 +0300 Subject: [PATCH 12/49] Rewrite iter.go docs --- iter.go | 30 +++++++++++++++++++----------- 1 file changed, 19 insertions(+), 11 deletions(-) diff --git a/iter.go b/iter.go index 60c2644..376b03d 100644 --- a/iter.go +++ b/iter.go @@ -5,11 +5,14 @@ import ( "sync" ) -// FromSeq converts an iterator into a stream. -// If err is not nil, the function ignores the passed seq and returns a stream with a single error. +// FromSeq converts an iterator into a stream. If err is not nil, +// FromSeq returns a stream with only that error and ignores seq. +// Otherwise, the values of seq are forwarded to the output, and the +// output is closed once seq ends. // -// Such function signature allows concise wrapping of functions that return an -// iterator and an error: +// This signature allows concise wrapping of functions that return an +// iterator and an error. FromSeq assumes a non-nil error means +// someFunc() could not construct the iterator. // // stream := rill.FromSeq(someFunc()) func FromSeq[A any](seq iter.Seq[A], err error) <-chan Try[A] { @@ -33,7 +36,9 @@ func FromSeq[A any](seq iter.Seq[A], err error) <-chan Try[A] { } // FromSeq2 converts an iterator of value-error pairs into a stream. -// For pairs with a non-nil error, FromSeq2 emits an error item and ignores the value. +// Each pair becomes one item. For pairs with a non-nil error, FromSeq2 +// emits an error item and ignores the value. The output is closed once +// seq ends. func FromSeq2[A any](seq iter.Seq2[A, error]) <-chan Try[A] { validateNilFunc(seq == nil) @@ -47,13 +52,16 @@ func FromSeq2[A any](seq iter.Seq2[A, error]) <-chan Try[A] { return out } -// ToSeq2 converts an input stream into an iterator of value-error pairs. -// Errors are yielded as ordinary pairs and do not stop the iteration; -// handle them inside the loop. +// ToSeq2 converts the stream into an iterator of value-error pairs, +// typically consumed with a for-range loop. Pairs are yielded until the +// stream is exhausted or the loop exits with break or return. Error +// items do not stop the iteration: they are yielded as ordinary pairs. // -// ToSeq2 returns a single-use iterator, which must be called for the pipeline -// to settle. If iteration stops early using break or return, ToSeq2 drains the -// remaining input in the background. +// On an early exit, ToSeq2 discards the rest of the stream in the +// background, the same way sinks do. +// +// The returned iterator is single-use and must be ranged for the +// pipeline to settle. If it is never ranged, the input is never drained. func ToSeq2[A any](in <-chan Try[A], options ...SinkOption) iter.Seq2[A, error] { // Unlike other sinks, ToSeq2 opens the options at the call site: its work // happens while the iterator is ranged, which can be arbitrarily far from From 45ca0f44ee94924984b70976115fc8bf41e0b267 Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 22 Aug 2026 12:16:11 +0300 Subject: [PATCH 13/49] Keep error and happy paths in one paragraph --- wrap.go | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/wrap.go b/wrap.go index 580ae11..45e94fa 100644 --- a/wrap.go +++ b/wrap.go @@ -98,10 +98,10 @@ func ToSlice[A any](in <-chan Try[A], options ...SinkOption) ([]A, error) { // FromChan converts a regular channel into a stream. If err is not nil, // FromChan returns a stream with only that error and ignores values. -// // Otherwise, values are forwarded to the output as they arrive, and the -// output is closed once values is exhausted. A nil input is never -// exhausted, so the output never closes. +// output is closed once the input is exhausted. +// +// A nil input is never exhausted, so the output never closes. // // This signature allows concise wrapping of functions that return a // channel and an error. FromChan assumes a non-nil error means @@ -132,10 +132,10 @@ func FromChan[A any](values <-chan A, err error) <-chan Try[A] { // FromChans converts separate value and error channels into a single // stream. Values and errors are forwarded to the output as they arrive, -// and nil errors are skipped. +// and nil errors are skipped. The output is closed only when both +// inputs are exhausted. // -// The output is closed only when both inputs are exhausted. A nil input -// is never exhausted, so the output never closes. +// A nil input is never exhausted, so the output never closes. // // This signature allows concise wrapping of functions that return two // channels. FromChans assumes someFunc() returns two channels that From 0ea7f5787b3ecc570fac2585fceb7149c0e5af30 Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 22 Aug 2026 13:49:36 +0300 Subject: [PATCH 14/49] Rewrite merge.go docs and generalize Tee to any channel --- merge.go | 52 ++++++++++++++++++++++++++++----------------------- merge_test.go | 2 +- 2 files changed, 30 insertions(+), 24 deletions(-) diff --git a/merge.go b/merge.go index d5a5074..72666c8 100644 --- a/merge.go +++ b/merge.go @@ -4,27 +4,31 @@ import ( "github.com/destel/rill/internal/core" ) -// Merge performs a fan-in operation on the list of input channels, returning a single output channel. -// The resulting channel will contain all items from all inputs, -// and will be closed when all inputs are exhausted. -// In particular, Merge with no arguments returns an already closed channel. +// Merge performs a fan-in, combining multiple channels into a single +// output channel. It returns immediately, and consumes the inputs +// simultaneously and independently, interleaving their items in the +// output as they arrive. Merge preserves the relative order of items +// from the same input. // -// This is a non-blocking function that processes items from each input sequentially. -// -// See the package documentation for more information on non-blocking functions and error handling. +// The output is closed only when all inputs are exhausted. +// A nil input is never exhausted, so the output never closes. +// Merge with no arguments immediately returns an empty closed channel. func Merge[A any](ins ...<-chan A) <-chan A { return core.Merge(ins...) } -// Split2 divides the input stream into two output streams based on the predicate function f: -// The splitting behavior is determined by the boolean return value of f. When f returns true, the item is sent to the outTrue stream, -// otherwise it is sent to the outFalse stream. In case of any error, the item is sent to both output streams. -// Both output streams must be consumed independently to avoid deadlocks. +// Split2 divides the stream into two streams: values that match the +// condition f go to outTrue, and the rest go to outFalse. Errors are +// sent to both outputs. +// +// The streams must be consumed concurrently to avoid a deadlock. // -// This is a non-blocking unordered function that processes items concurrently using n goroutines. -// An ordered version of this function, [OrderedSplit2], is also available. +// The argument n bounds the number of concurrent calls to f. Items are +// written to the outputs as they become ready, so their order can +// differ from the input order when n > 1. Use [OrderedSplit2] to +// preserve the order. // -// See the package documentation for more information on non-blocking unordered functions and error handling. +// See the package documentation for the behaviors that all stages share. // // Deprecated: Split2 will be removed in v1.0. Since the introduction of [Tee] // in v0.8, splitting no longer needs a dedicated operation — it can be composed @@ -64,7 +68,8 @@ func Split2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (outTrue <- return } -// OrderedSplit2 is the ordered version of [Split2]. +// OrderedSplit2 is the ordered version of [Split2]: the outputs +// preserve the input order, for values and errors alike. // // Deprecated: OrderedSplit2 will be removed in v1.0. Since the introduction of // [Tee] in v0.8, splitting no longer needs a dedicated operation — it can be @@ -104,25 +109,26 @@ func OrderedSplit2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (out return } -// Tee returns two streams that are identical to the input stream (both errors and values). -// Both output streams must be consumed independently to avoid deadlocks. +// Tee duplicates the input channel into two identical channels. It +// returns immediately, forwards each item to both outputs as it +// arrives, and closes both once the input is exhausted. // -// This is a non-blocking function that processes items in a single goroutine. -// See the package documentation for more information on non-blocking functions and error handling. +// The outputs must be consumed concurrently to avoid a deadlock. // -// If deep copying of values is needed, use [Map] on one or both outputs: +// If deep copying of values is needed, use [Map] on one or both +// outputs: // // out1, out2 := rill.Tee(in) // out2 = rill.Map(out2, 1, func(x A) (A, error) { // return deepCopy(x), nil // }) -func Tee[A any](in <-chan Try[A]) (<-chan Try[A], <-chan Try[A]) { +func Tee[A any](in <-chan A) (<-chan A, <-chan A) { if in == nil { return nil, nil } - out1 := make(chan Try[A]) - out2 := make(chan Try[A]) + out1 := make(chan A) + out2 := make(chan A) go func() { defer close(out1) diff --git a/merge_test.go b/merge_test.go index 944e5fa..c38112c 100644 --- a/merge_test.go +++ b/merge_test.go @@ -126,7 +126,7 @@ func TestSplit2(t *testing.T) { func TestTee(t *testing.T) { t.Run("nil", func(t *testing.T) { - out1, out2 := Tee[int](nil) + out1, out2 := Tee[Try[int]](nil) th.ExpectValue(t, out1, nil) th.ExpectValue(t, out2, nil) }) From 3fa68b8af622896d438d8c7c5d84d15de0c2f73e Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 22 Aug 2026 17:10:50 +0300 Subject: [PATCH 15/49] Rewrite batch.go docs --- batch.go | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/batch.go b/batch.go index 2460505..f99ff3a 100644 --- a/batch.go +++ b/batch.go @@ -4,23 +4,30 @@ import ( "time" ) -// Batch takes a stream of items and returns a stream of batches based on a maximum size and a timeout. +// Batch groups consecutive values of the stream into batches. In its +// simplest form, with timeout = -1 and no errors in the input, Batch +// accumulates values into a pending batch and emits it as soon as it +// reaches the target size. // -// A batch is emitted when one of the following conditions is met: -// - The batch reaches the maximum size -// - The time since the first item was added to the batch exceeds the timeout -// - An error is encountered in the input stream -// - The input stream is closed +// A positive timeout is the time each batch has to fill, starting from +// its first value. When it expires, the pending batch is emitted even +// if it is not full. This trades batch size for latency: batches can be +// smaller when the input is sparse, but no value is ever held longer +// than timeout. // -// Errors are never included in batches. Each error is forwarded to the output as a separate item, -// preserving the relative order of values and errors. +// A zero timeout panics: the expected behavior would be to accumulate +// until reading from the input blocks, but in practice, with an +// unbuffered input, that often produces a flood of one-item batches. +// Use a small positive timeout instead. // -// This function never emits empty batches. To disable the timeout and emit batches only based on the size, -// set the timeout to -1. Setting the timeout to zero is not supported and will result in a panic +// Input errors become batch boundaries: the pending batch, if not +// empty, is emitted first, and the error follows as a separate item. // -// This is a non-blocking ordered function that processes items sequentially. +// When the end of the input is reached, whatever has accumulated is +// emitted as a final batch. This function never emits empty batches, +// regardless of what triggered the emission. // -// See the package documentation for more information on non-blocking ordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[[]A] { validateMinSize(size, 1) if timeout == 0 { @@ -113,10 +120,10 @@ func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[ return out } -// Unbatch is the inverse of [Batch]. It takes a stream of batches and returns a stream of individual items. +// Unbatch flattens a stream of slices into a stream of their values. +// This function is the inverse of [Batch]. // -// This is a non-blocking ordered function that processes items sequentially. -// See the package documentation for more information on non-blocking ordered functions and error handling. +// See the package documentation for the behaviors that all stages share. func Unbatch[A any](in <-chan Try[[]A]) <-chan Try[A] { if in == nil { return nil From 88ea289059d48786c8cb2afe27fee4d8af1b0407 Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 22 Aug 2026 17:58:59 +0300 Subject: [PATCH 16/49] Rewrite util.go docs --- util.go | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/util.go b/util.go index 1b65b6c..0704dae 100644 --- a/util.go +++ b/util.go @@ -6,12 +6,14 @@ import ( "github.com/destel/rill/internal/core" ) -// Drain consumes and discards all items from an input channel, blocking until the channel is exhausted. +// Drain consumes and discards all items of the channel, blocking until +// it is exhausted. func Drain[A any](in <-chan A) { core.Drain(in) } -// Discard returns immediately, consumes the input channel in the background, and discards all its items. +// Discard returns immediately, then consumes and discards all items of +// the channel in the background. func Discard[A any](in <-chan A, options ...SinkOption) { opts := collectSinkOptions(options) @@ -36,24 +38,22 @@ func Discard[A any](in <-chan A, options ...SinkOption) { }() } -// DrainNB returns immediately, consumes the input channel in the background, and discards all its items. +// DrainNB is a non-blocking version of [Drain]. // -// Deprecated: use [Discard] instead. DrainNB will be removed in v1.0. +// Deprecated: use [Discard] instead, which is identical. DrainNB will +// be removed in v1.0. func DrainNB[A any](in <-chan A) { Discard(in) } -// Buffer takes a channel of items and returns a buffered channel of exact same items in the same order. -// This can be useful for preventing write operations on the input channel from blocking, especially if subsequent stages -// in the processing pipeline are slow. -// Up to size items can be buffered before back pressure is applied to the upstream producer. -// -// Typical usage of Buffer might look like this: +// Buffer forwards all input items to a new channel with a capacity of +// size. It returns immediately and closes the output once the input is +// exhausted. // // users := getUsers(ctx, companyID) // users = rill.Buffer(users, 100) -// // Now work with the users channel as usual. -// // Up to 100 users can be buffered if subsequent stages of the pipeline are slow. +// // Up to 100 users can be buffered if subsequent stages of the +// // pipeline are slow. func Buffer[A any](in <-chan A, size int) <-chan A { validateMinSize(size, 0) return core.Buffer(in, size) From 55def86014ac82c23d2aeb4dc0fadf8af727b717 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 28 Aug 2026 14:54:14 +0300 Subject: [PATCH 17/49] Rewrite the package doc opening --- doc.go | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/doc.go b/doc.go index 6c34aae..172f27a 100644 --- a/doc.go +++ b/doc.go @@ -1,11 +1,15 @@ -// Package rill provides composable, type-safe concurrency primitives for Go: -// functions that transform, filter, batch, reduce, and consume channel-based -// streams, with bounded concurrency and first-class error handling. -// -// Rill operates on ordinary Go channels - no custom stream abstraction, -// interfaces, or runtime. Its functions can be used standalone or composed -// into multi-stage pipelines that integrate easily with existing -// channel-based code. +// Package rill provides composable primitives for streaming pipelines +// over plain Go channels: functions that transform, filter, batch, reduce, +// and consume data streams, with bounded concurrency per stage, optional order +// preservation, and centralized error handling. +// +// The model follows the Go blog's "Pipelines and cancellation" +// (https://go.dev/blog/pipelines). Rill ships stages, sinks, and sources as +// generic functions, accepts hand-written ones of the same shape, and +// standardizes two conventions across all of them. Errors travel through +// the stream together with values, the way a Go function returns (value, +// error). Cancellation is a single rule: the caller stops the source, and +// the sink drains the rest. // // # Streams // From 89977f5c23dbbb4ee23553c160bc1c5702a362c3 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 28 Aug 2026 15:58:14 +0300 Subject: [PATCH 18/49] Compress the package doc opening --- doc.go | 29 +++++++++++++---------------- 1 file changed, 13 insertions(+), 16 deletions(-) diff --git a/doc.go b/doc.go index 172f27a..6eec2ed 100644 --- a/doc.go +++ b/doc.go @@ -1,22 +1,19 @@ -// Package rill provides composable primitives for streaming pipelines -// over plain Go channels: functions that transform, filter, batch, reduce, -// and consume data streams, with bounded concurrency per stage, optional order -// preservation, and centralized error handling. -// -// The model follows the Go blog's "Pipelines and cancellation" -// (https://go.dev/blog/pipelines). Rill ships stages, sinks, and sources as -// generic functions, accepts hand-written ones of the same shape, and -// standardizes two conventions across all of them. Errors travel through -// the stream together with values, the way a Go function returns (value, -// error). Cancellation is a single rule: the caller stops the source, and -// the sink drains the rest. +// Package rill provides composable primitives for building streaming +// pipelines over plain Go channels: functions that transform, filter, batch, +// reduce, and consume data streams, with bounded concurrency per stage, +// optional order preservation, centralized error handling, and minimal +// boilerplate. +// +// The model is similar to the Go blog's "Pipelines and cancellation" +// (https://go.dev/blog/pipelines), but it unifies error handling and +// cancellation, by letting errors travel downstream along with values. // // # Streams // -// In this package, a stream is a channel of [Try] structs. Such structs hold -// either a value or an error: this simplifies error propagation when multiple -// rill functions are composed together. Instances of [Try] are called items -// in this documentation. +// In this package, a stream is a plain channel of [Try] structs, each +// holding either a value or an error. This is Go's (value, error) return +// convention, carried over to channels. Such structs are often referred to +// as items below. // // # Composition and Stages // From e6409a682b53e75c254409ee70113e22d3c60a4b Mon Sep 17 00:00:00 2001 From: destel Date: Sat, 29 Aug 2026 11:37:33 +0300 Subject: [PATCH 19/49] WIP: draft new cancellation and settlement chapter --- doc.go | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/doc.go b/doc.go index 6eec2ed..5f7fffc 100644 --- a/doc.go +++ b/doc.go @@ -1,12 +1,12 @@ // Package rill provides composable primitives for building streaming // pipelines over plain Go channels: functions that transform, filter, batch, // reduce, and consume data streams, with bounded concurrency per stage, -// optional order preservation, centralized error handling, and minimal +// centralized error handling, optional order preservation, and minimal // boilerplate. // // The model is similar to the Go blog's "Pipelines and cancellation" // (https://go.dev/blog/pipelines), but it unifies error handling and -// cancellation, by letting errors travel downstream along with values. +// cancellation by letting errors travel downstream along with values. // // # Streams // @@ -116,7 +116,30 @@ // // When errors need to be handled mid-pipeline, use [Catch]. // -// # Context and cancellation +// # Context, cancellation and settlement +// +// Rill is context-agnostic: none of its functions take a [context.Context]. +// No scope object owns the callbacks, and a stage knows nothing besides its +// own input and output channels. The stopping mechanism is the caller's +// choice - a context, a done channel, or any other signal the source and the +// callbacks understand. +// +// Everything a stage has to say travels downstream, on the same path: values, +// errors as ordinary items, and closure. A stage closes its output only when +// it will do no more work: its input is consumed and every callback it +// started has returned. Such a stage is called settled. +// +// Closure accumulates. A stage cannot settle before its input closes, so a +// closed channel means that every stage feeding it has settled as well, and +// a single signal at the end of a linear pipeline covers all of them. Sinks +// are the exception: with no output channel, they have nothing to close. +// [Settlement] gives them an equivalent signal. +// +// Nothing travels the other way. A sink cannot reach the stages feeding it, +// so stopping the source is the caller's job, and settlement only reports +// that the work has ended - it never ends it. +// +// # Context and cancellation (Old) // // Rill itself is context-agnostic: none of its functions take a // [context.Context]. The stopping mechanism is the user's choice - a From 24317cef6f03c1ba99e5db498de46b68e9eff59d Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 3 Sep 2026 15:51:36 +0300 Subject: [PATCH 20/49] Refine ToSeq2 settlement docs --- iter.go | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/iter.go b/iter.go index 376b03d..0c8272b 100644 --- a/iter.go +++ b/iter.go @@ -57,11 +57,9 @@ func FromSeq2[A any](seq iter.Seq2[A, error]) <-chan Try[A] { // stream is exhausted or the loop exits with break or return. Error // items do not stop the iteration: they are yielded as ordinary pairs. // -// On an early exit, ToSeq2 discards the rest of the stream in the -// background, the same way sinks do. -// // The returned iterator is single-use and must be ranged for the -// pipeline to settle. If it is never ranged, the input is never drained. +// pipeline to settle. If the loop exits early with break or return, +// ToSeq2 drains the input in the background before reporting settlement. func ToSeq2[A any](in <-chan Try[A], options ...SinkOption) iter.Seq2[A, error] { // Unlike other sinks, ToSeq2 opens the options at the call site: its work // happens while the iterator is ranged, which can be arbitrarily far from From 1b49a44c5c4c9ab6da1ac8c36c9966ad8342c473 Mon Sep 17 00:00:00 2001 From: destel Date: Mon, 7 Sep 2026 14:17:55 +0300 Subject: [PATCH 21/49] Restructure the package doc --- doc.go | 278 +++++++++++++++++++++++++-------------------------------- 1 file changed, 121 insertions(+), 157 deletions(-) diff --git a/doc.go b/doc.go index 5f7fffc..fd26f4c 100644 --- a/doc.go +++ b/doc.go @@ -10,15 +10,14 @@ // // # Streams // -// In this package, a stream is a plain channel of [Try] structs, each -// holding either a value or an error. This is Go's (value, error) return -// convention, carried over to channels. Such structs are often referred to -// as items below. +// In this package, a stream is a plain channel that carries both values and errors. +// Each strream item is an instance of a [Try] struct that represents either a value or an error. +// This is Go's (value, error) return convention, carried over to channels. // -// # Composition and Stages +// # Stages, composition and pipelines // // Most functions in this package, such as [Map] or [Filter], take a -// stream as input and return a new stream as output. These functions: +// stream as input and return a new stream as output. These functions are called stages and they: // // - do not block, and return the output stream immediately // - process input values as they arrive @@ -27,27 +26,14 @@ // - write processing errors to the output as they occur // - close the output stream after the input is fully consumed and processed // -// Such functions are generic and can be composed into multi-stage pipelines, -// where the output of one stage is the input to the next, and the functions -// themselves are called stages. +// Such functions, along with sources and sinks described below, are generic and can +// be used either standalone or composed into multi-stage pipelines, +// where the output of one stage is the input to the next. // +// ids := rill.FromSlice(userIDs) // filtered := rill.Filter(input, ...) // batches := rill.Batch(filtered, ...) -// results := rill.Map(batches, ...) -// -// # Sinks -// -// A sink is a special type of stage that takes a stream as input, but returns -// a plain value and/or an error instead of a stream. Sinks, such as [ForEach] -// or [MapReduce], are usually the final stage of a pipeline. Sinks can also -// do processing on their input stream, but the lifecycle is different. Sinks: -// -// - block until the final result (successful or not) is known -// - return the first observed error immediately, regardless of where it -// came from (input or processing) -// - on early return (because of an error or the sink's internal logic), -// keep consuming and discarding the remaining input items (including -// late errors) in the background +// err := rill.ForEach(batches, ...) // // # Sources // @@ -57,16 +43,18 @@ // third-party library, or hand-written code. This first stream, together // with the code feeding it, is called the source. // -// # Extending rill +// # Sinks // -// Stages, sinks, and sources are ordinary functions that receive and/or -// return channels. Any user function of a similar shape is fully compatible with -// rill. +// A sink is function that takes a stream as input, but returns +// a a regular Go value and/or an error. Sinks, such as [ForEach] +// or [MapReduce], are usually the final stage of a pipeline. // -// For example, it's trivial to create a source that streams rows from a -// database table (just remember to close the channel when the data ends), -// or a sink that collects all observed errors into a slice. Custom reusable -// stages can also be created by composing existing rill functions. +// - block, until the final outcome (successful or not) is known +// - return early (before the input is fully consumed) on the first observed error, regardless of where it came from - upstream or the sink iteself +// - can return early because of the sink's internal logic, for example [Any] returns as soon as it finds a match +// - on early return keep consuming and discarding the remaining input items (including +// late errors) in the background, so upstream stages do not block and leak their goroutines +// - can optionally report pipeline settlement (see below) via the [Scope] API // // # Concurrency // @@ -76,156 +64,136 @@ // given enough input, reaches it. With n = 1, the callback is never // invoked concurrently: items are processed one by one, in input order. // -// Concurrency is per stage, not per pipeline: each stage enforces its own -// limit, so an I/O-bound stage can use a much larger n than a CPU-bound -// one. -// -// # Backpressure -// -// In the context of Go channels, backpressure means that sending to an -// unbuffered channel blocks until the receiver on the other end is ready to -// receive. Rill naturally inherits this property: a slow stage in the -// pipeline blocks the previous stage, and it in turn blocks the stage before that, -// and so on, until the slow stage catches up. -// -// In cases when this is not desirable, use [Buffer] to add slack between stages. -// // # Ordered stages // -// By default, results are written to the output stream as they are ready, so +// By default, results are written to the output stream in completion order, so // the order of outputs depends on how the Go runtime schedules the goroutines // in the worker pool, and how much time each individual item takes to // process. This is the normal behavior of a worker pool, but sometimes the // order of outputs matters. // -// One solution is to disable concurrency within the stage by setting n = 1. -// Another is to use ordered functions, such as [OrderedMap]. These functions -// stay concurrent, but each worker holds its result until all earlier -// results are sent, so the output order matches the input order at the cost +// Rill shiprovides ordered functions, such as [OrderedMap] or [OrderedFilter]. They stay concurrent, +// but each worker holds its result until all earlier results are sent, +// so the output order matches the input order at the cost // of some latency. This ordering guarantee holds for both values and errors. // -// Some stages, such as [Batch] or [Tee], process items sequentially and -// are naturally ordered. -// // # Error handling // -// Stages forward errors they encounter downstream: user callbacks never see -// them. As a result, every error, no matter where it originates, eventually -// reaches the sink, which returns the first one it observes to the user code, -// where it can be handled. +// Every error, wherever in the pipeline it originates, eventually reaches +// the sink, and the sink returns the first one it observes to the caller. // -// When errors need to be handled mid-pipeline, use [Catch]. +// To handle errors mid-pipeline, use [Catch]: a stage whose callback sees +// errors rather, and can handle, keep, or rewrite them. // -// # Context, cancellation and settlement +// [Catch] is also handy for tracking where errors come from. The snippet +// below tags every error coming out of the source, so that later they +// can be told apart from errors raised in the stages: // -// Rill is context-agnostic: none of its functions take a [context.Context]. -// No scope object owns the callbacks, and a stage knows nothing besides its -// own input and output channels. The stopping mechanism is the caller's -// choice - a context, a done channel, or any other signal the source and the -// callbacks understand. +// var errSource = errors.New("source failed") // -// Everything a stage has to say travels downstream, on the same path: values, -// errors as ordinary items, and closure. A stage closes its output only when -// it will do no more work: its input is consumed and every callback it -// started has returned. Such a stage is called settled. +// source = rill.Catch(source, 1, func(err error) error { +// return fmt.Errorf("%w: %w", errSource, err) +// }) // -// Closure accumulates. A stage cannot settle before its input closes, so a -// closed channel means that every stage feeding it has settled as well, and -// a single signal at the end of a linear pipeline covers all of them. Sinks -// are the exception: with no output channel, they have nothing to close. -// [Settlement] gives them an equivalent signal. +// # Context, cancellation and pipeline lifecycle // -// Nothing travels the other way. A sink cannot reach the stages feeding it, -// so stopping the source is the caller's job, and settlement only reports -// that the work has ended - it never ends it. +// Rill's lifecycle and cancellation model follows from two design decisions: // -// # Context and cancellation (Old) +// - don't become a framework: pipelines are not first class objects, but +// compositions of simpler functions that know nothing about each other +// - streams are plain channels: data and errors can only travel downstream // -// Rill itself is context-agnostic: none of its functions take a -// [context.Context]. The stopping mechanism is the user's choice - a -// context, a done channel, or any other signal the source and the callbacks -// understand. +// Together these force background draining: nothing travels upstream, so a sink +// that returns early cannot stop the stages feeding it, and abandoning them +// would block their sends forever. // -// The cancellation model is cooperative and follows from three properties -// of the library: +// A pipeline goes through the following lifecycle phases: // -// - pipelines are not first-class objects, but compositions of simpler -// stages, which know nothing about other stages or their in-flight -// callbacks -// - streams are plain channels: data and errors can only travel downstream -// - a source can be infinite, and no stage or sink can know whether it is -// -// The entire model is built around one idea: return control to the user -// code as soon as possible, and let it stop the source from producing new -// items. All other behaviors serve this idea: -// -// - stages forward all errors downstream -// - the sink returns the first error it observes, without waiting for -// in-flight callbacks to complete or for its input to end, which might -// not even be possible if the source is infinite -// - the sink keeps draining and discarding its input in the background, -// so that nothing upstream is blocked during the cancellation -// -// A sink can also return early without any error - for example, [Any] does -// so when it finds a match. The model and the responsibilities stay the -// same. -// -// While this may sound complicated, in typical use cases it boils down to at -// most one deferred call, as shown in the examples below. -// -// A pipeline doing I/O. Create a cancellable context before building the -// pipeline, and defer cancel(). Stages doing database or network calls are -// typically context-aware. When the sink returns and the deferred cancel -// fires, the source and all in-flight I/O stop quickly, while the sink's -// background drain disposes of whatever the pipeline still produces, late -// errors included. -// -// ctx, cancel := context.WithCancel(ctx) -// defer cancel() -// -// // context-aware source -// ids := streamUserIDs(ctx) -// -// // context-aware stage -// users := rill.Map(ids, 5, func(id int) (*User, error) { -// return db.GetUser(ctx, id) -// }) +// - active: processing is in progress, sink is blocked +// - result known: sink returned result to the caller; upstream stages might still be working, but sink drains and discards their results in the background +// - cancelled: caller can optionally cancel a context to stop the source from producing more work and stages from doing it +// - settled: all work is done, all user callbacks across the pipeline have returned // -// // context-aware sink -// return rill.ForEach(users, 5, func(u *User) error { -// // do something with the user -// return db.Save(ctx, u) -// }) +// If you do not need to wait for settlement, the cancellation model often +// collapses to nothing. Heavy network and database calls are context-aware +// by design; the sink immediately returns the first observed error to the caller; +// the caller passes that error up the stack until something handles it and +// cancels the context. That stops the remaining network calls, and the +// pipeline settles on its own in the background. // -// A sink-only pipeline over a finite source - for example, a standalone -// [ForEach] over a slice. Here even defer cancel() is not strictly -// necessary: after the return, the sink switches into drain mode and -// discards the remaining input items without invoking the user's callback. +// When the caller must wait for settlement, use a [Scope]. It has an +// errgroup-like shape, applied to pipelines: [NewScope] derives a context, +// and [Scope.Wait] cancels it and blocks until the pipeline has settled. +// Cancellation is cooperative: rill cancels the context, and callbacks that +// captured it stop the heavy work currently in progress. [Scope.Wait] +// returns even when the source is infinite, as long as the source watches +// the context too. // -// err := rill.ForEach(rill.FromSlice(finiteSource), 5, func(x int) error { -// return doSomething(x) -// }) +// scope, ctx := rill.NewScope(ctx) +// defer scope.Cancel() // -// Manual consumption. Add a deferred [Discard] call before the loop. With no -// sink, there is no one to drain the stream on early exit, so it becomes the -// caller's job - otherwise the goroutines feeding the stream leak: +// // source and other pipeline stages go here // -// defer rill.Discard(results) -// for res := range results { -// if res.Error != nil { -// return res.Error -// } -// // process res.Value -// } +// err := rill.ForEach(stream, 5, func(x int) error { +// return process(ctx, x) +// }, scope) +// +// // result known +// +// scope.Wait() // cancel and wait for settlement // -// [ToSeq2] handles this automatically and does not require a deferred call: +// Scope API can also be used for cancellation only, without calling [Scope.Wait] and waiting for settlement. +// In this case it becomes equivalent to [context.WithCancel]. // -// for value, err := range rill.ToSeq2(results) { +// When the sink consumes its input to the end - no errors across the pipeline, +// no short-circuit - the pipeline is already settled by the time the sink +// returns. This property makes both [Scope] and the context redundant for computation-only pipelines that can never fail. +// +// In one special case the cost of not cancelling is bounded: a pipeline where +// all the heavy work lives in the sink's callback. Once the sink returns and +// switches to background draining, it quickly stops calling its own callback. +// This is not settlement - O(concurrency) calls may still happen, but that +// number does not depend on the remaining input size. +// +// ids := rill.FromSlice(userIDs, nil) +// exists, err := rill.Any(ids, 5, func(id int) (bool, error) { +// user, err := getUser(ctx, id) // if err != nil { -// return err +// return false, err // } -// // process value -// } +// return user.Age > 35, nil +// }) +// +// # Extending rill +// +// Sources, stages, and sinks are ordinary functions that receive and/or +// return channels, so any user function of a similar shape works with the +// rest of the library. +// +// For example, it's easy to write a source that streams rows from +// a database table, or a sink that collects all observed errors into a +// slice. +// +// There are a few rules custom functions must follow to be compatible with rill's lifecycle. +// These rules are usually satisfied by construction. +// +// - sources must eventually close their output stream; a source that can +// run forever must watch a context +// - stages must close their output stream only after the input is fully +// consumed and all workers have returned +// - sinks must start with a deferred rill.Discard(in, options...), followed by a +// for-range loop that returns as soon as the sink's outcome is known +// +// Custom stages and sinks can also be built by composing existing +// rill functions, which is often the simplest way. +// +// # Backpressure +// +// In the context of Go channels, backpressure means that sending to an +// unbuffered channel blocks until the receiver on the other end is ready to +// receive. Rill naturally inherits this property: a slow stage in the +// pipeline blocks the previous stage, and it in turn blocks the stage before that, +// and so on, until the slow stage catches up. // // # Nil handling // @@ -240,11 +208,7 @@ // // # Panics // -// Rill validates its arguments and panics on misuse - a concurrency level -// below one, a nil callback, an invalid batch size. Such panics happen when -// the function is called, before any item is processed, and never depend on -// the data flowing through the pipeline. -// +// Rill validates arguments of its functions and panics on misuse, such that zero or negative concurrency level. // Rill does not automatically recover panics in user callbacks: a panicking // callback can crash the process, as it would in any hand-written concurrent // code. From 96184e2cb34884241e07a757bef6ffecac526d4f Mon Sep 17 00:00:00 2001 From: destel Date: Mon, 7 Sep 2026 14:22:01 +0300 Subject: [PATCH 22/49] Restore the Buffer pointer in the backpressure chapter --- doc.go | 2 ++ 1 file changed, 2 insertions(+) diff --git a/doc.go b/doc.go index fd26f4c..5717f93 100644 --- a/doc.go +++ b/doc.go @@ -195,6 +195,8 @@ // pipeline blocks the previous stage, and it in turn blocks the stage before that, // and so on, until the slow stage catches up. // +// When this is not desirable, use [Buffer] to add slack between stages. +// // # Nil handling // // Nil channels are valid in Go. They never emit values and are never closed. From 5f282e0c6ebcb2e7038f47e0b6846d13749a38b4 Mon Sep 17 00:00:00 2001 From: destel Date: Mon, 7 Sep 2026 14:38:09 +0300 Subject: [PATCH 23/49] Refine the ordered stages chapter --- doc.go | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/doc.go b/doc.go index 5717f93..4ff3039 100644 --- a/doc.go +++ b/doc.go @@ -66,17 +66,21 @@ // // # Ordered stages // -// By default, results are written to the output stream in completion order, so -// the order of outputs depends on how the Go runtime schedules the goroutines -// in the worker pool, and how much time each individual item takes to -// process. This is the normal behavior of a worker pool, but sometimes the -// order of outputs matters. -// -// Rill shiprovides ordered functions, such as [OrderedMap] or [OrderedFilter]. They stay concurrent, +// By default, results and errors are written to the output stream as soon +// as they are ready, so their order depends on how the Go runtime schedules +// the goroutines in the stage's worker pool, and how much time each individual +// item takes to process. +// +// This is the normal behavior of a worker pool. For cases where +// the order of outputs matters, rill provides ordered functions, +// such as [OrderedMap] or [OrderedFilter]. They stay concurrent, // but each worker holds its result until all earlier results are sent, // so the output order matches the input order at the cost // of some latency. This ordering guarantee holds for both values and errors. // +// Some stages, such as [Batch] or [Unbatch], process items sequentially and +// are naturally ordered. +// // # Error handling // // Every error, wherever in the pipeline it originates, eventually reaches From 3372b3878a4d89570d2e5d082018550c91fe9850 Mon Sep 17 00:00:00 2001 From: destel Date: Mon, 7 Sep 2026 22:36:07 +0300 Subject: [PATCH 24/49] Unify ordering wording and package doc trailers --- batch.go | 4 ++-- consume.go | 22 ++++++++++---------- doc.go | 15 +++++++------- merge.go | 2 +- reduce.go | 4 ++-- transform.go | 58 ++++++++++++++++++++++++---------------------------- wrap.go | 2 +- 7 files changed, 51 insertions(+), 56 deletions(-) diff --git a/batch.go b/batch.go index f99ff3a..34c8e1d 100644 --- a/batch.go +++ b/batch.go @@ -27,7 +27,7 @@ import ( // emitted as a final batch. This function never emits empty batches, // regardless of what triggered the emission. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[[]A] { validateMinSize(size, 1) if timeout == 0 { @@ -123,7 +123,7 @@ func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[ // Unbatch flattens a stream of slices into a stream of their values. // This function is the inverse of [Batch]. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func Unbatch[A any](in <-chan Try[[]A]) <-chan Try[A] { if in == nil { return nil diff --git a/consume.go b/consume.go index be3a4ae..b84bd10 100644 --- a/consume.go +++ b/consume.go @@ -6,8 +6,9 @@ import ( ) // ForEach calls f on the stream's values, like a concurrent for-range -// loop. It immediately returns the first observed error. Otherwise, it returns nil after f has been -// called on every value and every call has returned. +// loop. It immediately returns the first observed error. Otherwise, it +// returns nil after the input is fully consumed and every call to f has +// returned. // // The argument n bounds the number of concurrent calls to f. When n = 1, // ForEach processes items sequentially in stream order, similar to a @@ -15,7 +16,7 @@ import ( // synchronization, and all its effects are visible to the caller after // ForEach returns. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func ForEach[A any](in <-chan Try[A], n int, f func(A) error, options ...SinkOption) error { validateN(n) validateNilFunc(f == nil) @@ -57,7 +58,7 @@ func ForEach[A any](in <-chan Try[A], n int, f func(A) error, options ...SinkOpt // Err immediately returns the first error of the stream. Otherwise, it // returns nil after the input is fully consumed. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func Err[A any](in <-chan Try[A], options ...SinkOption) error { defer Discard(in, options...) @@ -74,8 +75,7 @@ func Err[A any](in <-chan Try[A], options ...SinkOption) error { // the item is a value, (zero, false, err) if it is an error, or // (zero, false, nil) if the stream is empty. // -// The rest of the stream is discarded. See the package documentation -// for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func First[A any](in <-chan Try[A], options ...SinkOption) (value A, found bool, err error) { defer Discard(in, options...) @@ -96,12 +96,12 @@ var errFound = errors.New("found") // Any reports whether the stream contains a value that matches the // condition f. It immediately returns (true, nil) or (false, err) on // the first observed match or error, respectively. Otherwise, it -// returns (false, nil) after f has been called on every value and every -// call has returned. +// returns (false, nil) after the input is fully consumed and every call +// to f has returned. // // The argument n bounds the number of concurrent calls to f. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func Any[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...SinkOption) (bool, error) { validateN(n) validateNilFunc(f == nil) @@ -126,12 +126,12 @@ func Any[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...Sin // All reports whether every value in the stream matches the condition // f. It immediately returns (false, nil) or (false, err) on the first // observed mismatch or error, respectively. Otherwise, it returns -// (true, nil) after f has been called on every value and every call has +// (true, nil) after the input is fully consumed and every call to f has // returned. // // The argument n bounds the number of concurrent calls to f. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func All[A any](in <-chan Try[A], n int, f func(A) (bool, error), options ...SinkOption) (bool, error) { validateN(n) validateNilFunc(f == nil) diff --git a/doc.go b/doc.go index 4ff3039..0c98876 100644 --- a/doc.go +++ b/doc.go @@ -66,20 +66,19 @@ // // # Ordered stages // -// By default, results and errors are written to the output stream as soon -// as they are ready, so their order depends on how the Go runtime schedules -// the goroutines in the stage's worker pool, and how much time each individual -// item takes to process. +// By default, results and errors are written to the output as soon as they +// are ready, in completion order. That order depends on how the Go runtime +// schedules the goroutines in the stage's worker pool, and on how much time +// each individual item takes to process. // -// This is the normal behavior of a worker pool. For cases where -// the order of outputs matters, rill provides ordered functions, +// For cases where the input order must be preserved, rill provides ordered functions, // such as [OrderedMap] or [OrderedFilter]. They stay concurrent, // but each worker holds its result until all earlier results are sent, // so the output order matches the input order at the cost // of some latency. This ordering guarantee holds for both values and errors. // -// Some stages, such as [Batch] or [Unbatch], process items sequentially and -// are naturally ordered. +// Some stages, such as [Batch] or [Unbatch], process items sequentially, so +// they are naturally ordered. // // # Error handling // diff --git a/merge.go b/merge.go index 72666c8..b1c5edf 100644 --- a/merge.go +++ b/merge.go @@ -28,7 +28,7 @@ func Merge[A any](ins ...<-chan A) <-chan A { // differ from the input order when n > 1. Use [OrderedSplit2] to // preserve the order. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. // // Deprecated: Split2 will be removed in v1.0. Since the introduction of [Tee] // in v0.8, splitting no longer needs a dedicated operation — it can be composed diff --git a/reduce.go b/reduce.go index 97bb18e..4dd00b5 100644 --- a/reduce.go +++ b/reduce.go @@ -25,7 +25,7 @@ import ( // // The argument n bounds the number of concurrent calls to f. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func Reduce[A any](in <-chan Try[A], n int, f func(A, A) (A, error), options ...SinkOption) (result A, hasResult bool, err error) { validateN(n) validateNilFunc(f == nil) @@ -265,7 +265,7 @@ func Reduce[A any](in <-chan Try[A], n int, f func(A, A) (A, error), options ... // The arguments nm and nr bound the number of concurrent calls to // mapper and reducer, respectively. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func MapReduce[A any, K comparable, V any](in <-chan Try[A], nm int, mapper func(A) (K, V, error), nr int, reducer func(V, V) (V, error), options ...SinkOption) (map[K]V, error) { validateN(nm) validateNilFunc(mapper == nil) diff --git a/transform.go b/transform.go index d16a6cf..ab2b5d6 100644 --- a/transform.go +++ b/transform.go @@ -8,11 +8,11 @@ import ( // type B, using f to transform each. When f returns an error, it's written // to the output instead of a value. // -// The argument n bounds the number of concurrent calls to f. Results are -// written to the output as they become ready, so their order can differ -// from the input order when n > 1. Use [OrderedMap] to preserve the order. +// The argument n bounds the number of concurrent calls to f. +// Results and errors are written to the output in completion order. +// Use [OrderedMap] to preserve the input order. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -31,8 +31,8 @@ func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] }) } -// OrderedMap is the ordered version of [Map]: the output preserves the -// input order, for values and errors alike. +// OrderedMap is the ordered version of [Map]: +// it writes results and errors in input order rather than completion order. func OrderedMap[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -54,13 +54,13 @@ func OrderedMap[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan // Filter takes a stream of values and returns a new stream, keeping // only the values that match the condition f. When f returns an error, // it's written to the output instead of the value. +// Errors are never filtered out. // -// The argument n bounds the number of concurrent calls to f. Results are -// written to the output as they become ready, so their order can differ -// from the input order when n > 1. Use [OrderedFilter] to preserve the -// order. +// The argument n bounds the number of concurrent calls to f. +// Results and errors are written to the output in completion order. +// Use [OrderedFilter] to preserve the input order. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func Filter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -79,8 +79,8 @@ func Filter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[ }) } -// OrderedFilter is the ordered version of [Filter]: the output preserves -// the input order, for values and errors alike. +// OrderedFilter is the ordered version of [Filter]: +// it writes results and errors in input order rather than completion order. func OrderedFilter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -102,14 +102,13 @@ func OrderedFilter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-ch // FilterMap takes a stream of values of type A and returns a stream of // values of type B, using f to transform each value and decide whether // to keep the result. When f returns an error, it's written to the -// output instead of a value. +// output instead of a value. Errors are never filtered out. // -// The argument n bounds the number of concurrent calls to f. Results are -// written to the output as they become ready, so their order can differ -// from the input order when n > 1. Use [OrderedFilterMap] to preserve -// the order. +// The argument n bounds the number of concurrent calls to f. +// Results and errors are written to the output in completion order. +// Use [OrderedFilterMap] to preserve the input order. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func FilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -128,8 +127,8 @@ func FilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <- }) } -// OrderedFilterMap is the ordered version of [FilterMap]: the output -// preserves the input order, for values and errors alike. +// OrderedFilterMap is the ordered version of [FilterMap]: +// it writes results and errors in input order rather than completion order. func OrderedFilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -159,7 +158,7 @@ func OrderedFilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, err // output. Use [OrderedFlatMap] to concatenate the sub-streams in the // input order. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -220,8 +219,6 @@ func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan // The two examples do the same thing: they buffer the lines, with or // without a bound. Without any buffering, the downloads would run one // at a time, and the stage would turn sequential. -// -// See the package documentation for the behaviors that all stages share. func OrderedFlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -254,12 +251,11 @@ func OrderedFlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) // to drop it from the stream, the same error to keep it, or a different // one to replace it. Values never reach f and are passed through as-is. // -// The argument n bounds the number of concurrent calls to f. Items are -// written to the output as they become ready, so their order can differ -// from the input order when n > 1. Use [OrderedCatch] to preserve the -// order. +// The argument n bounds the number of concurrent calls to f. +// Items are written to the output in completion order. +// Use [OrderedCatch] to preserve the input order. // -// See the package documentation for the behaviors that all stages share. +// See the [rill] package documentation for the full contract shared by all stages. func Catch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -278,8 +274,8 @@ func Catch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { }) } -// OrderedCatch is the ordered version of [Catch]: the output preserves -// the input order, for values and errors alike. +// OrderedCatch is the ordered version of [Catch]: +// it writes items in input order rather than completion order. func OrderedCatch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) diff --git a/wrap.go b/wrap.go index 45e94fa..0cdd5db 100644 --- a/wrap.go +++ b/wrap.go @@ -82,7 +82,7 @@ func FromSlice[A any](slice []A, err error) <-chan Try[A] { // partial slice. Otherwise, it consumes the stream to the end and // returns a slice of all values. // -// See the package documentation for the behaviors that all sinks share. +// See the [rill] package documentation for the full contract shared by all sinks. func ToSlice[A any](in <-chan Try[A], options ...SinkOption) ([]A, error) { defer Discard(in, options...) From 631d2feb7454e5c8e323157f01721049bf919fdb Mon Sep 17 00:00:00 2001 From: destel Date: Tue, 8 Sep 2026 00:54:52 +0300 Subject: [PATCH 25/49] Refine the stages and sinks definitions --- doc.go | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/doc.go b/doc.go index 0c98876..7b99f9b 100644 --- a/doc.go +++ b/doc.go @@ -11,13 +11,13 @@ // # Streams // // In this package, a stream is a plain channel that carries both values and errors. -// Each strream item is an instance of a [Try] struct that represents either a value or an error. +// Each stream item is an instance of a [Try] struct that represents either a value or an error. // This is Go's (value, error) return convention, carried over to channels. // // # Stages, composition and pipelines // -// Most functions in this package, such as [Map] or [Filter], take a -// stream as input and return a new stream as output. These functions are called stages and they: +// Many functions in this package take a stream as input and return a new stream as output. +// [Map], [Filter], and other such functions are called stages. They: // // - do not block, and return the output stream immediately // - process input values as they arrive @@ -26,9 +26,9 @@ // - write processing errors to the output as they occur // - close the output stream after the input is fully consumed and processed // -// Such functions, along with sources and sinks described below, are generic and can +// Stages (along with sources and sinks described below) are generic and can // be used either standalone or composed into multi-stage pipelines, -// where the output of one stage is the input to the next. +// where the output of one function becomes the input to the next. // // ids := rill.FromSlice(userIDs) // filtered := rill.Filter(input, ...) @@ -45,12 +45,11 @@ // // # Sinks // -// A sink is function that takes a stream as input, but returns -// a a regular Go value and/or an error. Sinks, such as [ForEach] -// or [MapReduce], are usually the final stage of a pipeline. +// Every pipeline ends with a function called a sink. Sinks, such as [ForEach] or [MapReduce], +// take a stream as input but return a regular Go value and/or an error. Such functions: // // - block, until the final outcome (successful or not) is known -// - return early (before the input is fully consumed) on the first observed error, regardless of where it came from - upstream or the sink iteself +// - return early (before the input is fully consumed) on the first observed error, regardless of where it came from - upstream or the sink itself // - can return early because of the sink's internal logic, for example [Any] returns as soon as it finds a match // - on early return keep consuming and discarding the remaining input items (including // late errors) in the background, so upstream stages do not block and leak their goroutines From 0bdc17a8f78cfe160ffc4869d81f7cd1283a004e Mon Sep 17 00:00:00 2001 From: destel Date: Tue, 8 Sep 2026 13:17:14 +0300 Subject: [PATCH 26/49] Typos --- doc.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc.go b/doc.go index 7b99f9b..06b3c41 100644 --- a/doc.go +++ b/doc.go @@ -30,8 +30,8 @@ // be used either standalone or composed into multi-stage pipelines, // where the output of one function becomes the input to the next. // -// ids := rill.FromSlice(userIDs) -// filtered := rill.Filter(input, ...) +// ids := rill.FromSlice(userIDs, nil) +// filtered := rill.Filter(ids, ...) // batches := rill.Batch(filtered, ...) // err := rill.ForEach(batches, ...) // From 2b2994882100d5c45090fefb51e2d2c484fc371d Mon Sep 17 00:00:00 2001 From: destel Date: Tue, 8 Sep 2026 18:10:57 +0300 Subject: [PATCH 27/49] Rewrite the pipeline lifecycle chapter --- doc.go | 94 ++++++++++++++++++++++++++++++---------------------------- 1 file changed, 48 insertions(+), 46 deletions(-) diff --git a/doc.go b/doc.go index 06b3c41..847078f 100644 --- a/doc.go +++ b/doc.go @@ -97,39 +97,44 @@ // return fmt.Errorf("%w: %w", errSource, err) // }) // -// # Context, cancellation and pipeline lifecycle +// # Pipeline lifecycle // -// Rill's lifecycle and cancellation model follows from two design decisions: +// Rill's lifecycle model follows from two design decisions: // -// - don't become a framework: pipelines are not first class objects, but +// - don't become a framework: pipelines are not first-class objects, but // compositions of simpler functions that know nothing about each other // - streams are plain channels: data and errors can only travel downstream // -// Together these force background draining: nothing travels upstream, so a sink -// that returns early cannot stop the stages feeding it, and abandoning them -// would block their sends forever. -// -// A pipeline goes through the following lifecycle phases: -// -// - active: processing is in progress, sink is blocked -// - result known: sink returned result to the caller; upstream stages might still be working, but sink drains and discards their results in the background -// - cancelled: caller can optionally cancel a context to stop the source from producing more work and stages from doing it -// - settled: all work is done, all user callbacks across the pipeline have returned -// -// If you do not need to wait for settlement, the cancellation model often -// collapses to nothing. Heavy network and database calls are context-aware -// by design; the sink immediately returns the first observed error to the caller; -// the caller passes that error up the stack until something handles it and -// cancels the context. That stops the remaining network calls, and the -// pipeline settles on its own in the background. -// -// When the caller must wait for settlement, use a [Scope]. It has an -// errgroup-like shape, applied to pipelines: [NewScope] derives a context, -// and [Scope.Wait] cancels it and blocks until the pipeline has settled. -// Cancellation is cooperative: rill cancels the context, and callbacks that -// captured it stop the heavy work currently in progress. [Scope.Wait] -// returns even when the source is infinite, as long as the source watches -// the context too. +// Together these force three things. A sink cannot stop or cancel the stages +// feeding it, only the caller can. A sink must pass control back to the caller +// as soon as the outcome is known, which can happen before the input +// is fully consumed. A sink must then drain the remaining input in the background, so +// upstream stages don't block forever and their callbacks can observe cancellation. +// +// A pipeline goes through three phases on its own. The caller can add two +// optional steps: +// +// - active: processing is in progress, the sink is blocked +// - result known: the sink has returned; upstream stages may +// still be working, but the sink drains and discards their results in +// the background +// - cancelled (optional): the caller cancels a context; the source +// stops producing new work, and the stages stop doing it +// - settled: no work remains; every user callback across the pipeline has +// returned +// - joined (optional): the caller has waited for settlement, and can now +// do what would otherwise conflict with callbacks in flight: release +// resources they used or read state they wrote +// +// In computation-only pipelines where nothing ever fails or short-circuits, +// the pipeline is already settled by the time the sink returns. +// +// To wait for settlement in pipelines that can return early (because of an +// error or any other reason), rill provides the [Scope] API that has a +// shape similar to errgroup. This API derives a context, manages its cancellation, +// and allows the caller to wait for settlement. And the same way as in errgroup, +// the already-submitted work can't be withdrawn, only cooperatively cancelled: +// heavy user callbacks must capture the derived context and respect its cancellation. // // scope, ctx := rill.NewScope(ctx) // defer scope.Cancel() @@ -142,30 +147,27 @@ // // // result known // -// scope.Wait() // cancel and wait for settlement +// scope.Wait() // cancel context and wait for settlement // -// Scope API can also be used for cancellation only, without calling [Scope.Wait] and waiting for settlement. -// In this case it becomes equivalent to [context.WithCancel]. +// // joined // -// When the sink consumes its input to the end - no errors across the pipeline, -// no short-circuit - the pipeline is already settled by the time the sink -// returns. This property makes both [Scope] and the context redundant for computation-only pipelines that can never fail. +// When joining is not needed, it's possible to use [Scope] +// in cancellation-only mode, or just use a regular [context.WithCancel]. +// Even that might not be necessary if the call site already has a cancellable context +// (which is often the case when heavy network calls are involved), so all context +// plumbing goes away: // -// In one special case the cost of not cancelling is bounded: a pipeline where -// all the heavy work lives in the sink's callback. Once the sink returns and -// switches to background draining, it quickly stops calling its own callback. -// This is not settlement - O(concurrency) calls may still happen, but that -// number does not depend on the remaining input size. +// // source and other pipeline stages go here // -// ids := rill.FromSlice(userIDs, nil) -// exists, err := rill.Any(ids, 5, func(id int) (bool, error) { -// user, err := getUser(ctx, id) -// if err != nil { -// return false, err -// } -// return user.Age > 35, nil +// err := rill.ForEach(stream, 5, func(x int) error { +// return process(ctx, x) // }) // +// if err != nil { +// // just return, the context will be cancelled up in the call stack +// return err +// } +// // # Extending rill // // Sources, stages, and sinks are ordinary functions that receive and/or From 10a7c5e8c55b8bd5a7262559ea5866a8bec15756 Mon Sep 17 00:00:00 2001 From: destel Date: Wed, 9 Sep 2026 14:32:47 +0300 Subject: [PATCH 28/49] tmp --- doc.go | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 87 insertions(+), 3 deletions(-) diff --git a/doc.go b/doc.go index 847078f..993f928 100644 --- a/doc.go +++ b/doc.go @@ -4,9 +4,93 @@ // centralized error handling, optional order preservation, and minimal // boilerplate. // +// # Pipelines and streams +// +// Rill functions can be used standalone or composed into multi-stage pipelines. // The model is similar to the Go blog's "Pipelines and cancellation" -// (https://go.dev/blog/pipelines), but it unifies error handling and -// cancellation by letting errors travel downstream along with values. +// (https://go.dev/blog/pipelines), but it unifies error handling by letting +// errors travel downstream along with values. In rill's terms, the post's +// definition of a pipeline becomes: +// +// A pipeline is a series of stages connected by streams - channels whose items +// are [Try] structs, each holding either a value or an error. Each stage is a +// group of goroutines running the same function; the argument n, where present, +// is the size of the group. In each stage, the goroutines: +// +// - receive values and errors from upstream via inbound streams +// - apply the function to the values, usually producing new values or errors +// - send the results downstream via outbound streams; upstream errors pass +// through unchanged +// +// Each stage can have any number of input and output streams, except the first stage +// that has no input streams and the last stage that has no output streams. These stages +// are called the source and the sink, respectively. +// +// ids := rill.FromSlice(userIDs, nil) // source +// filtered := rill.Filter(ids, 5, ...) // stage, 5 goroutines +// batches := rill.Batch(filtered, ...) // stage, 1 goroutine +// transformed := rill.Map(batches, 3, ...) // stage, 3 goroutines +// err := rill.ForEach(transformed, 2, ...) // sink, 2 goroutines +// +// The source and intermediate stages do not block: they spawn goroutines then immediately return their output stream. +// Those goroutines keep working until all input streams are fully consumed and processed. When all work is done, +// the output stream is closed. +// +// Sinks are different, they block, then return as soon as pipeline's outcome is known, which +// can happen before the input is fully consumed and all work across the pipeline is done. In particular, +// a sinks return early when: +// +// - it observes an error, whether it came from upstream or from the sink itself +// - its internal short-circuit condition is met, for example [Any] returns as soon as it finds a match, and [First] returns after +// consuming a single item from its input stream +// +// # Context, cancellation and settlement +// +// When sink returns early it keeps draining and discarding the remaining input in the background to prevent upstream +// stages from blocking forever and leaking their goroutines. Expensive cancellable work is usually context aware, so +// caller can cancel anything that remains after the sink's early return: +// +// ctx, cancel := context.WithCancel(ctx) +// defer cancel() +// +// // source and other pipeline stages go here; +// // they might be context aware +// +// err := rill.ForEach(stream, 5, func(x int) error { +// return process(ctx, x) +// }) +// +// // result known, cancel the context +// // or rely on the deferred call above +// cancel() +// +// When the caller wants not only to request cancellation but also to wait +// for the pipeline to settle (no work remains, every user callback has +// returned), rill provides the [Scope] API, which is like errgroup for pipelines. +// +// scope, ctx := rill.NewScope(ctx) +// defer scope.Cancel() +// +// // source and other pipeline stages go here; +// // they might be context aware +// +// err := rill.ForEach(stream, 5, func(x int) error { +// return process(ctx, x) +// }, scope) +// +// // result known +// +// scope.Wait() // cancel context and wait for settlement +// +// ------ the end so far ------ +// +// # If sink returned early +// +// which defines pipeline as a series of stages, +// connected by channels, where +// +// The post defines pipeline as a series of stages, +// connected by channels, // // # Streams // @@ -124,7 +208,7 @@ // returned // - joined (optional): the caller has waited for settlement, and can now // do what would otherwise conflict with callbacks in flight: release -// resources they used or read state they wrote +// resources they used, or read state they wrote // // In computation-only pipelines where nothing ever fails or short-circuits, // the pipeline is already settled by the time the sink returns. From 7595fd80af4f086858d8b28cc2527dc0ae28f65c Mon Sep 17 00:00:00 2001 From: destel Date: Wed, 9 Sep 2026 15:19:14 +0300 Subject: [PATCH 29/49] iteration --- doc.go | 53 ++++++++++++++++++++++++++++------------------------- 1 file changed, 28 insertions(+), 25 deletions(-) diff --git a/doc.go b/doc.go index 993f928..c391bc4 100644 --- a/doc.go +++ b/doc.go @@ -27,42 +27,43 @@ // are called the source and the sink, respectively. // // ids := rill.FromSlice(userIDs, nil) // source -// filtered := rill.Filter(ids, 5, ...) // stage, 5 goroutines -// batches := rill.Batch(filtered, ...) // stage, 1 goroutine -// transformed := rill.Map(batches, 3, ...) // stage, 3 goroutines -// err := rill.ForEach(transformed, 2, ...) // sink, 2 goroutines +// filtered := rill.Filter(ids, 5, ...) // stage, cconcurrency=5 +// batches := rill.Batch(filtered, ...) // stage +// transformed := rill.Map(batches, 3, ...) // stage, concurrency=3 +// err := rill.ForEach(transformed, 2, ...) // sink, concurrency=2 // // The source and intermediate stages do not block: they spawn goroutines then immediately return their output stream. // Those goroutines keep working until all input streams are fully consumed and processed. When all work is done, // the output stream is closed. // -// Sinks are different, they block, then return as soon as pipeline's outcome is known, which -// can happen before the input is fully consumed and all work across the pipeline is done. In particular, -// a sinks return early when: +// Sinks are different, they block until the pipeline's outcome is known, which +// can happen before the input is fully consumed and all work across the pipeline is done. +// What "outcome known" means depends on the sink, for example: // -// - it observes an error, whether it came from upstream or from the sink itself -// - its internal short-circuit condition is met, for example [Any] returns as soon as it finds a match, and [First] returns after -// consuming a single item from its input stream +// - [ForEach] immediately returns the first error it observes, otherwise fully consumes the input +// - [Any] can additionally short-circuit on the first match it finds +// - [First] never consumes more than one item from its input +// +// On early return, a sink drains and discards the remaining input in the +// background, so upstream stages don't block forever and leak their goroutines. // // # Context, cancellation and settlement // -// When sink returns early it keeps draining and discarding the remaining input in the background to prevent upstream -// stages from blocking forever and leaking their goroutines. Expensive cancellable work is usually context aware, so -// caller can cancel anything that remains after the sink's early return: +// That drain still pays for whatever work is already in flight upstream; +// only the caller can bound it further, since a sink can't reach the +// stages feeding it. Expensive cancellable work is usually context aware, +// so the caller can cancel anything that remains after the sink's early +// return: // // ctx, cancel := context.WithCancel(ctx) // defer cancel() // -// // source and other pipeline stages go here; -// // they might be context aware +// // source and other pipeline stages go here // -// err := rill.ForEach(stream, 5, func(x int) error { +// err := rill.ForEach(transformed, 5, func(x int) error { // return process(ctx, x) // }) -// -// // result known, cancel the context -// // or rely on the deferred call above -// cancel() +// // result known; the deferred cancel stops the rest // // When the caller wants not only to request cancellation but also to wait // for the pipeline to settle (no work remains, every user callback has @@ -71,16 +72,18 @@ // scope, ctx := rill.NewScope(ctx) // defer scope.Cancel() // -// // source and other pipeline stages go here; -// // they might be context aware +// // source and other pipeline stages go here; the source must also +// // watch ctx, or Wait below never returns (see [NewScope]'s example) // -// err := rill.ForEach(stream, 5, func(x int) error { +// err := rill.ForEach(transformed, 5, func(x int) error { // return process(ctx, x) // }, scope) -// // // result known // -// scope.Wait() // cancel context and wait for settlement +// scope.Wait() // cancel and wait for settlement +// +// In computation-only pipelines that never fail or short-circuit, everything +// settles by the time the sink returns, so [Scope] is not needed. // // ------ the end so far ------ // From 8554774c8a02f91c0ed479441be1f4912164a238 Mon Sep 17 00:00:00 2001 From: destel Date: Wed, 9 Sep 2026 22:09:37 +0300 Subject: [PATCH 30/49] Finish the package doc rewrite --- doc.go | 249 +++++++++++---------------------------------------------- 1 file changed, 48 insertions(+), 201 deletions(-) diff --git a/doc.go b/doc.go index c391bc4..e81c60a 100644 --- a/doc.go +++ b/doc.go @@ -1,4 +1,4 @@ -// Package rill provides composable primitives for building streaming +// Package rill provides composable primitives for building concurrent streaming // pipelines over plain Go channels: functions that transform, filter, batch, // reduce, and consume data streams, with bounded concurrency per stage, // centralized error handling, optional order preservation, and minimal @@ -13,28 +13,31 @@ // definition of a pipeline becomes: // // A pipeline is a series of stages connected by streams - channels whose items -// are [Try] structs, each holding either a value or an error. Each stage is a -// group of goroutines running the same function; the argument n, where present, -// is the size of the group. In each stage, the goroutines: +// are [Try] structs, each holding either a value or an error. Under the hood, each stage +// runs one or more goroutines that: // -// - receive values and errors from upstream via inbound streams -// - apply the function to the values, usually producing new values or errors -// - send the results downstream via outbound streams; upstream errors pass -// through unchanged +// - receive values and errors from upstream via input streams +// - process the received values, usually producing new values or errors +// - send the results downstream via output streams +// - forward upstream errors to the output streams ([Catch] is the only exception) // -// Each stage can have any number of input and output streams, except the first stage -// that has no input streams and the last stage that has no output streams. These stages -// are called the source and the sink, respectively. +// Usually, in a pipeline most stages have one input stream and one output stream, +// except the first stage that has no input stream and the last stage that has no output stream. +// These stages are called the source and the sink, respectively. [Merge] and [Tee] functions +// have more inputs/outputs and can be used to build DAG pipelines. // // ids := rill.FromSlice(userIDs, nil) // source -// filtered := rill.Filter(ids, 5, ...) // stage, cconcurrency=5 +// filtered := rill.Filter(ids, 5, ...) // stage, concurrency = 5 // batches := rill.Batch(filtered, ...) // stage -// transformed := rill.Map(batches, 3, ...) // stage, concurrency=3 -// err := rill.ForEach(transformed, 2, ...) // sink, concurrency=2 +// transformed := rill.Map(batches, 3, ...) // stage, concurrency = 3 +// err := rill.ForEach(transformed, 2, ...) // sink, concurrency = 2 // -// The source and intermediate stages do not block: they spawn goroutines then immediately return their output stream. -// Those goroutines keep working until all input streams are fully consumed and processed. When all work is done, -// the output stream is closed. +// Intermediate stages never block: they return their output streams +// immediately, while the goroutines they started stay working in +// the background. These stages always fully consume and process their +// inputs, before closing their outputs. This closure becomes an "all +// upstream work is done" signal that travels downstream along with values +// and errors. // // Sinks are different, they block until the pipeline's outcome is known, which // can happen before the input is fully consumed and all work across the pipeline is done. @@ -49,11 +52,9 @@ // // # Context, cancellation and settlement // -// That drain still pays for whatever work is already in flight upstream; -// only the caller can bound it further, since a sink can't reach the -// stages feeding it. Expensive cancellable work is usually context aware, -// so the caller can cancel anything that remains after the sink's early -// return: +// It's up to the caller whether to cancel the extra work that happens +// after an early return. Expensive work and large/infinite sources are usually +// context-aware, so all that's needed is to cancel the context they captured: // // ctx, cancel := context.WithCancel(ctx) // defer cancel() @@ -63,7 +64,9 @@ // err := rill.ForEach(transformed, 5, func(x int) error { // return process(ctx, x) // }) -// // result known; the deferred cancel stops the rest +// +// // result known; cancel manually or rely on deferred cancel +// cancel() // // When the caller wants not only to request cancellation but also to wait // for the pipeline to settle (no work remains, every user callback has @@ -82,80 +85,18 @@ // // scope.Wait() // cancel and wait for settlement // +// Under the hood, [Scope.Wait] waits for the sink's own work to finish, +// and for the "all upstream work is done" signal carried by the sink's +// input streams. +// // In computation-only pipelines that never fail or short-circuit, everything // settles by the time the sink returns, so [Scope] is not needed. // -// ------ the end so far ------ -// -// # If sink returned early -// -// which defines pipeline as a series of stages, -// connected by channels, where -// -// The post defines pipeline as a series of stages, -// connected by channels, -// -// # Streams -// -// In this package, a stream is a plain channel that carries both values and errors. -// Each stream item is an instance of a [Try] struct that represents either a value or an error. -// This is Go's (value, error) return convention, carried over to channels. -// -// # Stages, composition and pipelines -// -// Many functions in this package take a stream as input and return a new stream as output. -// [Map], [Filter], and other such functions are called stages. They: -// -// - do not block, and return the output stream immediately -// - process input values as they arrive -// - write processing results to the output as they are ready -// - forward input error items to the output as-is -// - write processing errors to the output as they occur -// - close the output stream after the input is fully consumed and processed -// -// Stages (along with sources and sinks described below) are generic and can -// be used either standalone or composed into multi-stage pipelines, -// where the output of one function becomes the input to the next. -// -// ids := rill.FromSlice(userIDs, nil) -// filtered := rill.Filter(ids, ...) -// batches := rill.Batch(filtered, ...) -// err := rill.ForEach(batches, ...) -// -// # Sources -// -// Every pipeline begins with a stream that is created rather than -// transformed. Any channel of [Try] structs can play this role, no matter -// where it comes from - a rill helper such as [FromSlice] or [Generate], a -// third-party library, or hand-written code. This first stream, together -// with the code feeding it, is called the source. -// -// # Sinks -// -// Every pipeline ends with a function called a sink. Sinks, such as [ForEach] or [MapReduce], -// take a stream as input but return a regular Go value and/or an error. Such functions: -// -// - block, until the final outcome (successful or not) is known -// - return early (before the input is fully consumed) on the first observed error, regardless of where it came from - upstream or the sink itself -// - can return early because of the sink's internal logic, for example [Any] returns as soon as it finds a match -// - on early return keep consuming and discarding the remaining input items (including -// late errors) in the background, so upstream stages do not block and leak their goroutines -// - can optionally report pipeline settlement (see below) via the [Scope] API -// -// # Concurrency -// -// Most stages and sinks are concurrent, and take the argument n, which -// acts as both an upper bound and a target for the number of concurrent -// invocations of the user callback. Rill never exceeds this bound, and, -// given enough input, reaches it. With n = 1, the callback is never -// invoked concurrently: items are processed one by one, in input order. -// // # Ordered stages // -// By default, results and errors are written to the output as soon as they -// are ready, in completion order. That order depends on how the Go runtime -// schedules the goroutines in the stage's worker pool, and on how much time -// each individual item takes to process. +// By default, stages write results to their output streams as soon as they +// are ready, in completion order. In concurrent stages, that order depends on how the Go runtime +// schedules the stages' goroutines, and on the time each result takes to produce. // // For cases where the input order must be preserved, rill provides ordered functions, // such as [OrderedMap] or [OrderedFilter]. They stay concurrent, @@ -163,126 +104,33 @@ // so the output order matches the input order at the cost // of some latency. This ordering guarantee holds for both values and errors. // -// Some stages, such as [Batch] or [Unbatch], process items sequentially, so -// they are naturally ordered. -// -// # Error handling -// -// Every error, wherever in the pipeline it originates, eventually reaches -// the sink, and the sink returns the first one it observes to the caller. -// -// To handle errors mid-pipeline, use [Catch]: a stage whose callback sees -// errors rather, and can handle, keep, or rewrite them. -// -// [Catch] is also handy for tracking where errors come from. The snippet -// below tags every error coming out of the source, so that later they -// can be told apart from errors raised in the stages: -// -// var errSource = errors.New("source failed") -// -// source = rill.Catch(source, 1, func(err error) error { -// return fmt.Errorf("%w: %w", errSource, err) -// }) -// -// # Pipeline lifecycle -// -// Rill's lifecycle model follows from two design decisions: -// -// - don't become a framework: pipelines are not first-class objects, but -// compositions of simpler functions that know nothing about each other -// - streams are plain channels: data and errors can only travel downstream -// -// Together these force three things. A sink cannot stop or cancel the stages -// feeding it, only the caller can. A sink must pass control back to the caller -// as soon as the outcome is known, which can happen before the input -// is fully consumed. A sink must then drain the remaining input in the background, so -// upstream stages don't block forever and their callbacks can observe cancellation. -// -// A pipeline goes through three phases on its own. The caller can add two -// optional steps: -// -// - active: processing is in progress, the sink is blocked -// - result known: the sink has returned; upstream stages may -// still be working, but the sink drains and discards their results in -// the background -// - cancelled (optional): the caller cancels a context; the source -// stops producing new work, and the stages stop doing it -// - settled: no work remains; every user callback across the pipeline has -// returned -// - joined (optional): the caller has waited for settlement, and can now -// do what would otherwise conflict with callbacks in flight: release -// resources they used, or read state they wrote -// -// In computation-only pipelines where nothing ever fails or short-circuits, -// the pipeline is already settled by the time the sink returns. -// -// To wait for settlement in pipelines that can return early (because of an -// error or any other reason), rill provides the [Scope] API that has a -// shape similar to errgroup. This API derives a context, manages its cancellation, -// and allows the caller to wait for settlement. And the same way as in errgroup, -// the already-submitted work can't be withdrawn, only cooperatively cancelled: -// heavy user callbacks must capture the derived context and respect its cancellation. -// -// scope, ctx := rill.NewScope(ctx) -// defer scope.Cancel() -// -// // source and other pipeline stages go here -// -// err := rill.ForEach(stream, 5, func(x int) error { -// return process(ctx, x) -// }, scope) -// -// // result known -// -// scope.Wait() // cancel context and wait for settlement -// -// // joined -// -// When joining is not needed, it's possible to use [Scope] -// in cancellation-only mode, or just use a regular [context.WithCancel]. -// Even that might not be necessary if the call site already has a cancellable context -// (which is often the case when heavy network calls are involved), so all context -// plumbing goes away: -// -// // source and other pipeline stages go here -// -// err := rill.ForEach(stream, 5, func(x int) error { -// return process(ctx, x) -// }) -// -// if err != nil { -// // just return, the context will be cancelled up in the call stack -// return err -// } -// // # Extending rill // -// Sources, stages, and sinks are ordinary functions that receive and/or -// return channels, so any user function of a similar shape works with the -// rest of the library. +// Rill is not a framework, but a collection of functions over plain Go +// channels. Almost any function that receives and/or returns such +// channels is compatible with rill. // -// For example, it's easy to write a source that streams rows from +// For example, it's easy to write a context-aware source that streams rows from // a database table, or a sink that collects all observed errors into a // slice. // -// There are a few rules custom functions must follow to be compatible with rill's lifecycle. -// These rules are usually satisfied by construction. +// The easiest way to write a custom stage is to compose it from existing +// functions rill provides. For manually written stages there are a few +// simple rules to follow. Most of them are satisfied by construction, +// and related to preserving background drain and settlement semantics: // // - sources must eventually close their output stream; a source that can // run forever must watch a context // - stages must close their output stream only after the input is fully -// consumed and all workers have returned +// consumed and processed // - sinks must start with a deferred rill.Discard(in, options...), followed by a // for-range loop that returns as soon as the sink's outcome is known // -// Custom stages and sinks can also be built by composing existing -// rill functions, which is often the simplest way. -// // # Backpressure // -// In the context of Go channels, backpressure means that sending to an -// unbuffered channel blocks until the receiver on the other end is ready to -// receive. Rill naturally inherits this property: a slow stage in the +// Backpressure means that sending to an unbuffered channel blocks until +// the receiver on the other end is ready to receive. Rill naturally +// inherits this property: a slow stage in the // pipeline blocks the previous stage, and it in turn blocks the stage before that, // and so on, until the slow stage catches up. // @@ -295,13 +143,12 @@ // forever. // // Rill does not introduce any special semantics for nil channels. If a stage -// receives a channel that blocks forever when read, it returns a channel that -// also blocks forever. If a sink receives such a channel, the sink itself -// hangs. +// receives a stream that's never closed, it never closes its output stream. +// If a sink receives such a stream, the sink itself blocks forever. // // # Panics // -// Rill validates arguments of its functions and panics on misuse, such that zero or negative concurrency level. +// Rill validates arguments of its functions and panics on misuse, such as zero or negative concurrency. // Rill does not automatically recover panics in user callbacks: a panicking // callback can crash the process, as it would in any hand-written concurrent // code. From e7e1286d8aba67f20912abb4bc5d4cbf947f8e5b Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 09:59:00 +0300 Subject: [PATCH 31/49] Split settlement into its own chapter, fix nil-handling claims --- doc.go | 64 +++++++++++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/doc.go b/doc.go index e81c60a..969fddf 100644 --- a/doc.go +++ b/doc.go @@ -45,12 +45,12 @@ // // - [ForEach] immediately returns the first error it observes, otherwise fully consumes the input // - [Any] can additionally short-circuit on the first match it finds -// - [First] never consumes more than one item from its input +// - [First] consumes one item and returns // // On early return, a sink drains and discards the remaining input in the // background, so upstream stages don't block forever and leak their goroutines. // -// # Context, cancellation and settlement +// # Context and cancellation // // It's up to the caller whether to cancel the extra work that happens // after an early return. Expensive work and large/infinite sources are usually @@ -68,6 +68,8 @@ // // result known; cancel manually or rely on deferred cancel // cancel() // +// # Structured concurrency +// // When the caller wants not only to request cancellation but also to wait // for the pipeline to settle (no work remains, every user callback has // returned), rill provides the [Scope] API, which is like errgroup for pipelines. @@ -75,15 +77,17 @@ // scope, ctx := rill.NewScope(ctx) // defer scope.Cancel() // -// // source and other pipeline stages go here; the source must also -// // watch ctx, or Wait below never returns (see [NewScope]'s example) +// // source and other pipeline stages go here // // err := rill.ForEach(transformed, 5, func(x int) error { // return process(ctx, x) // }, scope) +// // // result known // -// scope.Wait() // cancel and wait for settlement +// scope.Wait() // cancel ctx and wait for settlement +// +// // it's now safe to release resources and observe side effects // // Under the hood, [Scope.Wait] waits for the sink's own work to finish, // and for the "all upstream work is done" signal carried by the sink's @@ -104,6 +108,29 @@ // so the output order matches the input order at the cost // of some latency. This ordering guarantee holds for both values and errors. // +// # Backpressure +// +// Backpressure means that sending to an unbuffered channel blocks until +// the receiver on the other end is ready to receive. Rill naturally +// inherits this property: a slow stage in the +// pipeline blocks the previous stage, and it in turn blocks the stage before that, +// and so on, until the slow stage catches up. +// +// When this is not desirable, use [Buffer] to add slack between stages. +// +// # Nil handling +// +// Rill relies on input streams eventually closing for pipelines to finish. +// Nil channels never emit values or close, so passing nil as an input +// can leak goroutines or leave a sink blocked forever. +// +// # Panics +// +// Rill validates arguments of its functions and panics on misuse, such as zero or negative concurrency. +// Rill does not automatically recover panics in user callbacks: a panicking +// callback can crash the process, as it would in any hand-written concurrent +// code. +// // # Extending rill // // Rill is not a framework, but a collection of functions over plain Go @@ -125,31 +152,4 @@ // consumed and processed // - sinks must start with a deferred rill.Discard(in, options...), followed by a // for-range loop that returns as soon as the sink's outcome is known -// -// # Backpressure -// -// Backpressure means that sending to an unbuffered channel blocks until -// the receiver on the other end is ready to receive. Rill naturally -// inherits this property: a slow stage in the -// pipeline blocks the previous stage, and it in turn blocks the stage before that, -// and so on, until the slow stage catches up. -// -// When this is not desirable, use [Buffer] to add slack between stages. -// -// # Nil handling -// -// Nil channels are valid in Go. They never emit values and are never closed. -// In practice, this means that an attempt to read from a nil channel blocks -// forever. -// -// Rill does not introduce any special semantics for nil channels. If a stage -// receives a stream that's never closed, it never closes its output stream. -// If a sink receives such a stream, the sink itself blocks forever. -// -// # Panics -// -// Rill validates arguments of its functions and panics on misuse, such as zero or negative concurrency. -// Rill does not automatically recover panics in user callbacks: a panicking -// callback can crash the process, as it would in any hand-written concurrent -// code. package rill From 9a0330293dec886959b95c98c585457b10463478 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 10:11:02 +0300 Subject: [PATCH 32/49] Drop the package-doc pointer trailer from stage doc comments --- batch.go | 4 ---- merge.go | 2 -- transform.go | 10 ---------- 3 files changed, 16 deletions(-) diff --git a/batch.go b/batch.go index 34c8e1d..b95491d 100644 --- a/batch.go +++ b/batch.go @@ -26,8 +26,6 @@ import ( // When the end of the input is reached, whatever has accumulated is // emitted as a final batch. This function never emits empty batches, // regardless of what triggered the emission. -// -// See the [rill] package documentation for the full contract shared by all stages. func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[[]A] { validateMinSize(size, 1) if timeout == 0 { @@ -122,8 +120,6 @@ func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[ // Unbatch flattens a stream of slices into a stream of their values. // This function is the inverse of [Batch]. -// -// See the [rill] package documentation for the full contract shared by all stages. func Unbatch[A any](in <-chan Try[[]A]) <-chan Try[A] { if in == nil { return nil diff --git a/merge.go b/merge.go index b1c5edf..60ff165 100644 --- a/merge.go +++ b/merge.go @@ -28,8 +28,6 @@ func Merge[A any](ins ...<-chan A) <-chan A { // differ from the input order when n > 1. Use [OrderedSplit2] to // preserve the order. // -// See the [rill] package documentation for the full contract shared by all stages. -// // Deprecated: Split2 will be removed in v1.0. Since the introduction of [Tee] // in v0.8, splitting no longer needs a dedicated operation — it can be composed // from existing ones. Unlike Split2, the composition is also not limited to two diff --git a/transform.go b/transform.go index ab2b5d6..1051a3f 100644 --- a/transform.go +++ b/transform.go @@ -11,8 +11,6 @@ import ( // The argument n bounds the number of concurrent calls to f. // Results and errors are written to the output in completion order. // Use [OrderedMap] to preserve the input order. -// -// See the [rill] package documentation for the full contract shared by all stages. func Map[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -59,8 +57,6 @@ func OrderedMap[A, B any](in <-chan Try[A], n int, f func(A) (B, error)) <-chan // The argument n bounds the number of concurrent calls to f. // Results and errors are written to the output in completion order. // Use [OrderedFilter] to preserve the input order. -// -// See the [rill] package documentation for the full contract shared by all stages. func Filter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) @@ -107,8 +103,6 @@ func OrderedFilter[A any](in <-chan Try[A], n int, f func(A) (bool, error)) <-ch // The argument n bounds the number of concurrent calls to f. // Results and errors are written to the output in completion order. // Use [OrderedFilterMap] to preserve the input order. -// -// See the [rill] package documentation for the full contract shared by all stages. func FilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, error)) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -157,8 +151,6 @@ func OrderedFilterMap[A, B any](in <-chan Try[A], n int, f func(A) (B, bool, err // When n > 1, items from different sub-streams can interleave in the // output. Use [OrderedFlatMap] to concatenate the sub-streams in the // input order. -// -// See the [rill] package documentation for the full contract shared by all stages. func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) @@ -254,8 +246,6 @@ func OrderedFlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) // The argument n bounds the number of concurrent calls to f. // Items are written to the output in completion order. // Use [OrderedCatch] to preserve the input order. -// -// See the [rill] package documentation for the full contract shared by all stages. func Catch[A any](in <-chan Try[A], n int, f func(error) error) <-chan Try[A] { validateN(n) validateNilFunc(f == nil) From 3e08db1e72a4c1cfefd121f21501cd186fad9f4b Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 10:20:54 +0300 Subject: [PATCH 33/49] Document Scope interaction on Discard --- util.go | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/util.go b/util.go index 0704dae..2a2cd6b 100644 --- a/util.go +++ b/util.go @@ -12,8 +12,9 @@ func Drain[A any](in <-chan A) { core.Drain(in) } -// Discard returns immediately, then consumes and discards all items of -// the channel in the background. +// Discard returns immediately, then drains and discards all items of +// the channel in the background. A [Scope] passed as an option can be +// used to find out when draining completes. func Discard[A any](in <-chan A, options ...SinkOption) { opts := collectSinkOptions(options) From 32e18611551993f28fffd8f8559f0541f7985910 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 15:07:24 +0300 Subject: [PATCH 34/49] Rewrite the package doc opening and trim the extending chapter --- doc.go | 38 ++++++++++++++++++-------------------- 1 file changed, 18 insertions(+), 20 deletions(-) diff --git a/doc.go b/doc.go index 969fddf..d729af4 100644 --- a/doc.go +++ b/doc.go @@ -1,16 +1,17 @@ -// Package rill provides composable primitives for building concurrent streaming -// pipelines over plain Go channels: functions that transform, filter, batch, -// reduce, and consume data streams, with bounded concurrency per stage, -// centralized error handling, optional order preservation, and minimal -// boilerplate. +// Package rill provides composable concurrency primitives: functions over +// plain channels that transform, filter, batch, reduce, and consume data +// streams while propagating errors and optionally preserving order. +// +// Rill is not a framework: its functions can be used on their own or +// composed into multi-stage pipelines. Either way, they are compatible +// with existing channel-based code. // // # Pipelines and streams // -// Rill functions can be used standalone or composed into multi-stage pipelines. -// The model is similar to the Go blog's "Pipelines and cancellation" -// (https://go.dev/blog/pipelines), but it unifies error handling by letting -// errors travel downstream along with values. In rill's terms, the post's -// definition of a pipeline becomes: +// The pipeline model in this package is similar to the one described in the +// Go blog's "Pipelines and cancellation" (https://go.dev/blog/pipelines), +// but unifies error handling by letting errors travel downstream along with +// values. In rill's terms, the post's definition of a pipeline becomes: // // A pipeline is a series of stages connected by streams - channels whose items // are [Try] structs, each holding either a value or an error. Under the hood, each stage @@ -133,22 +134,19 @@ // // # Extending rill // -// Rill is not a framework, but a collection of functions over plain Go -// channels. Almost any function that receives and/or returns such -// channels is compatible with rill. -// -// For example, it's easy to write a context-aware source that streams rows from -// a database table, or a sink that collects all observed errors into a -// slice. +// Almost any custom function that takes or returns streams is compatible +// with rill. For example, it's easy to write a context-aware source that +// streams rows from a database table, or a sink that collects all observed +// errors into a slice. // // The easiest way to write a custom stage is to compose it from existing // functions rill provides. For manually written stages there are a few // simple rules to follow. Most of them are satisfied by construction, -// and related to preserving background drain and settlement semantics: +// and are related to preserving background drain and settlement semantics: // // - sources must eventually close their output stream; a source that can -// run forever must watch a context -// - stages must close their output stream only after the input is fully +// run forever must watch a context and be cancellable +// - stages must close their output stream, but only after the input is fully // consumed and processed // - sinks must start with a deferred rill.Discard(in, options...), followed by a // for-range loop that returns as soon as the sink's outcome is known From a45cd252b0b9a17ceb6b62577dd09e3778745a90 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 15:10:22 +0300 Subject: [PATCH 35/49] Use the doc's own term in the sink examples --- doc.go | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc.go b/doc.go index d729af4..8af4e63 100644 --- a/doc.go +++ b/doc.go @@ -66,7 +66,7 @@ // return process(ctx, x) // }) // -// // result known; cancel manually or rely on deferred cancel +// // outcome known; cancel manually or rely on deferred cancel // cancel() // // # Structured concurrency @@ -84,7 +84,7 @@ // return process(ctx, x) // }, scope) // -// // result known +// // outcome known // // scope.Wait() // cancel ctx and wait for settlement // From d05d85662f56f731844b169c4f3f4098598c5969 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 15:24:53 +0300 Subject: [PATCH 36/49] Copy-edit the package doc --- doc.go | 39 ++++++++++++++++++++------------------- 1 file changed, 20 insertions(+), 19 deletions(-) diff --git a/doc.go b/doc.go index 8af4e63..fcb8d3a 100644 --- a/doc.go +++ b/doc.go @@ -10,7 +10,7 @@ // // The pipeline model in this package is similar to the one described in the // Go blog's "Pipelines and cancellation" (https://go.dev/blog/pipelines), -// but unifies error handling by letting errors travel downstream along with +// but it unifies error handling by letting errors travel downstream along with // values. In rill's terms, the post's definition of a pipeline becomes: // // A pipeline is a series of stages connected by streams - channels whose items @@ -22,10 +22,11 @@ // - send the results downstream via output streams // - forward upstream errors to the output streams ([Catch] is the only exception) // -// Usually, in a pipeline most stages have one input stream and one output stream, -// except the first stage that has no input stream and the last stage that has no output stream. -// These stages are called the source and the sink, respectively. [Merge] and [Tee] functions -// have more inputs/outputs and can be used to build DAG pipelines. +// Usually, most stages in a pipeline have one input stream and one output stream. +// The exceptions are the first stage, which has no input stream, and the last +// stage, which has no output stream. These stages are called the source and +// the sink, respectively. The [Merge] and [Tee] functions have more inputs/outputs +// and can be used to build DAG pipelines. // // ids := rill.FromSlice(userIDs, nil) // source // filtered := rill.Filter(ids, 5, ...) // stage, concurrency = 5 @@ -34,21 +35,21 @@ // err := rill.ForEach(transformed, 2, ...) // sink, concurrency = 2 // // Intermediate stages never block: they return their output streams -// immediately, while the goroutines they started stay working in +// immediately, while the goroutines they started continue working in // the background. These stages always fully consume and process their -// inputs, before closing their outputs. This closure becomes an "all +// inputs before closing their outputs. This closure becomes an "all // upstream work is done" signal that travels downstream along with values // and errors. // -// Sinks are different, they block until the pipeline's outcome is known, which +// Sinks are different: they block until the pipeline's outcome is known, which // can happen before the input is fully consumed and all work across the pipeline is done. -// What "outcome known" means depends on the sink, for example: +// What "outcome known" means depends on the sink. For example: // -// - [ForEach] immediately returns the first error it observes, otherwise fully consumes the input +// - [ForEach] immediately returns the first error it observes; otherwise, it fully consumes the input // - [Any] can additionally short-circuit on the first match it finds // - [First] consumes one item and returns // -// On early return, a sink drains and discards the remaining input in the +// On an early return, a sink drains and discards the remaining input in the // background, so upstream stages don't block forever and leak their goroutines. // // # Context and cancellation @@ -66,13 +67,13 @@ // return process(ctx, x) // }) // -// // outcome known; cancel manually or rely on deferred cancel +// // outcome known; cancel manually or rely on the deferred cancel // cancel() // // # Structured concurrency // // When the caller wants not only to request cancellation but also to wait -// for the pipeline to settle (no work remains, every user callback has +// for the pipeline to settle (no work remains and every user callback has // returned), rill provides the [Scope] API, which is like errgroup for pipelines. // // scope, ctx := rill.NewScope(ctx) @@ -90,7 +91,7 @@ // // // it's now safe to release resources and observe side effects // -// Under the hood, [Scope.Wait] waits for the sink's own work to finish, +// Under the hood, [Scope.Wait] waits for the sink's own work to finish // and for the "all upstream work is done" signal carried by the sink's // input streams. // @@ -101,7 +102,7 @@ // // By default, stages write results to their output streams as soon as they // are ready, in completion order. In concurrent stages, that order depends on how the Go runtime -// schedules the stages' goroutines, and on the time each result takes to produce. +// schedules the stages' goroutines and on the time it takes to produce each result. // // For cases where the input order must be preserved, rill provides ordered functions, // such as [OrderedMap] or [OrderedFilter]. They stay concurrent, @@ -127,8 +128,8 @@ // // # Panics // -// Rill validates arguments of its functions and panics on misuse, such as zero or negative concurrency. -// Rill does not automatically recover panics in user callbacks: a panicking +// Rill validates the arguments to its functions and panics on misuse, such as zero or negative concurrency. +// Rill does not automatically recover from panics in user callbacks: a panicking // callback can crash the process, as it would in any hand-written concurrent // code. // @@ -140,8 +141,8 @@ // errors into a slice. // // The easiest way to write a custom stage is to compose it from existing -// functions rill provides. For manually written stages there are a few -// simple rules to follow. Most of them are satisfied by construction, +// functions rill provides. For manually written stages, there are a few +// simple rules to follow. Most of them are satisfied by construction // and are related to preserving background drain and settlement semantics: // // - sources must eventually close their output stream; a source that can From aa79d74c23d4a42302fe1a13775e31b5ab6b188b Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 17:36:03 +0300 Subject: [PATCH 37/49] Refine exported doc comments --- consume.go | 2 +- iter.go | 2 +- merge.go | 35 +++++++++++++++++------------------ options.go | 4 ++-- scope.go | 2 +- transform.go | 20 ++++++++++---------- util.go | 5 ++--- wrap.go | 13 ++++++------- 8 files changed, 40 insertions(+), 43 deletions(-) diff --git a/consume.go b/consume.go index b84bd10..9fa37f7 100644 --- a/consume.go +++ b/consume.go @@ -11,7 +11,7 @@ import ( // returned. // // The argument n bounds the number of concurrent calls to f. When n = 1, -// ForEach processes items sequentially in stream order, similar to a +// ForEach processes items sequentially in stream order, the same as in a // regular for-range loop: f can safely read and modify shared state without // synchronization, and all its effects are visible to the caller after // ForEach returns. diff --git a/iter.go b/iter.go index 0c8272b..74b85f6 100644 --- a/iter.go +++ b/iter.go @@ -57,7 +57,7 @@ func FromSeq2[A any](seq iter.Seq2[A, error]) <-chan Try[A] { // stream is exhausted or the loop exits with break or return. Error // items do not stop the iteration: they are yielded as ordinary pairs. // -// The returned iterator is single-use and must be ranged for the +// The returned iterator is single-use and must be ranged over for the // pipeline to settle. If the loop exits early with break or return, // ToSeq2 drains the input in the background before reporting settlement. func ToSeq2[A any](in <-chan Try[A], options ...SinkOption) iter.Seq2[A, error] { diff --git a/merge.go b/merge.go index 60ff165..dd91e1b 100644 --- a/merge.go +++ b/merge.go @@ -4,15 +4,14 @@ import ( "github.com/destel/rill/internal/core" ) -// Merge performs a fan-in, combining multiple channels into a single -// output channel. It returns immediately, and consumes the inputs -// simultaneously and independently, interleaving their items in the -// output as they arrive. Merge preserves the relative order of items -// from the same input. +// Merge performs a fan-in: it returns a channel that carries the items from +// all inputs, interleaved as they arrive, with the relative order of items +// from the same input preserved. Merge consumes its inputs simultaneously +// and independently. // -// The output is closed only when all inputs are exhausted. -// A nil input is never exhausted, so the output never closes. -// Merge with no arguments immediately returns an empty closed channel. +// The output is closed only when all inputs are exhausted. A nil input is +// never exhausted, so the output never closes. Merge with no arguments +// returns an empty closed channel. func Merge[A any](ins ...<-chan A) <-chan A { return core.Merge(ins...) } @@ -33,15 +32,15 @@ func Merge[A any](ins ...<-chan A) <-chan A { // from existing ones. Unlike Split2, the composition is also not limited to two // branches. // -// Quite often the predicate is a simple pure check (field comparison, type -// switch, etc). In such cases splitting is just [Tee] plus a [Filter] on each +// Quite often, the predicate is a simple, pure check (a field comparison, a type +// switch, etc.). In such cases, splitting is just [Tee] plus a [Filter] on each // branch: // // adults, minors := rill.Tee(users) // adults = rill.Filter(adults, 1, func(u User) (bool, error) { return u.Age >= 18, nil }) // minors = rill.Filter(minors, 1, func(u User) (bool, error) { return u.Age < 18, nil }) // -// If the predicate is expensive, stateful, or can fail, it must be evaluated +// If the predicate is expensive or stateful, or if it can fail, it must be evaluated // once per item, before [Tee]: inline this function's implementation, which // tags each item with the decision and routes on the tag. The same pattern // extends to n-way splitting by tagging with an index or key instead of a bool. @@ -67,22 +66,22 @@ func Split2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (outTrue <- } // OrderedSplit2 is the ordered version of [Split2]: the outputs -// preserve the input order, for values and errors alike. +// preserve the input order for values and errors alike. // // Deprecated: OrderedSplit2 will be removed in v1.0. Since the introduction of // [Tee] in v0.8, splitting no longer needs a dedicated operation — it can be // composed from existing ones. Unlike OrderedSplit2, the composition is also // not limited to two branches. // -// Quite often the predicate is a simple pure check (field comparison, type -// switch, etc). In such cases splitting is just [Tee] plus an [OrderedFilter] +// Quite often, the predicate is a simple, pure check (a field comparison, a type +// switch, etc.). In such cases, splitting is just [Tee] plus an [OrderedFilter] // on each branch: // // adults, minors := rill.Tee(users) // adults = rill.OrderedFilter(adults, 1, func(u User) (bool, error) { return u.Age >= 18, nil }) // minors = rill.OrderedFilter(minors, 1, func(u User) (bool, error) { return u.Age < 18, nil }) // -// If the predicate is expensive, stateful, or can fail, it must be evaluated +// If the predicate is expensive or stateful, or if it can fail, it must be evaluated // once per item, before [Tee]: inline this function's implementation, which // tags each item with the decision and routes on the tag. The same pattern // extends to n-way splitting by tagging with an index or key instead of a bool. @@ -107,9 +106,9 @@ func OrderedSplit2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (out return } -// Tee duplicates the input channel into two identical channels. It -// returns immediately, forwards each item to both outputs as it -// arrives, and closes both once the input is exhausted. +// Tee duplicates the input: it returns two channels that both carry every +// item from the input, forwarded as it arrives. Both outputs are closed +// once the input is exhausted. // // The outputs must be consumed concurrently to avoid a deadlock. // diff --git a/options.go b/options.go index b89bfa6..866490a 100644 --- a/options.go +++ b/options.go @@ -10,8 +10,8 @@ func (o sinkOptions) settle() { } } -// A SinkOption is an optional argument accepted by every sink, such as a -// [Scope]. The interface cannot be implemented outside this package. +// A SinkOption is an optional argument accepted by every sink. +// [Scope] implements this interface. type SinkOption interface { apply(options *sinkOptions) } diff --git a/scope.go b/scope.go index 59f8c3e..89231fc 100644 --- a/scope.go +++ b/scope.go @@ -9,7 +9,7 @@ import ( // all of its work is done - including any work that happens after a sink's // early return. // -// A scope passed to a sink as a [SinkOption] tracks not only that sink, but +// A scope passed to a sink as a [SinkOption] tracks not only that sink but // also the whole pipeline behind it. // // For branching pipelines (see [Tee]), multiple sinks can be attached to the diff --git a/transform.go b/transform.go index 1051a3f..7c79f85 100644 --- a/transform.go +++ b/transform.go @@ -176,21 +176,21 @@ func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan return out } -// OrderedFlatMap is the ordered version of [FlatMap]: the output is the -// sub-streams concatenated in the input order. +// OrderedFlatMap is the ordered version of [FlatMap]: the output consists +// of the sub-streams concatenated in the input order. // // The argument n bounds the number of concurrent calls to f. The -// sub-streams are prepared concurrently, but - unlike in [FlatMap] - -// consumed one at a time and in order: nothing reads from a sub-stream +// sub-streams are prepared concurrently but - unlike in [FlatMap] - +// are consumed one at a time and in order: nothing reads from a sub-stream // before its turn. In practice, to keep the stage concurrent, a // sub-stream must do all or part of its expensive work ahead of its // turn. // // Consider a stream of URLs: each file should be downloaded, and its -// lines streamed to the output, all in order. Downloading is the +// lines should be streamed to the output, all in order. Downloading is the // expensive work here. // -// Example 1: f downloads the whole file into memory, then streams the +// Example 1: f downloads the whole file into memory and then streams the // lines from there. Up to 5 downloads run concurrently. // // rill.OrderedFlatMap(urls, 5, func(u string) <-chan rill.Try[string] { @@ -198,10 +198,10 @@ func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan // return rill.FromSlice(lines, err) // }) // -// Example 2: f streams the lines as the file is being downloaded, +// Example 2: as the file is being downloaded, f streams the lines // through a [Buffer] that lets the sub-stream run ahead of its turn. -// Again up to 5 concurrent downloads, but each pauses after the first -// 100 lines, until its turn comes. +// Again, up to 5 downloads run concurrently, but a download that runs +// ahead of its turn pauses after its first 100 lines until the turn comes. // // rill.OrderedFlatMap(urls, 5, func(u string) <-chan rill.Try[string] { // lines := streamFileLines(u) @@ -210,7 +210,7 @@ func FlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan // // The two examples do the same thing: they buffer the lines, with or // without a bound. Without any buffering, the downloads would run one -// at a time, and the stage would turn sequential. +// at a time, and the stage would become sequential. func OrderedFlatMap[A, B any](in <-chan Try[A], n int, f func(A) <-chan Try[B]) <-chan Try[B] { validateN(n) validateNilFunc(f == nil) diff --git a/util.go b/util.go index 2a2cd6b..0a0a568 100644 --- a/util.go +++ b/util.go @@ -47,9 +47,8 @@ func DrainNB[A any](in <-chan A) { Discard(in) } -// Buffer forwards all input items to a new channel with a capacity of -// size. It returns immediately and closes the output once the input is -// exhausted. +// Buffer returns a channel of the specified capacity and forwards all input +// items to it. The output is closed once the input is exhausted. // // users := getUsers(ctx, companyID) // users = rill.Buffer(users, 100) diff --git a/wrap.go b/wrap.go index 0cdd5db..03fe67d 100644 --- a/wrap.go +++ b/wrap.go @@ -8,7 +8,7 @@ type Try[A any] struct { } // Stream is a type alias for a receive-only channel of [Try] structs. -// Using it is optional, but improves readability. +// Using it is optional but improves readability. // // Before: // @@ -45,7 +45,7 @@ func Wrap[A any](value A, err error) Try[A] { // // This signature allows concise wrapping of functions that return a // slice and an error. FromSlice assumes that a non-empty slice along -// with an error is a partial result, and preserves both. +// with an error is a partial result and preserves both. // // stream := rill.FromSlice(someFunc()) func FromSlice[A any](slice []A, err error) <-chan Try[A] { @@ -179,10 +179,9 @@ func FromChans[A any](values <-chan A, errs <-chan error) <-chan Try[A] { return out } -// ToChans splits the stream into two channels: one for values and one -// for errors. It returns immediately, forwards each item to the -// appropriate channel as it arrives, and closes both channels once the -// input is exhausted. +// ToChans splits the stream into two channels, one for values and one for +// errors, and forwards each item to the appropriate one. Both channels are +// closed once the input is exhausted. // // The channels must be consumed concurrently to avoid a deadlock. func ToChans[A any](in <-chan Try[A]) (<-chan A, <-chan error) { @@ -209,7 +208,7 @@ func ToChans[A any](in <-chan Try[A]) (<-chan A, <-chan error) { return out, errs } -// Generate is a shorthand for creating streams: it manages the +// Generate is shorthand for creating streams: it manages the // goroutine and channel lifecycle. Inside f, send writes a value to the // stream, and sendError writes an error unless it is nil. // From ff86745642093730b0e0be42ed320057c9c24562 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 17:36:50 +0300 Subject: [PATCH 38/49] Copy-edit mockapi doc comments --- mockapi/files.go | 2 +- mockapi/users.go | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/mockapi/files.go b/mockapi/files.go index 17a60c4..548f583 100644 --- a/mockapi/files.go +++ b/mockapi/files.go @@ -6,7 +6,7 @@ import ( "time" ) -// DownloadFile simulates a file download. It returns the whole content as []byte. +// DownloadFile simulates a file download. It returns the entire contents as a byte slice. func DownloadFile(ctx context.Context, url string) ([]byte, error) { if err := simulateWork(ctx, 1000*time.Millisecond); err != nil { return nil, err diff --git a/mockapi/users.go b/mockapi/users.go index 0d26b98..59465ae 100644 --- a/mockapi/users.go +++ b/mockapi/users.go @@ -1,6 +1,6 @@ // Package mockapi provides a very basic mock API for examples and demos. // It's intentionally kept public to enable running and experimenting with examples in the Go Playground. -// The implementation is naive and uses full scan for all operations. +// The implementation is naive and uses a full scan for all operations. package mockapi import ( @@ -79,7 +79,7 @@ func GetUser(ctx context.Context, id int) (*User, error) { return &user, nil } -// GetUsers returns a list of users by IDs. +// GetUsers returns a list of users by their IDs. // If a user is not found, nil is returned in the corresponding position. func GetUsers(ctx context.Context, ids []int) ([]*User, error) { if err := simulateWork(ctx, 1000*time.Millisecond); err != nil { From 5b82bcf0274d4ae29b19623fa375246932c6abb5 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 19:01:55 +0300 Subject: [PATCH 39/49] Revise examples: trim descriptions, rename the parallel streaming example --- example_test.go | 89 ++++++++++++++++--------------------------------- 1 file changed, 29 insertions(+), 60 deletions(-) diff --git a/example_test.go b/example_test.go index 40a716b..28178d3 100644 --- a/example_test.go +++ b/example_test.go @@ -19,8 +19,8 @@ import ( // --- Package examples --- -// This example demonstrates a Rill pipeline that fetches users from an API, -// updates their status to active and saves them back. +// This example demonstrates a rill pipeline that fetches users from an API, +// updates their status to active, and saves them back. // Both operations are performed concurrently. // [ForEach] returns on the first error, and context cancellation via defer stops all remaining fetches. func Example() { @@ -58,8 +58,8 @@ func Example() { fmt.Println("Error:", err) } -// This example demonstrates a Rill pipeline that fetches users from an API, -// and updates their status to active and saves them back. +// This example demonstrates a rill pipeline that fetches users from an API, +// updates their status to active, and saves them back. // Users are fetched concurrently and in batches to reduce the number of API calls. func Example_batching() { ctx, cancel := context.WithCancel(context.Background()) @@ -107,12 +107,14 @@ func Example_batching() { // This example demonstrates how batching can be used to group similar concurrent database updates into a single query. // The UpdateUserTimestamp function is used to update the last_active_at column in the users table. Updates are not -// executed immediately, but are rather queued and then sent to the database in batches of up to 5. +// executed immediately but are instead queued and then sent to the database in batches of up to 5. // -// When updates are sparse, it can take some time to collect a full batch. In this case the [Batch] function +// When updates are sparse, it can take some time to collect a full batch. In this case, the [Batch] function // emits partial batches, ensuring that updates are delayed by at most 100ms. // -// For simplicity, this example does not have retries, error handling and synchronization +// For simplicity, this example does not include retries, error handling, or synchronization. +// A more complete version of this pattern, with context support, error handling, and +// synchronization, is described at https://destel.dev/blog/real-time-batching-in-go. func Example_batchingRealTime() { // Start the background worker that processes the updates go updateUserTimestampWorker() @@ -137,13 +139,13 @@ func Example_batchingRealTime() { // This is the queue of user IDs to update. var userIDsToUpdate = make(chan int) -// UpdateUserTimestamp is the public API for updating the last_active_at column in the users table +// UpdateUserTimestamp is the public API for updating the last_active_at column in the users table. func UpdateUserTimestamp(userID int) { userIDsToUpdate <- userID } // This is a background worker that sends queued updates to the database in batches. -// For simplicity, there are no retries, error handling and synchronization +// For simplicity, this worker does not include retries, error handling, or synchronization. func updateUserTimestampWorker() { // convert the channel of user IDs into a stream ids := rill.FromChan(userIDsToUpdate, nil) @@ -164,12 +166,12 @@ func updateUserTimestampWorker() { // hosted online. // // Downloading all files at once would consume too much memory, while processing -// them one-by-one would take too long. And traditional concurrency patterns do not preserve the order of files, +// them one by one would take too long. Traditional concurrency patterns do not preserve the order of files // and would make it challenging to find the first match. // -// The combination of [OrderedFilter] and [First] functions solves the problem, +// The combination of the [OrderedFilter] and [First] functions solves the problem // while downloading and holding in memory at most 5 files at the same time. -// [First] returns on the first match, this triggers the context cancellation via defer, +// [First] returns on the first match; this triggers context cancellation via defer, // stopping URL generation and file downloads. func Example_orderingAndContext() { ctx, cancel := context.WithCancel(context.Background()) @@ -216,9 +218,12 @@ func Example_orderingAndContext() { } } -// This example demonstrates using [FlatMap] to fetch users from multiple departments concurrently. -// Additionally, it demonstrates how to write a reusable streaming wrapper over paginated API calls - the StreamUsers function -func Example_flatMap() { +// This example demonstrates the parallel streaming pattern: [FlatMap] turns each +// department into its own stream of users and merges these streams into one, +// fetching from several departments concurrently. +// Additionally, it demonstrates how to write a reusable streaming wrapper over paginated API calls - +// the StreamUsers function. +func Example_parallelStreaming() { ctx, cancel := context.WithCancel(context.Background()) defer cancel() @@ -270,8 +275,8 @@ func StreamUsers(ctx context.Context, query *mockapi.UserQuery) <-chan rill.Try[ } // This example demonstrates how to gracefully stop a pipeline on the first error. -// The CheckAllUsersExist uses several concurrent workers and returns an error as soon as it encounters a non-existent user. -// Such early return triggers the context cancellation, which in turn stops all remaining users fetches. +// The CheckAllUsersExist function uses several concurrent workers and returns an error as soon as it encounters a non-existent user. +// Such an early return triggers context cancellation, which in turn stops all remaining user fetches. func Example_context() { ctx := context.Background() @@ -280,7 +285,7 @@ func Example_context() { fmt.Printf("Check result: %v\n", err) } -// CheckAllUsersExist uses several concurrent workers to check if all users with given IDs exist. +// CheckAllUsersExist uses several concurrent workers to check if all users with the given IDs exist. func CheckAllUsersExist(ctx context.Context, concurrency int, ids []int) error { ctx, cancel := context.WithCancel(ctx) defer cancel() // cancel the remaining requests after the first error @@ -334,7 +339,7 @@ func ExampleAny() { fmt.Println("Error: ", err) } -// Also check out the package level examples to see Batch in action +// See the package-level examples for more realistic uses of Batch. func ExampleBatch() { // Generate a stream of numbers 0 to 49, where a new number is emitted every 50ms numbers := rill.Generate(func(send func(int), sendError func(error)) { @@ -374,7 +379,6 @@ func ExampleCatch() { printStream(ids) } -// The same example as for the [Catch], but using ordered versions of functions. func ExampleOrderedCatch() { // Convert a slice of strings into a stream strs := rill.FromSlice([]string{"1", "2", "3", "4", "5", "not a number 6", "7", "8", "9", "10"}, nil) @@ -436,7 +440,6 @@ func ExampleFilter() { printStream(primes) } -// The same example as for the [Filter], but using ordered versions of functions. func ExampleOrderedFilter() { // Convert a slice of numbers into a stream numbers := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) @@ -467,7 +470,6 @@ func ExampleFilterMap() { printStream(squares) } -// The same example as for the [FilterMap], but using ordered versions of functions. func ExampleOrderedFilterMap() { // Convert a slice of numbers into a stream numbers := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) @@ -521,7 +523,6 @@ func ExampleFlatMap() { printStream(result) } -// The same example as for the [FlatMap], but using ordered versions of functions. func ExampleOrderedFlatMap() { // Convert a slice of numbers into a stream numbers := rill.FromSlice([]int{1, 2, 3, 4, 5}, nil) @@ -557,31 +558,6 @@ func ExampleForEach() { fmt.Println("Error:", err) } -// There is no ordered version of the ForEach function. To achieve ordered processing, use concurrency set to 1. -// If you need a concurrent and ordered ForEach, then do all processing with the [OrderedMap], -// and then use ForEach with concurrency set to 1 at the final stage. -func ExampleForEach_ordered() { - // Convert a slice of numbers into a stream - numbers := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) - - // Square each number - // Concurrency = 3; Ordered - squares := rill.OrderedMap(numbers, 3, func(x int) (int, error) { - return square(x), nil - }) - - // Print results. - // Concurrency = 1; Ordered - err := rill.ForEach(squares, 1, func(y int) error { - fmt.Println(y) - return nil - }) - - // Handle errors - fmt.Println("Error:", err) -} - -// Generate a stream of URLs from https://example.com/file-0.txt to https://example.com/file-9.txt func ExampleGenerate() { urls := rill.Generate(func(send func(string), sendError func(error)) { for i := range 10 { @@ -592,12 +568,11 @@ func ExampleGenerate() { printStream(urls) } -// Generate an infinite stream of natural numbers (1, 2, 3, ...). -// New numbers are sent to the stream every 500ms until the context is canceled func ExampleGenerate_context() { ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) defer cancel() + // Keep generating numbers until the context is canceled after 5s. numbers := rill.Generate(func(send func(int), sendError func(error)) { for i := 1; ctx.Err() == nil; i++ { send(i) @@ -621,7 +596,6 @@ func ExampleMap() { printStream(squares) } -// The same example as for the [Map], but using ordered versions of functions. func ExampleOrderedMap() { // Convert a slice of numbers into a stream numbers := rill.FromSlice([]int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}, nil) @@ -813,10 +787,6 @@ func ExampleToSeq2() { } } -// This example demonstrates how to wait until the pipeline has no callbacks left to run. -// [ForEach] returns as soon as the error is known, while the source and the remaining -// workers are still going. [Scope.Wait] cancels the scope's Context and blocks -// until all of them have stopped. func ExampleNewScope() { scope, ctx := rill.NewScope(context.Background()) defer scope.Cancel() // extra cancel to make sure the context doesn't leak @@ -868,8 +838,8 @@ func ExampleNewScope() { // --- Helpers --- -// helper function that checks if a number is prime -// and simulates some additional work using sleep +// isPrime checks if a number is prime +// and simulates some additional work by sleeping. func isPrime(n int) bool { simulateWork(500 * time.Millisecond) @@ -884,14 +854,13 @@ func isPrime(n int) bool { return true } -// helper function that squares the number -// and simulates some additional work using sleep +// square returns the square of x and simulates some additional work by sleeping. func square(x int) int { simulateWork(500 * time.Millisecond) return x * x } -// printStream prints all items from a stream (one per line) and an error if any. +// printStream prints all items from a stream (one per line) and an error, if any. func printStream[A any](stream <-chan rill.Try[A]) { fmt.Println("Result:") err := rill.ForEach(stream, 1, func(x A) error { From b90db08c0c95d196dbe0af3b594091c891ec5beb Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 19:06:13 +0300 Subject: [PATCH 40/49] Copy-edit internal/core and internal/list comments --- internal/core/durable_mutex.go | 10 +++++----- internal/core/loops.go | 22 +++++++++++++--------- internal/core/pool.go | 8 ++++---- internal/list/list.go | 18 +++++++++--------- 4 files changed, 31 insertions(+), 27 deletions(-) diff --git a/internal/core/durable_mutex.go b/internal/core/durable_mutex.go index e4d4c5d..177821d 100644 --- a/internal/core/durable_mutex.go +++ b/internal/core/durable_mutex.go @@ -8,16 +8,16 @@ import ( // DurableMutex is a mutual exclusion lock whose contending callers block // durably when used inside a [testing/synctest] bubble. // -// Most critical sections do plain non-blocking work. For them a stock +// Most critical sections do plain non-blocking work. For them, a stock // [sync.Mutex] is the right choice, with or without synctest. // DurableMutex exists for one specific pattern: critical sections that do -// durably blocking operations while holding the lock -for example, +// durably blocking operations while holding the lock - for example, // atomically receiving values from a channel and appending them to a slice. // -// Under synctest the stock mutex breaks this pattern: while the holder is +// Under synctest, the stock mutex breaks this pattern: while the holder is // durably blocked on a channel, other goroutines are non-durably blocked on the Lock() calls, // so the bubble can neither advance time nor report a deadlock, and the test freezes. -// With DurableMutex the test runs normally, and a genuine deadlock is caught and reported by the framework. +// With DurableMutex, the test runs normally, and a genuine deadlock is caught and reported by the framework. // // DurableMutex does not attempt to prevent starvation. Under contention, // acquisition order depends on runtime scheduling, and the same goroutine @@ -27,7 +27,7 @@ import ( // A DurableMutex used within a synctest bubble is local to that bubble: // every Lock and Unlock call on it must occur in the same bubble. // -// The zero value is ready to use, unlocking an unlocked mutex +// The zero value is ready to use. Unlocking an unlocked mutex // panics, and a DurableMutex must not be copied after first use. type DurableMutex struct { state atomic.Int32 diff --git a/internal/core/loops.go b/internal/core/loops.go index ca63e44..3cd3e8b 100644 --- a/internal/core/loops.go +++ b/internal/core/loops.go @@ -4,8 +4,8 @@ import ( "sync" ) -// Loop allows to process items from the input channel concurrently using n goroutines. -// If done channel is not nil, it will be closed after all items are processed. +// Loop processes items from the input channel concurrently using n goroutines. +// If the done channel is not nil, it will be closed after all items are processed. func Loop[A, B any](in <-chan A, done chan<- B, n int, f func(A)) { if n == 1 { go func() { @@ -38,13 +38,17 @@ func Loop[A, B any](in <-chan A, done chan<- B, n int, f func(A)) { } } -// OrderedLoop is similar to Loop, but it allows to write results to some channel in the same order as items were read from the input. -// If done channel is not nil, it will be closed after all items are processed. -// Special "canWrite" channel is passed to user's function f. Typical f function looks like this: -// - Do some processing (this part is executed concurrently). -// - Read from canWrite channel exactly once. This step is required. Otherwise, behavior is undefined. -// - Write result of the processing somewhere. This step is optional. -// This way processing is done concurrently, but results are written in order. +// OrderedLoop is similar to Loop, but it allows the caller to write results to a +// channel in the same order as the items were read from the input. +// If the done channel is not nil, it will be closed after all items are processed. +// A special "canWrite" channel is passed to the user's function f. +// A typical implementation of f performs the following steps: +// +// - Do some processing (this part is executed concurrently). +// - Read from the canWrite channel exactly once. This step is required. Otherwise, behavior is undefined. +// - Write the result of the processing somewhere. This step is optional. +// +// This way, processing is done concurrently, but results are written in order. func OrderedLoop[A, B any](in <-chan A, done chan<- B, n int, f func(a A, canWrite <-chan struct{})) { if n == 1 { canWrite := make(chan struct{}, 1) diff --git a/internal/core/pool.go b/internal/core/pool.go index 9e2308d..49831aa 100644 --- a/internal/core/pool.go +++ b/internal/core/pool.go @@ -2,10 +2,10 @@ package core import "sync" -// Pool is a pool of reusable values. It grows on demand and never shrinks, -// it's caller's responsibility to ensure that the pool stays bounded. +// Pool is a pool of reusable values. It grows on demand and never shrinks; +// it's the caller's responsibility to ensure that the pool stays bounded. // -// A Pool is safe for concurrent use, unless Unsynchronized is set. Its fields +// A Pool is safe for concurrent use unless Unsynchronized is set. Its fields // must not be changed once the pool is in use. type Pool[T any] struct { // New creates a value when the pool is empty. It is required. @@ -23,7 +23,7 @@ type Pool[T any] struct { items []T } -// Get returns a value from the pool, or a new one if the pool is empty. +// Get returns a value from the pool or a new one if the pool is empty. func (p *Pool[T]) Get() T { if !p.Unsynchronized { p.mu.Lock() diff --git a/internal/list/list.go b/internal/list/list.go index 6796adc..7d4cbeb 100644 --- a/internal/list/list.go +++ b/internal/list/list.go @@ -11,7 +11,7 @@ type Node[T any] struct { list *List[T] } -// Next returns the next node, or nil if n is the last one or detached. +// Next returns the next node or nil if n is the last one or is detached. func (n *Node[T]) Next() *Node[T] { if l := n.list; l != nil && n.next != &l.root { return n.next @@ -19,7 +19,7 @@ func (n *Node[T]) Next() *Node[T] { return nil } -// Prev returns the previous node, or nil if n is the first one or detached. +// Prev returns the previous node or nil if n is the first one or is detached. func (n *Node[T]) Prev() *Node[T] { if l := n.list; l != nil && n.prev != &l.root { return n.prev @@ -54,7 +54,7 @@ func New[T any]() *List[T] { return l } -// Front returns the first node, or nil if the list is empty. +// Front returns the first node or nil if the list is empty. func (l *List[T]) Front() *Node[T] { if n := l.root.next; n != &l.root { return n @@ -62,7 +62,7 @@ func (l *List[T]) Front() *Node[T] { return nil } -// Back returns the last node, or nil if the list is empty. +// Back returns the last node or nil if the list is empty. func (l *List[T]) Back() *Node[T] { if n := l.root.prev; n != &l.root { return n @@ -77,21 +77,21 @@ func (l *List[T]) PushBack(n *Node[T]) { l.insertAfter(n, l.root.prev) } -// PushFront inserts n at the front of the list. Same contract for n as -// [List.PushBack]. +// PushFront inserts n at the front of the list. It has the same contract +// for n as [List.PushBack]. func (l *List[T]) PushFront(n *Node[T]) { l.insertAfter(n, &l.root) } -// InsertAfter inserts n after mark, which must be a node of l. Same contract -// for n as [List.PushBack]. +// InsertAfter inserts n after mark, which must be a node of l. It has the same +// contract for n as [List.PushBack]. func (l *List[T]) InsertAfter(n, mark *Node[T]) { if mark.list == l { l.insertAfter(n, mark) } } -// InsertBefore inserts n before mark. Same contract as [List.InsertAfter]. +// InsertBefore inserts n before mark. It has the same contract as [List.InsertAfter]. func (l *List[T]) InsertBefore(n, mark *Node[T]) { if mark.list == l { l.insertAfter(n, mark.prev) From c59ef2a449f57d34b1c391c4b2876a103ca50113 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 19:16:14 +0300 Subject: [PATCH 41/49] Copy-edit test and test-helper comments --- benchmark_test.go | 12 ++++++------ helpers_test.go | 4 ++-- internal/th/assertions.go | 6 +++--- internal/th/helpers.go | 12 ++++++------ merge_test.go | 2 +- 5 files changed, 18 insertions(+), 18 deletions(-) diff --git a/benchmark_test.go b/benchmark_test.go index fe56edc..20e52d0 100644 --- a/benchmark_test.go +++ b/benchmark_test.go @@ -10,9 +10,9 @@ import ( ) // The stock B/op and allocs/op columns are the per-item metrics reported by -// benchmarkThroughput, integer divided, so anything under one allocation per -// item reads as zero. We use custom floating point metrics and suppress the -// stock ones - even when -benchmem is passed explicitly. +// benchmarkThroughput, calculated using integer division, so anything under +// one allocation per item reads as zero. We use custom floating-point metrics +// and suppress the stock ones - even when -benchmem is passed explicitly. // This can only be done by mutating the flag. func TestMain(m *testing.M) { flag.Parse() @@ -54,7 +54,7 @@ func benchmarkThroughput(b *testing.B, definePipeline func(in <-chan Try[int])) } // benchmarkThroughputForLevels runs benchmarkThroughput once per concurrency -// level, as a subtest named name/n. +// level as a subtest named name/n. func benchmarkThroughputForLevels(b *testing.B, name string, levels []int, definePipeline func(in <-chan Try[int], n int)) { for _, n := range levels { b.Run(fmt.Sprintf("%s/%d", name, n), func(b *testing.B) { @@ -65,8 +65,8 @@ func benchmarkThroughputForLevels(b *testing.B, name string, levels []int, defin } } -// This benchmark acts as baseline. A single drainer is -// as simple as pipeline can get. +// This benchmark acts as a baseline. A single drainer is +// as simple as a pipeline can get. func BenchmarkDrain(b *testing.B) { benchmarkThroughput(b, func(in <-chan Try[int]) { Drain(in) diff --git a/helpers_test.go b/helpers_test.go index 5817a66..f8d685c 100644 --- a/helpers_test.go +++ b/helpers_test.go @@ -5,7 +5,7 @@ import ( ) // Item[A] is a comparable variation of Try[A] that's used for testing. -// Stores errors as messages +// It stores errors as strings. type Item[A any] struct { Value A Error string @@ -48,7 +48,7 @@ func toItemSlice[A any](in <-chan Try[A]) []Item[A] { return s } -// Converts a stream in a slice of values and a slice of error messages. +// Converts a stream into a slice of values and a slice of error messages. func toSliceAndErrors[A any](in <-chan Try[A]) ([]A, []string) { var values []A var errors []string diff --git a/internal/th/assertions.go b/internal/th/assertions.go index 428660c..eef7a61 100644 --- a/internal/th/assertions.go +++ b/internal/th/assertions.go @@ -175,10 +175,10 @@ func ExpectLeak(t *testing.T, f func(t *testing.T)) { } } -// outcomes: +// Outcomes: // 0 - nothing blocked -// 1 - some goroutines blocked, but not main -// 2 - main blocked and possibly some goroutines also blocked +// 1 - some goroutines blocked, but not the main goroutine +// 2 - the main goroutine blocked, possibly along with other goroutines func checkBlock(t *testing.T, f func(t *testing.T)) (outcome int) { mainBlocked := true diff --git a/internal/th/helpers.go b/internal/th/helpers.go index 30d3403..bb93db0 100644 --- a/internal/th/helpers.go +++ b/internal/th/helpers.go @@ -47,14 +47,14 @@ func DontClose[A any](in <-chan A) <-chan A { } // ExpectNoRace is a semantic name for a bare unsynchronized read. -// Tests sometimes need to do an unsynchronized access to a variable, to +// Tests sometimes need to perform an unsynchronized read of a variable to // have the race detector confirm that all writes in other goroutines // happen before this read. It's enough to do a no-op read: // // _ = myVariable // -// This works, but requires an explaining comment at every site. -// ExpectNoRace is also a no-op, but makes the call site clearer: +// This works but requires an explanatory comment at every site. +// ExpectNoRace is also a no-op but makes the call site clearer: // // th.ExpectNoRace(myVariable) // @@ -65,8 +65,8 @@ func ExpectNoRace[T any](value T) { } // DelayEach forwards items, sleeping for the given duration before each one. -// Under synctest this makes it impossible to consume the channel in zero fake -// time, hence one goroutine (main) can observe the intermediate state of another +// Under synctest, this makes it impossible to consume the channel in zero fake +// time, so one goroutine (main) can observe the intermediate state of another // goroutine (drain) consuming the stream. This function is usually // paired with [ExpectOpenChan]. // @@ -139,7 +139,7 @@ func TestLevels(t *testing.T, levels []int, f func(t *testing.T, n int)) { } // RunSynctest runs a subtest in a synctest bubble. -// It panics if any unless all goroutines started from f exit cleanly. +// It panics unless all goroutines started from f exit cleanly. func RunSynctest(t *testing.T, name string, f func(t *testing.T)) { t.Run(name, func(t *testing.T) { synctest.Test(t, f) diff --git a/merge_test.go b/merge_test.go index c38112c..26eee7e 100644 --- a/merge_test.go +++ b/merge_test.go @@ -9,7 +9,7 @@ import ( "github.com/destel/rill/internal/th" ) -// Full behavior of Merge is tested in the internal/core package. +// The full behavior of Merge is tested in the internal/core package. // This test only pins the wrapper wiring. func TestMerge(t *testing.T) { synctest.Test(t, func(t *testing.T) { From 32380a87d0b36d4fc4942c5c16086b160f0c1386 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 19:42:05 +0300 Subject: [PATCH 42/49] Scope Batch's latency claim to the no-backpressure case --- batch.go | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/batch.go b/batch.go index b95491d..e201c6c 100644 --- a/batch.go +++ b/batch.go @@ -13,7 +13,7 @@ import ( // its first value. When it expires, the pending batch is emitted even // if it is not full. This trades batch size for latency: batches can be // smaller when the input is sparse, but no value is ever held longer -// than timeout. +// than timeout, assuming there's no backpressure. // // A zero timeout panics: the expected behavior would be to accumulate // until reading from the input blocks, but in practice, with an From 8ddd24927170609a6f5a477df44db816eac778ef Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 22:53:02 +0300 Subject: [PATCH 43/49] Restructure Batch doc; say how ToSeq2 reports settlement --- batch.go | 33 ++++++++++++++------------------- iter.go | 3 ++- 2 files changed, 16 insertions(+), 20 deletions(-) diff --git a/batch.go b/batch.go index e201c6c..efc25d6 100644 --- a/batch.go +++ b/batch.go @@ -4,28 +4,23 @@ import ( "time" ) -// Batch groups consecutive values of the stream into batches. In its -// simplest form, with timeout = -1 and no errors in the input, Batch -// accumulates values into a pending batch and emits it as soon as it -// reaches the target size. +// Batch groups consecutive values of the stream into batches. With +// timeout = -1, it accumulates values until the batch reaches size, +// then emits it. When the input closes, any remaining values are +// emitted as a final batch. // -// A positive timeout is the time each batch has to fill, starting from -// its first value. When it expires, the pending batch is emitted even -// if it is not full. This trades batch size for latency: batches can be -// smaller when the input is sparse, but no value is ever held longer -// than timeout, assuming there's no backpressure. +// Input errors create batch boundaries: any pending batch is emitted +// first, followed by the error as a separate item. This function never +// emits empty batches. // -// A zero timeout panics: the expected behavior would be to accumulate -// until reading from the input blocks, but in practice, with an -// unbuffered input, that often produces a flood of one-item batches. -// Use a small positive timeout instead. +// A positive timeout adds another trigger: each batch has that much +// time to fill, starting from its first value. When the timeout expires, +// the pending batch is emitted even if it is not full. This trades batch +// size for latency: sparse input produces smaller batches, but no value +// is ever held longer than timeout. Backpressure can still delay delivery +// beyond the timeout. // -// Input errors become batch boundaries: the pending batch, if not -// empty, is emitted first, and the error follows as a separate item. -// -// When the end of the input is reached, whatever has accumulated is -// emitted as a final batch. This function never emits empty batches, -// regardless of what triggered the emission. +// A zero timeout panics. Use a small positive timeout instead. func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[[]A] { validateMinSize(size, 1) if timeout == 0 { diff --git a/iter.go b/iter.go index 74b85f6..0fcb21f 100644 --- a/iter.go +++ b/iter.go @@ -59,7 +59,8 @@ func FromSeq2[A any](seq iter.Seq2[A, error]) <-chan Try[A] { // // The returned iterator is single-use and must be ranged over for the // pipeline to settle. If the loop exits early with break or return, -// ToSeq2 drains the input in the background before reporting settlement. +// ToSeq2 drains the input in the background before reporting settlement +// via a [Scope]. func ToSeq2[A any](in <-chan Try[A], options ...SinkOption) iter.Seq2[A, error] { // Unlike other sinks, ToSeq2 opens the options at the call site: its work // happens while the iterator is ranged, which can be arbitrarily far from From ecdf958f6a86dae0b85b3c5dcb9cb338a0cd27d0 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 23:32:01 +0300 Subject: [PATCH 44/49] Scope Batch's latency claim in one sentence --- batch.go | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/batch.go b/batch.go index efc25d6..75fc35b 100644 --- a/batch.go +++ b/batch.go @@ -17,8 +17,7 @@ import ( // time to fill, starting from its first value. When the timeout expires, // the pending batch is emitted even if it is not full. This trades batch // size for latency: sparse input produces smaller batches, but no value -// is ever held longer than timeout. Backpressure can still delay delivery -// beyond the timeout. +// is ever held longer than timeout, assuming there's no backpressure. // // A zero timeout panics. Use a small positive timeout instead. func Batch[A any](in <-chan Try[A], size int, timeout time.Duration) <-chan Try[[]A] { From 96b11d38400fcecdc16bb8130e77739fa0966921 Mon Sep 17 00:00:00 2001 From: destel Date: Thu, 10 Sep 2026 23:32:03 +0300 Subject: [PATCH 45/49] Say "closed output", not "closure", in the package doc --- doc.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/doc.go b/doc.go index fcb8d3a..9913304 100644 --- a/doc.go +++ b/doc.go @@ -37,9 +37,9 @@ // Intermediate stages never block: they return their output streams // immediately, while the goroutines they started continue working in // the background. These stages always fully consume and process their -// inputs before closing their outputs. This closure becomes an "all -// upstream work is done" signal that travels downstream along with values -// and errors. +// inputs before closing their outputs. A closed output becomes an +// "all upstream work is done" signal that travels downstream along with +// values and errors. // // Sinks are different: they block until the pipeline's outcome is known, which // can happen before the input is fully consumed and all work across the pipeline is done. From 5a06019d7b5a2dbb919fa48dc02545d3c4dec7c8 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 11 Sep 2026 12:22:57 +0300 Subject: [PATCH 46/49] Fold the consume-concurrently rule into the Tee and ToChans contracts --- merge.go | 5 ++--- wrap.go | 5 ++--- 2 files changed, 4 insertions(+), 6 deletions(-) diff --git a/merge.go b/merge.go index dd91e1b..4684faa 100644 --- a/merge.go +++ b/merge.go @@ -108,9 +108,8 @@ func OrderedSplit2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (out // Tee duplicates the input: it returns two channels that both carry every // item from the input, forwarded as it arrives. Both outputs are closed -// once the input is exhausted. -// -// The outputs must be consumed concurrently to avoid a deadlock. +// once the input is exhausted. They must be consumed concurrently to avoid +// a deadlock. // // If deep copying of values is needed, use [Map] on one or both // outputs: diff --git a/wrap.go b/wrap.go index 03fe67d..e5e0714 100644 --- a/wrap.go +++ b/wrap.go @@ -181,9 +181,8 @@ func FromChans[A any](values <-chan A, errs <-chan error) <-chan Try[A] { // ToChans splits the stream into two channels, one for values and one for // errors, and forwards each item to the appropriate one. Both channels are -// closed once the input is exhausted. -// -// The channels must be consumed concurrently to avoid a deadlock. +// closed once the input is exhausted. They must be consumed concurrently +// to avoid a deadlock. func ToChans[A any](in <-chan Try[A]) (<-chan A, <-chan error) { if in == nil { return nil, nil From 9abab36365ec82e878c7a77ab8da2caa5b9e6fe9 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 11 Sep 2026 14:55:12 +0300 Subject: [PATCH 47/49] Apply doc review: Merge opener, Batch wording, Extending rules --- batch.go | 2 +- doc.go | 25 ++++++++++++------------- merge.go | 13 +++++++------ 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/batch.go b/batch.go index 75fc35b..7ed6152 100644 --- a/batch.go +++ b/batch.go @@ -6,7 +6,7 @@ import ( // Batch groups consecutive values of the stream into batches. With // timeout = -1, it accumulates values until the batch reaches size, -// then emits it. When the input closes, any remaining values are +// then emits it. When the input is exhausted, any pending values are // emitted as a final batch. // // Input errors create batch boundaries: any pending batch is emitted diff --git a/doc.go b/doc.go index 9913304..c6b193f 100644 --- a/doc.go +++ b/doc.go @@ -34,12 +34,11 @@ // transformed := rill.Map(batches, 3, ...) // stage, concurrency = 3 // err := rill.ForEach(transformed, 2, ...) // sink, concurrency = 2 // -// Intermediate stages never block: they return their output streams -// immediately, while the goroutines they started continue working in -// the background. These stages always fully consume and process their -// inputs before closing their outputs. A closed output becomes an -// "all upstream work is done" signal that travels downstream along with -// values and errors. +// Intermediate stages return their output streams immediately, while their +// goroutines continue working in the background. These stages always fully +// consume and process their inputs before closing their outputs. A closed +// output becomes an "all upstream work is done" signal that travels +// downstream along with values and errors. // // Sinks are different: they block until the pipeline's outcome is known, which // can happen before the input is fully consumed and all work across the pipeline is done. @@ -141,14 +140,14 @@ // errors into a slice. // // The easiest way to write a custom stage is to compose it from existing -// functions rill provides. For manually written stages, there are a few -// simple rules to follow. Most of them are satisfied by construction -// and are related to preserving background drain and settlement semantics: +// functions rill provides. For manually written stages, a few simple rules +// keep background draining and settlement working. Ordinary Go channel +// code usually satisfies most of them: // // - sources must eventually close their output stream; a source that can // run forever must watch a context and be cancellable -// - stages must close their output stream, but only after the input is fully -// consumed and processed -// - sinks must start with a deferred rill.Discard(in, options...), followed by a -// for-range loop that returns as soon as the sink's outcome is known +// - intermediate stages must close their output stream, but only after the +// input is fully consumed and processed +// - non-concurrent sinks must start with a deferred rill.Discard(in, options...), +// followed by a for-range loop that returns as soon as the sink's outcome is known package rill diff --git a/merge.go b/merge.go index 4684faa..7ddaeaf 100644 --- a/merge.go +++ b/merge.go @@ -4,10 +4,11 @@ import ( "github.com/destel/rill/internal/core" ) -// Merge performs a fan-in: it returns a channel that carries the items from -// all inputs, interleaved as they arrive, with the relative order of items -// from the same input preserved. Merge consumes its inputs simultaneously -// and independently. +// Merge performs a fan-in: it combines multiple input channels into +// one output channel. +// It reads all inputs concurrently, so a slow input does not delay +// forwarding items from the others. Items are interleaved as they arrive, +// preserving the relative order of items from each input. // // The output is closed only when all inputs are exhausted. A nil input is // never exhausted, so the output never closes. Merge with no arguments @@ -28,7 +29,7 @@ func Merge[A any](ins ...<-chan A) <-chan A { // preserve the order. // // Deprecated: Split2 will be removed in v1.0. Since the introduction of [Tee] -// in v0.8, splitting no longer needs a dedicated operation — it can be composed +// in v0.8, splitting no longer needs a dedicated operation - it can be composed // from existing ones. Unlike Split2, the composition is also not limited to two // branches. // @@ -69,7 +70,7 @@ func Split2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (outTrue <- // preserve the input order for values and errors alike. // // Deprecated: OrderedSplit2 will be removed in v1.0. Since the introduction of -// [Tee] in v0.8, splitting no longer needs a dedicated operation — it can be +// [Tee] in v0.8, splitting no longer needs a dedicated operation - it can be // composed from existing ones. Unlike OrderedSplit2, the composition is also // not limited to two branches. // From d98ae66cf73fb98d694e602c591cee5d8b088058 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 11 Sep 2026 15:06:55 +0300 Subject: [PATCH 48/49] Wrap long doc comment lines in the package doc and merge.go --- doc.go | 72 ++++++++++++++++++++++++++++++-------------------------- merge.go | 35 ++++++++++++++------------- scope.go | 1 - 3 files changed, 57 insertions(+), 51 deletions(-) diff --git a/doc.go b/doc.go index c6b193f..abf0db0 100644 --- a/doc.go +++ b/doc.go @@ -14,19 +14,20 @@ // values. In rill's terms, the post's definition of a pipeline becomes: // // A pipeline is a series of stages connected by streams - channels whose items -// are [Try] structs, each holding either a value or an error. Under the hood, each stage -// runs one or more goroutines that: +// are [Try] structs, each holding either a value or an error. Under the hood, +// each stage runs one or more goroutines that: // // - receive values and errors from upstream via input streams // - process the received values, usually producing new values or errors // - send the results downstream via output streams -// - forward upstream errors to the output streams ([Catch] is the only exception) +// - forward upstream errors to the output streams ([Catch] is the only +// exception) // -// Usually, most stages in a pipeline have one input stream and one output stream. -// The exceptions are the first stage, which has no input stream, and the last -// stage, which has no output stream. These stages are called the source and -// the sink, respectively. The [Merge] and [Tee] functions have more inputs/outputs -// and can be used to build DAG pipelines. +// Usually, most stages in a pipeline have one input stream and one output +// stream. The exceptions are the first stage, which has no input stream, and +// the last stage, which has no output stream. These stages are called the +// source and the sink, respectively. The [Merge] and [Tee] functions have more +// inputs/outputs and can be used to build DAG pipelines. // // ids := rill.FromSlice(userIDs, nil) // source // filtered := rill.Filter(ids, 5, ...) // stage, concurrency = 5 @@ -41,10 +42,12 @@ // downstream along with values and errors. // // Sinks are different: they block until the pipeline's outcome is known, which -// can happen before the input is fully consumed and all work across the pipeline is done. -// What "outcome known" means depends on the sink. For example: +// can happen before the input is fully consumed and all work across the +// pipeline is done. What "outcome known" means depends on the sink. For +// example: // -// - [ForEach] immediately returns the first error it observes; otherwise, it fully consumes the input +// - [ForEach] immediately returns the first error it observes; otherwise, it +// fully consumes the input // - [Any] can additionally short-circuit on the first match it finds // - [First] consumes one item and returns // @@ -71,9 +74,10 @@ // // # Structured concurrency // -// When the caller wants not only to request cancellation but also to wait -// for the pipeline to settle (no work remains and every user callback has -// returned), rill provides the [Scope] API, which is like errgroup for pipelines. +// When the caller wants not only to request cancellation but also to wait for +// the pipeline to settle (no work remains and every user callback has +// returned), rill provides the [Scope] API, which is like errgroup for +// pipelines. // // scope, ctx := rill.NewScope(ctx) // defer scope.Cancel() @@ -99,23 +103,24 @@ // // # Ordered stages // -// By default, stages write results to their output streams as soon as they -// are ready, in completion order. In concurrent stages, that order depends on how the Go runtime -// schedules the stages' goroutines and on the time it takes to produce each result. +// By default, stages write results to their output streams as soon as they are +// ready, in completion order. In concurrent stages, that order depends on how +// the Go runtime schedules the stages' goroutines and on the time it takes to +// produce each result. // -// For cases where the input order must be preserved, rill provides ordered functions, -// such as [OrderedMap] or [OrderedFilter]. They stay concurrent, -// but each worker holds its result until all earlier results are sent, -// so the output order matches the input order at the cost -// of some latency. This ordering guarantee holds for both values and errors. +// For cases where the input order must be preserved, rill provides ordered +// functions, such as [OrderedMap] or [OrderedFilter]. They stay concurrent, but +// each worker holds its result until all earlier results are sent, so the +// output order matches the input order at the cost of some latency. This +// ordering guarantee holds for both values and errors. // // # Backpressure // -// Backpressure means that sending to an unbuffered channel blocks until -// the receiver on the other end is ready to receive. Rill naturally -// inherits this property: a slow stage in the -// pipeline blocks the previous stage, and it in turn blocks the stage before that, -// and so on, until the slow stage catches up. +// Backpressure means that sending to an unbuffered channel blocks until the +// receiver on the other end is ready to receive. Rill naturally inherits this +// property: a slow stage in the pipeline blocks the previous stage, and it in +// turn blocks the stage before that, and so on, until the slow stage +// catches up. // // When this is not desirable, use [Buffer] to add slack between stages. // @@ -127,10 +132,10 @@ // // # Panics // -// Rill validates the arguments to its functions and panics on misuse, such as zero or negative concurrency. -// Rill does not automatically recover from panics in user callbacks: a panicking -// callback can crash the process, as it would in any hand-written concurrent -// code. +// Rill validates the arguments to its functions and panics on misuse, such as +// zero or negative concurrency. Rill does not automatically recover from panics +// in user callbacks: a panicking callback can crash the process, as it would in +// any hand-written concurrent code. // // # Extending rill // @@ -148,6 +153,7 @@ // run forever must watch a context and be cancellable // - intermediate stages must close their output stream, but only after the // input is fully consumed and processed -// - non-concurrent sinks must start with a deferred rill.Discard(in, options...), -// followed by a for-range loop that returns as soon as the sink's outcome is known +// - non-concurrent sinks must start with a deferred +// rill.Discard(in, options...), followed by a for-range loop that returns +// as soon as the sink's outcome is known package rill diff --git a/merge.go b/merge.go index 7ddaeaf..84cbd39 100644 --- a/merge.go +++ b/merge.go @@ -4,9 +4,8 @@ import ( "github.com/destel/rill/internal/core" ) -// Merge performs a fan-in: it combines multiple input channels into -// one output channel. -// It reads all inputs concurrently, so a slow input does not delay +// Merge performs a fan-in: it combines multiple input channels into one output +// channel. It reads all inputs concurrently, so a slow input does not delay // forwarding items from the others. Items are interleaved as they arrive, // preserving the relative order of items from each input. // @@ -33,18 +32,19 @@ func Merge[A any](ins ...<-chan A) <-chan A { // from existing ones. Unlike Split2, the composition is also not limited to two // branches. // -// Quite often, the predicate is a simple, pure check (a field comparison, a type -// switch, etc.). In such cases, splitting is just [Tee] plus a [Filter] on each -// branch: +// Quite often, the predicate is a simple, pure check (a field comparison, a +// type switch, etc.). In such cases, splitting is just [Tee] plus a [Filter] on +// each branch: // // adults, minors := rill.Tee(users) // adults = rill.Filter(adults, 1, func(u User) (bool, error) { return u.Age >= 18, nil }) // minors = rill.Filter(minors, 1, func(u User) (bool, error) { return u.Age < 18, nil }) // -// If the predicate is expensive or stateful, or if it can fail, it must be evaluated -// once per item, before [Tee]: inline this function's implementation, which -// tags each item with the decision and routes on the tag. The same pattern -// extends to n-way splitting by tagging with an index or key instead of a bool. +// If the predicate is expensive or stateful, or if it can fail, it must be +// evaluated once per item, before [Tee]: inline this function's implementation, +// which tags each item with the decision and routes on the tag. The same +// pattern extends to n-way splitting by tagging with an index or key instead of +// a bool. func Split2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (outTrue <-chan Try[A], outFalse <-chan Try[A]) { validateN(n) validateNilFunc(f == nil) @@ -74,18 +74,19 @@ func Split2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (outTrue <- // composed from existing ones. Unlike OrderedSplit2, the composition is also // not limited to two branches. // -// Quite often, the predicate is a simple, pure check (a field comparison, a type -// switch, etc.). In such cases, splitting is just [Tee] plus an [OrderedFilter] -// on each branch: +// Quite often, the predicate is a simple, pure check (a field comparison, a +// type switch, etc.). In such cases, splitting is just [Tee] plus an +// [OrderedFilter] on each branch: // // adults, minors := rill.Tee(users) // adults = rill.OrderedFilter(adults, 1, func(u User) (bool, error) { return u.Age >= 18, nil }) // minors = rill.OrderedFilter(minors, 1, func(u User) (bool, error) { return u.Age < 18, nil }) // -// If the predicate is expensive or stateful, or if it can fail, it must be evaluated -// once per item, before [Tee]: inline this function's implementation, which -// tags each item with the decision and routes on the tag. The same pattern -// extends to n-way splitting by tagging with an index or key instead of a bool. +// If the predicate is expensive or stateful, or if it can fail, it must be +// evaluated once per item, before [Tee]: inline this function's implementation, +// which tags each item with the decision and routes on the tag. The same +// pattern extends to n-way splitting by tagging with an index or key instead of +// a bool. func OrderedSplit2[A any](in <-chan Try[A], n int, f func(A) (bool, error)) (outTrue <-chan Try[A], outFalse <-chan Try[A]) { validateN(n) validateNilFunc(f == nil) diff --git a/scope.go b/scope.go index 89231fc..70a6ced 100644 --- a/scope.go +++ b/scope.go @@ -8,7 +8,6 @@ import ( // A Scope tracks the lifecycle of a pipeline and lets the caller wait until // all of its work is done - including any work that happens after a sink's // early return. -// // A scope passed to a sink as a [SinkOption] tracks not only that sink but // also the whole pipeline behind it. // From 1dfda250baaaf83ef8c28ae4cb388c59a0d0b958 Mon Sep 17 00:00:00 2001 From: destel Date: Fri, 11 Sep 2026 15:24:38 +0300 Subject: [PATCH 49/49] Tighten the sinks paragraph in the package doc --- doc.go | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/doc.go b/doc.go index abf0db0..6befffe 100644 --- a/doc.go +++ b/doc.go @@ -41,17 +41,16 @@ // output becomes an "all upstream work is done" signal that travels // downstream along with values and errors. // -// Sinks are different: they block until the pipeline's outcome is known, which -// can happen before the input is fully consumed and all work across the -// pipeline is done. What "outcome known" means depends on the sink. For -// example: +// Sinks are different: they block until their outcome is known, then return +// even if work remains in the pipeline. What "outcome known" means depends on +// the sink. For example: // // - [ForEach] immediately returns the first error it observes; otherwise, it // fully consumes the input // - [Any] can additionally short-circuit on the first match it finds // - [First] consumes one item and returns // -// On an early return, a sink drains and discards the remaining input in the +// On an early return, a sink drains and discards any remaining input in the // background, so upstream stages don't block forever and leak their goroutines. // // # Context and cancellation