Sink Merge operations use Lua to describe the read-modify-write rule for one
record. A business service sends an incoming document and a Lua program. Sink
reads the current document, runs the program, and uses the storage revision to
apply the result atomically.
Every program must return one function that accepts exactly two arguments:
return function(current, incoming)
current = current or json.object()
current.stock = incoming.stock
return current
endcurrentis the stored object. It isnilwhen the record does not exist; every Merge can create the record returned by the Lua function.incomingis the object carried by this Merge and is always present.currentandincomingmust have the same document encoding.- The function must return an object. It cannot return
nil, an array, or a scalar. - The result keeps the incoming encoding: BSON for MongoDB and JSON for search.
- A storage revision conflict makes Sink read the latest
currentdocument and run the same function again, so the script must be deterministic.
Use sink.v1.time.now() when a script needs the execution time.
The following script updates non-empty fields, merges tags, retains the latest 20 history entries, and records the Merge time:
local array = sink.v1.array
local object = sink.v1.object
return function(current, incoming)
current = current or json.object()
object.replace_nonempty_string(current, incoming, "title")
object.replace_nonempty_array(current, incoming, "images")
current.tags = array.union_strings(current.tags, incoming.tags)
local history = current.history or json.array()
array.append_all(history, incoming.history)
history = array.deduplicate(history, function(item)
return item.id
end)
current.history = array.keep_tail(history, 20)
current.updated_at = sink.v1.time.now()
return current
endCreate local aliases for frequently used submodules at the top of the program. This reduces both source size and repeated global table lookups.
sink.v1 is Sink's versioned, deterministic utility API. Missing or extra
arguments, wrong types, sparse arrays, invalid callback results, and invalid
limits fail the current Merge explicitly instead of silently changing data.
Every array argument must be a JSON array: a value from current or incoming,
or an array created with json.array(). Do not use an empty {} as an array.
targetis a required JSON array.sourceis a JSON array ornil.- Appends every
sourceitem totargetin its original order. - Mutates and returns the same
target; it does not mutatesource. - Does nothing when
sourceisnil.
itemsis a required JSON array.key_function(item)is required and must return exactly one string, number, or boolean.- Keeps the first item for each key in the original order and returns a new JSON array.
- Does not mutate
items. A callback error fails the Merge. - A key cannot be
nil, a table, or a function. Return a stable string when an object needs a composite key.
local unique = sink.v1.array.deduplicate(incoming.offers, function(item)
return (item.platform or "") .. ":" .. (item.id or "")
end)itemsis a required JSON array.limitis an integer greater than or equal to0.- Returns a new JSON array containing only the last
limititems. - A zero limit returns an empty array. The function does not mutate
items.
leftandrightare string JSON arrays ornil.- Combines and deduplicates them in left-then-right order and returns a new JSON array.
- Does not mutate either input. A non-string item in either array fails the Merge.
The target and source parameters must be JSON objects. The functions are
convenient when the same rule applies to a list of fields:
local fields = {"title", "description", "country"}
for index = 1, #fields do
sink.v1.object.replace_nonempty_string(current, incoming, fields[index])
endfieldmust be a string.- A missing or empty
source[field]leavestargetunchanged. - A non-empty string is assigned to
target[field]. - Any other source type fails the Merge instead of writing an invalid value.
- The function has no return value.
fieldmust be a string.- A missing or empty
source[field]leavestargetunchanged. - A non-empty JSON array is assigned to
target[field]. - Any other type or a sparse array fails the Merge.
- The function has no return value.
- Accepts no arguments.
- Returns a UTC RFC3339Nano string such as
2026-08-30T09:08:07.654321Z. - Returns one fixed observation time for the Merge. Multiple calls in the same script and reruns caused by storage revision conflicts return the same value.
- A synchronous Merge uses the time at which the server starts processing the operation. An asynchronous Merge uses the time at which the Worker starts processing the Kafka record, not the client submission time.
- In a BSON merge result the value is encoded as a native BSON datetime. In a JSON merge result it remains an RFC3339Nano string.
Sink does not expose os.time, the host clock, or mutable time zones because
they could produce different results during a retry.
JSON values and the JSON-compatible view of BSON values use this mapping:
| JSON | Lua |
|---|---|
| object | table |
| array | table with a JSON array marker |
| string | string |
| integer | 64-bit Lua integer |
| decimal | number |
| boolean | boolean |
| null | json.null |
JSON document bytes and Lua result strings and object keys must be valid UTF-8. Invalid bytes are rejected instead of being replaced with U+FFFD during JSON conversion, which can otherwise change values or collapse distinct object keys. Binary data must use an explicit encoding or a BSON binary value.
JSON integers must fit the signed 64-bit range, including integral values written with a decimal point or exponent. Sink rejects an out-of-range integer instead of rounding it through a floating-point conversion. Fractional JSON numbers use Lua floating-point numbers; values that underflow to zero are rejected.
BSON doubles remain doubles, including integral doubles such as 1.0. Integer
fields in an existing input table retain their BSON width through integer
updates; an int32 field promotes to int64 if its new value no longer fits.
A copied integer in a new field retains its input width when that value has one
unambiguous BSON width across the inputs. If equal int32 and int64 values make
that width ambiguous, Sink rejects the result; retain the original table/field
when exact integer widths matter. New integer values use int32 when they fit,
and int64 otherwise. Lua itself has one integer type and cannot distinguish the
BSON width of equal scalar values.
BSON datetimes retain their type when directly copied or moved, including into new tables and arrays. An equal ordinary string remains a string, even when it replaces a datetime at the same field. String transformations produce ordinary strings unless they return the original value unchanged.
BSON-only scalar types such as ObjectID, Binary, Timestamp, MinKey, and MaxKey
retain their native type when their Extended JSON view is passed through or
moved to another field. Sink tracks that type separately from the view's field
names. Editing a typed view must still produce the same valid BSON scalar type.
Ordinary BSON documents and newly constructed Lua objects remain documents even
when they contain keys such as $numberInt, $date, or $oid; these keys do not
create a BSON scalar. BSON inputs with duplicate document fields or more than
256 nested value levels are rejected before conversion.
The common JSON helpers are:
json.object()creates an explicitly typed empty JSON object.json.array()creates an explicitly typed empty JSON array.json.nullrepresents JSON null. Assigning Luanilremoves a table field.json.is_null(value)reports whether a value isjson.null.
An empty Lua table {} is encoded as an object. Use json.array() whenever an
empty array is required.
Sink provides deterministic base, string, table, math, and UTF-8 functions.
utf8.upper(value) performs Unicode-aware uppercasing; string.upper is only
suitable for ASCII text.
Scripts cannot use host file or network I/O, operating-system APIs,
require/package, dynamic code loading, coroutines, debug APIs, random numbers,
output, metatable mutation, or unbounded string repetition. A script cannot
access Sink configuration, environment variables, credentials, or another
tenant's data.
- Concurrent Merges for the same record use revision preconditions. A conflict reruns the script with the newest document.
- Kafka provides at-least-once delivery. If a Worker applies a write and exits before committing the offset, it may execute the same Merge again.
- Deduplication keys, append rules, and counters must account for replaying the
same input.
current.count = current.count + 1is not inherently idempotent. - Do not depend on random values, external state, or a time value that changes on every call.
sink.v1.time.now()is stable only within one Merge operation and its storage revision retries. Consuming a Kafka record again is a new execution, so the business rule must still provide its required idempotency.
Business idempotence is an application requirement for all retries and replays. Sink performs no business-operation deduplication. An ambiguous write acknowledgement may be followed by another execution of the same Lua, even in the same worker. Use business identifiers and atomic document updates where repeat effects are unsafe. CAS alone does not provide this guarantee.
Each execution has limits for wall-clock time, instructions, call depth, VM
stack size, source size, and result size. The time budget includes input
decoding and result encoding. Go document converters cannot be interrupted
mid-call, but a result completed after the deadline is rejected before storage
commit. Native sink.v1 array loops also enforce the execution deadline and
work limit. Array validation and copying consume the shared instruction budget
across all sink.v1 calls, including discarded intermediate results. Script
syntax, arguments, types, callbacks, resources, and result errors fail only the
corresponding Write operation and return a structured failure.
Pattern captures from string.find, string.match, and string.gmatch reserve
their full return stack and obey the stack limit even if the caller discards them.
Chunk initialization during compilation also observes the enclosing request's deadline and cancellation. A canceled request stops validation before publishing or committing writes; it is not reported as an invalid script. Parsing and bytecode compilation check cancellation between their synchronous phases.
Before rollout, business tests should cover at least:
- Existing and missing records.
- Missing fields, empty strings, empty arrays, and
json.null. - Duplicate elements, history limits, and invalid field types.
- Replaying the same incoming document twice.
- Final state after concurrent updates cause a storage revision conflict.
- Real Sink integration tests for both synchronous and Kafka modes.
Use sink lua test for fast local and CI contract tests. It
executes this production Lua engine directly, supports JSON and BSON fixtures,
fixes the observation time, and compares type-aware results without starting a
Sink server or backend.
Keep the script with the business application's source and create one reusable
LuaProgram in the process:
source := []byte(`
return function(current, incoming)
current = current or json.object()
current.stock = incoming.stock
current.updated_at = sink.v1.time.now()
return current
end`)
program, err := sink.NewLuaProgram(source)
if err != nil {
return err
}
options := sink.MergeOptions{
Incoming: incomingDocument,
Program: program,
}
operation, err := sink.NewMerge(address, options)
if err != nil {
return err
}Create incomingDocument with sink.DocumentEncodingBSON for a MongoDB
address or sink.DocumentEncodingJSON for an Elasticsearch or OpenSearch
address. Sink rejects mixed encodings and backend/encoding mismatches.
Identical source is declared only once within one Write RPC; Merge operations refer to it by SHA-256. An asynchronous Kafka record still carries the complete business source so a Worker can process it independently. The common utilities are built into Sink and therefore are not retransmitted with every request.
Scripts must select sink.v1 explicitly. The names and semantics of v1
functions remain stable; a future incompatible contract will use a new version
namespace. Do not probe or call undocumented globals, internal fields, or a
higher API version.
Result conversion first checks the expanded table graph, including each use of
a shared table. The limit includes a maximum depth of 256 and at most
max(64, min(max_result_bytes / 8, 1000000)) visited values/keys, followed by the
encoded byte check. This prevents small alias graphs from creating exponential
Go maps, slices, or serialized output. Current/incoming payloads are also limited
by max_result_bytes.
The embedded VM still has no strict allocation-byte quota during execution.
utf8.upper bounds the converted UTF-8 byte length by max_result_bytes before
each append, including characters whose uppercase form requires more bytes.
It checks cancellation and charges an instruction checkpoint every 1024 input
characters and at completion, even for unchanged strings or discarded results.
string.pack preflights the size of each intermediate string against
max_result_bytes before allocating it. Both string.pack and string.packsize
check cancellation and charge instruction checkpoints for format bytes before
processing, including whitespace and options that produce no output.
table.concat checks each append
against the same bound without retaining a separate list of string fragments,
even if the script discards the result or returns only its length. Valid packing
formats, alignment, variable strings, and concatenation ranges retain their
ordinary behavior within that bound. table.concat, table.move, table.insert,
table.remove, table.pack and table.unpack check cancellation and charge
instruction checkpoints while traversing elements, including empty strings and
nil values. table.sort checks each comparison, including default comparisons
that execute no Lua bytecode, while retaining the runtime's sorting algorithm.
Overlapping moves preserve their copy direction. Interrupted table mutations
may leave the execution's Lua table partially updated; failed merges do not
commit that intermediate state to storage.
string.gsub checks each output append, including capture expansion and
function/table replacements, against the same limit. Patterns use string.find
capture validation, so unfinished captures are rejected. Substitution loops also
check cancellation and instruction limits. The runtime's pattern syntax is
retained by a local copy of its matcher with VM checkpoints.
string.find, string.match, string.gmatch and the matching phase of
string.gsub check cancellation and charge instruction checkpoints during
search, backtracking, balanced matches, backreferences, and character-set
parsing and matching. A checkpoint runs at entry and every 256 units of native
matching work, including searches that produce no matches. Timeouts and
instruction exhaustion therefore stop an active pattern search without waiting
for the entire native call to finish. Literal string.find searches retain the
runtime's direct substring-search path.
string.format bounds the combined output and checks string/quoted-string
expansion before each conversion, including strings used repeatedly. Formatting
loops also check cancellation and instruction limits. Numeric widths and
precision retain the runtime's existing limits.
string.unpack reserves its complete return frame, including the next-position
value, within the VM stack limit before decoding. Its format preflight also
checks cancellation and charges instruction checkpoints for format bytes,
including padding and zero-width fields.
These limits do not make arbitrary Lua safe to run in a shared trusted process.
Use reviewed business scripts, isolate stores/workers into containers with
memory limits, and test the largest permitted documents. See
reliability boundaries for deployment acceptance criteria.