Skip to content

Add RETURNING * capture and callback polling support#22

Merged
jeffreyaven merged 4 commits intomainfrom
claude/returning-callback-support-DbI9G
Feb 27, 2026
Merged

Add RETURNING * capture and callback polling support#22
jeffreyaven merged 4 commits intomainfrom
claude/returning-callback-support-DbI9G

Conversation

@jeffreyaven
Copy link
Contributor

Summary

This PR adds support for capturing RETURNING * results from DML operations (INSERT/UPDATE/DELETE) and implements callback polling for tracking long-running asynchronous operations. This is particularly useful for providers like AWS Cloud Control API (awscc) that return operation tracking handles requiring polling until completion.

Key Changes

Core Utilities (src/core/utils.rs)

  • has_returning_clause() — Detects if a query contains a RETURNING clause (case-insensitive)
  • run_stackql_dml_returning() — Executes DML commands and captures the first row of RETURNING * results as a HashMap<String, String>, with retry logic and error handling
  • flatten_returning_row() — Flattens a RETURNING row into dotted context keys:
    • callback.{field} (shorthand, scoped to current resource)
    • {resource_name}.callback.{field} (fully-qualified, accessible downstream)
  • flatten_value_into_context() and flatten_json_into_context() — Recursively expand JSON objects in RETURNING results into nested dotted keys (e.g., ProgressEvent.OperationStatus)
  • check_short_circuit() — Checks if a short-circuit condition is met in captured callback data to skip polling when operation status is already terminal
  • run_callback_poll() — Polls a callback query until success or count column returns a truthy value, with configurable retries and delays
  • Comprehensive unit tests for all new functions

Template Engine (src/template/engine.rs)

  • preprocess_this_prefix() — Preprocesses this. prefix in Tera templates, replacing it with {resource_name}. to enable resource-scoped variable references within .iql files
  • insert_nested_key() — Recursively builds nested JSON objects from dotted keys to support multi-level property access (e.g., {{ callback.ProgressEvent.RequestToken }})
  • Enhanced build_tera_context() to handle arbitrary-depth dotted keys instead of just two-level nesting
  • Unit tests validating three and four-level dotted key access

Templating Configuration (src/core/templating.rs)

  • Extended QueryOptions struct with:
    • short_circuit_field: Option<String> — Dot-path into RETURNING result to check before polling
    • short_circuit_value: Option<String> — Expected value that allows skipping polling
  • Updated parse_anchor() to distinguish between numeric and string-valued options
  • Modified load_sql_queries() to return separate maps for uint and string options
  • Updated get_queries() to populate new callback options from parsed anchors
  • Added documentation for callback anchor syntax and options

Command Runner (src/commands/base.rs)

  • create_resource() — Now returns (bool, Option<HashMap<String, String>>) to capture RETURNING data
  • update_resource() — Now returns (bool, Option<HashMap<String, String>>) to capture RETURNING data
  • delete_resource() — Now returns Option<HashMap<String, String>> to capture RETURNING data
  • store_callback_data() — Stores flattened RETURNING row in global context for downstream access
  • run_callback() — Executes callback polling with short-circuit support
  • All three methods now detect RETURNING clauses and use run_stackql_dml_returning() when present
  • Dry-run mode properly logs RETURNING intent without executing captures

Build Command (src/commands/build.rs)

  • After successful create or update, captures RETURNING data and stores it
  • Executes callback:create or callback:update blocks (or bare callback) if present
  • Passes short-circuit field/value to callback runner
  • Handles both operation-specific (callback:create) and generic (callback) anchors

Teardown Command (src/commands/teardown.rs)

https://claude.ai/code/session_01HcAKNRjjQRn7TywY28zv1Y

Add two new features to stackql-deploy-rs:

1. RETURNING Result Capture: DML statements with `RETURNING *` now
   automatically capture the first row of the provider response into
   the template context under two namespaces:
   - `callback.{field}` — shorthand, scoped to current resource
   - `{resource_name}.callback.{field}` — fully-qualified, available
     to downstream resources for the rest of the stack run

2. Callback Blocks: New `/*+ callback:create */` (`:update`, `:delete`,
   or bare `callback`) anchor in `.iql` files that polls a query after
   a DML operation to track asynchronous operation completion.
   Supports `retries`, `retry_delay`, `short_circuit_field`, and
   `short_circuit_value` hint parameters.

Key implementation details:
- Recursive `build_tera_context` / `insert_nested_key` in engine.rs
  to support arbitrary-depth dotted keys (e.g. `a.b.c.d`)
- Extended `QueryOptions` and `parse_anchor` to handle string options
  (short_circuit_field, short_circuit_value) alongside existing u32 opts
- New utilities: `has_returning_clause`, `run_stackql_dml_returning`,
  `flatten_returning_row`, `check_short_circuit`, `run_callback_poll`
- `create_resource` / `update_resource` / `delete_resource` now return
  the captured RETURNING * row alongside their existing boolean result
- Callbacks only run during `build` and `teardown`; dry-run logs intent
- 58 tests pass; website docs updated (resource-query-files.md, manifest-file.md)

https://claude.ai/code/session_01HcAKNRjjQRn7TywY28zv1Y
…variables

Introduces preprocess_this_prefix(), a pre-rendering step that rewrites
{{ this.X }} to {{ resource_name.X }} inside Tera expression/tag blocks
before the template reaches the engine. This lets template authors in a
resource's .iql file reference their own resource-scoped variables
unambiguously without spelling out the full resource name.

Resolution rules:
- {{ fred }}               - existing precedence (global/last-writer-wins)
- {{ resource_name.fred }} - fully-qualified, works from anywhere (unchanged)
- {{ this.fred }}          - alias for the current resource's namespace only

Behaviour:
- Rewrites {{ this.X }} -> {{ resource_name.X }} and {% this.X %} -> {% resource_name.X %}
- Consistent with the callback shorthand: {{ this.callback.ProgressEvent.RequestToken }}
  resolves identically to {{ resource_name.callback.ProgressEvent.RequestToken }}
- If this. appears outside a resource context (empty resource_name) a clear
  diagnostic error is raised rather than silently resolving incorrectly

Adds 12 tests covering: basic rewrite, noop, multiple occurrences, deep
dotted paths, filter expressions, tag blocks, error when no resource context,
and end-to-end rendering for the key disambiguation scenarios from the design.

https://claude.ai/code/session_01HcAKNRjjQRn7TywY28zv1Y
- core/utils.rs: remove unreachable thread::sleep + attempt increment after
  a match where every arm diverges (return/continue/exit)
- commands/base.rs: annotate run_callback with #[allow(clippy::too_many_arguments)]
  (pre-existing 10-param signature; refactor is out of scope here)
- core/templating.rs: introduce SqlQueriesResult type alias to satisfy
  clippy::type_complexity on load_sql_queries return type

https://claude.ai/code/session_01HcAKNRjjQRn7TywY28zv1Y
@jeffreyaven jeffreyaven merged commit fa6cc01 into main Feb 27, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants