Repository navigation
Description
Feature Summary
A workflow can be built and read in the editor, but there is no form of it that runs anywhere else. A user who wants to keep a pipeline after the fact, hand it to a colleague who does not run Texera, or step through it in a notebook has nothing to take away.
What is missing is an export: given a plan, produce a single Python file that reads the same sources, applies the same operators in the same order, and prints its results. The operators already describe their work as Python for the engine, so the pieces exist; what is absent is a form of that description that stands on its own, without the runtime around it, and something to stitch the pieces into one script.
Proposed Solution or Design
An operator says how it reads outside the engine by implementing a StandaloneCodeGenerator trait, returning a block of pandas that names its inputs and outputs by position: in1df, in2df, out1df. A translator walks the plan in topological order, gives every output port a variable, substitutes those placeholders for the variables its upstreams were given, and prints the leaves. A variadic port, of which Union is the one example, needs the whole list of upstreams rather than a fixed count, since any count the operator states would be wrong for some workflow.
The compiling service exposes the result as an endpoint, so the editor can offer the script for a plan the user has open.
An operator that has no generator yet leaves a commented placeholder in the script rather than a line that looks like it works, so the export is useful before every operator implements it.
The set, in order
- 1. Translate a workflow into a standalone Python script: the trait and the translator #8407, the export itself: the trait and the translator. PR feat(workflow-compiling-service): translate a workflow into a standalone Python script #8327
- 2. Export the first five operators as Python, and serve the script #8501, the first five operators, and the endpoint that serves the script. PR feat(workflow-operator): export the first five operators, and serve the script #8502
- 3. Run an operator through the engine and keep what it wrote #8408, running an operator through the engine. PR test(verify): run an operator through the engine and keep what it wrote #8356
- 4. Run a Python operator the way the engine runs it #8409, running a Python operator the way the engine runs it. PR test(verify): run a Python operator the way the engine runs it #8357
- 5. Run the script the export produces #8410, running the script the export produces. PR test(verify): run the script the export produces #8358
- 6. Compare what the two paths produced #8411, comparing the two tables the paths produced. PR test(verify): compare what the two paths produced #8359
- 7. Compare the charts the two paths drew #8508, comparing the two charts the paths drew. PR test(verify): compare the charts the two paths drew #8509
- 8. Give every operator the same table to read #8412, the canonical table every operator reads. PR test(verify): give every operator the same table to read #8360
- 9. Hand-write the configurations a schema cannot describe #8413, the configurations a schema cannot describe. PR test(verify): hand-write the configurations a schema cannot describe #8363
- 10. Export the base transform operators as Python #8414, the operators that work on text. PR feat(workflow-operator): export the operators that work on text #8340
- 11. The relational and sampling operators, in three parts:
- 11a. Export the join and set operators as Python #8505, the join and set operators. PR feat(workflow-operator): export the join and set operators as Python #8506
- 11b. Export Aggregate and the sort operators as Python #8707, Aggregate and the sort operators. PR feat(workflow-operator): export Aggregate and the sort operators as Python #8709
- 11c. Export the samplers, Split and the control-flow operators as Python #8708, the samplers, Split and the control-flow operators. PR feat(workflow-operator): export the samplers, Split and the control-flow operators as Python #8710
- 12. The sources, in three parts:
- 12a. Export the CSV sources as Python #8415, the CSV family. PR feat(workflow-operator): export the CSV sources as Python #8341
- 12b. Export the JSON Lines and Arrow sources as Python #8702, JSON Lines and Arrow. PR feat(workflow-operator): export the JSON Lines and Arrow sources as Python #8704
- 12c. Export the line sources and the URL fetcher as Python #8703, the line sources and the URL fetcher. PR feat(workflow-operator): export the line sources and the URL fetcher as Python #8705
- 13. Export the hierarchy and graph visualizations as Python #8416, visualizations: hierarchies and graphs. PR feat(visualization): export the hierarchy and graph charts as Python #8344
- 14. Export the charts that draw over a table's columns #8417, visualizations: charts over a table's columns. PR feat(visualization): export the charts that draw over a table's columns #8365
- 15. Export the tables, ranges and embedded documents #8418, visualizations: tables, ranges and embedded documents. PR feat(visualization): export the tables, ranges and embedded documents #8366
- 16. Export the coordinate-system and domain plots as Python #8419, visualizations: charts drawn on a coordinate system. PR feat(visualization): export the charts drawn on a coordinate system #8346
- 17. Export the charts drawn over a field or a sequence #8503, visualizations: charts drawn over a field or a sequence. PR feat(visualization): export the charts drawn over a field or a sequence #8504
- 18. Export the four advanced trainers, and write their hyperparameter rules into the schema #8420, the four advanced trainers. PR feat(machine-learning): export the four advanced trainers #8348
- 19. Export the Machine Learning Scorer #8421, the Machine Learning Scorer. PR feat(machine-learning): export the Machine Learning Scorer #8367
- 20. Export the scikit-learn estimators and the Hugging Face models #8422, the estimators and the Hugging Face models. PR feat(operator): export the estimators and the Hugging Face models #8368
- 21. Derive an operator's configuration from its schema #8423, deriving a configuration from an operator's schema. PR test(verify): derive an operator's configuration from its schema #8361
- 22. Sweep a configuration into the variants a parity run needs #8424, the sweep that turns one configuration into the variants a parity run needs. PR test(verify): sweep a configuration into the variants a parity run needs #8388
- 23. Run every operator both ways and report what it did #8425, running every operator both ways and reporting what it did. PR test(verify): run every operator both ways and report what it did #8362
- 24. Run the source operators, and report what is covered #8426, the source operators, and the coverage report. PR test(verify): run the source operators, and report what is covered #8364
- 25. Export a workflow as Python from the editor, and read the schema's constraints in the property panel #8427, the editor: the export button. PR feat(gui): export a workflow as Python, and read the schema's constraints in the panel #8350
- 26. Read the schema's constraints in the property panel #8510, the editor: the property panel reads the schema's constraints. PR feat(gui): read the schema's constraints in the property panel #8507
- 27. Say what a new operator owes beyond its own code #8458, the guide a contributor reads before adding an operator. PR docs: say what a new operator owes beyond its own code #8459
Each is sized to be read in one sitting. Eight entries were split for growing past that: the export itself into the trait and the operators that use it, the text operators from the relational ones, the coordinate-system charts from the domain ones, the table comparison from the chart comparison, the editor's export button from the property panel, deriving a configuration into the entry that produces one and the entry that sweeps it into variants, the relational operators into joins and sets, Aggregate with the sort family, and the samplers with control flow, and the sources into the CSV family, JSON Lines with Arrow, and the line sources. The numbering is the order to read the set in: the export first, then the machinery that runs an operator both ways, then the operator families that implement the trait, then the two changes that read every operator at once, and the editor last. Those two come after the families so their assertions hold as written. Merging follows what each entry calls, not its number. This issue stays open until the guide lands.
Related, outside the set
Three new operators read the trait but make no existing operator exportable, so they are not among the 27:
- feat(operator): read the whole-number parts out of a timestamp column #8480 reads the whole-number parts out of a timestamp column.
- feat(operator): cut a numeric column into bins #8481 cuts a numeric column into bins.
- feat(operator): read a Parquet file as a source #8512 reads a Parquet file as a source.
Bugs found along the way, each with its own issue:
- A file scan's Limit and Offset accept a negative value #8513, a file scan's Limit and Offset accepted a negative value. PR feat(source): bound a file scan's window at zero #8514
- Casting text to a timestamp empties a moment the engine reads #8595, casting text to a timestamp emptied a moment the engine reads. PR feat(operator): read a timestamp cast over the years the engine reaches #8597
- A file scan reads every file as UTF-8, whatever its Encoding field says #8596, a file scan read every file as UTF-8, whatever its Encoding field said. PR feat(workflow-operator): honor the File Scan Encoding field #8605
- A file scan extracting an archive cannot also include the filename #8598, a file scan extracting an archive could not also include the filename. PR feat(workflow-operator): export the line sources and the URL fetcher as Python #8705
- A file scan with Limit 0 loses the file's columns, and the CSVOld scan throws #8602, a file scan with Limit 0 lost the file's columns, and the CSVOld scan threw. PR feat(workflow-operator): export the JSON Lines and Arrow sources as Python #8704
- isSink folds an operator type with the browser's locale #8616, isSink folded an operator type with the browser's locale. PR fix(frontend): use locale-independent lowercasing in isSink #8648
- JSONL File Scan reads a JSON null as the text "null" #8674, JSONL File Scan read a JSON null as the text "null". PR fix(amber): stop JSONL scan reading null as text #8740
- Should a table with no rows keep its declared schema? #8735, a port that carried no rows lost its declared columns. PR feat: keep a table's declared columns when it has no rows #8766
- Exported Python for Distinct drops rows the engine keeps when a DOUBLE holds NaN or -0.0 #8974, exported Distinct dropped rows the engine keeps when a DOUBLE holds NaN or -0.0. PR feat(workflow-operator): keep the rows Distinct's engine keeps for NaN and signed zero #8975
- Read the generated module as UTF-8 in three Python test drivers #8976, three Python test drivers did not read the generated module as UTF-8. PR feat(workflow-operator): read the generated module as UTF-8 in three test drivers #8988
- Exported Python for Filter disagrees with the engine on numbers in text, booleans, NaN and -0.0 #8977, exported Filter disagreed with the engine on numbers in text, booleans, NaN and -0.0. PR feat(workflow-operator): compare Filter's exported predicates the way the engine does #8980
- Exported Python for Regular Expression and Substring Search matches non-ASCII text differently from the engine #8989, exported Regular Expression and Substring Search matched non-ASCII text differently from the engine. PR feat(workflow-operator): match non-ASCII text in Regex and Substring Search exports as the engine does #8990
- Interval Join treats a null numeric key as 0 and fails on a null timestamp key #9002, Interval Join treated a null numeric key as 0 and failed on a null timestamp key. PR feat(operator): let a null Interval Join key match nothing #9003
- Exported Python looks up a column with its exact case where the engine ignores case #9004, exported Python looked up a column with its exact case where the engine ignores case. PR feat(workflow-operator): read a column named in a different case the way the engine does #9035
- Type Casting matches a column by its exact name where other operators ignore case #9036, Type Casting matched a column by its exact name where other operators ignore case.
- Result storage moves a timestamp before 1883 by the zone's local mean time #9074, result storage moved a timestamp before 1883 by the zone's local mean time. PR feat(workflow-core): store a timestamp before 1883 at the wall clock it holds #9076
- A Python operator pauses the run on a LONG column that holds a null and a value past 2^53 #9077, a Python operator paused the run on a LONG column that held a null and a value past 2^53.
Two improvements, each with its own issue:
- Run Workflow-to-Python verification only when relevant files change #9523, the Workflow-to-Python verification ran on PRs that could not change its result and skipped some that could. PR feat(ci): run Workflow-to-Python verification only when relevant files change #9524
- Exported Python scripts define every helper inline, burying the workflow code #9525, an exported script defined every helper inline, burying the workflow code. PR feat: export Python scripts that import their helpers from texera-helpers #9240
Affected Area
Workflow Engine (Amber), Workflow UI
Activity
kz930 commentedon Sep 3, 2026
@carloea2 @mengw15 — this is now open as a set of twenty changes rather than one. The list above gives the order; each is sized to be read in one sitting, most under a thousand lines.
The place to start is #8327, the export itself: an operator says how it reads outside the engine, a translator stitches the blocks into a script, and an endpoint serves it. Nothing else can land before it. After it come the pieces that run an operator both ways and compare the answers, then the operator families that implement the trait, then the two changes that read every operator at once — those sit after the families so their assertions hold as written — and the editor last.
They are not independent: each is cut from main and does not compile until the ones before it land, so the diffs read best in order.
kz930 commentedon Sep 3, 2026
One thing worth saying about the risk: a partial merge leaves nothing broken.
The export is usable as soon as #8327 lands. Each operator family after it is independent of the others, so every one that merges adds the operators it carries, and the ones left out simply stay as they are today. Nothing has to be taken whole, and nothing half-landed leaves the tree in a worse state than it is now.
The verification is what needs the whole set: it reads every operator at once, which is why the two changes that do that sit near the end.
kz930 commentedon Sep 4, 2026
/sub-issue #8407 #8408 #8409 #8410 #8411 #8412 #8413 #8414 #8415 #8416 #8417 #8418 #8419 #8420 #8421 #8422 #8423 #8424 #8425 #8426 #8427
60 remaining items
feat(workflow-operator): export the line sources and the URL fetcher …
feat(workflow-operator): export the join and set operators as Python (a…
test(verify): run a Python operator the way the engine runs it (apach…
feat(operator): read the whole-number parts out of a timestamp column (…
feat(operator): read a Parquet file as a source (apache#8512)
test(verify): derive an operator's configuration from its schema (apa…
test(verify): run the script the export produces (apache#8358)
test(verify): hand-write the configurations a schema cannot describe (a…
feat(operator): cut a numeric column into bins (apache#8481)
test(verify): sweep a configuration into the variants a parity run ne…
test(verify): compare what the two paths produced (apache#8359)