diff --git a/docs/concepts/_index.md b/docs/concepts/_index.md index 78b7338b3a1..3e3ee13248b 100644 --- a/docs/concepts/_index.md +++ b/docs/concepts/_index.md @@ -37,21 +37,18 @@ For contributors, it offers insight into the design principles and architecture ### What’s in this section -The **Concepts** section introduces the core ideas that define Texera’s design and operation: +- **[How Texera runs your workflow](how-texera-runs-your-workflow/):** pipelining, blocking operators, regions, workers and execution mode. -- **Workflows:** How users visually build and manage data pipelines. -- **Operators:** The modular units that perform data transformations. -- **Execution Engine:** The core component that executes workflows efficiently. -- **Data Model:** How Texera represents, stores, and streams data. -- **Architecture:** The high-level structure connecting frontend, backend, and execution layers. +More concept pages are still being written. Until then, these pages cover the same ground: -Each page below explores one of these areas in more depth, explaining how Texera’s internal components work together to support flexible, scalable, and interactive data analytics. +- **Workflows and operators:** [Guide for how to use Texera](/docs/tutorials/guide-for-how-to-use-texera/) and the [operator reference](/docs/reference/operators/). +- **Execution engine internals:** [Engine](/docs/reference/engine/). +- **Data storage:** [Storage](/docs/reference/storage/). +- **Architecture:** [Project structure](/docs/reference/project-structure/) and the [Overview](/docs/overview/). --- ### When to read this section -If you’re new to Texera, start with the **[Overview](/docs/overview/)** page to understand what the platform does. +If you’re new to Texera, start with the **[Overview](/docs/overview/)** page to understand what the platform does. Then come here to learn *how it works under the hood*. - -If you’re contributing to Texera or integrating it with other systems, the detailed concept pages — such as **Engine**, **Operator Framework**, and **Architecture** — will help you understand Texera’s internal design and extension points. diff --git a/docs/concepts/how-texera-runs-your-workflow/index.md b/docs/concepts/how-texera-runs-your-workflow/index.md new file mode 100644 index 00000000000..13c5e887c25 --- /dev/null +++ b/docs/concepts/how-texera-runs-your-workflow/index.md @@ -0,0 +1,69 @@ + + +--- +title: "How Texera Runs Your Workflow" +weight: 10 +--- + +This page explains why some operators start early, some wait, and some finish first. You don't need it to use Texera, but it helps when a run looks "stuck". + +## Data flows like an assembly line + +Texera doesn't wait for one operator to finish before starting the next. Rows flow down the arrows in small batches, so a Filter can work on the first rows while the CSV reader is still reading the rest. This is called **pipelining**. + +## Some operators must see everything first + +Some operators can't output anything until they have seen **all** their input. You can't sort a list you haven't finished reading. These are **blocking** operators: + +- **Sort**, **Aggregate**, **Distinct** +- Set operators: **Intersect**, **Difference**, **Symmetric Difference** +- **Join**: it reads one input fully first (the *build* side) and then streams the other +- Machine learning **training** operators + +While a blocking operator reads, the operators after it show 0 rows. That's normal. Python UDFs written with `UDFTableOperator` behave the same way: they output only after an input is complete. + +## Regions + +Texera splits a workflow into **regions** at blocking points. Operators in one region run together; a region starts after the regions it depends on finish. Between regions, results are saved (**materialized**) so the next region can read them. + +To see regions, click the **Layers** button in the editor toolbar and check **Regions**. On a narrow window the button can be hidden; widen the window if you don't see it. + +Layers menu with Grid, Regions, Workers, Status and Performance + +After you click **Run**, each region is drawn as a shaded area. Here Sort splits the workflow into two regions: + +![Two regions: CSV reader and Sort in one, Filter in another](regions.png) + +| Shade | Region is… | +|---|---| +| Gray | Waiting | +| Blue or yellow | Running (blue: reading an input that others wait for) | +| Green | Finished | + +## Workers + +An operator can run as several copies (**workers**) in parallel, each handling part of the data. For Python and other UDFs, set **Worker count** in the property panel. To show the count on each box, check **Workers** in the **Layers** menu. + +## Execution mode + +The **Settings** tab in the editor's left panel has **Execution Mode**: + +- **Pipelined** (default): rows flow between operators as described above. +- **Materialized**: every operator finishes and saves its output before the next one starts. Usually slower, because operators don't overlap. diff --git a/docs/concepts/how-texera-runs-your-workflow/layers-menu.png b/docs/concepts/how-texera-runs-your-workflow/layers-menu.png new file mode 100644 index 00000000000..3c1de294af0 Binary files /dev/null and b/docs/concepts/how-texera-runs-your-workflow/layers-menu.png differ diff --git a/docs/concepts/how-texera-runs-your-workflow/regions.png b/docs/concepts/how-texera-runs-your-workflow/regions.png new file mode 100644 index 00000000000..fd93e3d7d2d Binary files /dev/null and b/docs/concepts/how-texera-runs-your-workflow/regions.png differ diff --git a/docs/examples/_index.md b/docs/examples/_index.md index c548cf74dc5..bc5aa558f48 100644 --- a/docs/examples/_index.md +++ b/docs/examples/_index.md @@ -27,51 +27,30 @@ weight: 80 This section showcases example workflows, use cases, and demonstrations to help you understand Texera in action. {{% /pageinfo %}} -Texera makes it easy to design and execute data analytics workflows visually. -Here, you’ll find a collection of **example workflows** that highlight Texera’s capabilities — from data ingestion and transformation to visualization and machine learning. +Texera makes it easy to design and execute data analytics workflows visually. --- ## 🧩 Example Workflows -Each example demonstrates how Texera operators can be combined to solve different types of data problems. +Step-by-step examples you can follow today: -- **Text Analytics Workflow** - Analyze text data using tokenization, keyword extraction, and word cloud visualization. - → See the [Text Analytics Example](/docs/examples/text-analytics/) +- **[Filter and aggregate a CSV file](/docs/tutorials/create-dataset-upload-data/)**: upload data, keep the rows you need, and compute averages. +- **[Add a column with Python](/docs/tutorials/guide-to-use-python-udf/)**: write a small Python UDF, and see what common errors mean. +- **[Bring a Jupyter notebook into Texera](/docs/tutorials/migrate-jupyter-notebook/)**: turn notebook cells into a workflow. -- **Join and Filter Workflow** - Combine multiple datasets using joins and filters to create complex data pipelines. - → See the [Join Operator Example](/docs/examples/join-and-filter/) - -- **Machine Learning Workflow** - Build end-to-end ML workflows with data preprocessing, model training, and evaluation operators. - → See the [Machine Learning Example](/docs/examples/machine-learning/) - -- **Visualization Workflow** - Explore Texera’s interactive visual operators like scatter plots, histograms, and word clouds. - → See the [Visualization Example](/docs/examples/visualization/) +Pages for text analytics, joins, machine learning and visualization examples are planned. Until then, see the [operator reference](/docs/reference/operators/). --- -## 💡 How to Run the Examples - -To try these examples on your local Texera instance: +## 💡 Ready-made examples -1. Launch Texera following the [Getting Started](/docs/getting-started/) guide. -2. Open the **Workflow Editor** in your browser at `http://localhost:4200`. -3. Import an example workflow file (`.json`) into the editor. -4. Run the workflow to see Texera’s operators and data visualizations in action. +- **Your own Texera ([Docker install](/docs/getting-started/installing-using-docker/))**: start it with `docker compose --profile examples up` to pre-create two example workflows and their datasets. +- **Any Texera**: open **Hub → Workflows** to browse workflows others have made public. +- **Import a workflow file**: on **Your Work → Workflows**, click the cloud upload icon next to **Create Workflow** and choose a workflow `.json` or `.zip` file. --- ## 🧠 Want to Contribute an Example? -If you’ve built your own workflow and want to share it: -- Fork the [Texera repository](https://github.com/apache/texera). -- Add your workflow under `examples/` with a short README. -- Submit a pull request following our [Contribution Guidelines](/docs/contribution-guidelines/). - ---- - -These examples are a great starting point for learning Texera’s **visual programming model** and understanding how different operators interact to form powerful data pipelines. +Write a tutorial page under `docs/tutorials/` with screenshots and a sample data file, then submit a pull request following our [Contribution Guidelines](/docs/contribution-guidelines/). diff --git a/docs/getting-started/ui-overview.md b/docs/getting-started/ui-overview.md index 5f4aaa8ff2b..c2f32a62ff1 100644 --- a/docs/getting-started/ui-overview.md +++ b/docs/getting-started/ui-overview.md @@ -39,12 +39,9 @@ Once logged in, you should see the following page: ### Navigation Bar -On the left sidebar, you can switch between different resource modules: +The left sidebar switches between Hub, Your Work (workflows, datasets, computing units and more) and, for admins, Admin. Each item is described in [The dashboard](/docs/tutorials/guide-for-how-to-use-texera/#the-dashboard). -- **Workflows** — manage workflow projects. -- **Datasets** — upload and manage data files. -- **Quota** — check usage statistics and resource consumption. -- **Admin** — manage system users (visible only to admins). +For a walkthrough of the workflow editor, see the [Guide for how to use Texera](/docs/tutorials/guide-for-how-to-use-texera/). {{< alert color="success" title="Tip" >}} Hover over the navigation icons to see quick tooltips for each section. diff --git a/docs/reference/operators/user-defined-functions/python/1-out-python-udf.md b/docs/reference/operators/user-defined-functions/python/1-out-python-udf.md index bcfab91e6b0..6996bcd059c 100644 --- a/docs/reference/operators/user-defined-functions/python/1-out-python-udf.md +++ b/docs/reference/operators/user-defined-functions/python/1-out-python-udf.md @@ -27,6 +27,8 @@ tags: [user-defined-functions, python] [Home](../../../) > [User Defined Functions](../../) > [Python](../) +> New to Python UDFs? Start with the [Guide to Use a Python UDF](/docs/tutorials/guide-to-use-python-udf/). It covers the required import, choosing one class, and common errors. + ### Input Properties | Property | Requirement | Type | Default | Description | diff --git a/docs/reference/operators/user-defined-functions/python/2-in-python-udf.md b/docs/reference/operators/user-defined-functions/python/2-in-python-udf.md index a1d9143c55d..d1a6a3ba407 100644 --- a/docs/reference/operators/user-defined-functions/python/2-in-python-udf.md +++ b/docs/reference/operators/user-defined-functions/python/2-in-python-udf.md @@ -27,6 +27,8 @@ tags: [user-defined-functions, python] [Home](../../../) > [User Defined Functions](../../) > [Python](../) +> New to Python UDFs? Start with the [Guide to Use a Python UDF](/docs/tutorials/guide-to-use-python-udf/). It covers the required import, choosing one class, and common errors. + ### Input Properties | Property | Requirement | Type | Default | Description | diff --git a/docs/reference/operators/user-defined-functions/python/python-udf.md b/docs/reference/operators/user-defined-functions/python/python-udf.md index 1680c9f3b7e..1dc6743ce12 100644 --- a/docs/reference/operators/user-defined-functions/python/python-udf.md +++ b/docs/reference/operators/user-defined-functions/python/python-udf.md @@ -27,6 +27,8 @@ tags: [user-defined-functions, python] [Home](../../../) > [User Defined Functions](../../) > [Python](../) +> New to Python UDFs? Start with the [Guide to Use a Python UDF](/docs/tutorials/guide-to-use-python-udf/). It covers the required import, choosing one class, and common errors. + ### Input Properties | Property | Requirement | Type | Default | Description | diff --git a/docs/tutorials/computing-units.md b/docs/tutorials/computing-units.md new file mode 100644 index 00000000000..e43841ea115 --- /dev/null +++ b/docs/tutorials/computing-units.md @@ -0,0 +1,67 @@ + + +--- +title: "Computing Units" +weight: 15 +--- + +A **computing unit** is the machine that runs your workflows. You pick one in the workflow editor's top bar before clicking **Run** ([how](../guide-for-how-to-use-texera/#run-a-workflow)). You can see all of yours under **Your Work → Compute**. + +## Create one + +Click **+ Computing Unit** in the top bar, or **Create Computing Unit** on the Compute page. What you see depends on how your Texera is set up: + +- **Create Computing Unit** (hosted Texera): Texera starts a new machine for you. Choose: + + | Field | What to pick | + |---|---| + | **Select RAM Size** | Memory. Start with the smallest; pick more if a run fails on large data. | + | **Select #CPU Core(s)** | More cores help only when operators run with several workers. | + | **Select #GPU(s)** | Only for deep learning in Python. Shown only if GPUs are available. | + | **Image** | The software installed on the unit. Keep **Default** unless told otherwise. Fixed for the unit's lifetime. | + | **Advanced Settings** | Shared memory (raise it for PyTorch) and JVM heap. Leave the defaults unless an admin tells you otherwise. | + +- **Connect to a Local Computing Unit** (Texera on your own computer): enter a name and keep the suggested address. + +A new unit shows a gold dot while it starts. You can select it once the dot turns green. + +## Status dots + +| Dot | Meaning | +|---|---| +| Green | Running: ready to use | +| Gold | Starting up, or shutting down | +| Red | Unavailable. Pick or create another unit | + +## Tips + +- **Reuse one unit** for all your workflows. You don't need one per workflow. +- **A new unit usually won't fix an error in your workflow**, such as a mistake in Python code or an operator setting. Read the error first ([where to find errors](../guide-for-how-to-use-texera/#where-to-find-errors)). A different unit *can* help when the problem is the unit itself: not enough memory, CPU or GPU, or a missing Python package or software image. +- **Terminating a unit stops it.** Your workflows and datasets are kept; the results of runs on it are lost. To get them back, run the workflow again on another unit. +- **Details**: click the eye icon next to a unit to see its CPU, memory and GPU limits. Local units have no limits and show `NaN`. +- **Python packages**: the **+** icon next to a unit manages its Python environments. + +## Common messages + +| Message | What to do | +|---|---| +| `You may only have N computing unit(s) running at the same time` | Reuse a running unit, terminate a Kubernetes unit, or disconnect a local unit you no longer need (trash icon). | +| `Insufficient CPU / memory / GPU available in the server` | Ask for less RAM, fewer cores or GPUs, or try again later. | +| **Run** is disabled | You only have read access to the selected unit. Pick a unit you own. | diff --git a/docs/tutorials/create-dataset-upload-data.md b/docs/tutorials/create-dataset-upload-data.md deleted file mode 100644 index 0e7237a67fe..00000000000 --- a/docs/tutorials/create-dataset-upload-data.md +++ /dev/null @@ -1,54 +0,0 @@ - - ---- -title: "Create Dataset, upload data to it and use it in Workflow" -weight: 20 ---- - -This tutorial goes through the process of preparing data by creating dataset and creating a workflow to analyze data resided in the dataset using Texera. - -More specifically, we are going to create a dataset named `Sales Dataset` which contains a file about the sales data of different types of merchandises for several countries. And the workflow will calculate the average sales per item type across different countries in Europe from the [CountrySalesData.csv](statics/files/CountrySalesData.csv) (Make sure the downloaded file is in `.csv` file extension). The sales data has been downloaded from [eforexcel.com](http://eforexcel.com/wp/downloads-18-sample-csv-files-data-sets-for-testing-sales/) and has 100 rows of data. - -We will first be creating a dataset and uploading the sales data to it. Then we will be creating a workflow on Texera Web UI to - 1. read the data from the file; - 2. filter the relevant data based on keywords; - 3. perform an aggregation. - -**1. Upload data by creating a Dataset** - * Go to the Dataset tab and click the `dataset creation` icon to start creating the datasaet - * Name the dataset as `Sales Dataset`, drag and drop the `CountrySalesData.csv` to the file uploading area - * Click `Create`, the dataset we just created, along with the preview of `CountrySalesData.csv` is shown. -![2024-03-05 22 00 43](/images/github-assets/e17631b3-bf58-442f-af19-00f0ab704acb.png) - -**2. Read data in Workflow** - * On the left panel, go to the `environment` tab and click `Add Dataset` to add the `Sales Dataset` to current workflow. `CountrySalesData.csv` will be available to be previewed and loaded to the workflow. -![2024-03-05 22 26 45](/images/github-assets/45e98e6b-fe6a-405c-bd24-22ee28ee3716.png)' - * Drag and drop a `CSV File Scan` operator. On the right panel, input the file name `CountrySalesData.csv` and select the path from the drop down menu - * Run the workflow, you should be able to see the loaded sales data. -![2024-03-05 22 46 11](/images/github-assets/77389a4c-dd73-4179-b8c0-ebf10241b182.png) - - -**3. Add operators to analyze data** - * Drag and drop a `Filter` operator to keep only the sales data in `Europe` -![2024-03-05 22 51 26](/images/github-assets/9b73fcaa-a7df-4efb-8189-4054a6bef527.png) - - * Drag and drop a `Aggregate` operator to get the average sold units group by `Item Type` -![2024-03-05 22 53 06](/images/github-assets/67ade74c-df20-44b1-a9fa-1b8edb4af0cf.png) - diff --git a/docs/tutorials/create-dataset-upload-data/CountrySalesData.csv b/docs/tutorials/create-dataset-upload-data/CountrySalesData.csv new file mode 100644 index 00000000000..34d1d7e323d --- /dev/null +++ b/docs/tutorials/create-dataset-upload-data/CountrySalesData.csv @@ -0,0 +1,101 @@ +Region,Country,Item Type,Sales Channel,Units Sold,Unit Price +Sub-Saharan Africa,Kenya,Cosmetics,Online,1196,362.97 +Europe,Spain,Fruits,Online,8323,101.53 +Europe,Italy,Cosmetics,Online,3953,48.1 +North America,Canada,Fruits,Online,3667,280.8 +Europe,Norway,Fruits,Offline,822,429.77 +Europe,Norway,Cereal,Offline,6877,71.17 +Europe,Norway,Clothes,Online,1698,259.67 +Asia,Japan,Baby Food,Online,9256,34.69 +Asia,Japan,Office Supplies,Offline,5156,209.67 +North America,Mexico,Clothes,Online,2955,310.27 +Asia,India,Fruits,Offline,8614,222.4 +Sub-Saharan Africa,Ghana,Clothes,Online,1944,229.64 +Asia,Japan,Cereal,Offline,6919,25.9 +Europe,Norway,Fruits,Offline,5582,308.67 +North America,Mexico,Baby Food,Online,4432,213.34 +Europe,France,Office Supplies,Offline,9479,437.02 +North America,Mexico,Office Supplies,Offline,5695,18.72 +North America,Mexico,Cereal,Online,8098,34.41 +Sub-Saharan Africa,Kenya,Office Supplies,Online,6529,177.5 +North America,Canada,Cereal,Offline,6590,245.81 +Asia,Japan,Fruits,Offline,6814,434.17 +North America,Canada,Cereal,Online,2897,74.21 +Asia,India,Cosmetics,Online,4314,130.51 +Asia,Japan,Fruits,Offline,9288,146.32 +Asia,Vietnam,Fruits,Online,7491,396.7 +North America,Mexico,Cosmetics,Offline,1706,216.54 +North America,Canada,Cereal,Online,3430,198.91 +Europe,Spain,Fruits,Online,1687,9.1 +Asia,Vietnam,Baby Food,Offline,427,39.31 +Asia,Vietnam,Cosmetics,Online,4142,420.81 +Sub-Saharan Africa,Ghana,Baby Food,Online,8006,437.03 +North America,Mexico,Cosmetics,Offline,1417,71.11 +Sub-Saharan Africa,Ghana,Cosmetics,Online,8469,18.95 +Sub-Saharan Africa,Kenya,Office Supplies,Online,8662,137.48 +Europe,Spain,Fruits,Offline,2746,162.31 +Asia,Vietnam,Fruits,Offline,3664,273.3 +Asia,India,Cosmetics,Online,3285,232.1 +Sub-Saharan Africa,Kenya,Baby Food,Offline,7747,120.7 +Sub-Saharan Africa,Ghana,Office Supplies,Offline,5984,43.71 +Europe,Germany,Cosmetics,Online,5543,97.08 +Europe,Italy,Office Supplies,Offline,1399,368.73 +Europe,Italy,Office Supplies,Online,7842,392.16 +North America,Mexico,Baby Food,Offline,7598,182.0 +Europe,Germany,Cereal,Online,461,74.15 +North America,Canada,Fruits,Offline,5751,76.2 +Asia,India,Baby Food,Online,8637,332.03 +Asia,Japan,Cereal,Online,468,117.54 +Sub-Saharan Africa,Kenya,Fruits,Offline,4259,243.62 +Asia,India,Office Supplies,Offline,7516,294.53 +North America,Canada,Fruits,Online,8587,229.05 +North America,Canada,Fruits,Online,2464,83.28 +North America,Canada,Fruits,Online,5350,303.08 +North America,Canada,Fruits,Online,4081,91.45 +Europe,France,Fruits,Offline,9213,21.01 +Europe,Italy,Clothes,Online,4551,203.96 +North America,Canada,Office Supplies,Offline,9177,393.78 +Asia,Japan,Cereal,Offline,2002,178.11 +Sub-Saharan Africa,Kenya,Office Supplies,Online,7027,40.52 +Sub-Saharan Africa,Kenya,Cereal,Offline,2352,118.09 +Asia,Japan,Cereal,Online,6535,390.41 +Asia,Vietnam,Cereal,Online,7080,437.45 +North America,Mexico,Cosmetics,Online,5852,146.28 +Sub-Saharan Africa,Kenya,Clothes,Offline,7226,312.06 +North America,Mexico,Fruits,Offline,8402,423.09 +Europe,Germany,Baby Food,Online,4361,126.2 +Asia,Japan,Cereal,Offline,4247,183.96 +North America,Mexico,Baby Food,Offline,952,353.62 +Asia,Japan,Baby Food,Offline,285,282.44 +Sub-Saharan Africa,Kenya,Fruits,Online,1101,122.98 +Europe,Italy,Baby Food,Offline,9071,189.05 +Sub-Saharan Africa,Kenya,Baby Food,Online,1803,426.73 +Sub-Saharan Africa,Kenya,Cereal,Online,5121,279.96 +Asia,Japan,Cosmetics,Online,4442,158.56 +Europe,Spain,Baby Food,Online,312,324.96 +Asia,Vietnam,Cosmetics,Online,7334,54.81 +North America,Mexico,Fruits,Offline,8311,141.65 +Asia,India,Clothes,Online,2299,183.42 +Sub-Saharan Africa,Kenya,Cereal,Online,1168,278.57 +Sub-Saharan Africa,Ghana,Cereal,Online,1394,295.71 +North America,Mexico,Fruits,Online,4811,28.5 +Asia,India,Clothes,Offline,69,122.46 +Sub-Saharan Africa,Ghana,Cereal,Online,5081,102.9 +Asia,India,Clothes,Offline,1384,213.57 +Asia,India,Fruits,Online,1498,122.86 +Europe,Germany,Cosmetics,Online,6464,18.69 +Sub-Saharan Africa,Kenya,Baby Food,Online,9784,176.88 +Sub-Saharan Africa,Ghana,Cereal,Offline,2381,27.87 +North America,Canada,Fruits,Online,9579,352.92 +Asia,India,Baby Food,Online,2190,283.6 +Europe,Italy,Cosmetics,Online,318,278.9 +Asia,Japan,Clothes,Online,7496,352.81 +Europe,Norway,Baby Food,Offline,4141,357.77 +Sub-Saharan Africa,Kenya,Office Supplies,Online,3790,327.87 +North America,Mexico,Cosmetics,Online,7858,401.41 +Sub-Saharan Africa,Kenya,Fruits,Online,1279,267.47 +Sub-Saharan Africa,Ghana,Office Supplies,Offline,9312,66.51 +North America,Canada,Cosmetics,Offline,1640,307.33 +North America,Mexico,Office Supplies,Offline,7623,209.8 +Europe,Norway,Cereal,Offline,1416,412.53 +Europe,Spain,Cosmetics,Online,8310,426.25 diff --git a/docs/tutorials/create-dataset-upload-data/create-dataset.png b/docs/tutorials/create-dataset-upload-data/create-dataset.png new file mode 100644 index 00000000000..cac67f3f180 Binary files /dev/null and b/docs/tutorials/create-dataset-upload-data/create-dataset.png differ diff --git a/docs/tutorials/create-dataset-upload-data/dataset-version.png b/docs/tutorials/create-dataset-upload-data/dataset-version.png new file mode 100644 index 00000000000..7a6725e58a1 Binary files /dev/null and b/docs/tutorials/create-dataset-upload-data/dataset-version.png differ diff --git a/docs/tutorials/create-dataset-upload-data/index.md b/docs/tutorials/create-dataset-upload-data/index.md new file mode 100644 index 00000000000..14a4360e293 --- /dev/null +++ b/docs/tutorials/create-dataset-upload-data/index.md @@ -0,0 +1,73 @@ + + +--- +title: "Create Dataset, upload data to it and use it in Workflow" +weight: 20 +--- + +In this tutorial you upload a CSV file to Texera and build a workflow that computes the **average units sold per item type in Europe**. + +Download the sample file first: [CountrySalesData.csv](CountrySalesData.csv) (100 rows of made-up sales data). + +## 1. Create a dataset + +A **dataset** is a folder of files stored in Texera. Every time you change its files, you save a new **version**. + +1. In the left menu, go to **Your Work → Datasets** and click **Create Dataset**. +2. Enter a name, e.g. `Sales Dataset`, and click **Create**. Texera turns the name into a lowercase id (`sales-dataset`). + +![Create New Dataset dialog](create-dataset.png) + +## 2. Upload the file and save a version + +1. Open the **Versions & Files** tab. +2. Under **Create New Version**, drag the CSV onto the upload area, or click **Upload Files**. +3. Optionally describe the change (e.g. `First upload`), then click **Submit**. + +Upload area with one finished file, a version description and the Submit button + +The version (`v1`) now appears under **Current Versions** with a preview of the file: + +![Dataset page showing version v1 and a preview of CountrySalesData.csv](dataset-version.png) + +{{< alert color="warning" title="Versions are fixed" >}} +A workflow reads one exact version of a file. Its path includes the version, e.g. +`/dataset/texera/sales-dataset/v1 - First upload of sales data/CountrySalesData.csv`. +When you upload a new version, existing workflows keep reading the old one. To use the new data, open the operator and select the file again, picking the new version. +{{< /alert >}} + +## 3. Read the file in a workflow + +1. Go to **Your Work → Workflows → Create Workflow**. +2. Open the **Operators** panel (grid icon at the top left of the canvas) and drag a **CSV File Scan** operator (under *Data Input*) onto the canvas. +3. In its property panel, click **Select File**. Choose the dataset, the version, then the file, and click **Select**. + +![File picker with dataset, version and file](select-file.png) + +## 4. Filter and aggregate + +1. Add a **Filter** operator and connect it after the CSV File Scan. Click the blue **+** next to **Predicates**, then set Attribute `Region`, Condition `=`, Value `Europe`. +2. Add an **Aggregate** operator after the Filter. Set: function `average`, attribute `Units Sold`, result attribute `avg_units`, group by key `Item Type`. +3. Click the Aggregate box and turn on the **eye icon** in the toolbar, then click **Run**. To pick a computing unit first, see [Run a workflow](../guide-for-how-to-use-texera/#run-a-workflow). +4. Click the Aggregate box. If the result panel is closed, open it with the square icon at the bottom left. The result: + +![Finished workflow: 100 rows read, 23 kept by the filter, 6 averages](workflow-result.png) + +The numbers on the arrows show how many rows passed through: 100 read, 23 in Europe, and 6 item types. diff --git a/docs/tutorials/create-dataset-upload-data/select-file.png b/docs/tutorials/create-dataset-upload-data/select-file.png new file mode 100644 index 00000000000..e4a98da5445 Binary files /dev/null and b/docs/tutorials/create-dataset-upload-data/select-file.png differ diff --git a/docs/tutorials/create-dataset-upload-data/upload-and-create-version.png b/docs/tutorials/create-dataset-upload-data/upload-and-create-version.png new file mode 100644 index 00000000000..9f05efc1b7e Binary files /dev/null and b/docs/tutorials/create-dataset-upload-data/upload-and-create-version.png differ diff --git a/docs/tutorials/create-dataset-upload-data/workflow-result.png b/docs/tutorials/create-dataset-upload-data/workflow-result.png new file mode 100644 index 00000000000..62eba86c5d3 Binary files /dev/null and b/docs/tutorials/create-dataset-upload-data/workflow-result.png differ diff --git a/docs/tutorials/guide-for-how-to-use-texera.md b/docs/tutorials/guide-for-how-to-use-texera.md deleted file mode 100644 index f41967a80bb..00000000000 --- a/docs/tutorials/guide-for-how-to-use-texera.md +++ /dev/null @@ -1,65 +0,0 @@ - - ---- -title: "Guide for how to use Texera" -weight: 10 ---- - -Texera is an open-source system that supports collaborative data science at scale using Web-based workflows. This page includes instructions on how to install the system as a developer and do a simple workflow. - -## Prerequisites -We assume you either went through [Installing Apache Texera using Docker](/docs/getting-started/installing-using-docker/), or the [Guide for Texera Developers](/docs/contribution-guidelines/guide-for-developers/). And Texera is up-and-running on your laptop. - -## Access Texera through Browser -Enter Texera's URL on your browser to access Texera. - -An admin account with username `texera` and password `texera` is pre-created by default. Input the username, password and click the `Sign in` button to login as the admin: -![Screenshot 2025-06-16 at 3 43 02 PM](/images/github-assets/ebcb10f7-3d25-4673-b908-7e2de2d3179e.png) - -### User Dashboard UI Overview -Once logged in, you should see the below page: -![Screenshot 2025-06-16 at 3 45 38 PM](/images/github-assets/baa3d599-99c6-401f-a772-4300cbc7c9a9.png) - -This is Texera's dashboard page. On the left navigation bar, you can switch between different resource modules, including -- `Workflows` for workflow management -- `Datasets` for dataset management -- `Quota` for checking the usage statistics -- `Admin` for managing users on the Texera system. This tab is only visible for system admins. - -### Workflow Workspace UI Overview - -![Screenshot 2025-06-16 at 3 52 40 PM](/images/github-assets/605df038-5335-4a65-bcf8-fa872ce79551.png) - -1. **Operator Library/Menu**: - - It is separated into multiple dropdown menus based on the operator type, e.g., Source Operator, Search Operator, etc. You can drag and drop an operator from these dropdown menus onto the Workflow Canvas. - -2. **Workflow Canvas**: - - It is the main playground, where you can drag and drop Operators from the Operator Library onto it. Each operator is shown as a square box and connected with other operators with arrowed links which indicates the data flow. - -3. **Properties Editor Panel**: - - The panel will show up when you highlight a specific operator (by clicking on it) in the Workflow Canvas. You can customize the properties of the selected operator, for example, set the keyword for a filter. When the selected operator is configured correctly, a green ring will surround it; while a red ring usually indicates an error in configuration or connection to other operators. - -4. **Result Panel**: - - By default or when there is no result, it is hidden. You can click on the little UP arrow to expand this panel. When a workflow is finished running, the result panel will pop up with the data. You may slide up and down or left and right to view the data inside the panel. - diff --git a/docs/tutorials/guide-for-how-to-use-texera/errors-all-operators.png b/docs/tutorials/guide-for-how-to-use-texera/errors-all-operators.png new file mode 100644 index 00000000000..d5b176a9713 Binary files /dev/null and b/docs/tutorials/guide-for-how-to-use-texera/errors-all-operators.png differ diff --git a/docs/tutorials/guide-for-how-to-use-texera/index.md b/docs/tutorials/guide-for-how-to-use-texera/index.md new file mode 100644 index 00000000000..ab422e713bd --- /dev/null +++ b/docs/tutorials/guide-for-how-to-use-texera/index.md @@ -0,0 +1,119 @@ + + +--- +title: "Guide for how to use Texera" +weight: 10 +--- + +Texera lets you analyze data by connecting boxes (**operators**) on a canvas instead of writing a program. Each box does one step, like "read a file", "keep rows where…", or "compute an average". The arrows carry data from one box to the next. + +## Sign in + +Open the Texera address you were given in your browser and sign in. + +On a Texera you installed yourself ([Docker guide](/docs/getting-started/installing-using-docker/)), the built-in admin account is username `texera`, password `texera`. + +![Sign-in page](/images/github-assets/ebcb10f7-3d25-4673-b908-7e2de2d3179e.png) + +## The dashboard + +After signing in you see the dashboard. The left menu has: + +- **Hub**: workflows and datasets other people have made public. +- **Your Work** + - **Workflows**: your workflows. Click **Create Workflow** to start one. + - **Datasets**: your data files ([how to upload](../create-dataset-upload-data/)). + - **Compute**: your computing units (see [below](#run-a-workflow)). + - **Warehouses** (on some setups): where your run results are stored. + - **Environments**: extra Python packages for Python operators. + - **Quota**: how much storage and compute you have used. +- **Admin**: users, executions, computing units and site settings. Admins only. + +## The workflow editor + +![Workflow editor](/images/github-assets/605df038-5335-4a65-bcf8-fa872ce79551.png) + +1. **Operators**: all available boxes, grouped by type. If the panel is closed, click the grid icon at the top left of the canvas. Drag a box onto the canvas, or type its name in the search box. Boxes added through search land on top of each other; drag them apart. +2. **Canvas**: connect the output dot on the right of one box to the input dot on the left of the next. +3. **Property panel**: click a box to set its options. A red outline means an option is missing or wrong. +4. **Result panel**: shows a box's output, printed messages and errors. If it's closed, open it with the square icon at the bottom left. + +## Run a workflow + +A **computing unit** is the machine that runs your workflow. Pick one in the top bar, then click **Run**. If the list is empty, click **+ Computing Unit** to create one. + +![Computing unit selector and Run button](select-computing-unit.png) + +Some Texera setups also need a **warehouse**, the place your run's results are stored. Pick one from the **Warehouse** menu next to the computing unit. If the Run button says **+ Warehouse** instead, click it to create one. The name can use letters, digits, `-` and `_`. + +- One computing unit can run all your workflows. Reuse it. +- A new unit usually won't fix an error in the workflow itself (code or settings). It can help when the run needs more memory, CPU or GPU, or different Python packages. +- Terminating a unit stops it. Your workflows and datasets are kept; the results of runs on it are lost. + +More in [Computing Units](../computing-units/). + +## Read a running workflow + +Each box's outline color shows its state. The number on each side is how many rows went in (left) and came out (right). + +![A finished CSV reader (green) feeding a running Python UDF (orange), 31 rows in and 30 out](operator-colors.png) + +| Color | Meaning | +|---|---| +| Gray | Not started yet | +| Yellow-green | Ready to start | +| Orange | Running | +| Magenta | Paused. You paused it, or a row caused an error | +| Green | Finished | + +Some operators (e.g. Sort, Aggregate, and the build side of Join) must read **all** their input before they output anything. They show 0 rows out until the operator before them finishes. That's normal. See [How Texera runs your workflow](/docs/concepts/how-texera-runs-your-workflow/). + +## See the results + +1. Before running, click the box and turn on its **eye icon** (top toolbar). Texera shows results only for boxes with the eye on. +2. After the run, click the box and open the result panel (square icon, bottom left) if it's closed. The **Result** tab shows its rows. + +The result panel only shows the box you clicked. If it says *No operator is selected*, click a box. If it says *This operator has no results yet*, turn on that box's eye icon and run again. + +## Where to find errors + +- **Click the box that failed, or an empty spot on the canvas.** If the run failed, the result panel shows a **Static Error** tab with the error, including code that failed to start. With nothing selected it lists every operator's errors. A run that stops right away with every box at 0 rows usually means this. + +![Static Error tab listing an error for all operators](errors-all-operators.png) + +- **Click the box** to see its **Console**: messages from `print(...)` and errors on a specific row. + +Python error messages are explained in the [Python UDF guide](../guide-to-use-python-udf/#finding-and-fixing-errors). + +## Pause and stop + +While a workflow runs, **Run** becomes **Pause**. **Pause** can be resumed. The red ⊗ button to its left **stops** the run for good. Stop a run before fixing and re-running it. + +![Stop button next to Pause](stop-button.png) + +## My workflow looks stuck + +The run button says *Submitting* or *Pause*, the timer keeps counting, and every box stays at 0 rows. Check these in order: + +1. **Click each colored box** and look at its **Console**. A magenta box is paused on an error. +2. **Is a box still orange, waiting for its input?** Operators such as Sort and Join wait for all their input. Check the box before it. +3. **Stop** the run (red ⊗), fix the problem, and **Run** again. +4. **Don't create a new computing unit just to retry.** Errors in code or settings happen on every unit. Switch units only if the error points at the unit, e.g. out of memory or a missing package. +5. Nothing in the Console? Ask your Texera administrator to check the computing unit logs. diff --git a/docs/tutorials/guide-for-how-to-use-texera/operator-colors.png b/docs/tutorials/guide-for-how-to-use-texera/operator-colors.png new file mode 100644 index 00000000000..a9b1259803b Binary files /dev/null and b/docs/tutorials/guide-for-how-to-use-texera/operator-colors.png differ diff --git a/docs/tutorials/guide-for-how-to-use-texera/select-computing-unit.png b/docs/tutorials/guide-for-how-to-use-texera/select-computing-unit.png new file mode 100644 index 00000000000..3aaef7c6471 Binary files /dev/null and b/docs/tutorials/guide-for-how-to-use-texera/select-computing-unit.png differ diff --git a/docs/tutorials/guide-for-how-to-use-texera/stop-button.png b/docs/tutorials/guide-for-how-to-use-texera/stop-button.png new file mode 100644 index 00000000000..7b3a5784e7c Binary files /dev/null and b/docs/tutorials/guide-for-how-to-use-texera/stop-button.png differ diff --git a/docs/tutorials/guide-to-use-python-udf.md b/docs/tutorials/guide-to-use-python-udf.md deleted file mode 100644 index b02c1a78daf..00000000000 --- a/docs/tutorials/guide-to-use-python-udf.md +++ /dev/null @@ -1,183 +0,0 @@ - - ---- -title: "Guide to Use a Python UDF" -weight: 30 ---- - -## What is Python UDF -User-defined Functions (UDFs) provide a means to incorporate custom logic into Texera. Texera offers comprehensive Python UDF APIs, enabling users to accomplish various tasks. This guide will delve into the usage of UDFs, breaking down the process step by step. - - -*** - - -## UDF UI and Editor - - -The UDF operator offers the following interface, requiring the user to provide the following inputs: `Python code`, `worker count`, and `output schema`. - -

-Screenshot 2023-07-04 at 12 51 37 -

- -- Screenshot 2023-07-04 at 13 25 59 Users can click on the "Edit code content" button to open the UDF code editor, where they can enter their custom Python code to define the desired operator. - -- Screenshot 2023-07-04 at 13 27 22 Users have the flexibility to adjust the parallelism of the UDF operator by modifying the number of workers. The engine will then create the corresponding number of workers to execute the same operator in parallel. - -- Screenshot 2023-07-04 at 13 27 29 Users need to provide the output schema of the UDF operator, which describes the output data's fields. - - The option `Retain input columns` allows users to include the input schema as the foundation for the output schema. - - The `Extra output column(s)` list allows users to define additional fields that should be included in the output schema. - -
-
-
- -- Screenshot 2023-07-04 at 13 04 31 _Optionally_, users can click on the pencil icon located next to the operator name to make modifications to the name of the operator. - - -*** - -## Operator Definition - -### Iterator-based operator -In Texera, all operators are implemented as iterators, including Python UDFs. -Concepturally, a defined operator is executed as: - -```python -operator = UDF() # initialize a UDF operator - -... # some other initialization logic - -# the main process loop -while input_stream.has_more(): - input_data = next_data() - output_iterator = operator.process(input_data) - for output_data in output_iterator: - send(output_data) - -... # some cleanup logic - -``` - -### Operator Life Cycle -The complete life cycle of a UDF operator consists of the following APIs: -1. `open() -> None` Open a context of the operator. Usually it can be used for loading/initiating some resources, such as a file, a model, or an API client. It will be invoked once per operator. -2. `process(data, port: int) -> Iterator[Optional[data]]` Process an input data from the given port, returning an iterator of optional data as output. It will be invoked once for every unit of data. -3. `on_finish(port: int) -> Iterator[Optional[data]]` Callback when one input port is exhausted, returning an iterator of optional data as output. It will be invoked once per port. -4. `close() -> None` Close the context of the operator. It will be invoked once per operator. - - -### Process Data APIs -There are three APIs to process the data in different units. - -1. Tuple API. - -```python - -class ProcessTupleOperator(UDFOperatorV2): - - def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: - yield tuple_ - -``` -Tuple API takes one input tuple from a port at a time. It returns an iterator of optional `TupleLike` instances. A `TupleLike` is any data structure that supports key-value pairs, such as `pytexera.Tuple`, `dict`, `defaultdict`, `NamedTuple`, etc. - -Tuple API is useful for implementing functional operations which are applied to tuples one by one, such as map, reduce, and filter. - -2. Table API. -```python - -class ProcessTableOperator(UDFTableOperator): - - def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]: - yield table -``` -Table API consumes a `Table` at a time, which consists of all the tuples from a port. It returns an iterator of optional `TableLike` instances. A `TableLike ` is a collection of `TupleLike`, and currently, we support `pytexera.Table` and `pandas.DataFrame` as a `TableLike` instance. More flexible types will be supported down the road. - -Table API is useful for implementing blocking operations that will consume all the data from one port, such as join, sort, and machine learning training. - -3. Batch API. -```python - -class ProcessBatchOperator(UDFBatchOperator): - - BATCH_SIZE = 10 - - def process_batch(self, batch: Batch, port: int) -> Iterator[Optional[BatchLike]]: - yield batch -``` -Batch API consumes a batch of tuples at a time. Similar to `Table`, a `Batch` is also a collection of `Tuple`s; however, its size is defined by the `BATCH_SIZE`, and one port can have multiple batches. It returns an iterator of optional `BatchLike` instances. A `BatchLike ` is a collection of `TupleLike`, and currently, we support `pytexera.Batch` and `pandas.DataFrame` as a `BatchLike` instance. More flexible types will be supported down the road. - -The Batch API serves as a hybrid API combining the features of both the Tuple and Table APIs. It is particularly valuable for striking a balance between time and space considerations, offering a trade-off that optimizes efficiency. - -_All three APIs can return an empty iterator by `yield None`._ - -### Schemas - -A UDF has an input Schema and an output Schema. The input schema is determined by the upstream operator's output schema and the engine will make sure the input data (tuple, table, or batch) matches the input schema. On the other hand, users are required to define the output schema of the UDF, and it is the user's responsibility to make sure the data output from the UDF matches the defined output schema. - -### Ports - -- Input ports: -A UDF can take zero, one or multiple input ports, different ports can have different input schemas. Each port can take in multiple links, as long as they share the same schema. - -- Output ports: -Currently, a UDF can only have exactly one output port. This means it cannot be used as a terminal operator (i.e., operator without output ports), or have more than one output port. - -#### 1-out UDF - -This UDF has zero input port and one output port. It is considered as a source operator (operator that produces data without an upstream). It has a special API: -```python - -class GenerateOperator(UDFSourceOperator): - - @overrides - def produce(self) -> Iterator[Union[TupleLike, TableLike, None]]: - yield -``` - -This `produce()` API returns an iterator of `TupleLike`, `TableLike`, or simply `None`. - -See [Generator Operator](https://github.com/apache/texera/blob/main/amber/src/main/python/pytexera/udf/examples/generator_operator_integer.py) for an example of 1-out UDF. - - -#### 2-in UDF - -This UDF has two input ports, namely `model` port and `tuples` port. The `tuples` port depends on the `model` port, which means that during the execution, the `model` port will execute first, and the `tuples` port will start after the `model` port consumes all its input data. -This dependency is particularly useful to implement machine learning inference operators, where a machine learning model is sent into the 2-in UDF through the `model` port, and becomes an operator state, then the tuples are coming in through the `tuples` port to be processed by the model. - -An example of 2-in UDF: -``` -class SVMClassifier(UDFOperatorV2): - - - @overrides - def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: - - if port == 0: # models port - self.model = tuple_['model'] - - else: # tuples port - tuple_['pred'] = self.model.predict(tuple_['text']) - yield tuple_ -``` - -_Currently, in 2-in UDF, "Retain input columns" will retain only the `tuples` port's input schema._ \ No newline at end of file diff --git a/docs/tutorials/guide-to-use-python-udf/index.md b/docs/tutorials/guide-to-use-python-udf/index.md new file mode 100644 index 00000000000..84ad523d93b --- /dev/null +++ b/docs/tutorials/guide-to-use-python-udf/index.md @@ -0,0 +1,216 @@ + + +--- +title: "Guide to Use a Python UDF" +weight: 30 +--- + +A **Python UDF** (user-defined function) is an operator that runs a small piece of Python code you write. Use it when no built-in operator does what you need. + +You don't need to be a programmer. Copy the example below and change one line at a time. + +## Which Python operator should I use? + +| You want to… | Use | +|---|---| +| Add or change a column with a one-line formula, e.g. `tuple_["Units Sold"] * 2` | **Python Lambda Function** (no class to write) | +| Turn a whole table into one summary row, e.g. `table["Units Sold"].mean()` | **Python Table Reducer** (no class to write) | +| Anything else with one or more inputs | **Python UDF** | +| Apply a model (input 0) to data rows (input 1) | **2-in Python UDF** | +| Create data from nothing (no input) | **1-out Python UDF** | + +## Your first UDF in 4 steps + +This example adds a `Revenue` column (units × price) to the sales data from [the dataset tutorial](../create-dataset-upload-data/). + +**1. Add the operator.** Drag **Python UDF** (under *User-defined Functions → Python*) onto the canvas and connect your data to it. + +**2. Write the code.** Click **Edit code content**. The editor opens with a template. Its classes are commented out (their lines start with `#`), so running it unchanged fails. Uncomment one class, or replace everything with: + +```python +from pytexera import * + + +class ProcessTupleOperator(UDFOperatorV2): + + @overrides + def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: + tuple_["Revenue"] = tuple_["Units Sold"] * tuple_["Unit Price"] + yield tuple_ +``` + +![Python UDF code editor with the Revenue example](udf-code-editor.png) + +**3. Declare the new column.** In the property panel, under **Extra output column(s)**, click **+** and add `Revenue` with type `double`. Leave **Retain input columns** checked to keep the original columns. + +Retain input columns and one extra output column named Revenue + +**4. Run.** Click the eye icon on the UDF, then **Run**. Click the UDF, then the **Result** tab in the result panel. + +## The 5 rules + +Almost every UDF problem comes from breaking one of these. + +| # | Rule | If you break it | +|---|---|---| +| 1 | `from pytexera import *` is at the top (the template already has it) | `name 'UDFOperatorV2' is not defined` | +| 2 | Exactly **one** class. Delete or comment out the others. | `There should be one and only one Operator defined` | +| 3 | The method name must match the class you inherit from (table below) | `... does not implement process_tuple` or `... No super class method found` | +| 4 | Send results with `yield`, never `return` a value | ``process_tuple must `yield` results, not `return` them`` | +| 5 | The columns you yield must match the output columns (step 3) | `expected but missing` or `unexpected field` | + +## How your code receives data + +Pick one row of this table. The class name in brackets and the method name must go together. + +| Class `(…)` | Method you write | You receive | Good for | +|---|---|---|---| +| `UDFOperatorV2` | `process_tuple(self, tuple_, port)` | one row at a time (use it like a dict: `tuple_["col"]`) | per-row formulas, filtering | +| `UDFBatchOperator` | `process_batch(self, batch, port)` | `BATCH_SIZE` rows at a time, as a pandas DataFrame | calling an API in chunks | +| `UDFTableOperator` | `process_table(self, table, port)` | **all** rows of one input, as a pandas DataFrame | sorting, statistics, ML training | +| `UDFSourceOperator` (1-out UDF only) | `produce(self)` | nothing; you create the data | generating or downloading data | + +What you can `yield`: + +- **A row**: a `Tuple` or a `dict`, e.g. `yield {"name": "a", "value": 1}`. +- **A table**: a pandas `DataFrame` or `Table`. Yield the DataFrame itself, not a dict that contains DataFrames. +- **`None`**: produce nothing for this input. + +To drop a row, just don't `yield` it. To output several rows, `yield` several times. + +## One-time setup (loading a model, opening a file) + +Put setup code in `open()`, which runs once before any data arrives: + +```python +from pytexera import * + + +class ProcessTupleOperator(UDFOperatorV2): + + @overrides + def open(self): + self.threshold = 100 + + @overrides + def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: + if tuple_["Units Sold"] > self.threshold: + yield tuple_ +``` + +Two more optional methods: + +- `on_finish(self, port)` runs once when each input ends, and can `yield` rows, e.g. a total after the last row. Use it with `UDFOperatorV2` or `UDFBatchOperator`. +- `close(self)` runs once at the very end. Use it to release what `open()` set up, e.g. close a file. + +If you write `__init__` instead of `open()`, make its first line `super().__init__()`. Older Texera versions fail without it (see the table below). + +To set values like `threshold` from the property panel instead of in code, see [UI parameters](/docs/reference/operators/user-defined-functions/python/#ui-parameters). + +## Output columns + +The UDF's output columns are **Retain input columns** (the input's columns, if checked) plus **Extra output column(s)**. Every row you `yield` must contain exactly these columns. + +- Adding a column? Add it under **Extra output column(s)**. +- Returning a completely new table (e.g. a summary)? Uncheck **Retain input columns** and list every output column. +- With several inputs, **Retain input columns** keeps the columns of input 0 (for the 2-in UDF: the `tuples` input, port 1). + +## Several inputs + +Add input ports with the **+** under the operator's left side. Ports are numbered from 0. Your method's `port` argument tells you which input a row came from. + +- **Inputs arrive in no fixed order.** Don't assume input 0 finishes before input 1. +- To combine inputs safely, use `UDFTableOperator`. It calls `process_table` once per input, when that input is complete. Keep each table and act when you have all of them: + +```python +from pytexera import * + +NUM_INPUTS = 2 + + +class ProcessTableOperator(UDFTableOperator): + + @overrides + def open(self): + self.tables = {} + + @overrides + def process_table(self, table: Table, port: int) -> Iterator[Optional[TableLike]]: + self.tables[port] = table + if len(self.tables) == NUM_INPUTS: + yield self.tables[0].merge(self.tables[1], on="Country") +``` + +- To force an order, click an input port's dot. Under **dependencies**, list the ports that must finish first. For example, port 1 with dependencies `0` waits for port 0. +- The **2-in Python UDF** has this built in: port 0 (`model`) is read completely before port 1 (`tuples`) starts. + +```python +from pytexera import * + + +class ProcessTupleOperator(UDFOperatorV2): + + @overrides + def process_tuple(self, tuple_: Tuple, port: int) -> Iterator[Optional[TupleLike]]: + if port == 0: # model input: remember the model + self.model = tuple_["model"] + else: # tuples input: use it + tuple_["pred"] = self.model.predict([tuple_["text"]])[0] + yield tuple_ +``` + +## Creating data (1-out Python UDF) + +```python +from pytexera import * + + +class GenerateOperator(UDFSourceOperator): + + @overrides + def produce(self) -> Iterator[Union[TupleLike, TableLike, None]]: + for i in range(10): + yield {"number": i} +``` + +List every column (here `number`, type `integer`) under **Columns** in the property panel. More examples: [pytexera examples](https://github.com/apache/texera/tree/main/amber/src/main/python/pytexera/udf/examples). + +## Finding and fixing errors + +- `print(...)` in your code shows up in the **Console** tab. Click the UDF to see it. +- If a row fails, the UDF turns **magenta** (paused) and the error appears in its **Console**. Fix the code, then stop the run (red ⊗ next to *Pause*) and run again. + +![Console showing an error on the paused UDF](udf-console-error.png) + +- If the code can't even load (missing import, wrong class), the error appears in **Static Error**. Click the UDF, or an empty spot on the canvas, to see it. The message includes the error type and your line number, e.g. `(NameError at line 2 of the UDF code)`. See [where to find errors](../guide-for-how-to-use-texera/#where-to-find-errors). + +| Error message contains | Cause | Fix | +|---|---|---| +| `name 'UDFOperatorV2' is not defined` (or `Tuple`, `Table`, …) | Missing import | Add `from pytexera import *` at the top | +| `There should be one and only one Operator defined` | No class, or more than one. The rest of the message names the classes. | Keep exactly one class, uncommented | +| `X subclasses UDFOperatorV2 but does not implement process_tuple` | The method doesn't match the class (e.g. `process_table` in a `UDFOperatorV2`) | Rename the method, or subclass the base the message suggests | +| `X does not subclass a pytexera base class` | The class has no pytexera base | Subclass one, e.g. `class ProcessTupleOperator(UDFOperatorV2):` (`UDFSourceOperator` for a 1-out UDF) | +| `No super class method found` | A method marked `@overrides` doesn't exist in the base class | Use the pair from [the table](#how-your-code-receives-data), or remove `@overrides` from a helper method | +| ``must `yield` results, not `return` them`` (older versions: `MatchError: '_' not provided` or `'NoneType' object is not iterable`) | `return` used instead of `yield` | Replace `return x` with `yield x` | +| `_TableOperator__table_data` (older versions) | `__init__` without `super().__init__()` | Add `super().__init__()`, or use `open()` | +| `A DataFrame was put in column 'X'` | A DataFrame is in a column whose type isn't binary | To output its rows, `yield` the DataFrame itself; to keep it as one value, make the column binary | +| `expected but missing in the Tuple` | A declared output column is missing from what you yield | Yield that column, or remove it from **Extra output column(s)** (**Columns** for a 1-out UDF) | +| `unexpected field` | You yield a column that isn't declared | Add it under **Extra output column(s)** (**Columns** for a 1-out UDF), or stop yielding it | +| `Column name X already exists!` | An extra output column has the same name as an input column | Rename it, or uncheck **Retain input columns** | diff --git a/docs/tutorials/guide-to-use-python-udf/udf-code-editor.png b/docs/tutorials/guide-to-use-python-udf/udf-code-editor.png new file mode 100644 index 00000000000..3ab3194d265 Binary files /dev/null and b/docs/tutorials/guide-to-use-python-udf/udf-code-editor.png differ diff --git a/docs/tutorials/guide-to-use-python-udf/udf-console-error.png b/docs/tutorials/guide-to-use-python-udf/udf-console-error.png new file mode 100644 index 00000000000..f118a87a8ec Binary files /dev/null and b/docs/tutorials/guide-to-use-python-udf/udf-console-error.png differ diff --git a/docs/tutorials/guide-to-use-python-udf/udf-output-columns.png b/docs/tutorials/guide-to-use-python-udf/udf-output-columns.png new file mode 100644 index 00000000000..03cc2b687f4 Binary files /dev/null and b/docs/tutorials/guide-to-use-python-udf/udf-output-columns.png differ diff --git a/docs/tutorials/sharing-and-permissions/index.md b/docs/tutorials/sharing-and-permissions/index.md new file mode 100644 index 00000000000..24a7ae20227 --- /dev/null +++ b/docs/tutorials/sharing-and-permissions/index.md @@ -0,0 +1,54 @@ + + +--- +title: "Sharing and Permissions" +weight: 25 +--- + +You can share **workflows**, **datasets** and, if your admin enabled it, **computing units**. + +## Share with a person + +1. Open the item and click **Share**: in the workflow editor's top bar, or the share icon on the item in your dashboard list. +2. Type the person's e-mail (they need a Texera account) and click **+**. You can add several. +3. Pick an **Access Level** and click **Share**. + +![Share dialog with Private/Public choice, e-mail field, access level and the access list](share-dialog.png) + +The **Access** list at the bottom shows who has access. Change someone's level there, or remove them with the trash icon. + +## Access levels + +| Level | Can do | +|---|---| +| **read** | Open and view. The workflow canvas is read-only. | +| **write** | Also edit, share with others, and make it public | +| **OWNER** | Everything. The creator. Can't be removed. | + +To **run** a workflow you need **write** access to a computing unit, even if you can open the workflow. + +## Datasets are shared separately + +Sharing a workflow does **not** share the datasets it reads. When someone else runs your workflow, Texera checks *their* access to each dataset. Share the datasets with them too, or make them public. + +## Public items + +- **Public** in the share dialog lets anyone on Texera find and view the item on the **Hub**. Anyone signed in can **Clone** a public workflow to get their own editable copy. +- For datasets, the dataset's **Settings** tab also has **Downloadable**. When it's off, other people can browse the files but can't download them. diff --git a/docs/tutorials/sharing-and-permissions/share-dialog.png b/docs/tutorials/sharing-and-permissions/share-dialog.png new file mode 100644 index 00000000000..c8b0a09e17d Binary files /dev/null and b/docs/tutorials/sharing-and-permissions/share-dialog.png differ