Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion agent-service/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,8 @@ LLM_ENDPOINT=http://localhost:9096
# Texera backend services
TEXERA_DASHBOARD_SERVICE_ENDPOINT=http://localhost:8080
WORKFLOW_COMPILING_SERVICE_ENDPOINT=http://localhost:9090
WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://localhost:8085
WORKFLOW_EXECUTION_SERVICE_ENDPOINT=http://localhost:8085

# Docs read by the agent's searchDocs/readDoc tools. The image copies them to ./docs;
# a local run from agent-service/ needs the repo's docs folder.
TEXERA_DOCS_DIR=../docs
115 changes: 115 additions & 0 deletions agent-service/scripts/docs-search-eval.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,115 @@
[
{
"group": "platform",
"question": "where do I find a workflow that someone shared with me",
"expected": "getting-started/ui-overview"
},
{
"group": "platform",
"question": "how do I upload a file to a dataset and use it in a workflow",
"expected": "tutorials/create-dataset-upload-data"
},
{
"group": "platform",
"question": "how do I log in to Texera for the first time",
"expected": "getting-started/AccessLogin"
},
{ "group": "platform", "question": "what is a computing unit and who can use one", "expected": "security/_index" },
{
"group": "platform",
"question": "how do I run a workflow and see the results in the browser",
"expected": "tutorials/guide-for-how-to-use-texera"
},
{
"group": "operator",
"question": "how do I remove duplicate rows",
"expected": "reference/operators/data-cleaning/distinct"
},
{
"group": "operator",
"question": "CSV file scan delimiter and header options",
"expected": "reference/operators/data-input/csv-file-scan"
},
{
"group": "operator",
"question": "join two tables on a key column",
"expected": "reference/operators/data-cleaning/join/hash-join"
},
{
"group": "operator",
"question": "group by a column and compute sum or average",
"expected": "reference/operators/data-cleaning/aggregate/aggregate"
},
{
"group": "operator",
"question": "keep only the first 100 rows",
"expected": "reference/operators/data-cleaning/limit"
},
{
"group": "operator",
"question": "search a string column with a regular expression",
"expected": "reference/operators/search/regular-expression"
},
{
"group": "operator",
"question": "how to write a Python UDF operator",
"expected": "tutorials/guide-to-use-python-udf"
},
{
"group": "operator",
"question": "convert a column from string to integer",
"expected": "reference/operators/data-cleaning/type-casting"
},
{
"group": "operator",
"question": "draw a word cloud from text",
"expected": "reference/operators/visualization/media/word-cloud"
},
{
"group": "operator",
"question": "sort rows by a column",
"expected": "reference/operators/data-cleaning/sort/sort"
},
{
"group": "deploy",
"question": "install Texera with docker compose",
"expected": "getting-started/installing-using-docker"
},
{
"group": "deploy",
"question": "run Texera on kubernetes with helm",
"expected": "getting-started/run-on-kubernetes"
},
{
"group": "deploy",
"question": "how do I enable the AI agent and configure LiteLLM",
"expected": "tutorials/guide-to-enable-llm-agent"
},
{
"group": "deploy",
"question": "stop and restart the Texera containers",
"expected": "getting-started/installing-using-docker"
},
{ "group": "platform", "question": "what are the user roles and access privileges", "expected": "security/_index" },
{
"group": "platform",
"question": "convert my Jupyter notebook into a workflow",
"expected": "tutorials/migrate-jupyter-notebook"
},
{
"group": "operator",
"question": "read data from a PostgreSQL database",
"expected": "reference/operators/database-connector/postgresql-source"
},
{
"group": "operator",
"question": "list the files in a dataset version",
"expected": "reference/operators/data-input/file-lister"
},
{
"group": "operator",
"question": "what does set snapshot output mode mean",
"expected": "reference/operators/output-modes"
},
{ "group": "platform", "question": "overview of Texera architecture and core concepts", "expected": "overview" }
]
69 changes: 69 additions & 0 deletions agent-service/scripts/docs-search-eval.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

// Measures recall@3 and recall@5 of the docs keyword index over a fixed question set.
// Usage: bun run scripts/docs-search-eval.ts [docsDir] (default: ../docs)

import { readFileSync } from "node:fs";
import { join } from "node:path";
import { loadDocsIndex } from "../src/agent/tools/docs-index";

interface EvalQuestion {
group: string;
question: string;
expected: string;
}

const docsDir = process.argv[2] ?? join(import.meta.dir, "..", "..", "docs");
const index = loadDocsIndex(docsDir);
if (!index) {
console.error(`docs directory not found: ${docsDir}`);
process.exit(1);
}

const questions: EvalQuestion[] = JSON.parse(readFileSync(join(import.meta.dir, "docs-search-eval.json"), "utf8"));

let hitAt3 = 0;
let hitAt5 = 0;
const misses: string[] = [];
const noHit: string[] = [];

for (const q of questions) {
const hits = index.search(q.question, 5);
const paths = [...new Set(hits.map(h => h.path))];
const rank = paths.indexOf(q.expected);
if (rank >= 0 && rank < 3) hitAt3++;
if (rank >= 0) hitAt5++;
if (hits.length === 0) noHit.push(q.question);
const mark = rank < 0 ? "MISS" : `#${rank + 1}`;
console.log(`${mark.padEnd(5)} [${q.group}] ${q.question}`);
if (rank < 0) {
misses.push(`${q.question} -> expected ${q.expected}, got ${paths.slice(0, 3).join(", ") || "(nothing)"}`);
}
}

console.log(`\npages indexed: ${index.pages().length}`);
console.log(`recall@3: ${hitAt3}/${questions.length} = ${(hitAt3 / questions.length).toFixed(2)}`);
console.log(`recall@5: ${hitAt5}/${questions.length} = ${(hitAt5 / questions.length).toFixed(2)}`);
if (misses.length > 0) {
console.log("\nmissed (expected page not in top 5):");
for (const m of misses) console.log(`- ${m}`);
}
console.log(`\nno hit at all (candidate docs gaps): ${noHit.length === 0 ? "none" : ""}`);
for (const q of noHit) console.log(`- ${q}`);
41 changes: 41 additions & 0 deletions agent-service/src/agent/prompts.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,47 @@ describe("buildSystemPrompt", () => {
expect(prompt).toContain("## R UDF Guide");
});

test("describes the Documentation Read section between the ongoing task and the dataflow", () => {
const prompt = buildSystemPrompt(makeMetadataStore());

const bullets = [
"- `# Ongoing Task` — the current task, including turns you've taken so far",
"- `# Documentation Read` — the output of your searchDocs and readDoc calls in the current task (omitted if none)",
"- `# Current Dataflow` — the live DAG: every operator's current state",
];
expect(prompt).toContain(bullets.join("\n"));
expect(prompt).not.toContain("three top-level sections");
});

test("shows the Documentation Read section in the layout the way the context renders it", () => {
const prompt = buildSystemPrompt(makeMetadataStore());

expect(prompt).toContain(
[
"# Documentation Read",
"Output of your searchDocs and readDoc calls in this task, oldest first.",
"",
'## Turn 1: searchDocs {"query":"<keywords>"}',
" <the tool's output>",
"",
"# Current Dataflow",
].join("\n")
);
});

test("keeps the rule about reading documentation and adds the docs-tools rule right after it", () => {
const prompt = buildSystemPrompt(makeMetadataStore());

const kept =
"- **Read documentation first**: When the task mentions abstract concepts, load documentation to understand exact definitions.";
const added = "- **Answer questions about Texera from its documentation**:";
expect(prompt).toContain(`${kept}\n${added}`);
expect(prompt).toContain("call searchDocs first, then readDoc on the best match, and answer with the page link.");
expect(prompt).toContain("Only cite links that these tools returned");
expect(prompt).toContain("say so when the documentation does not cover the question");
expect(prompt).toContain("Do not edit the workflow for a question.");
});

test("reports that no operators are available when a restricted type is absent from metadata", () => {
const prompt = buildSystemPrompt(makeMetadataStore(), ["MissingOperator"]);

Expand Down
10 changes: 9 additions & 1 deletion agent-service/src/agent/prompts.ts
Original file line number Diff line number Diff line change
Expand Up @@ -163,10 +163,11 @@ Dataflow represents data analysis as a DAG (directed acyclic graph) where:

## Context Format

Your conversation context is a single message with three top-level sections, in this order:
Your conversation context is a single message with these top-level sections, in this order:

- \`# Completed Tasks\` — previous tasks you've already finished (omitted if none)
- \`# Ongoing Task\` — the current task, including turns you've taken so far
- \`# Documentation Read\` — the output of your searchDocs and readDoc calls in the current task (omitted if none)
- \`# Current Dataflow\` — the live DAG: every operator's current state

**Overall layout:**
Expand Down Expand Up @@ -215,6 +216,12 @@ Thought: ...
- Error:
<full error trace, possibly multi-line>

# Documentation Read
Output of your searchDocs and readDoc calls in this task, oldest first.

## Turn 1: searchDocs {"query":"<keywords>"}
<the tool's output>

# Current Dataflow
## Operators

Expand All @@ -241,6 +248,7 @@ Result:
- **One operation per operator**: Each operator does one task (join, filter, aggregate, etc.). Use links to connect them.
- **Build incrementally**: Link new operators to existing ones. Never recreate data already in the workflow.
- **Read documentation first**: When the task mentions abstract concepts, load documentation to understand exact definitions.
- **Answer questions about Texera from its documentation**: If the searchDocs tool is available and the user asks how Texera or one of its features works, call searchDocs first, then readDoc on the best match, and answer with the page link. Only cite links that these tools returned, and say so when the documentation does not cover the question. Do not edit the workflow for a question. If the user asks you to build or change something, you may still consult the docs before acting.
- **Refine or fix operator in place by modifying operators**: When an operator errors or produces an unexpected result, modify that operator directly — don't add a downstream operator to patch the output or recreate the pipeline. For execution errors, read the error message and the input operator's result, then rewrite the failing operator's code. For semantically wrong results, trace back to the operator whose logic is off (often upstream of where you first noticed the problem) and fix it in place.
- **Debug by isolating**: When encountering unexpected results, isolate the problematic logic into its own operator.
- **Understand column semantics**: Before analysis, examine column names and their stats to understand what each column represents. Columns may carry semantic meaning that affects how data should be filtered or interpreted — respect these signals and apply appropriate preprocessing before computing results.
Expand Down
Loading
Loading