Skip to content

feat: Implemented search annotations for runs #68

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
yuechao-qin wants to merge 4 commits into
base: master
Choose a base branch
from
+1,266 −53
Open

feat: Implemented search annotations for runs #68

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bef4c47
Implemented search annotations for runs
yuechao-qin Jan 13, 2026
2e69b8f
Remove key from ValueFilter
yuechao-qin Jan 14, 2026
87e12b2
Run black formatter and added configuration to the pyproject.toml
yuechao-qin Jan 20, 2026
ac77dde
Fixed the group logic so that only 1 SQL EXISTS appear per group query
yuechao-qin Jan 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
91 changes: 91 additions & 0 deletions cloud_pipelines_backend/api_router.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,6 +269,97 @@ def stream_container_log(
router.get("/api/pipeline_runs/", tags=["pipelineRuns"], **default_config)(
inject_session_dependency(list_pipeline_runs_func)
)
router.post(
"/api/pipeline_runs/search/",
Copy link

@morgan-wowk morgan-wowk Jan 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a piece of feedback on REST semantics mainly. Typically you wouldn't see a POST request for searching for resources, in other words getting resources. If it were a GraphQL API then things would be different.

Here's the suggestion:

Turn this into a GET endpoint:

GET /api/pipeline_runs

This is clear that you get retrieving a list of pipeline runs, which would be an unfiltered, paginated response by default. Then add the search capability after the foundation (unfiltered search) is established.

From loading the UI, I can see there is already a request being made:

curl 'http://localhost:8000/api/pipeline_runs?include_pipeline_names=true&include_execution_stats=true'

and upon asking cursor, I know that it has a pagination implementation already. We can add the filter options to this existing endpoint rather than creating a new endpoint. Which would set a good precedence for all future search capabilities on other resources.

Here is an example of how I've seen this implemented:

curl 'http://localhost:8000/api/pipeline_runs?filter=in(annotations.env:staging,prod)+eq(annotations.app:agenticsearch)

With other operators like gte (greater than or equal), lte, etc.

Accounting for complex AND / OR combinations is something that would require some thought but I wouldn't go there unless that's something we see people using. The above example is a basic OR and AND.

Copy link
Contributor

@Ark-kun Ark-kun Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main reason for using POST is that GET requests do not support body. Sending complex query structures via query parameters can be problematic in certain cases. If we can overcome that, then we can use GET piepline_runs.

Copy link

@morgan-wowk morgan-wowk Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will dig up some documentation on how I've seen this achieved previously then we can review if it meets our needs

Copy link

@morgan-wowk morgan-wowk Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some helpful resources:

Copy link

@morgan-wowk morgan-wowk Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if we solve this once, we can use the same solution for all future APIs and won't need an extra /search endpoint to maintain

tags=["pipelineRuns"],
summary="Search Pipeline Runs",
description="""Search pipeline runs with annotation filters.

## What is an Annotation?

An annotation is a **key-value pair** attached to a pipeline run.
For example the following annotations (i.e. key = value):
- `environment` = `production`
- `team` = `backend`
- `priority` = `high`

## Filter Types

### KeyFilter - Search by annotation key
| Operator | Description |
|----------|-------------|
| `exists` | Key exists (any value) |
| `equals` | Key exactly matches string |
| `contains` | Key contains substring |
| `in_set` | Key matches one of multiple values |

### ValueFilter - Search by annotation value (across ALL annotations)
| Operator | Description |
|----------|-------------|
| `equals` | Value exactly matches string |
| `contains` | Value contains substring |
| `in_set` | Value matches one of multiple values |

### FilterGroup - Combine filters with logic
| Operator | Description |
|----------|-------------|
| `and` | ALL filters must match |
| `or` | ANY filter must match |

All filters support `negate: true` to invert the condition (e.g., NOT equals).

---

## Examples

### 1. Key equals a string
Find runs where annotation key equals "environment":
```json
{
"annotation_filters": {
"filters": [
{"operator": "equals", "key": "environment"}
]
}
}
```

### 2. Key contains substring AND value in set
Find runs where key contains "env" AND value is "prod" or "staging":
```json
{
"annotation_filters": {
"filters": [
{"operator": "contains", "key": "env"},
{"operator": "in_set", "values": ["prod", "staging"]}
],
"operator": "and"
}
}
```

### 3. Complex: (key contains OR value contains) AND key NOT contains
Find runs where (key contains "env" OR any value contains "prod") AND key NOT contains "deprecated":
```json
{
"annotation_filters": {
"filters": [
{
"filters": [
{"operator": "contains", "key": "env"},
{"operator": "contains", "value": "prod"}
],
"operator": "or"
},
{"operator": "contains", "value": "deprecated", "negate": true}
],
"operator": "and"
}
}
```
""",
**default_config,
)(inject_session_dependency(pipeline_run_service.search))
router.get("/api/pipeline_runs/{id}", tags=["pipelineRuns"], **default_config)(
inject_session_dependency(pipeline_run_service.get)
)
Expand Down
Loading