Skip to content

feat(feedback): add hydradb feedback, and surface the request id it needs - #37

Merged
abhishekt-hydra merged 2 commits into
mainfrom
gunmay/feedback-command
Sep 11, 2026
Merged

abhishekt-hydra merged 2 commits into
mainfrom
gunmay/feedback-command

Conversation

@gunmay-hydradb

@gunmay-hydradb gunmay-hydradb commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

What

hydradb feedback REQUEST_ID — report whether a query's results were actually useful.

It correlates on one key, the request_id from that query's response meta. Nothing else about the original query is re-sent, so nothing has to be trusted from the client.

hydradb query "contract terms"
#   … results …
#   request_id: 8f1c0e8a-…  ·  rate it: hydradb feedback 8f1c0e8a-… --feedback "..."

hydradb feedback 8f1c0e8a-… -f "returned the 2023 policy, not the current one" --rating negative

This is two changes, because a feedback command alone would have shipped unusable

The request id was unreachable from the CLI. _unwrap returns .data and drops meta, so a successful query discarded its own request id before any caller saw it — you could run a query and then have nothing to give feedback on.

So context.query now carries the id into its payload. That is additive — /query's data has no request_id of its own — so the documented --output json shape gains a key and loses none:

hydradb --output json query "contract terms" | jq -r .request_id

The human output prints a copy-pasteable line, including on an empty result. A query that found nothing is the case most worth reporting, and the one with no chunk ids to fall back on.

FeedbackSubmitRequestFeedback.model_fields    # ['feedback']
FeedbackSubmitRequestGroundTruth.model_fields # ['ground_truth']

It does work today, and I verified that against staging — the models are extra="allow", so undeclared fields ride along to the wire. But that means every field this endpoint needs travels as an accident of pydantic config, at two levels of nesting, and sending an answer together with source ids means picking one branch and smuggling the other past it. A generated model that cannot name its own required field is equivalent to no model.

When the spec is fixed so the branches keep their siblings, this is a one-file change back to the SDK.

Note: the TS SDK has the same defect but strips unknown fields instead of allowing them, so hydradb-mcp had no choice at all there. Same spec bug, two different failure modes.

Testing

11 new tests, including that request_id actually reaches the wire and that an answer and source ids travel together — the two things the SDK model cannot do.

Verified end to end against staging: query → request id surfaced → feedback recorded, both the prose and ground-truth paths, with real feedback_ids returned.

  • pytest — 342 passed
  • pytest conformance — 33 passed
  • make lint — clean

Related

…eeds

POST /feedback records whether a query's results were actually useful. It
correlates on ONE key -- the request_id from that query's response meta --
and nothing else about the original query is re-sent, so nothing has to be
trusted from the client.

That key was unreachable from the CLI. `_unwrap` returns `.data` and drops
`meta`, so a successful query discarded its own request id before any
caller saw it. A feedback command alone would therefore have shipped
unusable: you could run a query and then have nothing to give feedback ON.

So this is two changes:

1. `context.query` carries the request id into its payload. Additive --
   /query's data has no `request_id` of its own -- so the documented
   `--output json` shape gains a key and loses none, and piping it to the
   feedback command works. The human output prints a copy-pasteable line,
   including on an EMPTY result: a query that found nothing is the case
   most worth reporting, and the one with no chunk ids to fall back on.

2. `hydradb feedback REQUEST_ID` itself, plus a _Feedback resource.

Hand-rolled rather than routed through sdk.feedback.submit (CONTRACT §2
rule 7). The SDK HAS the resource, but its generated model cannot express a
valid request: submit takes an undiscriminated union of {feedback} and
{ground_truth}, and ground_truth is itself a union of {answer} and
{source_ids}. Both unions dropped every field their branches share,
request_id included -- the correlation key the endpoint exists for is on
neither model.

It does work today, and I verified that against staging: the models are
extra="allow", so undeclared fields ride along to the wire. But that means
every field this endpoint needs travels as an accident of pydantic config,
at two levels of nesting, and sending an answer together with source ids
means picking one branch and smuggling the other past it. A generated model
that cannot name its own required field is equivalent to no model. When the
spec is fixed, this is a one-file change back to the SDK.

Details worth knowing:

- Ground-truth source ids are trimmed and de-duplicated, because they are
  scored: the same document listed twice would weight one piece of evidence
  as two. The 100-id cap is checked AFTER that, since the server also
  de-duplicates first -- 150 ids collapsing to 80 is a request it accepts,
  and refusing it locally would be worse than the round trip.
- A submission with neither prose nor ground truth is refused HERE. The
  server refuses it too, but after a round trip.
- `--source` defaults to "user", not the "agent" the MCP client sends. A
  person at a terminal is a user; MCP's caller is always a model. The two
  populations are separated at write time, so the default has to follow who
  is actually calling. `--source agent` is there for scripted runs.
- `recorded: false` is reported as "accepted but NOT durably stored".
  Calling it success would tell an eval run it was captured when it was not.

11 tests, and verified end to end against staging: query -> request id
surfaced -> feedback recorded, prose and ground-truth paths both.

342 tests + 33 conformance pass, ruff clean.

Signed-off-by: gun29may <gunmayjhingran29@gmail.com>
@greptile-apps

greptile-apps Bot commented Sep 11, 2026 •

Copy link
Copy Markdown

Greptile Summary

This PR adds a complete feedback workflow to the CLI and exposes each query’s request ID so users can correlate feedback with an existing query.

  • Registers hydradb feedback with prose, rating, ground-truth, source, and scope options.
  • Adds a raw authenticated POST /feedback client resource with request normalization, source-ID deduplication, limits, and consistent response/error handling.
  • Preserves query metadata by adding meta.request_id to the query payload and displays a copyable feedback hint, including for empty results.
  • Adds wrapper and command-level tests for request encoding, validation, result rendering, and request-ID propagation.
  • Greptile automatically discovered a related ticket that helped explain the purpose of this PR: provide CLI and MCP feedback support while surfacing the query request ID needed for correlation.

Confidence Score: 5/5

The PR appears safe to merge; the previous findings are fully addressed and no new actionable defects were identified.

Source validation now rejects unsupported values before network access, and the new command-level tests cover the previously untested public feedback surface. The additional validation and tests introduced since the previous review preserve the shared CLI execution, output, scope, and error contracts.

Important Files Changed

Filename Overview
src/hydradb_cli/hydra/client.py Adds request-ID propagation and a normalized raw feedback resource with authentication, scope, validation, and response handling.
src/hydradb_cli/commands/_impl.py Adds feedback validation, execution, result rendering, and query feedback hints; the previous source-validation concern is fixed.
src/hydradb_cli/commands/canonical.py Defines the public feedback command options and maps them through the shared command implementation.
src/hydradb_cli/main.py Registers the new top-level feedback command.
tests/test_cli_commands.py Adds command-level coverage that resolves the previous testing finding and protects validation, mapping, output, and exit behavior.
tests/test_wrapper.py Covers feedback wire payloads, request normalization, source-ID handling, defaults, limits, and query request-ID propagation.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    User[CLI user] --> Query[hydradb query]
    Query --> Context[HydraDB context client]
    Context --> API[HydraDB API]
    API --> Envelope[Query data plus meta.request_id]
    Envelope --> Payload[Query payload with request_id]
    Payload --> User
    User --> Feedback[hydradb feedback REQUEST_ID]
    Feedback --> Validation[Validate signal, rating, source, and scope]
    Validation --> FeedbackClient[HydraDB feedback client]
    FeedbackClient --> Post[POST /feedback]
    Post --> Result[Recorded or not-durably-stored result]
    Result --> User
Loading

Reviews (2): Last reviewed commit: "fix(feedback): validate --source, and re..." | Re-trigger Greptile

Comment thread src/hydradb_cli/commands/canonical.py
Comment thread tests/test_wrapper.py
…the network

Two Greptile P2s on #37, both correct.

1. --source forwarded any string. --rating was validated locally but its
   sibling was not, so `--source agnet` cost a round trip and a server
   rejection to learn what a local message could say immediately.

2. The command itself was untested. The wrapper tests call feedback.submit()
   directly, which left registration, option-to-wrapper mapping, both
   renderings and the exit code unproven -- and unlike the dashboard PR,
   this repo already has the harness for it (108 CliRunner invocations in
   test_cli_commands.py), so there was no reason not to use it.

Eight command-level tests cover what the wrapper tests could not: the flags
are registered, every option reaches the wrapper, an invalid rating or
source is refused BEFORE the call goes out, `recorded: false` does not
render as success, and --output json is the payload.

Writing them surfaced a third thing, found by driving the real CLI: an
empty submission reported

  ✗ Error: Connection error: feedback needs something to record: ...

The wrapper raises HydraDBClientError(0, ...) for a local refusal, but
status 0 is this codebase's marker for a TRANSPORT failure -- errors.py
uses it only for connect/timeout/network -- and handle_api_error renders it
as "Connection error:". So a local refusal blamed the network. The
pre-existing guards avoid this by validating in the command layer
(require_tenant_id), which is why `No database specified` reads cleanly;
the feedback guards now do the same. The wrapper keeps its checks for
anyone importing it as a library, but the command layer catches first.

350 tests + 33 conformance pass, ruff clean. Re-verified end to end against
staging: every command, all four feedback shapes, and all three guards.

Signed-off-by: gun29may <gunmayjhingran29@gmail.com>
@abhishekt-hydra
abhishekt-hydra merged commit 930b511 into main Sep 11, 2026
7 checks passed
gunmay-hydradb pushed a commit that referenced this pull request Sep 11, 2026
Two conflicts, both from #37 (feedback) landing on main first.

1. src/hydradb_cli/hydra/client.py — BOTH sides independently added a
   `_raw_post` helper: this branch for the titles field, #37 for /feedback.
   The implementations are identical below the signature, so the duplicate
   collapses to one. Kept main's: `json_body: Any` rather than `dict` (which
   still accepts this branch's call sites) and the docstring naming CONTRACT
   §2 rule 7. httpx sets Content-Type from `json=`, so dropping the explicit
   header changes nothing on the wire.

2. CHANGELOG.md — both sides appended under Unreleased. Kept both entries.

Everything else merged cleanly, and both features are intact: `titles` on
context.query and the _Feedback resource, each with their own tests.

354 tests + 33 conformance pass, ruff clean.

Signed-off-by: gun29may <gunmayjhingran29@gmail.com>
gunmay-hydradb pushed a commit that referenced this pull request Sep 11, 2026
Minor, not patch: the command surface gains a capability. `hydradb query`
accepts repeatable `--title` for exact multi-title filtering, and main has
since taken `hydradb feedback` (#37) without a bump of its own, so this
release carries both. Nothing was removed or renamed.

Both places, because this package keeps the version twice -- pyproject.toml
for packaging and __init__.py for `hydradb --version` -- and nothing in the
tests or the build asserts they agree. They would drift silently, and the
one users see is the one that is easiest to forget. Verified equal after
the change, and `hydradb --version` reports 0.3.0.

Signed-off-by: gun29may <gunmayjhingran29@gmail.com>
gunmay-hydradb pushed a commit that referenced this pull request Sep 11, 2026
Patch bump covering repeatable `--title` on hydradb query, and the
`hydradb feedback` command (#37) which main took without a bump of its own.
Nothing was removed or renamed, and no existing invocation changes.

Both places, because this package keeps the version twice -- pyproject.toml
for packaging and __init__.py for `hydradb --version` -- and nothing in the
tests or the build asserts they agree. They would drift silently, and the
one users see is the one that is easiest to forget. Verified equal after
the change, and `hydradb --version` reports 0.2.1.

Signed-off-by: gun29may <gunmayjhingran29@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants