Skip to content

fix: skip eval invocations without user content to prevent ValidationError#6134

Open
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/eval-user-content-validation-error
Open

fix: skip eval invocations without user content to prevent ValidationError#6134
tcconnally wants to merge 3 commits into
google:mainfrom
Perseus-Computing-LLC:fix/eval-user-content-validation-error

Conversation

@tcconnally

Copy link
Copy Markdown

Problem

When a session contains invocations without user-authored events (author != "user"), convert_events_to_eval_invocations() creates an Invocation with an empty Content(parts=[]). This causes a Pydantic ValidationError because Invocation.user_content is typed as genai_types.Content and empty Content triggers validation failure.

Error when calling POST /apps/{app_name}/eval-sets/{eval_set_id}/add-session:

ValidationError on Invocation.user_content

Fix

Skip invocations entirely when no user event is found. Evaluations without user input are not meaningful, so filtering them out is both a correctness fix and an efficiency improvement.

Before

for invocation_id, events in events_by_invocation_id.items():
    user_content = Content(parts=[])
    ...
    invocations.append(Invocation(user_content=user_content, ...))

After

for invocation_id, events in events_by_invocation_id.items():
    user_content = Content(parts=[])
    ...
    if not user_content.parts:  # No user event found
        continue
    invocations.append(Invocation(user_content=user_content, ...))

Closes #3760

…Error

When a session contains invocations without user-authored events,
convert_events_to_eval_invocations() created an Invocation with an
empty Content(parts=[]). This caused a Pydantic ValidationError because
Invocation.user_content is typed as genai_types.Content (non-optional)
and an empty Content with no parts triggered validation failure.

Fix: skip invocations entirely when no user event is found in the
event list, since evaluations without user input are not meaningful.

Closes google#3760
@rohityan rohityan self-assigned this Jun 15, 2026
@haranrk haranrk assigned haranrk and unassigned rohityan Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Evaluation Invocation.user_content ValidationError when session has invocations without user events

3 participants