feat(eval): expose user_simulator_config in generate_responses by primenko-v · Pull Request #5733 · google/adk-python

primenko-v · 2026-05-17T16:34:29Z

Link to Issue or Description of Change

1. Link to an existing issue (if applicable):

N/A

2. Or, if no issue exists, describe the change:

Problem:
EvaluationGenerator.generate_responses constructs a UserSimulatorProvider() with no arguments, so the LLM-backed path always runs with the default BaseUserSimulatorConfig. There is no way for a caller to override the user-simulation model, max-allowed invocations, or custom instructions when driving multi-turn conversations through LlmBackedUserSimulator.

Solution:
Add an optional user_simulator_config parameter to generate_responses and forward it to UserSimulatorProvider(...). Callers can now pass an LlmBackedUserSimulatorConfig to customize the LLM-backed simulator.

The behavior is backward compatible:

When the argument is omitted, UserSimulatorProvider falls back to BaseUserSimulatorConfig() exactly as before.
Static eval cases are unaffected: the config is ignored by StaticUserSimulator.

Testing Plan

Unit Tests:

I have added or updated unit tests for my change.
All unit tests pass locally.

A unit test for the proposed change was added to tests/unittests/evaluation/test_evaluation_generator.py: TestGenerateResponses::test_generate_responses_forwards_llm_backed_user_simulator_config

All tests pass:

> uv run pytest tests/unittests/ -rs
...
================================== short test summary info ================================== 
SKIPPED [1] tests/unittests/integrations/crewai/test_crewai_tool.py:20: Requires Python 3.10+
================  5770 passed, 1 skipped, 2358 warnings in 129.40s (0:02:09) ================

The skipped test is not related to this change — it skips on main as well.

Manual End-to-End (E2E) Tests:

A reference setup lives at https://github.com/primenko-v/adk-x-mlflow (tag pr-demo/user-simulator-config).

It loads an LlmBackedUserSimulatorConfig from YAML and forwards it to EvaluationGenerator.generate_responses via the new user_simulator_config parameter — see src/mlflow_adk/simulate.py.

To reproduce (requires GOOGLE_CLOUD_PROJECT and ADC via gcloud auth application-default login):

git clone --recurse-submodules --branch pr-demo/user-simulator-config \
    https://github.com/primenko-v/adk-x-mlflow.git
cd adk-x-mlflow
cp .env.example .env  # fill in GOOGLE_CLOUD_PROJECT
uv sync
uv run python -m mlflow_adk.simulate --no-mlflow --output-traces traces.jsonl

Checklist

I have read the CONTRIBUTING.md document.
I have performed a self-review of my own code.
I have commented my code, particularly in hard-to-understand areas.
I have added tests that prove my fix is effective or that my feature works.
New and existing unit tests pass locally with my changes.
I have manually tested my changes end-to-end.
Any dependent changes have been merged and published in downstream modules.

rohityan · 2026-05-20T23:28:51Z

Hi @primenko-v , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Please fix formatting errors by running autoformat.sh.

primenko-v · 2026-05-21T06:38:16Z

Hi @primenko-v , Thank you for your contribution! We appreciate you taking the time to submit this pull request. Please fix formatting errors by running autoformat.sh.

Thank you for the review @rohityan ! I installed and ran the pre-commit hook, as the autoformat.sh doesn't seem to exist anymore.

primenko-v · 2026-05-28T22:07:08Z

@rohityan let me know if there's anything else needed!

rohityan · 2026-05-29T18:53:20Z

Hi @ankursharmas , can you please review this.

ankursharmas · 2026-06-02T19:03:55Z

+class TestGenerateResponses:
+  """Test cases for EvaluationGenerator.generate_responses method."""


We don't need this new class. The newly added test case can be a part of the existing class TestGenerateInferencesFromRootAgent

Thank you for the feedback @ankursharmas !

The newly added test exercises generate_responses rather than _generate_inferences_from_root_agent. It seems to me that test_evaluation_generator.py mostly follows a one-class-per-method convention, where each test class targets a single method. For example:

Test class Method under test

TestConvertEventsToEvalInvocation convert_events_to_eval_invocations

TestGetAppDetailsByInvocationId _get_app_details_by_invocation_id

TestGenerateInferencesForSingleUserInvocation _generate_inferences_for_single_user_invocation

TestGenerateInferencesForSingleUserInvocationLive _generate_inferences_for_single_user_invocation_live

Following that, wouldn't a separate TestGenerateResponses class for generate_responses fit better here?

While looking at this I also noticed that test_generates_inferences_with_user_simulator_live lives in TestGenerateInferencesFromRootAgent even though it tests _generate_inferences_from_root_agent_live, so the convention is already a bit mixed there.

On a separate note: there was a careless merge on my side which I have now fixed.

Ah makes sense! Thank you for pointing that out.

@ankursharmas do you consider the conversation resolved?

@ankursharmas this PR is currently blocked by this open thread. Let me know if you need any further changes from my side; otherwise, could you please resolve the conversation?

primenko-v · 2026-06-04T07:28:26Z

I can see that the pre-commit fails on the formatting now, and surprisingly not on the files changed by this PR:

uv run pre-commit run --all-files
check yaml...............................................................Passed
fix end of files.........................................................Failed
- hook id: end-of-file-fixer
- exit code: 1
- files were modified by this hook

Fixing tests/unittests/flows/llm_flows/test_base_llm_flow.py

[Manually removed trim trailing whitespace errors on src/google/adk/cli/browser/*.js files from the output]

pyproject-fmt............................................................Passed
isort....................................................................Failed
- hook id: isort
- files were modified by this hook

Fixing /home/prmnk/proj/adk-python-main/src/google/adk/flows/llm_flows/base_llm_flow.py
Fixing /home/prmnk/proj/adk-python-main/tests/unittests/flows/llm_flows/test_base_llm_flow.py

pyink....................................................................Failed
- hook id: pyink
- files were modified by this hook

reformatted src/google/adk/flows/llm_flows/basic.py
reformatted src/google/adk/models/gemini_llm_connection.py
reformatted tests/unittests/flows/llm_flows/test_base_llm_flow.py
reformatted tests/unittests/models/test_gemini_llm_connection.py

All done! ✨ 🍰 ✨
4 files reformatted, 1510 files left unchanged.

I have tried syncing my branch with the latest main, but the formatting issue is still there.

It looks like the reformatted files were brought by this commit that was pushed to main directly. The thing is, pre-commit/action runs --all-files, so any PR that touches .py files now inherits this failure.

I have opened #5962 to fix the formatting introduced by that commit, and #5963 to try to prevent these formatting issues.

primenko-v · 2026-06-09T15:20:25Z

The formatting issues were fixed on main and merged into this branch.

Merge #5733 ### Link to Issue or Description of Change **1. Link to an existing issue (if applicable):** N/A **2. Or, if no issue exists, describe the change:** **Problem:** `EvaluationGenerator.generate_responses` constructs a `UserSimulatorProvider()` with no arguments, so the LLM-backed path always runs with the default `BaseUserSimulatorConfig`. There is no way for a caller to override the user-simulation model, max-allowed invocations, or custom instructions when driving multi-turn conversations through `LlmBackedUserSimulator`. **Solution:** Add an optional `user_simulator_config` parameter to `generate_responses` and forward it to `UserSimulatorProvider(...)`. Callers can now pass an `LlmBackedUserSimulatorConfig` to customize the LLM-backed simulator. The behavior is backward compatible: - When the argument is omitted, `UserSimulatorProvider` falls back to `BaseUserSimulatorConfig()` exactly as before. - Static eval cases are unaffected: the config is ignored by `StaticUserSimulator`. ### Testing Plan **Unit Tests:** - [x] I have added or updated unit tests for my change. - [x] All unit tests pass locally. A unit test for the proposed change was added to `tests/unittests/evaluation/test_evaluation_generator.py`: `TestGenerateResponses::test_generate_responses_forwards_llm_backed_user_simulator_config` All tests pass: ``` > uv run pytest tests/unittests/ -rs ... ================================== short test summary info ================================== SKIPPED [1] tests/unittests/integrations/crewai/test_crewai_tool.py:20: Requires Python 3.10+ ================ 5770 passed, 1 skipped, 2358 warnings in 129.40s (0:02:09) ================ ``` The skipped test is not related to this change — it skips on `main` as well. **Manual End-to-End (E2E) Tests:** A reference setup lives at https://github.com/primenko-v/adk-x-mlflow (tag `pr-demo/user-simulator-config`). It loads an `LlmBackedUserSimulatorConfig` from YAML and forwards it to `EvaluationGenerator.generate_responses` via the new `user_simulator_config` parameter — see [`src/mlflow_adk/simulate.py`](https://github.com/primenko-v/adk-x-mlflow/blob/pr-demo/user-simulator-config/src/mlflow_adk/simulate.py#L74-L79). To reproduce (requires GOOGLE_CLOUD_PROJECT and ADC via `gcloud auth application-default login`): ```bash git clone --recurse-submodules --branch pr-demo/user-simulator-config \ https://github.com/primenko-v/adk-x-mlflow.git cd adk-x-mlflow cp .env.example .env # fill in GOOGLE_CLOUD_PROJECT uv sync uv run python -m mlflow_adk.simulate --no-mlflow --output-traces traces.jsonl ``` ### Checklist - [x] I have read the [CONTRIBUTING.md](https://github.com/google/adk-python/blob/main/CONTRIBUTING.md) document. - [x] I have performed a self-review of my own code. - [x] I have commented my code, particularly in hard-to-understand areas. - [x] I have added tests that prove my fix is effective or that my feature works. - [x] New and existing unit tests pass locally with my changes. - [x] I have manually tested my changes end-to-end. - [x] Any dependent changes have been merged and published in downstream modules. Co-authored-by: Ankur Sharma <ankusharma@google.com> COPYBARA_INTEGRATE_REVIEW=#5733 from primenko-v:propagate-user-simulator-config 24209b6 PiperOrigin-RevId: 933503403

adk-bot · 2026-06-17T05:41:49Z

Thank you @primenko-v for your contribution! 🎉

Your changes have been successfully imported and merged via Copybara in commit e7a673c.

Closing this PR as the changes are now in the main branch.

primenko-v added 6 commits April 29, 2026 16:38

Configure UserSimulatorProvider

ffc4361

Merge 'main' into propagate-user-simulator-config

e5892f8

Test UserSimulatorProvider configuration

caf69f2

Add docstring for user_simulator_config param

780a640

Merge branch 'main' into propagate-user-simulator-config

6945d43

Remove the stale evaluation_generator comment

2301b07

rohityan self-assigned this May 18, 2026

rohityan added 2 commits May 18, 2026 11:40

Merge branch 'main' into propagate-user-simulator-config

c50bc32

Merge branch 'main' into propagate-user-simulator-config

9a4526f

rohityan added eval [Component] This issue is related to evaluation request clarification [Status] The maintainer need clarification or more information from the author labels May 20, 2026

Fix formatting

1248a64

Merge branch 'main' into propagate-user-simulator-config

17e6dfb

rohityan added needs review [Status] The PR/issue is awaiting review from the maintainer and removed request clarification [Status] The maintainer need clarification or more information from the author labels May 29, 2026

rohityan requested a review from ankursharmas May 29, 2026 18:53

ankursharmas suggested changes Jun 2, 2026

View reviewed changes

primenko-v added 3 commits June 3, 2026 18:40

Merge 'main' into propagate-user-simulator-config

109a0fd

Clean up test_evaluation_generator

ad00b9b

Add back TestGenerateResponses

942b263

ankursharmas approved these changes Jun 3, 2026

View reviewed changes

ankursharmas requested a review from GWeale June 3, 2026 20:13

GWeale approved these changes Jun 3, 2026

View reviewed changes

ankursharmas self-assigned this Jun 4, 2026

Merge branch 'main' into propagate-user-simulator-config

dcfbbe1

Merge branch 'google:main' into propagate-user-simulator-config

24209b6

adk-bot added the merged [Status] This PR is merged label Jun 17, 2026

adk-bot closed this Jun 17, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(eval): expose user_simulator_config in generate_responses#5733

feat(eval): expose user_simulator_config in generate_responses#5733
primenko-v wants to merge 15 commits into
google:mainfrom
primenko-v:propagate-user-simulator-config

primenko-v commented May 17, 2026

Uh oh!

rohityan commented May 20, 2026

Uh oh!

primenko-v commented May 21, 2026

Uh oh!

primenko-v commented May 28, 2026

Uh oh!

rohityan commented May 29, 2026

Uh oh!

ankursharmas Jun 2, 2026

Uh oh!

primenko-v Jun 3, 2026

Uh oh!

ankursharmas Jun 3, 2026

Uh oh!

primenko-v Jun 9, 2026

Uh oh!

primenko-v Jun 12, 2026

Uh oh!

primenko-v commented Jun 4, 2026 •

edited

Loading

Uh oh!

primenko-v commented Jun 9, 2026

Uh oh!

adk-bot commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		class TestGenerateResponses:
		"""Test cases for EvaluationGenerator.generate_responses method."""

Test class	Method under test
`TestConvertEventsToEvalInvocation`	`convert_events_to_eval_invocations`
`TestGetAppDetailsByInvocationId`	`_get_app_details_by_invocation_id`
`TestGenerateInferencesForSingleUserInvocation`	`_generate_inferences_for_single_user_invocation`
`TestGenerateInferencesForSingleUserInvocationLive`	`_generate_inferences_for_single_user_invocation_live`

Conversation

primenko-v commented May 17, 2026

Link to Issue or Description of Change

Testing Plan

Checklist

Uh oh!

rohityan commented May 20, 2026

Uh oh!

primenko-v commented May 21, 2026

Uh oh!

primenko-v commented May 28, 2026

Uh oh!

rohityan commented May 29, 2026

Uh oh!

ankursharmas Jun 2, 2026

Choose a reason for hiding this comment

Uh oh!

primenko-v Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

ankursharmas Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

primenko-v Jun 9, 2026

Choose a reason for hiding this comment

Uh oh!

primenko-v Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

primenko-v commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

primenko-v commented Jun 9, 2026

Uh oh!

adk-bot commented Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

primenko-v commented Jun 4, 2026 •

edited

Loading