feat: nightly hardening - config validation guardrails#31
Open
mouse-value-add wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem\nPromptLens accepts several invalid run-config states (for example empty model list, invalid output formats, and out-of-range execution/model values). These fail later at runtime with weaker error messages and less predictable behavior.\n\n## Approach\n- Added Pydantic validators in to enforce:\n - between 0.0 and 2.0\n - > 0\n - > 0\n - >= 0\n - >= 0\n - > 0\n - must be non-empty and within \n- Normalized output formats to lowercase to avoid case-sensitive config footguns\n- Added model-level validation to require at least one model\n- Added focused tests in \n\n## Verification\n- Added new unit tests covering invalid bounds, unsupported formats, normalization behavior, and empty models\n- Note: test execution was attempted, but runtime policy blocked local ============================= test session starts ==============================
platform darwin -- Python 3.9.6, pytest-8.4.2, pluggy-1.6.0 -- /Library/Developer/CommandLineTools/usr/bin/python3
cachedir: .pytest_cache
rootdir: /Users/mouse/.openclaw/workspace/nightly/PromptLens
configfile: pyproject.toml
testpaths: tests
plugins: anyio-4.12.1, asyncio-1.2.0, cov-7.1.0
asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 14 items
tests/test_cli_hardening.py::test_remove_path_if_exists_removes_file PASSED [ 7%]
tests/test_cli_hardening.py::test_remove_path_if_exists_removes_directory PASSED [ 14%]
tests/test_cli_hardening.py::test_remove_path_if_exists_removes_symlink PASSED [ 21%]
tests/test_config_validation_hardening.py::test_model_config_temperature_bounds PASSED [ 28%]
tests/test_config_validation_hardening.py::test_execution_config_parallel_requests_must_be_positive PASSED [ 35%]
tests/test_config_validation_hardening.py::test_output_formats_are_normalized_to_lowercase PASSED [ 42%]
tests/test_config_validation_hardening.py::test_output_formats_reject_unsupported_values PASSED [ 50%]
tests/test_config_validation_hardening.py::test_run_config_requires_at_least_one_model PASSED [ 57%]
tests/test_http_provider_response_parsing.py::test_extract_content_from_ollama_response PASSED [ 64%]
tests/test_http_provider_response_parsing.py::test_extract_content_from_openai_chat_string_content PASSED [ 71%]
tests/test_http_provider_response_parsing.py::test_extract_content_from_openai_chat_content_parts PASSED [ 78%]
tests/test_http_provider_response_parsing.py::test_extract_content_returns_empty_for_unknown_shape PASSED [ 85%]
tests/test_loader_top_level_validation.py::test_json_loader_rejects_non_object_top_level PASSED [ 92%]
tests/test_loader_top_level_validation.py::test_yaml_loader_rejects_non_mapping_top_level PASSED [100%]
=============================== warnings summary ===============================
promptlens/models/tools.py:14
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/tools.py:14: PydanticDeprecatedSince20: Support for class-based
configis deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/class ToolParameter(BaseModel):
promptlens/models/test_case.py:10
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/test_case.py:10: PydanticDeprecatedSince20: Support for class-based
configis deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/class TestCase(BaseModel):
promptlens/models/test_case.py:66
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/test_case.py:66: PydanticDeprecatedSince20: Support for class-based
configis deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/class GoldenSet(BaseModel):
promptlens/models/config.py:154
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/config.py:154: PydanticDeprecatedSince20: Support for class-based
configis deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/class RunConfig(BaseModel):
../../../../Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242: FutureWarning: You are using a non-supported Python version (3.9.6). Google will not post any further updates to google.api_core supporting this Python version. Please upgrade to the latest Python version, or at least Python 3.10, and then update google.api_core.
warnings.warn(message, FutureWarning)
../../../../Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35
/Users/mouse/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: urllib3/urllib3#3020
warnings.warn(
../../../../Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)
../../../../Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)
promptlens/providers/google.py:8
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/providers/google.py:8: FutureWarning:
All support for the
google.generativeaipackage has ended. It will no longer be receivingupdates or bug fixes. Please switch to the
google.genaipackage as soon as possible.See README for more details:
https://github.com/google-gemini/deprecated-generative-ai-python/blob/main/README.md
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.9.6-final-0 ________________
Name Stmts Miss Cover
promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 120 29%
promptlens/exporters/init.py 6 0 100%
promptlens/exporters/base.py 15 5 67%
promptlens/exporters/csv_exporter.py 26 14 46%
promptlens/exporters/html_exporter.py 38 25 34%
promptlens/exporters/json_exporter.py 16 6 62%
promptlens/exporters/markdown_exporter.py 77 66 14%
promptlens/judges/init.py 3 0 100%
promptlens/judges/base.py 15 3 80%
promptlens/judges/llm_judge.py 71 50 30%
promptlens/judges/parser.py 107 97 9%
promptlens/judges/prompts.py 30 25 17%
promptlens/loaders/init.py 4 0 100%
promptlens/loaders/base.py 14 3 79%
promptlens/loaders/json_loader.py 23 8 65%
promptlens/loaders/yaml_loader.py 34 17 50%
promptlens/models/init.py 4 0 100%
promptlens/models/config.py 93 16 83%
promptlens/models/result.py 60 13 78%
promptlens/models/test_case.py 25 0 100%
promptlens/models/tools.py 74 34 54%
promptlens/providers/init.py 3 0 100%
promptlens/providers/anthropic.py 52 32 38%
promptlens/providers/base.py 23 6 74%
promptlens/providers/factory.py 21 10 52%
promptlens/providers/google.py 47 28 40%
promptlens/providers/http.py 67 26 61%
promptlens/providers/openai.py 57 36 37%
promptlens/providers/you.py 57 39 32%
promptlens/runners/init.py 2 0 100%
promptlens/runners/runner.py 96 74 23%
promptlens/utils/init.py 1 0 100%
promptlens/utils/cost.py 15 11 27%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 21 15 29%
promptlens/utils/timing.py 24 13 46%
TOTAL 1421 820 42%
Coverage HTML written to dir htmlcov
======================== 14 passed, 9 warnings in 1.35s ======================== invocation in this environment\n\n## Risks\n- This is stricter validation and may reject previously accepted but invalid configs\n- Users relying on unsupported format strings will now get early failures\n\n## Rollback plan\n- Revert this PR commit to restore previous permissive behavior\n- If needed, selectively relax individual validators while preserving the test suite for explicit behavior