Skip to content

feat: nightly hardening - config validation guardrails#31

Open
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260519-config-validation-guards
Open

feat: nightly hardening - config validation guardrails#31
mouse-value-add wants to merge 1 commit into
brainsparker:mainfrom
mouse-value-add:chore/nightly-hardening-20260519-config-validation-guards

Conversation

@mouse-value-add

Copy link
Copy Markdown
Contributor

Problem\nPromptLens accepts several invalid run-config states (for example empty model list, invalid output formats, and out-of-range execution/model values). These fail later at runtime with weaker error messages and less predictable behavior.\n\n## Approach\n- Added Pydantic validators in to enforce:\n - between 0.0 and 2.0\n - > 0\n - > 0\n - >= 0\n - >= 0\n - > 0\n - must be non-empty and within \n- Normalized output formats to lowercase to avoid case-sensitive config footguns\n- Added model-level validation to require at least one model\n- Added focused tests in \n\n## Verification\n- Added new unit tests covering invalid bounds, unsupported formats, normalization behavior, and empty models\n- Note: test execution was attempted, but runtime policy blocked local ============================= test session starts ==============================

platform darwin -- Python 3.9.6, pytest-8.4.2, pluggy-1.6.0 -- /Library/Developer/CommandLineTools/usr/bin/python3
cachedir: .pytest_cache
rootdir: /Users/mouse/.openclaw/workspace/nightly/PromptLens
configfile: pyproject.toml
testpaths: tests
plugins: anyio-4.12.1, asyncio-1.2.0, cov-7.1.0
asyncio: mode=strict, debug=False, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collecting ... collected 14 items

tests/test_cli_hardening.py::test_remove_path_if_exists_removes_file PASSED [ 7%]
tests/test_cli_hardening.py::test_remove_path_if_exists_removes_directory PASSED [ 14%]
tests/test_cli_hardening.py::test_remove_path_if_exists_removes_symlink PASSED [ 21%]
tests/test_config_validation_hardening.py::test_model_config_temperature_bounds PASSED [ 28%]
tests/test_config_validation_hardening.py::test_execution_config_parallel_requests_must_be_positive PASSED [ 35%]
tests/test_config_validation_hardening.py::test_output_formats_are_normalized_to_lowercase PASSED [ 42%]
tests/test_config_validation_hardening.py::test_output_formats_reject_unsupported_values PASSED [ 50%]
tests/test_config_validation_hardening.py::test_run_config_requires_at_least_one_model PASSED [ 57%]
tests/test_http_provider_response_parsing.py::test_extract_content_from_ollama_response PASSED [ 64%]
tests/test_http_provider_response_parsing.py::test_extract_content_from_openai_chat_string_content PASSED [ 71%]
tests/test_http_provider_response_parsing.py::test_extract_content_from_openai_chat_content_parts PASSED [ 78%]
tests/test_http_provider_response_parsing.py::test_extract_content_returns_empty_for_unknown_shape PASSED [ 85%]
tests/test_loader_top_level_validation.py::test_json_loader_rejects_non_object_top_level PASSED [ 92%]
tests/test_loader_top_level_validation.py::test_yaml_loader_rejects_non_mapping_top_level PASSED [100%]

=============================== warnings summary ===============================
promptlens/models/tools.py:14
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/tools.py:14: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class ToolParameter(BaseModel):

promptlens/models/test_case.py:10
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/test_case.py:10: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class TestCase(BaseModel):

promptlens/models/test_case.py:66
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/test_case.py:66: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class GoldenSet(BaseModel):

promptlens/models/config.py:154
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/models/config.py:154: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class RunConfig(BaseModel):

../../../../Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/api_core/_python_version_support.py:242: FutureWarning: You are using a non-supported Python version (3.9.6). Google will not post any further updates to google.api_core supporting this Python version. Please upgrade to the latest Python version, or at least Python 3.10, and then update google.api_core.
warnings.warn(message, FutureWarning)

../../../../Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35
/Users/mouse/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: urllib3/urllib3#3020
warnings.warn(

../../../../Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/auth/init.py:54: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)

../../../../Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40
/Users/mouse/Library/Python/3.9/lib/python/site-packages/google/oauth2/init.py:40: FutureWarning: You are using a Python version 3.9 past its end of life. Google will update google-auth with critical bug fixes on a best-effort basis, but not with any other fixes or features. Please upgrade your Python version, and then update google-auth.
warnings.warn(eol_message.format("3.9"), FutureWarning)

promptlens/providers/google.py:8
/Users/mouse/.openclaw/workspace/nightly/PromptLens/promptlens/providers/google.py:8: FutureWarning:

All support for the google.generativeai package has ended. It will no longer be receiving
updates or bug fixes. Please switch to the google.genai package as soon as possible.
See README for more details:

https://github.com/google-gemini/deprecated-generative-ai-python/blob/main/README.md

import google.generativeai as genai

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform darwin, python 3.9.6-final-0 ________________

Name Stmts Miss Cover

promptlens/init.py 3 0 100%
promptlens/main.py 3 3 0%
promptlens/cli.py 169 120 29%
promptlens/exporters/init.py 6 0 100%
promptlens/exporters/base.py 15 5 67%
promptlens/exporters/csv_exporter.py 26 14 46%
promptlens/exporters/html_exporter.py 38 25 34%
promptlens/exporters/json_exporter.py 16 6 62%
promptlens/exporters/markdown_exporter.py 77 66 14%
promptlens/judges/init.py 3 0 100%
promptlens/judges/base.py 15 3 80%
promptlens/judges/llm_judge.py 71 50 30%
promptlens/judges/parser.py 107 97 9%
promptlens/judges/prompts.py 30 25 17%
promptlens/loaders/init.py 4 0 100%
promptlens/loaders/base.py 14 3 79%
promptlens/loaders/json_loader.py 23 8 65%
promptlens/loaders/yaml_loader.py 34 17 50%
promptlens/models/init.py 4 0 100%
promptlens/models/config.py 93 16 83%
promptlens/models/result.py 60 13 78%
promptlens/models/test_case.py 25 0 100%
promptlens/models/tools.py 74 34 54%
promptlens/providers/init.py 3 0 100%
promptlens/providers/anthropic.py 52 32 38%
promptlens/providers/base.py 23 6 74%
promptlens/providers/factory.py 21 10 52%
promptlens/providers/google.py 47 28 40%
promptlens/providers/http.py 67 26 61%
promptlens/providers/openai.py 57 36 37%
promptlens/providers/you.py 57 39 32%
promptlens/runners/init.py 2 0 100%
promptlens/runners/runner.py 96 74 23%
promptlens/utils/init.py 1 0 100%
promptlens/utils/cost.py 15 11 27%
promptlens/utils/diff.py 25 25 0%
promptlens/utils/retry.py 21 15 29%
promptlens/utils/timing.py 24 13 46%

TOTAL 1421 820 42%
Coverage HTML written to dir htmlcov
======================== 14 passed, 9 warnings in 1.35s ======================== invocation in this environment\n\n## Risks\n- This is stricter validation and may reject previously accepted but invalid configs\n- Users relying on unsupported format strings will now get early failures\n\n## Rollback plan\n- Revert this PR commit to restore previous permissive behavior\n- If needed, selectively relax individual validators while preserving the test suite for explicit behavior

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant