diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index 6faefd1..f08e81a 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -45,10 +45,10 @@ jobs: - name: Install dependencies run: | pip install uv - uv sync --group dev + uv sync --all-extras --group dev - name: Run benchmark tests run: | - uv run pytest -m benchmark -v + uv run pytest -m "benchmark and not (slow or live)" -v test-all: name: All Tests (With Optional Deps) @@ -86,14 +86,6 @@ jobs: run: | pip install uv uv sync --all-extras --group dev - - name: Cache benchmark data - uses: actions/cache@v4 - with: - path: | - maseval/benchmark/tau2/data/ - maseval/benchmark/macs/data/ - maseval/benchmark/macs/prompt_templates/ - key: benchmark-data-${{ hashFiles('maseval/benchmark/tau2/data_loader.py', 'maseval/benchmark/macs/data_loader.py') }} - name: Run slow tests run: | uv run pytest -m "slow and not credentialed" -v diff --git a/AGENTS.md b/AGENTS.md index abd250c..2ba2a93 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -494,3 +494,50 @@ MASEval provides a seeding system for reproducible benchmark runs. Seeds cascade - Focus on getting it right, not keeping it the same We have zero obligation to maintain backwards compatibility. If you find code messy, propose a fix. + +## Scientific Integrity + +MASEval is a scientific library. Scientific integrity is paramount. **Never introduce defaults that could silently alter benchmark behavior or experimental outcomes.** + +### The Boundary + +**Guiding principle:** If a researcher would need to report a parameter in a paper's "Experimental Setup" section, **do not invent a default for it.** + +**Acceptable (infrastructure/convenience):** `TaskQueue(limit=None)`, `Logger(verbose=False)`, `num_workers=1`, `print_results(color=True)` — these don't affect scientific results. + +**Unacceptable (experimental parameters):** Temperature, seed, model version, prompt format, simulation duration, agent limits, dataset splits, scoring functions — these alter what's being measured. + +### Reproducing Benchmarks + +When integrating external benchmarks, match the source implementation exactly. Never invent fallback values. + +```python +# BAD: Invented defaults +config = EnvironmentConfig( + duration=getattr(scenario, "duration", 86400), # Made-up fallback! +) +start_time = getattr(scenario, "start_time", None) # Hides missing attributes + +# GOOD: Pass through directly, let errors surface +config = EnvironmentConfig( + duration=scenario.duration, # Trust the source +) +start_time = scenario.start_time # AttributeError if missing + +# GOOD: Copy source defaults with documentation +# Default value copied from original_library/evaluator.py:L45 +EVAL_TEMPERATURE = 0.7 + +class Evaluator: + def run(self, temperature: Optional[float] = None): + if temperature is None: + temperature = EVAL_TEMPERATURE # From source:L45 + +# also good: +class Evaluator: + # default temperature from source:L45 + def run(self, temperature: Optional[float] = 0.7): + ... +``` + +**Rule:** Only copy defaults that exist in the source. If the original doesn't provide a default, neither should you. Always document the source file and line number. \ No newline at end of file diff --git a/BENCHMARKS.md b/BENCHMARKS.md index bcb15d0..8224456 100644 --- a/BENCHMARKS.md +++ b/BENCHMARKS.md @@ -33,11 +33,14 @@ MultiAgentBench is a comprehensive benchmark suite for evaluating multi-agent co ### Source and License -- **Original Repository:** [https://github.com/ulab-uiuc/MARBLE](https://github.com/ulab-uiuc/MARBLE) +- **Original Repository:** [https://github.com/ulab-uiuc/MARBLE](https://github.com/ulab-uiuc/MARBLE) (where the original work was done) +- **Fork Used:** [https://github.com/cemde/MARBLE](https://github.com/cemde/MARBLE) (contains bug fixes for MASEval integration) - **Paper:** [MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents](https://arxiv.org/abs/2503.01935) - **Code License:** MIT - **Data License:** MIT +> **Note**: MASEval uses a fork with bug fixes. All credit for the original work goes to the MARBLE team (Haofei Yu et al.). + --- ## 4. GAIA2 diff --git a/CHANGELOG.md b/CHANGELOG.md index c6b0989..dc7611f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- GAIA2 default agent now uses ARE's `parse_json_tool_call` and `get_offset_from_time_config_mode` directly instead of reimplementing them locally, reducing code duplication and ensuring correctness as ARE evolves (PR: #PR_NUMBER_PLACEHOLDER) + +### Fixed + +- Fixed GAIA2 default agent failing on reasoning models (o1, o3, GPT-5) that reject `stop` and `temperature` parameters. Client-side stop-token truncation (matching ARE's reference implementation) is now always applied, and `llm_args` values set to `None` are omitted from API calls (PR: #PR_NUMBER_PLACEHOLDER) +- Fixed GAIA2 multi-turn notification loop: `wait_for_notification()` no longer terminates the agent prematurely, enabling correct behavior for `time` and `adaptability` scenarios that require the agent to wait for simulation events and resume (PR: #PR_NUMBER_PLACEHOLDER) +- Fixed GAIA2 environment using wrong simulation clock: `start_time` and `time_increment_in_seconds` from the scenario are now passed to ARE's `EnvironmentConfig`, matching ARE's `scenario_runner.py`. Previously the agent environment started at epoch (1970-01-01), causing failures in time-dependent capabilities like `adaptability` and `time` (PR: #PR_NUMBER_PLACEHOLDER) +- Added `Gaia2Environment.poll_notifications()` convenience method for custom agent implementations to drain the notification queue without needing ARE-internal imports (PR: #PR_NUMBER_PLACEHOLDER) +- Fixed GAIA2 environment exposing 4 extra AgentUserInterface tools (`get_last_message_from_user`, `get_last_message_from_agent`, `get_last_unread_messages`, `get_all_messages`) that ARE filters out in `remove_aui_irrelevant_tools()`. These tools are now excluded, and `wait_for_user_response` is set to `False` on the AUI app, matching ARE's reference behavior (PR: #PR_NUMBER_PLACEHOLDER) +- Fixed GAIA2 default agent swallowing tool errors as `Observation:` messages instead of `ERROR:` messages. Errors now raise and are formatted as `ERROR:` with the tool description reminder, matching ARE's `json_action_executor.py` error propagation (PR: #PR_NUMBER_PLACEHOLDER) +- Fixed GAIA2 default agent step counter not incrementing on errors, causing duplicate `[OUTPUT OF STEP N]` numbers. Step counter now increments for both observations and errors, matching ARE's `base_agent.py` (PR: #PR_NUMBER_PLACEHOLDER) +- Fixed GAIA2 default agent not normalizing Python-style booleans (`True`/`False`) to JSON-valid (`true`/`false`) in LLM output, causing `json.loads()` failures during action parsing. Now matches ARE's LLM engine preprocessing (PR: #PR_NUMBER_PLACEHOLDER) + ### Added **Benchmarks** @@ -16,11 +31,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - `DefaultAgentGaia2Benchmark` with ReAct-style agent for direct comparison with ARE reference implementation (PR: #26) - Tool wrapper (`AREToolWrapper`) for MASEval tracing of ARE tools with simulation time tracking (PR: #26) - Data loading utilities: `load_tasks()`, `configure_model_ids()` for loading scenarios from HuggingFace (PR: #26) + - `Gaia2JudgeEngineConfig` for configuring the judge's LLM model and provider (e.g., switching from HuggingFace to OpenRouter) via `configure_model_ids(tasks, judge_engine_config=...)` (PR: #PR_NUMBER_PLACEHOLDER) - Metrics: `compute_gaia2_metrics()` for GSR (Goal Success Rate) computation by capability type (PR: #26) - - Support for 7 capability dimensions: execution, search, adaptability, time, ambiguity, agent2agent, noise (PR: #26) + - Support for 5 capability dimensions: execution, search, adaptability, time, ambiguity (PR: #26) - Added `gaia2` optional dependency: `pip install maseval[gaia2]` (PR: #26) -- MultiAgentBench Benchmark: Integration with MARBLE MultiAgentBench for evaluating multi-agent collaboration across research, bargaining, coding, and database domains (PR: #25) +- MultiAgentBench Benchmark: Integration with MARBLE MultiAgentBench for evaluating multi-agent collaboration across all 6 paper-defined domains: research, bargaining, coding, database, werewolf, and minecraft (PR: #25) - `MultiAgentBenchBenchmark` abstract base class for framework-agnostic multi-agent evaluation with seeding support for evaluators and agents (PR: #25) - `MarbleMultiAgentBenchBenchmark` for exact MARBLE reproduction mode using native MARBLE agents (note: MARBLE's internal LLM calls bypass MASEval seeding) (PR: #25) - `MultiAgentBenchEnvironment` and `MultiAgentBenchEvaluator` components (PR: #25) @@ -52,14 +68,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Composable pytest markers (`live`, `credentialed`, `slow`, `smoke`) for fine-grained test selection; default runs exclude slow, credentialed, and smoke tests (PR: #29) - Marker implication hook: `credentialed` implies `live`, so `-m "not live"` always gives a fully offline run (PR: #29) - Skip decorators (`requires_openai`, `requires_anthropic`, `requires_google`) for tests needing API keys (PR: #29) -- Data integrity tests for Tau2 and MACS benchmarks validating download pipelines, file structures, and database content (PR: #29) +- Data integrity tests for Tau2, MACS, GAIA2, and MultiAgentBench benchmarks validating download pipelines, file structures, and data content (PR: #29) - HTTP-level API contract tests for model adapters (OpenAI, Anthropic, Google GenAI, LiteLLM) using `respx` mocks — no API keys needed (PR: #29) - Live API round-trip tests for all model adapters (`-m credentialed`) (PR: #29) -- CI jobs for slow tests (with benchmark data caching) and credentialed tests (behind GitHub Environment approval) (PR: #29) +- Real-data integration tests for GAIA2 (ARE environments, tools, evaluator pipeline) and MultiAgentBench (MARBLE data loading, environments, evaluation, pipeline smoke tests) (PR: #PR_NUMBER_PLACEHOLDER) +- CI jobs for slow tests (with benchmark data caching for Tau2, MACS, GAIA2, and MultiAgentBench) and credentialed tests (behind GitHub Environment approval) (PR: #29) - Added `respx` dev dependency for HTTP-level mocking (PR: #29) ### Changed +**Benchmarks** + +- MultiAgentBench: Full alignment with MARBLE paper — all 6 domains now fully supported end-to-end. Removed `web` and `worldsimulation` (not in paper, no task data). Added `werewolf` domain with config-based task loading and LLM evaluation. Added `minecraft` domain evaluation. Fixed `bargaining` environment mapping to use `WorldSimulationEnvironment`. Fixed `WerewolfEnv` constructor handling. Removed hard-coded minecraft infrastructure block. (PR: #PR_NUMBER_PLACEHOLDER) + **Core** - Simplified seeding API: `seed_generator` parameter in setup methods is now always non-None (`SeedGenerator` instead of `Optional[SeedGenerator]`). When seeding is disabled (`seed=None`), `derive_seed()` returns `None` instead of raising an error. This eliminates all `if seed_generator is not None:` conditional checks - the same code path works whether seeding is enabled or disabled. (PR: #27) @@ -89,6 +110,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +**Benchmarks** + +- GAIA2: Fixed multi-turn scenario evaluation always failing due to missing intermediate judge calls. The evaluator now calls `judge(env)` for each intermediate turn before `judge.validate(env)`, matching ARE's intended evaluation flow. Single-turn scenarios were unaffected. (PR: #PR_NUMBER_PLACEHOLDER) +- GAIA2: Fixed data loader producing unusable tasks — `VALID_CAPABILITIES` included nonexistent HF configs (`agent2agent`, `noise`), `DEFAULT_CONFIG` referenced nonexistent `"validation"` config, `task.query` used fabricated `task_instruction` field (doesn't exist in ARE), and `oracle_events` were stored but never used. `load_tasks()` now correctly iterates HF capability configs with a pinned revision, leaves `query` empty (GAIA2 is event-driven), and omits oracle events from evaluation_data. (PR: #PR_NUMBER_PLACEHOLDER) +- Tau2: Fixed tool result serialization in `DefaultTau2Agent` — now uses `model_dump()` + `json.dumps()` (matching original tau2-bench) instead of Python `str()`/`repr()`, which produced noisy formats with raw enum values, Pydantic repr strings, and `datetime.date(...)` literals that degraded agent accuracy (PR: #PR_NUMBER_PLACEHOLDER) - Fixed incorrect return type annotations on `DB.load()` and `DB.copy_deep()` in Tau2 benchmark — now use `Self` instead of `"DB"`, so subclass methods return the correct type (PR: #29) ### Removed diff --git a/FIXES.md b/FIXES.md new file mode 100644 index 0000000..3b9f6df --- /dev/null +++ b/FIXES.md @@ -0,0 +1,226 @@ +## GAIA2 Faithfulness Report: maseval vs ARE (AREFork) + +Here are all instances where the maseval implementation diverges from the original ARE implementation: + +--- + +### 1. **Scenario Duration Not Set (Critical)** + +**ARE:** `preprocess_scenario()` sets `scenario.duration` to `max_scenario_duration` (default `1800` seconds / 30 minutes) before the simulation runs. For "time" capability, it uses `MAX_TIME_SCENARIO_DURATION = 420` (7 minutes). (`config.py:18-20`, `utils.py:69-76`) + +**maseval:** Never calls `preprocess_scenario()`. The `scenario.duration` stays at whatever was loaded from JSON (possibly `None`). The fallback `86400` in `data_loader.py:203` is stored in `environment_data["duration"]` but never used — `setup_state()` reads `scenario.duration` directly. If `scenario.duration` is `None`, ARE's `EnvironmentConfig` defaults to 60 seconds. + +**Impact:** Simulations may run with wrong duration — either 60s (too short) or whatever the JSON contains, rather than the correct 1800s/420s. + +--- + +### 2. **Task Timeout: 600s vs 1860s** + +**ARE:** `DEFAULT_SCENARIO_TIMEOUT = 1860` (31 minutes). (`config.py:20`) + +**maseval:** `DEFAULT_TIMEOUT_SECONDS = 600.0` (10 minutes). (`data_loader.py:40`) + +**Impact:** Tasks may time out prematurely in maseval, especially complex scenarios that need up to 30 minutes. + +--- + +### 3. **Iteration Counting Logic (Behavioral)** + +**ARE:** `iterations` counter is incremented in the `finally` block on EVERY iteration, including errors and invalid formats. Termination checks `agent.iterations >= agent.max_iterations`. (`base_agent.py:849`) + +**maseval:** `_iteration_count` only incremented on successful action parse. Format retries tracked separately in `_format_retry_count`. (`gaia2.py:601`) + +**Impact:** ARE's agent terminates after 80 total loop iterations (errors included). maseval's agent can do 80 successful iterations PLUS up to 10 format retries per iteration = potentially far more LLM calls before terminating. + +--- + +### 4. **Max Iterations Termination Behavior** + +**ARE:** When `max_iterations` reached, the agent calls `send_message_to_user` with "Max iterations (80) reached. Stopping." through the actual tool, recording it in the event log. Then logs `MaxIterationsAgentError`. (`are_simulation.py:109-116`) + +**maseval:** Returns the string `"Max iterations (80) reached."` as a Python return value. Does NOT call the `send_message_to_user` tool. (`gaia2.py:632`) + +**Impact:** The judge evaluates completed events in the simulation. ARE's max-iteration message is recorded as an event; maseval's is not. This may affect evaluation results. + +--- + +### 5. **System Prompt: Missing Current Time** + +**ARE:** Injects `"Today's date in 'YYYY-MM-DD HH' format is {date_str}"` from `scenario.start_time` into the system prompt. (`are_simulation_main.py:156-164`) + +**maseval:** Does not include any current time information in the system prompt. (`environment_instructions.txt`) + +**Impact:** The agent doesn't know the starting simulation time, which is critical for time-sensitive tasks. + +--- + +### 6. **System Prompt: Agent Hints Included vs Excluded** + +**ARE:** Default agent uses `DEFAULT_ARE_SIMULATION_REACT_JSON_SYSTEM_PROMPT` with `json_agent_hints=""` (empty). (`system_prompt.py:182-190`) + +**maseval:** `agent_instructions.txt` includes the `JSON_AGENT_HINTS` block ("EXECUTION GUIDELINES: Take one action at a time..."). (`agent_instructions.txt:53-56`) + +**Impact:** Different agent behavior due to extra instructions in the prompt. + +--- + +### 7. **System Prompt: Notification System Description** + +**ARE:** Dynamically generates notification policy from `get_notification_system_prompt()` based on the actual notification system config and scenario apps. (`are_simulation_main.py:147-154`) + +**maseval:** Hardcoded generic notification description. (`environment_instructions.txt:20-22`) + +**Impact:** Agent receives different (less specific) notification policy information. + +--- + +### 8. **Tool Description Format** + +**ARE:** Uses Jinja2 template: `- {{ tool.name }}: {{ tool.description }}\n Takes inputs: {{tool.inputs}}\n Returns an output of type: {{tool.output_type}}`. Tool inputs are rendered as raw dict. (`tool_box.py:16-20`) + +**maseval:** Custom format: `Tool: {name}\nDescription: {desc}\nParameters:\n - {param}: {type} (required/optional) - {desc}`. (`gaia2.py:331-351`) + +**Impact:** LLM sees different tool description formatting, which can affect how it constructs tool calls. + +--- + +### 9. **Message History Format** + +**ARE:** Uses log-based message construction with specific templates: `[TASK]: \n{content}\n`, `[OUTPUT OF STEP {i}] Observation:\n***\n{content}\n***\n`, error messages with "Now let's retry" suffix. (`base_agent.py:93-113`) + +**maseval:** Simple `{"role": "user/assistant", "content": "..."}` format. Observations formatted as `"Observation: {result}"`. (`gaia2.py:559-629`) + +**Impact:** Significant difference in how conversation history is presented to the LLM. + +--- + +### 10. **Pre-step Notification Polling Missing** + +**ARE:** Has `get_are_simulation_update_pre_step()` as a conditional pre-step that polls for environment notifications before each agent step. (`agent_factory.py:37`) + +**maseval:** No pre-step functions. Notifications are only received when the agent explicitly calls `wait_for_notification`. (`gaia2.py:562-633`) + +**Impact:** Agent may miss asynchronous notifications (e.g., incoming messages) that arrive between iterations. + +--- + +### 11. **Environment Stop Message Not Checked** + +**ARE:** Termination condition checks `agent.notification_system.message_queue.has_environment_stop_message()`. (`are_simulation.py:105-107`) + +**maseval:** No environment stop message checking. (`gaia2.py:568`) + +**Impact:** Agent may continue running after the environment signals it should stop. + +--- + +### 12. **JSON Parsing: Different Error Handling** + +**ARE:** On JSONDecodeError, raises `JsonParsingAgentError` with detailed error. No trailing comma fix. (`json_action_executor.py:33-57`) + +**maseval:** On JSONDecodeError, tries to fix trailing commas and retry. Returns `None` instead of raising. (`gaia2.py:414-424`) + +**Impact:** maseval is more lenient, accepting malformed JSON that ARE would reject. This changes which agent outputs count as valid actions vs errors. + +--- + +### 13. **`action_input` Default Value** + +**ARE:** Missing `action_input` defaults to empty string `""`. (`json_action_executor.py:64-70`) + +**maseval:** Missing `action_input` defaults to empty dict `{}`. (`gaia2.py:470`) + +**Impact:** Tools receiving `""` vs `{}` may behave differently. + +--- + +### 14. **Evaluation: Exceptions Scored as 0.0 vs Excluded** + +**ARE:** Exceptions and "no_validation" get `score=None` and are EXCLUDED from success rate calculations. (`hf_upload_utils.py:33-52`, `report_stats.py`) + +**maseval:** Evaluation exceptions result in `gsr=0.0, passed=False`. They are counted as failures in metrics. (`evaluator.py:153-163`) + +**Impact:** maseval inflates failure rates by counting infrastructure errors as agent failures. + +--- + +### 15. **Partial GSR Always Equals GSR** + +**ARE:** `GraphPerEventJudge` can produce partial success rates based on fraction of matched oracle events. + +**maseval:** Sets `partial_gsr = gsr` unconditionally (always 0.0 or 1.0). (`evaluator.py:146`) + +**Impact:** Partial success information is lost. + +--- + +### 16. **LLM Judge Not Implemented** + +**maseval:** Stores `use_llm_judge` and `model` but never references them in `__call__()`. Always creates `GraphPerEventJudgeConfig()` regardless. (`evaluator.py:55-56`, `evaluator.py:131-132`) + +**Impact:** LLM-based judging is advertised but non-functional. + +--- + +### 17. **Turn Initialization Skipped** + +**ARE:** Calls `scenario.initialize_turns()` with trigger conditions for online validation during simulation. (`utils.py:145-150`) + +**maseval:** Does not call `initialize_turns()`. Only calls `build_event_id_to_turn_idx()`. (`environment.py:125`) + +**Impact:** Online validation during simulation is skipped. May cause issues with judge trigger conditions. + +--- + +### 18. **Duration Fallback in data_loader is Invented** + +**maseval:** `"duration": getattr(scenario, "duration", 86400)` — the `86400` (24 hours) fallback is invented and doesn't exist anywhere in ARE. (`data_loader.py:203`) + +**ARE:** Duration defaults to `None` in `Scenario` class, then gets set to `1800` or `420` during preprocessing. + +**Impact:** Violates AGENTS.md scientific integrity guidelines: "Only copy defaults that exist in the source." + +--- + +### 19. **`judge_type` Stored But Ignored** + +**maseval:** Stores `judge_type` from scenario metadata but always uses `GraphPerEventJudgeConfig()`. (`evaluator.py:61`, `evaluator.py:131`) + +**Impact:** Scenarios requiring different judge types (e.g., `InContextJudge`) would use the wrong judge. + +--- + +### 20. **Simulated Generation Time Not Implemented** + +**ARE:** Pauses/resumes the environment during LLM generation to simulate realistic generation times. Configurable via `SimulatedGenerationTimeConfig`. (`base_agent.py:623-689`) + +**maseval:** No simulated generation time support. + +**Impact:** Simulation time advances differently, which could affect time-sensitive scenarios. + +--- + +### Summary of Severity + +| # | Issue | Severity | Done | +| --- | ------------------------------------------------------- | ------------ | ---- | +| 1 | Scenario duration not set (preprocess_scenario skipped) | **Critical** | ✓ | +| 2 | Task timeout 600s vs 1860s | **High** | ✓ | +| 3 | Iteration counting (errors excluded vs included) | **High** | ✓ | +| 4 | Max iterations doesn't call send_message_to_user tool | **High** | ✓ | +| 5 | Missing current time in system prompt | **High** | ✓ | +| 6 | Agent hints included vs excluded in prompt | **Medium** | ✓ | +| 7 | Notification system description differs | **Medium** | ✓ | +| 8 | Tool description format differs | **Medium** | ✓ | +| 9 | Message history format differs | **Medium** | ✓ | +| 10 | Pre-step notification polling missing | **Medium** | ✓ | +| 11 | Environment stop message not checked | **Medium** | ✓ | +| 12 | JSON parsing more lenient | **Low** | ✓ | +| 13 | action_input default "" vs {} | **Low** | ✓ | +| 14 | Exceptions scored as 0.0 vs excluded | **High** | ✓ | +| 15 | Partial GSR always equals GSR | **Medium** | ✓ | +| 16 | LLM judge not implemented | **Medium** | ✓ | +| 17 | Turn initialization skipped | **Medium** | ✓ | +| 18 | Invented 86400 duration fallback | **Medium** | ✓ | +| 19 | judge_type stored but ignored | **Low** | ✓ | +| 20 | Simulated generation time not implemented | **Low** | ✓ | diff --git a/GAIA2BUG.md b/GAIA2BUG.md new file mode 100644 index 0000000..e69de29 diff --git a/MULTIAGENTBENCHITERATIONBUG.md b/MULTIAGENTBENCHITERATIONBUG.md new file mode 100644 index 0000000..99cfe24 --- /dev/null +++ b/MULTIAGENTBENCHITERATIONBUG.md @@ -0,0 +1,65 @@ +# MultiAgentBench: MARBLE Engine Coordination Loop Bypassed + +## Bug + +`run_agents()` ([multiagentbench.py:274-321](maseval/benchmark/multiagentbench/multiagentbench.py#L274-L321)) calls each agent's `run(query)` **once**. `max_iterations` and `coordinate_mode` are stored but never used for loop control or dispatch. + +## Expected Behavior + +MARBLE's `Engine.start()` dispatches to one of four coordination modes, each running `while current_iteration < max_iterations` with an LLM-based `EnginePlanner` that assigns tasks, summarizes results, and decides when to stop: + +- **Star**: Planner assigns tasks to agents each iteration (database domain) +- **Graph**: All agents act, then self-plan via `plan_task()` in subsequent rounds (research, bargaining) +- **Chain**: One agent acts, picks next agent via `plan_next_agent()`, chain limit = `max_iterations * len(agents)` +- **Tree**: Root delegates recursively via `plan_tasks_for_children()` (coding domain) + +## What MASEval Does vs. Should Do + +| Step | MARBLE Engine | MASEval | +| ------------------------------- | ------------------------------------------------ | ----------------------------------------------------------- | +| Agents, Environment, AgentGraph | Created and wired up | **Same** (via `create_marble_agents`, `_setup_agent_graph`) | +| SharedMemory | Created | **Missing** | +| EnginePlanner | Created; assigns tasks, summarizes, decides stop | **Missing** | +| `max_iterations` | Controls loop bound | Stored, **unused** | +| `coordinate_mode` | Selects coordination method | Stored, **unused** | +| Coordination loop | 4 mode-specific multi-iteration loops | **Each agent acts once** | + +MASEval creates MARBLE agents, wraps them in `MarbleAgentAdapter` for tracing, and sets up `AgentGraph` with relationships — but never instantiates `Engine`, `EnginePlanner`, or `SharedMemory`. The `raw_marble_config` needed to build a MARBLE `Config` is already stored in `environment_data["raw_marble_config"]`. + +## Proposed Fixes + +### Option A: Subclass Engine, Inject Pre-Created Components (~150 lines) + +Create `MASEvalEngine(Engine)` that skips `__init__`'s factory methods and uses the agents/environment MASEval already created. Call `engine.start()` to run MARBLE's native coordination. + +- **Pro**: Uses MARBLE's exact logic for all 4 modes; low drift risk +- **Con**: Fragile coupling to Engine internals (attribute names, init order). Engine calls `agent.act()` directly, bypassing `MarbleAgentAdapter._run_agent()` — traces must be extracted post-hoc. Must suppress Engine's `_write_to_jsonl()` side effect and internal `Evaluator`. + +### Option B: Reimplement Coordination Loops (~500-800 lines) + +Port all 4 coordination modes into MASEval, calling agents through adapters. Use MARBLE's `EnginePlanner` and `SharedMemory` directly. + +- **Pro**: Full MASEval tracing on every `act()` call; no Engine internals dependency; no side effects +- **Con**: Largest effort. Must faithfully port ~800 lines including edge cases (Minecraft `block_hit_rate`, tree recursion, chain agent selection). Drift risk if MARBLE updates coordination logic. + +### Option C: Let Engine Run Natively (~80-120 lines) + +Build a MARBLE `Config` from `raw_marble_config`, instantiate `Engine(config)`, call `engine.start()`. Extract results afterward. + +- **Pro**: Simplest; zero drift risk; guaranteed correctness for all modes +- **Con**: Engine creates its own agents/environment, so `setup_agents()`/`setup_environment()` become stubs (strains base class contract). MASEval tracing fully bypassed — traces only from post-hoc extraction. Same `_write_to_jsonl()` and internal `Evaluator` side effects as Option A. + +### Comparison + +| Criteria | A (Subclass) | B (Reimplement) | C (Native) | +| ------------------- | ------------- | ---------------- | ------------- | +| Correctness | High | High if faithful | Highest | +| MASEval tracing | Post-hoc | Full | Post-hoc | +| Effort | Medium | High | Low | +| Drift risk | Medium | High | Low | +| Side effects | Must suppress | None | Must suppress | +| Base class contract | OK | OK | Strained | + +## Further point to validate: + +data_loader.py:266 falls back to 10 when the JSONL lacks environment.max_iterations, which is incorrect for research (3), bargaining (3), and coding/database (5) diff --git a/TESTINGPLAN.md b/TESTINGPLAN.md new file mode 100644 index 0000000..222e5f0 --- /dev/null +++ b/TESTINGPLAN.md @@ -0,0 +1,371 @@ +# Testing Plan: GAIA2 & MultiAgentBench Alignment + +## Context + +Commit `e6d8a03` ("Improved Testing Infrastructure") established a two-tier testing +pattern for benchmarks (Tau2 and MACS). GAIA2 and MultiAgentBench do not yet follow +this pattern. This plan brings them into alignment. + +### What Tau2 & MACS Have That GAIA2 & MultiAgentBench Don't + +| Capability | Tau2 | MACS | GAIA2 | MABench | +| --------------------------------------------------- | ---- | ---- | ------ | ------- | +| `@benchmark` marker on all tests | Yes | Yes | Yes | **No** | +| `@live` marker on network/data tests | Yes | Yes | **No** | **No** | +| `@slow` marker on heavy download tests | Yes | Yes | **No** | **No** | +| `test_data_integrity.py` (tmp-dir, self-contained) | Yes | Yes | **No** | **No** | +| Session-scoped conftest fixture for domain tests | Yes | Yes | **No** | **No** | +| Real-data integration tests | Yes | Yes | **No** | **No** | +| Parametrized across domains/capabilities | Yes | Yes | **No** | **No** | +| Descriptive data-availability assertions | Yes | Yes | **No** | **No** | + +### Two Patterns for Real-Data Tests + +The improved infrastructure uses two distinct patterns for tests that touch real data. +Both are described in `tests/README.md` under "Benchmark tests": + +**Pattern A — Self-contained tmp-dir download (data integrity & integration tests)** + +`test_data_integrity.py` and `test_*_integration_real_data.py` define their own +module/class-scoped fixture that downloads into `tmp_path_factory`. They validate +upstream data freshness without relying on cached state. Marked +`live + slow + benchmark`. + +Examples: +- `test_tau2/test_data_integrity.py` → class-scoped `_download_data` into tmp dir +- `test_macs/test_data_integrity.py` → module-scoped `macs_data_dir` into tmp dir +- `test_macs/test_macs_integration_real_data.py` → module-scoped `real_macs_data` + +**Pattern B — Session-scoped conftest fixture (domain tool & environment tests)** + +A session-scoped fixture in `conftest.py` downloads to the **package's default data +directory** (cached across runs). Downstream fixtures cascade from it. Tests that +depend on this fixture are marked `live + benchmark`. + +Examples: +- `test_tau2/conftest.py::ensure_tau2_data` → `retail_db` → `retail_toolkit` +- `test_macs/conftest.py::ensure_macs_templates` + +--- + +## Plan: GAIA2 + +### Existing State + +GAIA2 has 7 test files with good mock coverage: +- `test_evaluator.py` — single/multi-turn judge, GSR metrics +- `test_environment.py` — scenario extraction, tool wrapping, cleanup +- `test_benchmark.py` — lifecycle, seeding, agent-agnostic design +- `test_default_agent.py` — ReAct loop, action parsing, termination +- `test_tool_wrapper.py` — invocation tracking, tracing +- `test_data_loader.py` — constants, validation, model ID config +- `conftest.py` — MockARETool, MockAREEnvironment, MockGraphPerEventJudge + +All test classes are marked `@pytest.mark.benchmark`. No tests are marked `@live` or +`@slow`. No real data is downloaded or tested. + +### Changes + +#### 1. New file: `test_data_integrity.py` (Pattern A) + +Self-contained data integrity tests that download from HuggingFace into a tmp dir. + +``` +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark, pytest.mark.gaia2] +``` + +Module-scoped fixture: +```python +@pytest.fixture(scope="module") +def gaia2_data(tmp_path_factory): + """Download GAIA2 validation split into a temporary directory.""" + from maseval.benchmark.gaia2.data_loader import load_tasks + tasks = load_tasks(split="validation") + return tasks +``` + +Test classes: + +- **TestGaia2DatasetIntegrity** + - `test_validation_split_loads` — `load_tasks("validation")` returns data + - `test_minimum_task_count` — dataset has >= expected number of tasks + - `test_required_fields_present` — every task has `scenario`, `oracle_events`, `capability` + - `test_oracle_events_non_empty` — every task has at least one oracle event + - `test_scenario_is_deserializable` — ARE scenario field is valid (not empty/null) for a sample + +- **TestGaia2CapabilityCoverage** + - `@pytest.mark.parametrize("capability", VALID_CAPABILITIES)` + - `test_capability_has_tasks` — every declared capability has >= 1 task in the dataset + +#### 2. New file: `test_integration.py` (Pattern A) + +Real-data integration tests exercising the GAIA2 pipeline with real tasks but a +DummyModelAdapter (no API keys needed). + +``` +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark, pytest.mark.gaia2] +``` + +Module-scoped fixture reuses the `gaia2_data` approach (download into tmp dir, or +reuse the data integrity fixture via a shared conftest fixture). + +Test classes: + +- **TestGaia2EnvironmentWithRealData** + - `test_environment_setup_from_real_task` — `Gaia2Environment` from a real task + creates without error, `setup_state()` succeeds + - `test_real_tools_are_wrapped` — tools created from a real scenario are + `Gaia2GenericTool` instances with name, description, and callable inputs + - `test_real_tools_have_valid_schema` — every wrapped tool's `inputs` is a dict + with expected structure (not empty, has descriptions) + +- **TestDefaultAgentWithRealTools** + - `test_agent_builds_system_prompt_with_real_tools` — `DefaultGaia2Agent` constructed + from real task tools has a system prompt that mentions real tool names + - `test_single_step_execution` — run agent for 1 iteration with a canned ReAct + response targeting a real tool name, verify tool invocation is recorded + +- **TestGaia2EvaluatorWithRealOracleEvents** + - `test_evaluator_processes_real_oracle_events` — `Gaia2Evaluator` with a real task's + `oracle_events` and a mock judge runs without error + - `test_evaluator_returns_scoreable_result` — result has expected fields (gsr, status) + +- **TestGaia2PipelineSmoke** + - `test_full_pipeline_single_task` — `Gaia2Benchmark.run()` on one real task with + `DummyModelAdapter` produces a `TaskResult` with expected structure + (status in known statuses, traces dict present, eval dict present) + +#### 3. Update `conftest.py` — Session-scoped fixture (Pattern B) + +Add a session-scoped fixture for tests that want real ARE tools/environments without +re-downloading each time: + +```python +@pytest.fixture(scope="session") +def ensure_gaia2_data(): + """Download GAIA2 validation data to the package's default cache. + + Tests that need real data should depend on this and be marked @pytest.mark.live. + """ + from maseval.benchmark.gaia2.data_loader import load_tasks + tasks = load_tasks(split="validation") + return tasks +``` + +This enables future domain-specific test files (e.g., testing specific ARE app tools) +to depend on `ensure_gaia2_data` without each file re-downloading. + +#### 4. No changes to existing test files + +The existing mock-based tests are solid Tier 1 tests. They should remain as-is with +their `@benchmark` marker and no `@live`/`@slow` markers. + +--- + +## Plan: MultiAgentBench + +### Existing State + +MultiAgentBench has 6 test files with extensive coverage (~3,200 lines): +- `test_evaluator.py` — domain-specific evaluation, parsing, metrics +- `test_benchmark.py` — lifecycle, seeding, MARBLE integration, coordination modes +- `test_data_loader.py` — JSONL loading, domain info, MARBLE download, werewolf config +- `test_environment.py` — infrastructure checks, MARBLE env delegation, tool wrapping +- `test_marble_adapter.py` — agent wrapping, action/communication logging +- `conftest.py` — task data fixtures, agent adapter, concrete benchmark + +**Critical gap:** No test classes are marked `@pytest.mark.benchmark`. This means: +- `pytest -m benchmark` misses all MultiAgentBench tests +- `pytest -m core` incorrectly includes them +- The CI `test-benchmark` job doesn't run them + +No tests are marked `@live` or `@slow`. No real MARBLE data is downloaded or tested. + +### Changes + +#### 1. Add `@pytest.mark.benchmark` to ALL existing test classes + +Every test class/function in `test_evaluator.py`, `test_benchmark.py`, +`test_data_loader.py`, `test_environment.py`, and `test_marble_adapter.py` needs +`@pytest.mark.benchmark`. This is the highest-priority change — it fixes the CI +pipeline visibility. + +Preferred approach: use `pytestmark` at file level where all tests in a file are +benchmark tests: + +```python +pytestmark = pytest.mark.benchmark +``` + +#### 2. New file: `test_data_integrity.py` (Pattern A) + +Self-contained data integrity tests that clone/verify MARBLE into a tmp dir. + +``` +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark] +``` + +Module-scoped fixture: +```python +@pytest.fixture(scope="module") +def marble_data(tmp_path_factory): + """Clone and verify MARBLE data into a temporary directory.""" + from maseval.benchmark.multiagentbench.data_loader import ( + ensure_marble_exists, + ) + data_dir = tmp_path_factory.mktemp("marble_data") + marble_dir = ensure_marble_exists(data_dir=data_dir, auto_download=True) + return marble_dir +``` + +Test classes: + +- **TestMarbleDataPresence** + - `@pytest.mark.parametrize("domain", VALID_DOMAINS)` + - `test_domain_directory_exists` — domain directory exists in MARBLE + - `test_domain_has_task_data` — JSONL/config files present per domain + +- **TestMarbleTaskStructure** + - `@pytest.mark.parametrize("domain", VALID_DOMAINS - {"werewolf"})` + - `test_minimum_task_count` — each domain has >= expected minimum tasks + - `test_required_fields` — each task has `scenario`, `task_id`, `task`, `agents`, + `relationships` + - `test_agent_structure` — each agent has `agent_id` + +- **TestMarbleWerewolfConfigs** + - `test_werewolf_config_files_exist` — config YAML files present + - `test_werewolf_configs_parse` — YAML files parse correctly + - `test_werewolf_configs_have_roles` — configs contain expected roles + +#### 3. New file: `test_integration_real_data.py` (Pattern A) + +Real-data integration tests, following the MACS `test_macs_integration_real_data.py` +pattern. + +``` +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark] +``` + +Module-scoped fixture (same as data integrity or shared via conftest). + +Test classes: + +- **TestMultiAgentBenchRealDataLoading** + - `@pytest.mark.parametrize("domain", VALID_DOMAINS)` + - `test_load_tasks_returns_tasks` — `load_tasks(domain)` returns non-empty TaskQueue + - `test_tasks_have_agents` — every loaded task has >= 1 agent in its config + - `test_configure_model_ids` — `configure_model_ids()` modifies tasks in place + +- **TestMultiAgentBenchRealEnvironment** + - `@pytest.mark.parametrize("domain", NON_INFRA_DOMAINS)` + where `NON_INFRA_DOMAINS = VALID_DOMAINS - {"database", "minecraft"}` + - `test_environment_setup` — `MultiAgentBenchEnvironment` initializes from real task + - `test_environment_state` — `setup_state()` extracts domain and max_iterations + - `test_environment_traces` — `gather_traces()` returns dict with expected keys + +- **TestMultiAgentBenchRealEvaluation** + - `@pytest.mark.parametrize("domain", VALID_DOMAINS - {"minecraft"})` + - `test_evaluator_creation` — evaluator created from real task's domain and + `DummyModelAdapter` without error + - `test_evaluator_processes_structure` — evaluator's `filter_traces()` processes + a synthetic trace structure without error + +- **TestMultiAgentBenchPipelineSmoke** + - `@pytest.mark.parametrize("domain", NON_INFRA_DOMAINS)` + - `test_full_pipeline_single_task` — benchmark `.run()` on one real task with + `DummyModelAdapter` produces results with expected structure. Uses descriptive + assertion messages: + ```python + assert len(results) > 0, ( + f"No results for domain '{domain}'. " + "Check test_data_integrity tests first." + ) + ``` + +#### 4. Update `conftest.py` — Session-scoped fixture (Pattern B) + +```python +@pytest.fixture(scope="session") +def ensure_marble_data(): + """Clone MARBLE data once per session. + + Tests that need real data should depend on this and be marked @pytest.mark.live. + """ + from maseval.benchmark.multiagentbench.data_loader import ensure_marble_exists + marble_dir = ensure_marble_exists(auto_download=True) + return marble_dir +``` + +#### 5. Descriptive assertion messages in existing tests + +Update assertion messages in existing environment/evaluator tests where data absence +could cause confusing failures, linking to data integrity tests. This is low priority +and can be done opportunistically. + +--- + +## CI Updates + +### `.github/workflows/test.yml` + +Add caching for GAIA2 and MARBLE data in the `test-slow` job: + +```yaml +- name: Cache GAIA2 data + uses: actions/cache@v4 + with: + path: ~/.cache/huggingface/ # or wherever HF datasets cache + key: gaia2-data-${{ hashFiles('maseval/benchmark/gaia2/data_loader.py') }} + +- name: Cache MARBLE data + uses: actions/cache@v4 + with: + path: maseval/benchmark/multiagentbench/marble/ + key: marble-data-${{ hashFiles('maseval/benchmark/multiagentbench/data_loader.py') }} +``` + +### `tests/README.md` + +Update the tree to include `test_multiagentbench/`: + +``` +└── test_benchmarks/ + ├── test_tau2/ # Tau2 benchmark + data integrity + ├── test_macs/ # MACS benchmark + data integrity + ├── test_gaia2/ # GAIA2 benchmark + data integrity + └── test_multiagentbench/ # MultiAgentBench + data integrity +``` + +--- + +## Summary of New Files + +| Benchmark | New File | Pattern | Markers | +| -------------- | ------------------------------ | ------- | ---------------------------- | +| GAIA2 | `test_data_integrity.py` | A | `live + slow + benchmark + gaia2` | +| GAIA2 | `test_integration.py` | A | `live + slow + benchmark + gaia2` | +| MultiAgentBench| `test_data_integrity.py` | A | `live + slow + benchmark` | +| MultiAgentBench| `test_integration_real_data.py` | A | `live + slow + benchmark` | + +## Summary of Modified Files + +| File | Change | +| ---- | ------ | +| `test_multiagentbench/test_evaluator.py` | Add `pytestmark = pytest.mark.benchmark` | +| `test_multiagentbench/test_benchmark.py` | Add `pytestmark = pytest.mark.benchmark` | +| `test_multiagentbench/test_data_loader.py` | Add `pytestmark = pytest.mark.benchmark` | +| `test_multiagentbench/test_environment.py` | Add `pytestmark = pytest.mark.benchmark` | +| `test_multiagentbench/test_marble_adapter.py` | Add `pytestmark = pytest.mark.benchmark` | +| `test_gaia2/conftest.py` | Add `ensure_gaia2_data` session fixture | +| `test_multiagentbench/conftest.py` | Add `ensure_marble_data` session fixture | +| `.github/workflows/test.yml` | Add data caching for GAIA2 + MARBLE | +| `tests/README.md` | Add `test_multiagentbench/` to tree | + +## Priority Order + +1. **Add `@benchmark` to all MultiAgentBench tests** — fixes CI pipeline visibility (broken now) +2. **`test_data_integrity.py` for both** — foundation for catching upstream data regressions +3. **`test_integration*.py` for both** — catches real-data issues mocks can't +4. **Session-scoped conftest fixtures** — enables future domain-specific test files +5. **CI caching** — makes the slow tier practical in CI +6. **README update** — keeps documentation aligned diff --git a/docs/benchmark/gaia2.md b/docs/benchmark/gaia2.md index 2e0e0ad..21a9af6 100644 --- a/docs/benchmark/gaia2.md +++ b/docs/benchmark/gaia2.md @@ -8,7 +8,7 @@ The **Gaia2 Benchmark** evaluates LLM-based agents on dynamic, multi-step scenar - **ARE simulation environment** with real-time dynamics and event scheduling - **Tool-based time control** via `wait_for_notification()` for temporal reasoning -- **7 capability dimensions**: execution, search, adaptability, time, ambiguity, agent2agent, noise +- **5 capability dimensions**: execution, search, adaptability, time, ambiguity - **Deterministic evaluation** via GraphPerEventJudge comparing completed vs expected events - **12 app tools**: Calendar, Email, Messaging, Contacts, Shopping, Cab, City, FileSystem, Browser, ChatsApp, SystemApp, Timer @@ -97,8 +97,6 @@ Gaia2 tasks are organized by capability dimension: | `adaptability` | Adapting to changing requirements | | `time` | Temporal reasoning tasks | | `ambiguity` | Handling ambiguous instructions | -| `agent2agent` | Multi-agent collaboration | -| `noise` | Handling noisy inputs | Load specific capabilities: @@ -110,6 +108,45 @@ tasks = load_tasks(capability="time", limit=10) tasks = load_tasks(limit=50) ``` +## Multi-Turn Notification Loop + +GAIA2 uses an **event-driven** multi-turn architecture, not user-turn interaction. Unlike Tau2 (where a user simulator drives multi-turn), GAIA2 scenarios have scheduled events (e.g., "calendar events added at t=240s", "friend replies at t=300s") that the agent must wait for and react to. + +The benchmark invokes the agent **once**. The agent handles multi-turn internally via the notification loop: + +1. Agent calls `SystemApp__wait_for_notification(timeout=N)` as a normal tool. +2. The ARE environment processes scheduled events, advances simulation time, and queues resulting notifications — all synchronously during the tool call. +3. The tool returns. The agent's loop continues (it does **not** terminate). +4. Before the next LLM call, the agent polls `environment.poll_notifications()` to retrieve messages that arrived during the wait. +5. The agent injects those messages into its context and continues reasoning. +6. Eventually the agent calls `AgentUserInterface__send_message_to_user` — the **only** termination signal. + +### What custom agents must implement + +The ARE tools handle all environment-side mechanics automatically (event processing, time advancement, notification queuing). No callbacks or hooks required. Custom agents must handle two things: + +**1. Do not terminate on `wait_for_notification`.** Treat it as a regular tool call. Only terminate on `AgentUserInterface__send_message_to_user`. + +**2. Poll notifications between steps.** After `wait_for_notification` returns, new messages are in the queue. Call `environment.poll_notifications()` to drain them: + +```python +# Between agent steps (e.g., before each LLM call): +user_msgs, env_notifs, has_stop = environment.poll_notifications() + +# Inject into agent context (format matches ARE's convention): +if user_msgs: + content = "\n".join(user_msgs) + messages.append({"role": "user", "content": f"User messages updates:\n***\n{content}\n***\n"}) +if env_notifs: + content = "\n".join(env_notifs) + messages.append({"role": "user", "content": f"Environment notifications updates:\n***\n{content}\n***\n"}) +if has_stop: + # Environment signalled simulation end — stop the agent loop + break +``` + +See `DefaultGaia2Agent` source for the canonical single-loop implementation. + ## Key Differences from Tau2 | Aspect | Gaia2 | Tau2 | @@ -132,7 +169,7 @@ tasks = load_tasks(limit=50) ::: maseval.benchmark.gaia2.DefaultGaia2Agent -::: maseval.benchmark.gaia2.AREToolWrapper +::: maseval.benchmark.gaia2.Gaia2GenericTool ::: maseval.benchmark.gaia2.load_tasks diff --git a/docs/benchmark/multiagentbench.md b/docs/benchmark/multiagentbench.md index 1ffabda..ec08796 100644 --- a/docs/benchmark/multiagentbench.md +++ b/docs/benchmark/multiagentbench.md @@ -2,9 +2,9 @@ The **MultiAgentBench** benchmark evaluates multi-agent collaboration and competition in LLM-based systems across diverse scenarios including research, negotiation, coding, and more. -[MultiAgentBench](https://github.com/ulab-uiuc/MARBLE) (from the MARBLE framework) is designed to evaluate how multiple LLM-based agents collaborate and compete to solve complex tasks. The benchmark features: +[MultiAgentBench](https://github.com/ulab-uiuc/MARBLE) (from the MARBLE framework, where the original work was done) is designed to evaluate how multiple LLM-based agents collaborate and compete to solve complex tasks. We use a [bug-fixed fork](https://github.com/cemde/MARBLE) for MASEval integration. The benchmark features: -- **7 diverse domains**: research, bargaining, coding, database, web, worldsimulation, minecraft +- **6 diverse domains**: research, bargaining, coding, database, werewolf, minecraft (minecraft is untested) - **Multiple coordination modes**: cooperative, star, tree, hierarchical - **LLM-based evaluation**: Matches MARBLE's evaluation methodology - **Framework-agnostic**: Use with any agent framework or MARBLE's native agents diff --git a/examples/gaia2_benchmark/gaia2_mwe.py b/examples/gaia2_benchmark/gaia2_mwe.py new file mode 100644 index 0000000..f77e3a6 --- /dev/null +++ b/examples/gaia2_benchmark/gaia2_mwe.py @@ -0,0 +1,46 @@ +import os +from typing import Any, Dict, Sequence + +from dotenv import load_dotenv +from openai import OpenAI + +from maseval import AgentAdapter +from maseval.benchmark.gaia2 import DefaultAgentGaia2Benchmark, compute_gaia2_metrics, load_tasks +from maseval.benchmark.gaia2.environment import Gaia2Environment +from maseval.core.seeding import SeedGenerator +from maseval.core.task import Task +from maseval.core.user import User +from maseval.interface.inference import OpenAIModelAdapter + +load_dotenv() + + +class MyGaia2(DefaultAgentGaia2Benchmark): + """Run GAIA2 benchmark using OpenAI.""" + + def get_model_adapter(self, model_id: str, **kwargs: Any) -> OpenAIModelAdapter: + client = OpenAI(api_key=os.environ["OPENAI_API_KEY"]) + adapter = OpenAIModelAdapter(client, model_id=model_id, seed=kwargs.get("seed")) + if "register_name" in kwargs: + self.register("models", kwargs["register_name"], adapter) + return adapter + + def setup_agents(self, agent_data: Dict[str, Any], environment: Gaia2Environment, task: Task, user: User, seed_generator: SeedGenerator): + """Set up your own agents here""" + pass + + def run_agents(self, agents: Sequence[AgentAdapter], task: Task, environment: Gaia2Environment, query: str = "") -> Any: + """How to run the agents on the task.""" + pass + + +if __name__ == "__main__": + tasks = load_tasks(capability="execution", limit=2) + + benchmark = MyGaia2( + agent_data={"model_id": "gpt-4o", "verbose": 1}, + progress_bar=True, + ) + + results = benchmark.run(tasks=tasks) + summary = compute_gaia2_metrics(results) diff --git a/maseval/benchmark/gaia2/PROVENANCE.md b/maseval/benchmark/gaia2/PROVENANCE.md index 3b5e063..b682171 100644 --- a/maseval/benchmark/gaia2/PROVENANCE.md +++ b/maseval/benchmark/gaia2/PROVENANCE.md @@ -44,16 +44,16 @@ MASEval provides: | MASEval Method | ARE Method/Component | Notes | | -------------------------------- | ------------------------------------- | ------------------------------------ | -| `Gaia2Environment.setup_state()` | `Environment.initialize_scenario()` | Initializes ARE simulation | +| `Gaia2Environment.setup_state()` | `Environment.run(scenario, wait_for_end=False)` | Starts ARE simulation in background | | `Gaia2Environment.create_tools()`| `App.get_tools()` for all apps | Wraps all app tools with tracing | | `Gaia2Environment.cleanup()` | `Environment.stop()` | Ensures proper resource cleanup | -| `get_simulation_time()` | `TimeManager.current_time` | Exposes simulation time for tracing | +| `get_simulation_time()` | `Environment.current_time` | Exposes simulation time for tracing | ### Evaluator Integration | MASEval Method | ARE Component | Notes | | ------------------------------- | --------------------------------------- | ------------------------------------ | -| `Gaia2Evaluator.__call__()` | `GraphPerEventJudge.evaluate()` | Delegates to ARE's deterministic judge | +| `Gaia2Evaluator.__call__()` | `GraphPerEventJudge.validate(env)` | Delegates to ARE's deterministic judge | | `filter_traces()` | N/A | MASEval-specific trace extraction | | `compute_gaia2_metrics()` | N/A | MASEval-specific metrics aggregation | @@ -73,16 +73,15 @@ Scenarios are loaded from HuggingFace: https://huggingface.co/datasets/meta-agents-research-environments/gaia2 ``` -| Config | Description | Split | -| ----------- | ------------------------------------------ | ---------- | -| `validation`| Full validation set (all capabilities) | validation | -| `execution` | Execution capability only | validation | -| `search` | Search capability only | validation | -| `adaptability` | Adaptability capability only | validation | -| `time` | Temporal reasoning only | validation | -| `ambiguity` | Ambiguity handling only | validation | -| `agent2agent` | Multi-agent collaboration only | validation | -| `noise` | Noise handling only | validation | +Revision: `78ea3bdbdeec2bdcd6afa5420915d8a22f23ed99` + +| Config | Description | Split | +| -------------- | ------------------------------ | ---------- | +| `execution` | Execution capability only | validation | +| `search` | Search capability only | validation | +| `adaptability` | Adaptability capability only | validation | +| `time` | Temporal reasoning only | validation | +| `ambiguity` | Ambiguity handling only | validation | ## MASEval-Specific Additions diff --git a/maseval/benchmark/gaia2/__init__.py b/maseval/benchmark/gaia2/__init__.py index e5d5be5..f4f17d2 100644 --- a/maseval/benchmark/gaia2/__init__.py +++ b/maseval/benchmark/gaia2/__init__.py @@ -13,8 +13,6 @@ - adaptability: Adapting to changing requirements - time: Temporal reasoning tasks - ambiguity: Handling ambiguous instructions - - agent2agent: Multi-agent collaboration - - noise: Handling noisy inputs Usage: from maseval.benchmark.gaia2 import ( @@ -52,6 +50,7 @@ def get_model_adapter(self, model_id, **kwargs): # Main benchmark components from maseval.benchmark.gaia2.gaia2 import ( Gaia2Benchmark, + Gaia2SimulatedGenerationTimeConfig, DefaultGaia2Agent, DefaultGaia2AgentAdapter, DefaultAgentGaia2Benchmark, @@ -68,17 +67,19 @@ def get_model_adapter(self, model_id, **kwargs): # Tool wrapper from maseval.benchmark.gaia2.tool_wrapper import ( - AREToolWrapper, + Gaia2GenericTool, wrap_are_tools, ) -# Data loading +# Data loading and configuration from maseval.benchmark.gaia2.data_loader import ( load_tasks, configure_model_ids, + Gaia2JudgeEngineConfig, VALID_CAPABILITIES, VALID_SPLITS, HF_DATASET_ID, + HF_DATASET_REVISION, ) @@ -86,6 +87,7 @@ def get_model_adapter(self, model_id, **kwargs): # Benchmark "Gaia2Benchmark", # Default agent implementation + "Gaia2SimulatedGenerationTimeConfig", "DefaultGaia2Agent", "DefaultGaia2AgentAdapter", "DefaultAgentGaia2Benchmark", @@ -95,12 +97,14 @@ def get_model_adapter(self, model_id, **kwargs): "Gaia2Evaluator", "compute_gaia2_metrics", # Tool wrapper - "AREToolWrapper", + "Gaia2GenericTool", "wrap_are_tools", - # Data loading + # Data loading and configuration "load_tasks", "configure_model_ids", + "Gaia2JudgeEngineConfig", "VALID_CAPABILITIES", "VALID_SPLITS", "HF_DATASET_ID", + "HF_DATASET_REVISION", ] diff --git a/maseval/benchmark/gaia2/data_loader.py b/maseval/benchmark/gaia2/data_loader.py index 32c834a..71d438a 100644 --- a/maseval/benchmark/gaia2/data_loader.py +++ b/maseval/benchmark/gaia2/data_loader.py @@ -3,7 +3,7 @@ This module provides functions to: 1. Load Gaia2 scenarios from HuggingFace 2. Convert scenarios to MASEval Task objects -3. Configure model IDs for benchmark components +3. Configure model IDs and judge engine for benchmark components Reference Paper: "GAIA-2: A Controllable Multi-Turn Conversational Benchmark for Agents" Data: https://huggingface.co/datasets/meta-agents-research-environments/gaia2 @@ -11,7 +11,7 @@ No side effects on import. Data download/processing must be explicitly called. """ -from pathlib import Path +from dataclasses import dataclass from typing import Any, Dict, List, Optional, Tuple, Union from maseval import Task, TaskQueue @@ -19,29 +19,80 @@ # ============================================================================= -# Constants +# Judge Engine Configuration # ============================================================================= -DEFAULT_DATA_DIR = Path(__file__).parent / "data" +@dataclass +class Gaia2JudgeEngineConfig: + """Configuration for the ARE judge's LLM engine used in semantic comparison. + + ARE's ``GraphPerEventJudge`` uses an LLM to semantically compare tool arguments + (e.g., email content, calendar event descriptions) between agent actions and oracle + (expected) actions. This config controls which model and provider the judge uses. + + Defaults match ARE's built-in defaults. + + ARE's ``LLMEngineConfig`` only supports ``model_name``, ``provider``, and + ``endpoint``. Provider-specific parameters (e.g., OpenRouter's ``fallbacks`` + or ``route``) are not supported by ARE's engine pipeline. + + ARE ``validation/configs.py:28-29`` + + Attributes: + model_name: LLM model identifier for the judge engine. + provider: LLM provider name (e.g., ``"huggingface"``, ``"openrouter"``, ``"openai"``). + Passed to LiteLLM as ``custom_llm_provider``. + endpoint: Optional custom API endpoint URL. + + Example:: + + from maseval.benchmark.gaia2 import ( + load_tasks, configure_model_ids, Gaia2JudgeEngineConfig, + ) + + tasks = load_tasks(capability="execution", limit=5) + + # Use OpenRouter instead of HuggingFace for judge LLM + configure_model_ids( + tasks, + judge_engine_config=Gaia2JudgeEngineConfig( + provider="openrouter", + ), + ) + """ + + # ARE validation/configs.py:28 + model_name: str = "meta-llama/Meta-Llama-3.3-70B-Instruct" + # ARE validation/configs.py:29 + provider: str = "huggingface" + endpoint: Optional[str] = None + + +# ============================================================================= +# Constants +# ============================================================================= + +# HuggingFace config names that exist on the dataset. +# Each config corresponds to a capability and contains ~160 scenarios. +# ARE simulation/types.py: CapabilityTag enum values VALID_CAPABILITIES: Tuple[str, ...] = ( "execution", "search", "adaptability", "time", "ambiguity", - "agent2agent", - "noise", ) VALID_SPLITS: Tuple[str, ...] = ("validation",) # Only validation has oracle events -DEFAULT_CONFIG = "validation" # Full dataset -DEFAULT_TIMEOUT_SECONDS = 600.0 # 10 minutes per task +# ARE scenarios/config.py:20: DEFAULT_SCENARIO_TIMEOUT = 1860 +DEFAULT_TIMEOUT_SECONDS = 1860.0 # 31 minutes per task (matching ARE) DEFAULT_MAX_RETRIES = 1 # HuggingFace dataset info HF_DATASET_ID = "meta-agents-research-environments/gaia2" +HF_DATASET_REVISION = "78ea3bdbdeec2bdcd6afa5420915d8a22f23ed99" # ============================================================================= @@ -58,24 +109,24 @@ def load_tasks( ) -> TaskQueue: """Load Gaia2 tasks from HuggingFace. + Each HuggingFace config corresponds to a capability (execution, search, + adaptability, time, ambiguity). When ``capability`` is None, all + capabilities are loaded and combined. + + GAIA2 is event-driven: the task query is delivered to agents via the + notification system at runtime (first ``send_message_to_agent`` event), + not as a static field. ``task.query`` is left empty. + Args: - capability: Filter by capability type (execution, search, adaptability, - time, ambiguity, agent2agent, noise). None loads all. + capability: Filter by capability type. None loads all capabilities. split: Dataset split (currently only "validation" available) - limit: Maximum number of tasks to load - timeout_seconds: Maximum execution time per task. Default 600 (10 minutes). - Set to None to disable timeout. + limit: Maximum number of tasks to load (across all capabilities) + timeout_seconds: Maximum execution time per task. Default 1860 (31 minutes, + matching ARE's DEFAULT_SCENARIO_TIMEOUT). Set to None to disable timeout. max_retries: Maximum retry attempts. Default 1 (skip on failure). Returns: - TaskQueue with Task objects containing: - - id: Unique scenario identifier - - query: Initial task instructions - - environment_data: {"scenario": BenchmarkScenario, "capability": str, ...} - - evaluation_data: {"oracle_events": [...], "judge_type": str} - - user_data: {} # Gaia2 uses event-based simulation, not user turns - - metadata: {"capability": str, "universe_id": str, ...} - - protocol: TaskProtocol with timeout and tags + TaskQueue with Task objects. Raises: ValueError: If capability or split is invalid @@ -97,7 +148,7 @@ def load_tasks( # Import dependencies (optional) try: - from datasets import load_dataset + from datasets import Dataset, load_dataset except ImportError as e: raise ImportError("HuggingFace datasets library is required for loading Gaia2 tasks.\nInstall with: pip install datasets") from e @@ -110,129 +161,117 @@ def load_tasks( "Or: uv add --optional gaia2 meta-agents-research-environments" ) from e - # Determine HuggingFace config name - config_name = capability if capability else DEFAULT_CONFIG + # When no capability specified, load all capabilities and combine + capabilities = [capability] if capability else list(VALID_CAPABILITIES) - # Load dataset from HuggingFace - dataset = load_dataset( - HF_DATASET_ID, - name=config_name, - split=split, - ) - - # Apply limit - if limit: - dataset = dataset.select(range(min(limit, len(dataset)))) - - # Convert to MASEval Task objects importer = JsonScenarioImporter() - tasks = [] - - for row in dataset: - # Parse scenario from JSON - scenario, oracle_events, _ = importer.import_from_json_to_benchmark(json_str=row["data"]) - - task = _convert_gaia2_to_maseval( - row=row, - scenario=scenario, - oracle_events=oracle_events, - timeout_seconds=timeout_seconds, - max_retries=max_retries, + tasks: List[Task] = [] + + for cap in capabilities: + # Each capability is a HuggingFace config name + # Passing `split` guarantees the return type is Dataset (not DatasetDict) + dataset = load_dataset( + HF_DATASET_ID, + name=cap, + split=split, + revision=HF_DATASET_REVISION, ) - tasks.append(task) + assert isinstance(dataset, Dataset) + + for row in dataset: + # Parse scenario from JSON + # import_from_json_to_benchmark returns (scenario, completed_events, _) + # completed_events are from previous runs, not oracle events. + # Oracle events are generated at runtime by preprocess_scenario(). + scenario, _, _ = importer.import_from_json_to_benchmark(json_str=row["data"]) + + task = _convert_gaia2_to_maseval( + row=row, + scenario=scenario, + timeout_seconds=timeout_seconds, + max_retries=max_retries, + config_capability=cap, + ) + tasks.append(task) + + if limit and len(tasks) >= limit: + break + + if limit and len(tasks) >= limit: + break return TaskQueue(tasks) -def _get_scenario_metadata(scenario: Any, key: str, default: Any = None) -> Any: - """Safely get metadata from an ARE scenario object. - - Args: - scenario: ARE BenchmarkScenario object - key: Metadata key to retrieve - default: Default value if key not found - - Returns: - The metadata value or default - """ - metadata = getattr(scenario, "metadata", None) - if metadata is None: - return default - if isinstance(metadata, dict): - return metadata.get(key, default) - # Try attribute access as fallback - return getattr(metadata, key, default) - - def _convert_gaia2_to_maseval( row: Dict[str, Any], scenario: Any, - oracle_events: List[Any], timeout_seconds: Optional[float], max_retries: int, + config_capability: str, ) -> Task: """Convert Gaia2 scenario to MASEval Task. + GAIA2 is event-driven: the task query is delivered via the notification + system at runtime (first ``send_message_to_agent`` event). There is no + static query field on ARE scenario objects. + ARE agents/default_agent/are_simulation_main.py:79-102 + + Oracle events are generated at runtime by ``preprocess_scenario()`` during + environment setup, not at data-load time. + Args: row: Raw row from HuggingFace dataset scenario: ARE BenchmarkScenario object - oracle_events: List of oracle events for evaluation timeout_seconds: Maximum execution time per task max_retries: Maximum retry attempts + config_capability: The capability from the HuggingFace config name Returns: MASEval Task object """ - # Extract query from scenario's task definition - query = getattr(scenario, "task_instruction", "") - - # Parse capability from scenario metadata or row - capability = row.get("category") or _get_scenario_metadata(scenario, "capability", "unknown") + scenario_id = getattr(scenario, "scenario_id", None) # Build environment_data + # Duration is NOT set here — ARE's preprocess_scenario() sets it during + # environment setup based on capability (1800s standard, 420s for Time). + # ARE scenarios/config.py:18-19, scenarios/scenario_imported_from_json/utils.py:69-76 environment_data: Dict[str, Any] = { "scenario": scenario, - "capability": capability, - "universe_id": _get_scenario_metadata(scenario, "universe_id"), - "duration": getattr(scenario, "duration", 86400), + "capability": config_capability, } - # Build evaluation_data with oracle events + # Evaluation uses scenario.judge.validate() at runtime (created by + # preprocess_scenario). No static evaluation data needed at load time. + # ARE scenarios/scenario_imported_from_json/utils.py:110-112 evaluation_data: Dict[str, Any] = { - "oracle_events": oracle_events, - "judge_type": _get_scenario_metadata(scenario, "judge_type", "graph_per_event"), + "judge_type": "graph_per_event", } - # Build metadata metadata: Dict[str, Any] = { - "scenario_id": row.get("scenario_id") or row.get("id"), - "capability": capability, - "universe_id": environment_data.get("universe_id"), + "scenario_id": scenario_id or row.get("scenario_id") or row.get("id"), + "capability": config_capability, } - # Build protocol protocol = TaskProtocol( timeout_seconds=timeout_seconds, max_retries=max_retries, - tags={"capability": capability, "benchmark": "gaia2"}, + tags={"capability": config_capability, "benchmark": "gaia2"}, ) - # Build task kwargs - task_kwargs: Dict[str, Any] = { - "query": query, - "environment_data": environment_data, - "evaluation_data": evaluation_data, - "user_data": {}, # Gaia2 uses event-based simulation - "metadata": metadata, - "protocol": protocol, - } - - # Include id if provided task_id = row.get("id") or row.get("scenario_id") - if task_id: - task_kwargs["id"] = str(task_id) - - return Task(**task_kwargs) + if not task_id: + raise ValueError("HuggingFace row missing both 'id' and 'scenario_id' fields") + + return Task( + id=str(task_id), + query="", # Event-driven: real query comes from notification system at runtime + environment_data=environment_data, + evaluation_data=evaluation_data, + user_data={}, + metadata=metadata, + protocol=protocol, + ) # ============================================================================= @@ -244,27 +283,36 @@ def configure_model_ids( tasks: Union[TaskQueue, List[Task]], *, evaluator_model_id: Optional[str] = None, + judge_engine_config: Optional[Gaia2JudgeEngineConfig] = None, ) -> Union[TaskQueue, List[Task]]: - """Configure model IDs for benchmark components in task data. + """Configure model IDs and judge engine for benchmark components. - Gaia2 uses ARE's deterministic judge by default, but can optionally - use an LLM-based judge for complex assertions. + Gaia2's ``GraphPerEventJudge`` uses an LLM for semantic comparison of tool + arguments (email content, calendar descriptions, etc.). By default it uses + ARE's built-in defaults (``meta-llama/Meta-Llama-3.3-70B-Instruct`` via + HuggingFace). Pass ``judge_engine_config`` to override the model/provider. Note: Unlike Tau2, Gaia2 doesn't have a user simulator (interactions happen through scheduled events), so there's no user_model_id. Args: - tasks: TaskQueue or list of Tasks to configure - evaluator_model_id: Optional model ID for LLM-based evaluation + tasks: TaskQueue or list of Tasks to configure. + evaluator_model_id: Optional model ID for LLM-based evaluation. + judge_engine_config: Optional judge engine configuration. Controls + which LLM model and provider the ARE judge uses for semantic + comparison. When ``None``, ARE's defaults are used. Returns: - The same collection (mutated in place for convenience) + The same collection (mutated in place for convenience). + + Example:: - Example: >>> tasks = load_tasks(capability="execution", limit=5) >>> configure_model_ids( ... tasks, - ... evaluator_model_id="gpt-4o", # Optional, for LLM-based judge + ... judge_engine_config=Gaia2JudgeEngineConfig( + ... provider="openrouter", + ... ), ... ) """ for task in tasks: @@ -277,6 +325,10 @@ def configure_model_ids( ) task.evaluation_data["model_id"] = evaluator_model_id + # Evaluation data: judge engine configuration (optional) + if judge_engine_config is not None: + task.evaluation_data["judge_engine_config"] = judge_engine_config + return tasks @@ -290,6 +342,6 @@ def configure_model_ids( tasks = load_tasks(limit=5) print(f"Loaded {len(tasks)} tasks") for task in tasks: - print(f" - {task.id}: {task.query[:50]}...") + print(f" - {task.id} (capability={task.metadata.get('capability')})") except ImportError as e: print(f"Error: {e}") diff --git a/maseval/benchmark/gaia2/environment.py b/maseval/benchmark/gaia2/environment.py index 4aec938..1da9923 100644 --- a/maseval/benchmark/gaia2/environment.py +++ b/maseval/benchmark/gaia2/environment.py @@ -5,11 +5,11 @@ Reference Paper: "GAIA-2: A Controllable Multi-Turn Conversational Benchmark for Agents" """ -from typing import Any, Dict, List, Optional +from typing import Any, Dict, List, Optional, Tuple from maseval import Environment -from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper +from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool class Gaia2Environment(Environment): @@ -33,6 +33,7 @@ def __init__( self, task_data: Dict[str, Any], callbacks: Optional[List[Any]] = None, + judge_engine_config: Optional[Any] = None, ): """Initialize Gaia2 environment. @@ -41,18 +42,31 @@ def __init__( - scenario: ARE BenchmarkScenario object - capability: Capability type (execution, search, etc.) - universe_id: Universe identifier - - duration: Scenario duration in seconds callbacks: Optional callbacks + judge_engine_config: Optional :class:`Gaia2JudgeEngineConfig` controlling + which LLM model and provider the ARE judge uses for semantic comparison. + Passed explicitly from ``setup_environment()`` (lives in ``evaluation_data``). """ self._scenario = task_data.get("scenario") + self._judge_engine_config = judge_engine_config self._are_env: Any = None - self._tool_wrappers: Dict[str, AREToolWrapper] = {} + self._tool_wrappers: Dict[str, Gaia2GenericTool] = {} super().__init__(task_data, callbacks) def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: """Initialize ARE scenario and start simulation. + Delegates to ARE's ``preprocess_scenario()`` for faithful preprocessing: + + 1. Ensure SystemApp is present. + 2. Set scenario duration from ARE defaults (1800s standard, 420s for Time). + 3. Initialize the scenario (populates apps, events). + 4. Run oracle mode to generate expected event log. + 5. Soft-reset so app state is clean for agent run. + 6. Create judge and initialize turns with trigger conditions. + 7. Start the agent-mode simulation. + Args: task_data: Task data with scenario, capability, universe_id @@ -63,6 +77,11 @@ def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: try: from are.simulation.environment import Environment as AREEnvironment # type: ignore[import-not-found] from are.simulation.environment import EnvironmentConfig # type: ignore[import-not-found] + from are.simulation.scenarios.scenario_imported_from_json.utils import ( # type: ignore[import-not-found] + get_scenario_duration, + preprocess_scenario, + ) + from are.simulation.validation import GraphPerEventJudgeConfig # type: ignore[import-not-found] except ImportError as e: raise ImportError( "ARE (Agent Research Environments) is required for Gaia2 benchmark.\n" @@ -70,47 +89,127 @@ def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: "Or: uv add --optional gaia2 meta-agents-research-environments" ) from e + # Scenario duration defaults from ARE scenarios/config.py:18-19 + from are.simulation.scenarios.config import ( # type: ignore[import-not-found] + MAX_SCENARIO_DURATION, + MAX_TIME_SCENARIO_DURATION, + ) + scenario = task_data.get("scenario") if scenario is None: raise ValueError("Task data must contain 'scenario' with ARE BenchmarkScenario") - # Create ARE environment with config + # Determine scenario duration (matching ARE's get_scenario_duration) + # ARE scenarios/config.py:18: MAX_SCENARIO_DURATION = 1800 (30 min) + # ARE scenarios/config.py:19: MAX_TIME_SCENARIO_DURATION = 420 (7 min) + max_duration = get_scenario_duration(scenario, MAX_TIME_SCENARIO_DURATION, MAX_SCENARIO_DURATION) + + # Use ARE's preprocess_scenario() for faithful preprocessing. + # This handles: SystemApp insertion, duration setting, scenario initialization, + # oracle run, soft reset, judge creation, turn initialization with trigger + # conditions, and judge state initialization. + # GraphPerEventJudge uses an LLM for semantic comparison of tool arguments + # (email content, calendar descriptions, etc.) via soft checkers. + # ARE scenarios/scenario_imported_from_json/utils.py:43-157 + if self._judge_engine_config is not None: + # User provided custom judge engine config — create engine explicitly + # ARE validation/configs.py:32-59 + from are.simulation.agents.are_simulation_agent_config import ( # type: ignore[import-not-found] + LLMEngineConfig, + ) + from are.simulation.validation.configs import create_judge_engine # type: ignore[import-not-found] + + llm_engine_config = LLMEngineConfig( + model_name=self._judge_engine_config.model_name, + provider=self._judge_engine_config.provider, + endpoint=self._judge_engine_config.endpoint, + ) + engine = create_judge_engine(llm_engine_config) + judge_config = GraphPerEventJudgeConfig(engine=engine) + else: + # Default: use ARE's built-in defaults (Llama 3.3 70B via HuggingFace) + # ARE validation/configs.py:28-29, 149 + judge_config = GraphPerEventJudgeConfig() + + preprocess_scenario( + scenario=scenario, + judge_config=judge_config, + max_scenario_duration=max_duration, + ) + + # Create ARE environment for the agent run + # Match ARE scenario_runner.py:267-282 + from are.simulation.notification_system import VerboseNotificationSystem # type: ignore[import-not-found] + config = EnvironmentConfig( oracle_mode=False, - duration=getattr(scenario, "duration", 86400), # Default 24 hours + duration=scenario.duration, + time_increment_in_seconds=scenario.time_increment_in_seconds, ) - self._are_env = AREEnvironment(config) - - # Initialize scenario (loads apps, events, state) - self._are_env.initialize_scenario(scenario) + if scenario.start_time and scenario.start_time > 0: + config.start_time = scenario.start_time + # Match ARE scenario_runner.py:281: VerboseNotificationSystem() defaults + # to VerbosityLevel.MEDIUM, which includes environment notifications + # (email, messaging, shopping, cab, calendar). Without this, the default + # is VerbosityLevel.LOW (no environment notifications). + self._are_env = AREEnvironment(config, notification_system=VerboseNotificationSystem()) + + # Run scenario (registers apps, schedules events, starts event loop) + # wait_for_end=False so control returns immediately for agent interaction + self._are_env.run(scenario, wait_for_end=False, schedule_events=True) return { "scenario_id": getattr(scenario, "scenario_id", None), - "duration": getattr(scenario, "duration", None), + "duration": scenario.duration, "capability": task_data.get("capability"), "universe_id": task_data.get("universe_id"), + "start_time": getattr(scenario, "start_time", None), } - def create_tools(self) -> Dict[str, AREToolWrapper]: - """Wrap all ARE app tools for MASEval tracing. + # Tools removed by ARE's remove_aui_irrelevant_tools() + # ARE agents/default_agent/are_simulation_main.py:206-228 + # User messages are delivered via the notification system, not via these tools. + _AUI_TOOLS_TO_REMOVE = { + "AgentUserInterface__get_last_message_from_user", + "AgentUserInterface__get_last_message_from_agent", + "AgentUserInterface__get_last_unread_messages", + "AgentUserInterface__get_all_messages", + } + + def create_tools(self) -> Dict[str, Gaia2GenericTool]: + """Wrap ARE app tools for MASEval tracing. - Includes critical tools: - - SystemApp.get_current_time(): Query simulation time - - SystemApp.wait_for_notification(timeout): Advance simulation time - - All domain app tools (calendar, email, messaging, etc.) + Creates framework-agnostic Gaia2GenericTool instances that provide + clean API with built-in tracing. + + Filters out AgentUserInterface message-retrieval tools that ARE removes + in ``remove_aui_irrelevant_tools()``, and sets ``wait_for_user_response`` + to ``False`` so the AUI does not block waiting for a response when the + agent sends a message. User messages are delivered via the notification + system instead. + + ARE agents/default_agent/are_simulation_main.py:206-228 Returns: - Dict mapping tool names to AREToolWrapper instances + Dict mapping tool names to Gaia2GenericTool instances """ - tools: Dict[str, AREToolWrapper] = {} + tools: Dict[str, Gaia2GenericTool] = {} if self._are_env is None: return tools - # Get all tools from all apps in the ARE environment + # Get all tools from all apps, filtering out AUI message-retrieval tools + # ARE agents/default_agent/are_simulation_main.py:221-227 for app in self._are_env.apps.values(): + # Set wait_for_user_response=False on AUI so it doesn't block + # ARE agents/default_agent/are_simulation_main.py:216 + if hasattr(app, "wait_for_user_response"): + app.wait_for_user_response = False + for tool in app.get_tools(): - wrapper = AREToolWrapper(tool, self) + if tool.name in self._AUI_TOOLS_TO_REMOVE: + continue + wrapper = Gaia2GenericTool(tool, self) tools[tool.name] = wrapper self._tool_wrappers[tool.name] = wrapper @@ -126,7 +225,7 @@ def get_simulation_time(self) -> float: return 0.0 try: - return self._are_env.time_manager.current_time + return self._are_env.current_time except AttributeError: return 0.0 @@ -148,6 +247,170 @@ def get_are_environment(self) -> Any: """ return self._are_env + def get_notification_system(self) -> Any: + """Get the ARE notification system. + + Used by agents that need to poll for messages between iterations, + matching ARE's pre-step notification polling behavior. + + Returns: + ARE NotificationSystem instance, or None if not available + """ + if self._are_env is None: + return None + return getattr(self._are_env, "notification_system", None) + + def poll_notifications(self) -> Tuple[List[str], List[str], bool]: + """Poll pending notifications from the ARE notification system. + + Drains all pending messages from the notification queue and returns + them as pre-formatted strings. Call this between agent steps to + receive messages that arrived during ``wait_for_notification()`` or + from background simulation events. + + GAIA2 uses an event-driven multi-turn architecture. When the agent + calls ``SystemApp__wait_for_notification``, the ARE environment + processes scheduled events, advances simulation time, and queues + notifications. After the tool returns, call this method to retrieve + those notifications and inject them into the agent's context before + the next LLM call. + + ARE agents/default_agent/steps/are_simulation.py:26-62 + + Returns: + Tuple of ``(user_messages, env_notifications, has_stop_message)``. + ``user_messages`` and ``env_notifications`` contain pre-formatted + strings ready to inject into agent context. ``has_stop_message`` + is True when the environment has signalled the simulation is over. + """ + notification_system = self.get_notification_system() + if notification_system is None: + return [], [], False + + try: + from datetime import datetime, timezone + + from are.simulation.notification_system import MessageType # type: ignore[import-not-found] + + # Use simulation time, not wall-clock time. Notifications are timestamped + # with simulation time (via TimeManager), so querying with wall-clock would + # drain all messages prematurely. Matches ARE agents/default_agent/steps/are_simulation.py:30-32. + sim_time = self.get_simulation_time() + timestamp = datetime.fromtimestamp(sim_time, tz=timezone.utc) + unhandled = notification_system.message_queue.get_by_timestamp(timestamp=timestamp) + + if not unhandled: + return [], [], False + + # Separate by message type, matching ARE steps/are_simulation.py:34-61 + user_messages: List[str] = [] + env_notifications: List[str] = [] + has_stop = False + + for notif in unhandled: + msg_type = getattr(notif, "message_type", None) + if msg_type == MessageType.USER_MESSAGE: + user_messages.append(notif.message) + elif msg_type == MessageType.ENVIRONMENT_NOTIFICATION: + ts = notif.timestamp.strftime("%Y-%m-%d %H:%M:%S") if notif.timestamp else "" + env_notifications.append(f"[{ts}] {notif.message}") + elif msg_type == MessageType.ENVIRONMENT_STOP: + has_stop = True + + return user_messages, env_notifications, has_stop + + except Exception: + return [], [], False + + def get_turn_notifications(self) -> Tuple[List[str], bool, bool]: + """Get notifications for turn transitions, re-queuing env notifications. + + Matches ARE's ``get_notifications()`` in ``are_simulation_main.py:331-359``: + drains the notification queue, separates by type, re-queues environment + notifications (so the inner loop's pre-step picks them up), and returns + user messages and status flags. + + Returns: + Tuple of ``(user_messages, has_env_notifications, has_stop)``. + ``user_messages`` are raw message strings for ``[TASK]`` formatting. + ``has_env_notifications`` is True when env notifications were re-queued. + ``has_stop`` is True when the environment signalled stop. + """ + notification_system = self.get_notification_system() + if notification_system is None: + return [], False, False + + try: + from datetime import datetime, timezone + + from are.simulation.notification_system import MessageType # type: ignore[import-not-found] + + sim_time = self.get_simulation_time() + timestamp = datetime.fromtimestamp(sim_time, tz=timezone.utc) + unhandled = notification_system.message_queue.get_by_timestamp(timestamp=timestamp) + + if not unhandled: + return [], False, False + + user_messages: List[str] = [] + has_env = False + has_stop = False + + for notif in unhandled: + msg_type = getattr(notif, "message_type", None) + if msg_type == MessageType.USER_MESSAGE: + user_messages.append(notif.message) + elif msg_type == MessageType.ENVIRONMENT_NOTIFICATION: + # Re-queue for inner loop's pre-step to pick up + # ARE are_simulation_main.py:349-352 + notification_system.message_queue.put(notif) + has_env = True + elif msg_type == MessageType.ENVIRONMENT_STOP: + has_stop = True + + return user_messages, has_env, has_stop + + except Exception: + return [], False, False + + def get_start_time(self) -> Optional[float]: + """Get the scenario start time. + + Returns: + Start time as Unix timestamp, or None if not available + """ + return self.state.get("start_time") + + def pause(self) -> None: + """Pause the ARE simulation environment. + + Stops time progression during LLM generation, matching ARE's + simulated generation time behavior. + ARE simulation/environment.py:262-272 + + No-op if environment is not available or not running. + """ + if self._are_env is not None: + try: + self._are_env.pause() + except Exception: + pass + + def resume_with_offset(self, offset: float) -> None: + """Resume the ARE simulation environment with a time offset. + + Advances simulation time by the given offset and resumes the event loop. + ARE simulation/environment.py:286-298 + + Args: + offset: Time in seconds to advance the simulation clock + """ + if self._are_env is not None: + try: + self._are_env.resume_with_offset(offset) + except Exception: + pass + def cleanup(self) -> None: """Stop ARE simulation when task completes. diff --git a/maseval/benchmark/gaia2/evaluator.py b/maseval/benchmark/gaia2/evaluator.py index 7b58382..c27f7fc 100644 --- a/maseval/benchmark/gaia2/evaluator.py +++ b/maseval/benchmark/gaia2/evaluator.py @@ -5,12 +5,15 @@ Reference Paper: "GAIA-2: A Controllable Multi-Turn Conversational Benchmark for Agents" """ +import logging from typing import Any, Dict, List, Optional from maseval import Evaluator, Task, TaskExecutionStatus from maseval.benchmark.gaia2.environment import Gaia2Environment +logger = logging.getLogger(__name__) + # Statuses where agent is accountable (included in scoring) SCOREABLE_STATUSES = frozenset( @@ -25,8 +28,9 @@ class Gaia2Evaluator(Evaluator): """Evaluates Gaia2 scenarios using ARE's judge system. - Uses ARE's GraphPerEventJudge for deterministic evaluation based on - the event DAG. Supports optional LLM-based judge for complex assertions. + Uses ARE's ``GraphPerEventJudge`` which combines deterministic hard checks + (exact value matching) with LLM-based soft checks (semantic comparison of + content like email bodies and calendar descriptions). The evaluator compares completed events in the simulation against oracle (expected) events to compute Goal Success Rate (GSR). @@ -57,7 +61,6 @@ def __init__( # Extract evaluation data from task eval_data = task.evaluation_data - self.oracle_events = eval_data.get("oracle_events", []) self.judge_type = eval_data.get("judge_type", "graph_per_event") def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: @@ -101,70 +104,120 @@ def __call__( ) -> Dict[str, Any]: """Evaluate using ARE's judge system. + Uses the judge created during ``preprocess_scenario()`` (attached to the + scenario object) rather than creating a new one. This ensures turn + initialization and judge state are consistent. + + Exceptions return ``gsr=None`` (excluded from scoring), matching ARE's + behavior where exceptions/no_validation get ``score=None``. + ARE benchmark/hf_upload_utils.py:33-52, benchmark/report_stats.py + Args: traces: Filtered execution traces final_answer: Final answer from agent (not used in Gaia2) Returns: - Dict with: - - gsr: Goal Success Rate (0.0 or 1.0) - - partial_gsr: Partial success rate - - passed: Boolean indicating full success - - event_results: Per-event evaluation results - - capability: Task capability type + Dict with evaluation results. ``gsr`` is None for evaluation errors + (excluded from scoring) or a float for valid results. """ - # Import ARE judge (required dependency for Gaia2) - from are.simulation.validation import JudgeFactory # type: ignore[import-not-found] - from are.simulation.validation.config import GraphPerEventJudgeConfig # type: ignore[import-not-found] - - # Create ARE judge - judge_config = GraphPerEventJudgeConfig() - judge = JudgeFactory.create(judge_config) - - # Get ARE environment and completed events + # Get ARE environment are_env = self.environment.get_are_environment() if are_env is None: + # Infrastructure error: return None score (excluded from scoring) + # ARE benchmark/hf_upload_utils.py:47-48 return { - "gsr": 0.0, - "partial_gsr": 0.0, + "gsr": None, + "partial_gsr": None, "passed": False, + "status": "no_validation", "error": "ARE environment not available", "capability": self.task.metadata.get("capability"), } try: - completed_events = are_env.get_completed_events() - except AttributeError: - completed_events = [] - - # Run ARE's judge - try: - result = judge.evaluate( - oracle_events=self.oracle_events, - completed_events=completed_events, - scenario=self.environment.get_scenario(), - ) - - # Convert ARE result to MASEval format - gsr = 1.0 if result.passed else 0.0 - partial_gsr = getattr(result, "partial_score", gsr) + # Use the scenario's judge (created during preprocess_scenario) + # ARE scenarios/scenario_imported_from_json/utils.py:112 + scenario = self.environment.get_scenario() + judge = getattr(scenario, "judge", None) + + if judge is None: + # Fallback: create judge if not available on scenario + from are.simulation.validation import GraphPerEventJudgeConfig, JudgeFactory # type: ignore[import-not-found] + + judge_engine_config = self.task.evaluation_data.get("judge_engine_config") + if judge_engine_config is not None: + from are.simulation.agents.are_simulation_agent_config import ( # type: ignore[import-not-found] + LLMEngineConfig, + ) + from are.simulation.validation.configs import create_judge_engine # type: ignore[import-not-found] + + llm_engine_config = LLMEngineConfig( + model_name=judge_engine_config.model_name, + provider=judge_engine_config.provider, + endpoint=judge_engine_config.endpoint, + ) + engine = create_judge_engine(llm_engine_config) + judge_cfg = GraphPerEventJudgeConfig(engine=engine) + else: + judge_cfg = GraphPerEventJudgeConfig() + + judge = JudgeFactory()(judge_cfg) + judge.initialize_state(scenario) + + # Ensure intermediate turns are judged before final validation. + # + # ARE's validate() checks (turn_idx + 1) == (nb_turns - 1) to + # confirm it's on the last turn before running final validation. + # ARE simulation/validation/base.py:104 + # + # MASEval uses a fixed run-then-evaluate cycle with no hook to + # call the judge between turns. ARE's online-mode trigger + # conditions run in the background event loop and MAY have + # already advanced turn_idx, but MASEval does not control this. + # We check turn_idx and only judge turns that haven't been + # processed yet — this is idempotent regardless of whether + # trigger conditions fired. + nb_turns = judge.state.nb_turns + last_intermediate_turn = nb_turns - 2 # turns 0..N-2 are intermediate + while judge.state.turn_idx < last_intermediate_turn: + turn = judge.state.turn_idx + 1 + judgment = judge(are_env) + if not judgment.success: + logger.info("Intermediate turn %d/%d failed: %s", turn, nb_turns - 1, judgment.failure) + break # validate() will return failure via last_turn_success check + + # Run ARE's judge validation for the final turn + result = judge.validate(are_env) + + # Convert ARE ScenarioValidationResult to MASEval format + passed = bool(result.success) + gsr = 1.0 if passed else 0.0 + + # Extract partial GSR from judge result if available + # ARE's judge can produce partial scores based on event matching + partial_gsr = getattr(result, "partial_success_rate", gsr) + if partial_gsr is None: + partial_gsr = gsr return { "gsr": gsr, "partial_gsr": partial_gsr, - "passed": result.passed, - "event_results": getattr(result, "event_results", []), + "passed": passed, + "status": "success" if passed else "failed", + "rationale": getattr(result, "rationale", None), "capability": self.task.metadata.get("capability"), "tool_call_count": len(traces.get("tool_invocations", [])), "final_simulation_time": traces.get("simulation_time", 0), } except Exception as e: - # Return failure on evaluation error + # Evaluation error: return None score (excluded from scoring) + # ARE benchmark/hf_upload_utils.py:42-46: exceptions get score=None return { - "gsr": 0.0, - "partial_gsr": 0.0, + "gsr": None, + "partial_gsr": None, "passed": False, + "status": "exception", "error": str(e), "capability": self.task.metadata.get("capability"), "tool_call_count": len(traces.get("tool_invocations", [])), @@ -175,20 +228,16 @@ def __call__( def compute_gaia2_metrics(results: List[Dict[str, Any]]) -> Dict[str, Any]: """Compute summary metrics across all Gaia2 benchmark results. - Infrastructure errors are excluded from scoring metrics. - Uses SCOREABLE_STATUSES to determine which results count toward agent score. + Matches ARE's scoring logic: + - Only validated runs (non-null GSR) count toward success rate + - Exceptions and no_validation results are excluded from scoring + - ARE benchmark/report_stats.py: success_rate calculated only from validated runs Args: results: List of result dicts from benchmark.run() Returns: - Dict with: - - total_tasks: Total number of tasks - - scored_tasks: Tasks included in scoring - - gsr: Overall Goal Success Rate - - partial_gsr: Average partial GSR - - by_capability: Metrics broken down by capability type - - status_counts: Count by status + Dict with metrics including total_tasks, scored_tasks, GSR, and per-capability breakdown. """ if not results: return { @@ -214,16 +263,21 @@ def compute_gaia2_metrics(results: List[Dict[str, Any]]) -> Dict[str, Any]: if status not in SCOREABLE_STATUSES: continue # Skip infrastructure errors - scored_tasks += 1 evals = res.get("eval") or [] for entry in evals: - gsr = entry.get("gsr", 0.0) - partial_gsr = entry.get("partial_gsr", 0.0) + gsr = entry.get("gsr") + partial_gsr = entry.get("partial_gsr") capability = entry.get("capability", "unknown") + # Skip entries with None score (exceptions, no_validation) + # ARE benchmark/report_stats.py: only validated runs count + if gsr is None: + continue + + scored_tasks += 1 total_gsr += gsr - total_partial_gsr += partial_gsr + total_partial_gsr += partial_gsr if partial_gsr is not None else gsr # Track by capability if capability not in by_capability: @@ -236,7 +290,7 @@ def compute_gaia2_metrics(results: List[Dict[str, Any]]) -> Dict[str, Any]: by_capability[capability]["count"] += 1 by_capability[capability]["gsr_sum"] += gsr - by_capability[capability]["partial_gsr_sum"] += partial_gsr + by_capability[capability]["partial_gsr_sum"] += partial_gsr if partial_gsr is not None else gsr if entry.get("passed", False): by_capability[capability]["passed"] += 1 diff --git a/maseval/benchmark/gaia2/gaia2.py b/maseval/benchmark/gaia2/gaia2.py index 0b42714..0196220 100644 --- a/maseval/benchmark/gaia2/gaia2.py +++ b/maseval/benchmark/gaia2/gaia2.py @@ -45,11 +45,12 @@ def get_model_adapter(self, model_id, **kwargs): results = benchmark.run(tasks) """ -import json import re +import time from abc import abstractmethod from pathlib import Path -from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple +from dataclasses import dataclass +from typing import Any, Callable, Dict, List, Literal, Optional, Sequence, Tuple from maseval import AgentAdapter, Benchmark, Evaluator, ModelAdapter, Task, User from maseval.core.callback import BenchmarkCallback @@ -71,14 +72,43 @@ class Gaia2Benchmark(Benchmark): MASEval orchestration, tracing, and agent flexibility. The ARE simulation runs internally; agents interact purely via tool calls. - Time control happens through SystemApp.wait_for_notification(). + Time control happens through ``SystemApp__wait_for_notification``. Subclasses must implement: - - setup_agents(): Create agents for the task - - get_model_adapter(): Provide model adapters + + - ``setup_agents()`` — Create agents for the task + - ``get_model_adapter()`` — Provide model adapters + + Multi-Turn Architecture: + GAIA2 uses ARE's **two-level loop** architecture: + + - **Outer loop** (turns): drains the notification queue, formats user + messages as ``[TASK]``, re-queues environment notifications, then + runs the inner step loop. + - **Inner loop** (steps): ReAct cycle. Terminates on + ``send_message_to_user`` (TERMINATED — turn complete) or + ``wait_for_notification`` (PAUSED — outer loop continues). + + ``ARE are_simulation_main.py:agent_loop()`` + + **What custom agents must do:** + + - **Terminate inner loop** on both ``send_message_to_user`` and + ``wait_for_notification``. The former completes a turn; the latter + pauses the agent while ARE processes events. + - **Between turns** (outer loop): drain notifications via + ``environment.get_turn_notifications()`` which re-queues environment + notifications and returns user messages for ``[TASK]`` formatting. + - **Within turns** (inner loop pre-step): poll notifications via + ``environment.poll_notifications()`` to pick up re-queued environment + notifications and new messages. + + See the default agent implementation for the reference two-level loop + approach. """ - # Single-turn by default (ARE handles time internally via tools) + # The benchmark invokes the agent once; multi-turn is the agent's + # responsibility (see "Multi-Turn Notification Loop" above). MAX_INVOCATIONS = 1 def __init__( @@ -138,7 +168,8 @@ def setup_environment( Returns: Gaia2Environment instance """ - return Gaia2Environment(task_data=task.environment_data) + judge_engine_config = task.evaluation_data.get("judge_engine_config") + return Gaia2Environment(task_data=task.environment_data, judge_engine_config=judge_engine_config) def setup_user( # type: ignore[override] self, @@ -262,7 +293,7 @@ def evaluate( - gsr: Goal Success Rate - partial_gsr: Partial success rate - passed: Boolean - - event_results: Per-event results + - rationale: Judge rationale (if available) Args: evaluators: List of evaluators @@ -288,23 +319,71 @@ def evaluate( # Prompt templates directory _PROMPT_TEMPLATES_DIR = Path(__file__).parent / "prompt_templates" -# ARE default parameters +# ARE default parameters (documented with source locations) +# ARE agents/are_simulation_agent_config.py:36 _DEFAULT_MAX_ITERATIONS = 80 +# ARE agents/llm/llm_engine.py:17 _DEFAULT_TEMPERATURE = 0.5 +# ARE agents/llm/llm_engine.py:16 _DEFAULT_MAX_TOKENS = 16384 +# ARE agents/default_agent/base_agent.py:347 _DEFAULT_INVALID_FORMAT_RETRIES = 10 + +@dataclass +class Gaia2SimulatedGenerationTimeConfig: + """Configuration for simulating the LLM's generation time. + + Matches ARE's ``SimulatedGenerationTimeConfig`` (ARE types.py:1574-1592). + + In "measured" mode, the simulation clock advances by the actual LLM + generation wall-clock time. In "fixed" mode, it advances by a fixed + number of seconds. + + Attributes: + mode: "measured" uses actual LLM wall-clock time; "fixed" uses ``seconds``. + seconds: Fixed offset in seconds. Used when mode is "fixed". + """ + + # ARE types.py:1586 + mode: Literal["fixed", "measured"] = "measured" + # ARE types.py:1587 + seconds: Optional[float] = 1.0 + + def __post_init__(self) -> None: + # ARE types.py:1589-1592 + if self.mode == "fixed": + if self.seconds is None: + raise ValueError("When mode is 'fixed', seconds must be provided and cannot be None.") + + # Stop sequences for text-based action parsing _STOP_SEQUENCES = ["", "Observation:"] -# Termination tool names - agent terminates when these are called +# Termination tool names — send_message_to_user terminates the inner step +# loop with TERMINATED state (turn is complete). +# ARE termination_methods/are_simulation.py:112-114 _TERMINATION_TOOLS = frozenset( { "AgentUserInterface__send_message_to_user", - "SystemApp__wait_for_notification", } ) +# Pause tool — wait_for_notification terminates the inner step loop with +# PAUSED state (agent is waiting for events; outer turn loop continues). +# ARE termination_methods/are_simulation.py:34-36, 116-121 +_PAUSE_TOOL = "SystemApp__wait_for_notification" + + +class _RunningState: + """Agent running states matching ARE's RunningState enum. + + ARE agents/default_agent/base_agent.py:RunningState + """ + + TERMINATED = "TERMINATED" + PAUSED = "PAUSED" + def _load_prompt_template(name: str) -> str: """Load a prompt template from file. @@ -319,8 +398,31 @@ def _load_prompt_template(name: str) -> str: return path.read_text() +def _get_tool_description_with_args(name: str, tool: Callable) -> str: + """Build a single tool's description with args, matching ARE's format. + + ARE tool_box.py:16-20 ``DEFAULT_TOOL_DESCRIPTION_TEMPLATE``. + + Args: + name: Tool name + tool: Callable with optional description/inputs/output_type attributes + + Returns: + Formatted tool description string + """ + desc = getattr(tool, "description", "") or "" + inputs = getattr(tool, "inputs", {}) or {} + output_type = getattr(tool, "output_type", "string") or "string" + return f"- {name}: {desc}\n Takes inputs: {inputs}\n Returns an output of type: {output_type}" + + def _build_tool_descriptions(tools: Dict[str, Callable]) -> str: - """Build tool descriptions for embedding in system prompt. + """Build tool descriptions matching ARE's Jinja2 template format. + + ARE tool_box.py:16-20 uses the template: + ``- {{ tool.name }}: {{ tool.description }}`` + `` Takes inputs: {{tool.inputs}}`` + `` Returns an output of type: {{tool.output_type}}`` Args: tools: Dict of tool name -> callable @@ -332,30 +434,106 @@ def _build_tool_descriptions(tools: Dict[str, Callable]) -> str: for name, tool in tools.items(): desc = getattr(tool, "description", "") or "" inputs = getattr(tool, "inputs", {}) or {} + output_type = getattr(tool, "output_type", "string") or "string" + + # Match ARE's format: raw dict representation for inputs + descriptions.append(f"- {name}: {desc}\n Takes inputs: {inputs}\n Returns an output of type: {output_type}") + + return "\n".join(descriptions) + + +def _get_notification_system_prompt(environment: Optional[Any]) -> str: + """Generate notification system prompt matching ARE's behavior. + + ARE agents/default_agent/prompts/notification_system.py:32-46 + + Args: + environment: Optional Gaia2Environment - # Format parameters - params = [] - properties = inputs.get("properties", {}) - required = set(inputs.get("required", [])) + Returns: + Notification system prompt string + """ + if environment is None: + # Fallback: basic notification policy (matches ARE's NotificationSystem default) + return ( + "Notification policy:\n" + "- All new messages from the User will be notified to you.\n" + "- The environment state may also change over time, but environment events will not be notified to you.\n" + "- You can also proactively check for any other update in an App by using the tools given to you.\n" + "- If a call to SystemApp__wait_for_notification times out, you will receive a notification." + ) + + try: + notification_system = environment.get_notification_system() + if notification_system is None: + # Same basic fallback + return ( + "Notification policy:\n" + "- All new messages from the User will be notified to you.\n" + "- The environment state may also change over time, but environment events will not be notified to you.\n" + "- You can also proactively check for any other update in an App by using the tools given to you.\n" + "- If a call to SystemApp__wait_for_notification times out, you will receive a notification." + ) + + # Use ARE's notification prompt generator + from are.simulation.agents.default_agent.prompts.notification_system import ( # type: ignore[import-not-found] + get_notification_system_prompt, + ) - for param_name, param_info in properties.items(): - param_type = param_info.get("type", "any") - param_desc = param_info.get("description", "") - req_marker = " (required)" if param_name in required else " (optional)" - params.append(f" - {param_name}: {param_type}{req_marker} - {param_desc}") + are_env = environment.get_are_environment() + scenario = environment.get_scenario() + apps = getattr(scenario, "apps", None) or (list(are_env.apps.values()) if are_env else None) + return get_notification_system_prompt(notification_system, apps) + except Exception: + # Graceful fallback + return ( + "Notification policy:\n" + "- All new messages from the User will be notified to you.\n" + "- The environment state may also change over time, but environment events will not be notified to you.\n" + "- You can also proactively check for any other update in an App by using the tools given to you.\n" + "- If a call to SystemApp__wait_for_notification times out, you will receive a notification." + ) - params_str = "\n".join(params) if params else " (no parameters)" - descriptions.append(f"Tool: {name}\nDescription: {desc}\nParameters:\n{params_str}") +def _get_current_time_description(environment: Optional[Any]) -> str: + """Generate current time description matching ARE's behavior. + + ARE agents/default_agent/are_simulation_main.py:156-164 + + Args: + environment: Optional Gaia2Environment + + Returns: + Current time description string + """ + if environment is None: + return "" + + try: + from datetime import datetime, timezone + + # ARE are_simulation_main.py:157: `scenario.start_time or 0` + # Defaults to Unix epoch (1970-01-01 00) when start_time is None + start_time = environment.get_start_time() or 0 + date_str = datetime.fromtimestamp(start_time, tz=timezone.utc).strftime("%Y-%m-%d %H") + return f"Today's date in 'YYYY-MM-DD HH' format is {date_str}" + except Exception: + return "" - return "\n\n".join(descriptions) +def _build_system_prompt(tools: Dict[str, Callable], environment: Optional[Any] = None) -> str: + """Build the full system prompt with tool descriptions and dynamic placeholders. -def _build_system_prompt(tools: Dict[str, Callable]) -> str: - """Build the full system prompt with tool descriptions. + Matches ARE's system prompt construction: + - Tool descriptions in ARE's Jinja2 format + - Dynamic notification system policy from ARE's notification_system.py + - Current time from scenario start_time (ARE are_simulation_main.py:156-164) + - Empty agent_reminder_description (ARE are_simulation_main.py:166-171) + - Scenario additional_system_prompt appended (ARE are_simulation_main.py:138-145) Args: tools: Dict of tool name -> callable + environment: Optional Gaia2Environment for dynamic prompt generation Returns: Complete system prompt string @@ -363,71 +541,59 @@ def _build_system_prompt(tools: Dict[str, Callable]) -> str: # Load templates general = _load_prompt_template("general_instructions") agent = _load_prompt_template("agent_instructions") - environment = _load_prompt_template("environment_instructions") + environment_template = _load_prompt_template("environment_instructions") template = _load_prompt_template("system_prompt") - # Build tool descriptions + # Build tool descriptions in ARE's format tool_descriptions = _build_tool_descriptions(tools) - # Format environment instructions with tool descriptions - environment_formatted = environment.format(tool_descriptions=tool_descriptions) + # Format environment instructions with tool descriptions and environment hints + # ARE system_prompt.py:187-189: environment_hints is always "" for default JSON agent + environment_formatted = environment_template.format(tool_descriptions=tool_descriptions, environment_hints="") + + # Replace dynamic placeholders (matching ARE's are_simulation_main.py:138-171) + # 1. Notification system description + notification_prompt = _get_notification_system_prompt(environment) + environment_formatted = environment_formatted.replace("<>", notification_prompt) + + # 2. Agent reminder description (always empty in ARE) + # ARE are_simulation_main.py:166-171 + environment_formatted = environment_formatted.replace("<>", "") + + # 3. Current time description + # ARE are_simulation_main.py:156-164 + time_description = _get_current_time_description(environment) + environment_formatted = environment_formatted.replace("<>", time_description) # Assemble full prompt - return template.format( + prompt = template.format( general_instructions=general, agent_instructions=agent, environment_instructions=environment_formatted, ) + # Append scenario's additional_system_prompt if present + # ARE are_simulation_main.py:138-145 + if environment is not None: + try: + scenario = environment.get_scenario() + additional = getattr(scenario, "additional_system_prompt", None) + if additional is not None: + prompt += "\n\n" + additional + except Exception: + pass -def _parse_json_blob(json_blob: str) -> Optional[Dict[str, Any]]: - """Parse JSON blob using the same approach as original ARE. - - Finds the first '{' and last '}' to correctly handle nested JSON. - - Args: - json_blob: String potentially containing JSON - - Returns: - Parsed dict or None if parsing fails - """ - try: - first_brace = json_blob.find("{") - if first_brace == -1: - return None - - # Find all closing braces and use the last one (handles nested JSON) - brace_positions = [m.start() for m in re.finditer(r"}", json_blob)] - if not brace_positions: - return None - - last_brace = brace_positions[-1] - json_str = json_blob[first_brace : last_brace + 1] - - # Handle escaped quotes - json_str = json_str.replace('\\"', "'") + return prompt - # Handle triple quotes - json_str = re.sub(r'"""(.*?)"""', r"'\1'", json_str, flags=re.DOTALL) - return json.loads(json_str, strict=False) - except json.JSONDecodeError: - # Try to fix common issues - try: - # Remove trailing commas - fixed = re.sub(r",\s*}", "}", json_str) - fixed = re.sub(r",\s*]", "]", fixed) - return json.loads(fixed, strict=False) - except (json.JSONDecodeError, UnboundLocalError): - return None - except Exception: - return None +def _parse_action_from_text(text: str) -> Optional[Tuple[str, str, Dict[str, Any] | str]]: + """Parse Thought and Action from LLM text output. + Uses ARE's ``parse_json_tool_call()`` for JSON parsing and action extraction. + The outer text parsing (Thought/Action section extraction) is custom. -def _parse_action_from_text(text: str) -> Optional[Tuple[str, str, Dict[str, Any]]]: - """Parse Thought and Action from LLM text output. + Expected format:: - Expected format: Thought: [reasoning] Action: @@ -442,6 +608,10 @@ def _parse_action_from_text(text: str) -> Optional[Tuple[str, str, Dict[str, Any Returns: Tuple of (thought, tool_name, tool_args) or None if parsing fails """ + from are.simulation.agents.default_agent.tools.json_action_executor import ( # type: ignore[import-not-found] + parse_json_tool_call, + ) + # Extract thought (everything before "Action:") thought = "" if "Thought:" in text: @@ -461,21 +631,36 @@ def _parse_action_from_text(text: str) -> Optional[Tuple[str, str, Dict[str, Any if end_action_pos != -1: action_text = action_text[:end_action_pos] - # Parse JSON using the robust method (matching original ARE) - action_data = _parse_json_blob(action_text) - if action_data is None: + # Parse JSON tool call using ARE's parser + # ARE handles: code fence removal, JSON extraction, action/action_input extraction + # Raises JsonParsingAgentError (not ValueError) on failure + try: + tool_name, tool_args = parse_json_tool_call(action_text) + return (thought, str(tool_name), tool_args) + except Exception: return None - tool_name = action_data.get("action", "") - tool_args = action_data.get("action_input", {}) - # Handle string action_input - pass through as-is (matching original ARE behavior) - # Original ARE passes string args directly to tools, we convert to dict with single arg - if isinstance(tool_args, str): - # Keep string as-is for tools that accept positional string args - pass +def _apply_stop_truncation(text: str, stop_sequences: List[str]) -> str: + """Apply client-side stop-sequence truncation. - return (thought, tool_name, tool_args) + Removes the first occurrence of any stop sequence and everything after it. + This matches ARE's LiteLLMEngine behavior (litellm_engine.py:126-127) and + serves as a universal fallback when stop sequences are not supported at the + API level (e.g., reasoning models like o1/o3/GPT-5). + + Always applied regardless of whether API-level ``stop`` is also used. + + Args: + text: Raw LLM response text. + stop_sequences: Tokens to truncate on. + + Returns: + Text truncated at the first matching stop sequence. + """ + for stop_token in stop_sequences: + text = text.split(stop_token)[0] + return text class DefaultGaia2Agent: @@ -485,23 +670,41 @@ class DefaultGaia2Agent: action parsing (Thought/Action/Observation cycle) rather than native function calling. - Key characteristics matching ARE: - - Text-based JSON action format with token - - Stop sequences: ["", "Observation:"] - - Default temperature: 0.5 - - Default max_tokens: 16384 - - Default max_iterations: 80 - - Invalid format retry: up to 10 times - - Terminates on send_message_to_user or wait_for_notification + Uses ARE's **two-level loop** architecture: + + - **Outer loop** (``_turn_loop``): iterates over turns, matching + ``are_simulation_main.py:agent_loop()``. Between turns, drains the + notification queue, formats user messages as ``[TASK]``, re-queues + environment notifications for the inner loop's pre-step. + - **Inner loop** (``_step_loop``): iterates over steps within a turn, + matching ``base_agent.py:execute_agent_loop()``. Terminates on BOTH + ``send_message_to_user`` (TERMINATED) and ``wait_for_notification`` + (PAUSED). + + Key characteristics matching ARE (base_agent.py, are_simulation.py): + + - Text-based JSON action format with `` token + - Stop sequences: ``["", "Observation:"]`` + - Default temperature: 0.5 (ARE llm_engine.py:17) + - Default max_tokens: 16384 (ARE llm_engine.py:16) + - Default max_iterations: 80 (ARE are_simulation_agent_config.py:36) + - Invalid format retry: up to 10 times (ARE base_agent.py:347) + - Iteration counter incremented EVERY loop (including errors) (ARE base_agent.py:849) + - Terminates inner loop on send_message_to_user (TERMINATED) or + wait_for_notification (PAUSED) + - Max-iterations sends message to user via tool (ARE are_simulation.py:109-116) + - Pre-step notification polling (ARE steps/are_simulation.py:26-62) """ def __init__( self, tools: Dict[str, Callable], model: ModelAdapter, + environment: Optional[Any] = None, llm_args: Optional[Dict[str, Any]] = None, max_iterations: int = _DEFAULT_MAX_ITERATIONS, invalid_format_retries: int = _DEFAULT_INVALID_FORMAT_RETRIES, + simulated_generation_time_config: Optional[Gaia2SimulatedGenerationTimeConfig] = None, verbose: int = 0, ): """Initialize the agent. @@ -509,20 +712,45 @@ def __init__( Args: tools: Dict of tool name -> callable model: ModelAdapter for LLM interactions - llm_args: Additional arguments for model calls. Defaults are applied - for temperature (0.5), max_tokens (16384), and stop sequences. + environment: Optional Gaia2Environment for notification polling + llm_args: Additional arguments for model calls, passed as kwargs + to ``model.chat()``. Defaults (from ARE source): + + - ``temperature``: 0.5 (ARE llm_engine.py:17) + - ``max_tokens``: 16384 (ARE llm_engine.py:16) + - ``stop``: ``["", "Observation:"]`` + + **Stop-token handling:** Client-side stop-token truncation + (ARE litellm_engine.py:126-127) is always applied to the + response, regardless of whether ``stop`` is also passed to + the API. When ``stop`` is passed, the API enforces it for + efficiency (saves tokens, precise cutoff). When ``stop`` is + ``None``, only client-side truncation runs — action parsing + still works correctly. + + **None filtering:** Parameters set to ``None`` are omitted + from the API call entirely. Use this to disable parameters + the model provider rejects:: + + llm_args={"stop": None, "temperature": None} max_iterations: Maximum iterations before stopping. Default 80. invalid_format_retries: Max retries for invalid format. Default 10. + simulated_generation_time_config: Optional config for simulated generation + time. When set, the simulation is paused during LLM generation and + resumed with a time offset. Default None (disabled). + ARE agents/are_simulation_agent_config.py:28-30 verbose: Verbosity level (0=quiet, 1=basic, 2=detailed) """ self.tools = tools self.model = model + self.environment = environment self.max_iterations = max_iterations self.invalid_format_retries = invalid_format_retries + self.simulated_generation_time_config = simulated_generation_time_config self.verbose = verbose # Build system prompt with tool descriptions - self.system_prompt = _build_system_prompt(tools) + self.system_prompt = _build_system_prompt(tools, environment) # Apply default LLM args, allowing user overrides self.llm_args = { @@ -538,6 +766,7 @@ def __init__( self._format_retry_count = 0 self._terminated = False self._final_message: Optional[str] = None + self._step_count = 0 # Counts all outputs: observations AND errors (ARE base_agent.py:450-451) def reset(self) -> None: """Reset agent state.""" @@ -546,105 +775,443 @@ def reset(self) -> None: self._format_retry_count = 0 self._terminated = False self._final_message = None + self._step_count = 0 def run(self, query: str) -> str: """Execute task and return final response. + GAIA2 is event-driven: the real task instruction is delivered via the + notification system (first ``send_message_to_agent`` event). The outer + turn loop (``_turn_loop``) drains the notification queue and formats + user messages as ``[TASK]``, matching ARE's ``agent_loop()``. + + When ``query`` is non-empty (e.g. standalone use), it is prepended as + a ``[TASK]`` message before entering the turn loop. + Args: - query: Task query/instructions + query: Task query/instructions (may be empty for GAIA2) Returns: Final text response from agent """ - self._messages.append({"role": "user", "content": query}) - return self._react_loop() + # Match ARE's message format: [TASK]: \n{content}\n + # ARE base_agent.py:96 + if query: + self._messages.append({"role": "user", "content": f"[TASK]: \n{query}\n"}) + return self._turn_loop() + + def _pull_notifications(self) -> None: + """Pull messages from the ARE notification system. + + Delegates to ``Gaia2Environment.poll_notifications()`` which drains + the notification queue and returns pre-formatted strings. + + Matches ARE's pre-step notification polling behavior. + ARE agents/default_agent/steps/are_simulation.py:26-62 + """ + if self.environment is None: + return + + user_messages, env_notifications, _ = self.environment.poll_notifications() + + # Inject into message history matching ARE's format + # ARE base_agent.py:107: "User messages updates:\n***\n{content}\n***\n" + if user_messages: + content = "\n".join(user_messages) + self._messages.append({"role": "user", "content": f"User messages updates:\n***\n{content}\n***\n"}) + + # ARE base_agent.py:110-112: "Environment notifications updates:\n***\n{content}\n***\n" + if env_notifications: + content = "\n".join(env_notifications) + self._messages.append({"role": "user", "content": f"Environment notifications updates:\n***\n{content}\n***\n"}) - def _react_loop(self) -> str: - """ReAct loop: Thought -> Action -> Observation -> repeat. + def _check_environment_stop(self) -> bool: + """Check if the environment has sent a stop message. + + Uses a lightweight peek via the ARE notification system's + ``has_environment_stop_message()`` when available, falling back to + ``poll_notifications()``. + + ARE agents/default_agent/termination_methods/are_simulation.py:105-107 + + Returns: + True if environment has signaled stop + """ + if self.environment is None: + return False + + # Prefer the non-draining peek when available + notification_system = self.environment.get_notification_system() + if notification_system is not None: + try: + return notification_system.message_queue.has_environment_stop_message() + except Exception: + pass + + return False + + def _pause_env(self) -> None: + """Pause the ARE environment before LLM generation. + + ARE base_agent.py:623-627 + """ + if self.environment is not None: + self.environment.pause() + + def _resume_env(self, offset: float) -> None: + """Resume the ARE environment after LLM generation with time offset. + + ARE base_agent.py:680-689 + + Args: + offset: Time in seconds to advance the simulation clock + """ + if self.environment is not None: + self.environment.resume_with_offset(offset) + + def _turn_loop(self) -> str: + """Outer turn loop matching ARE's ``agent_loop()``. + + ARE ``are_simulation_main.py:230-326``: iterates over turns. Between + turns, drains the notification queue, separates user messages (→ + ``[TASK]`` format) from environment notifications (→ re-queued for + inner loop's pre-step), then runs the inner step loop. + + - ``send_message_to_user`` → TERMINATED: increments turn count + - ``wait_for_notification`` → PAUSED: outer loop continues without + incrementing turns Returns: Final text response """ - while self._iteration_count < self.max_iterations and not self._terminated: - # Build messages for LLM - messages = [{"role": "system", "content": self.system_prompt}] + self._messages + import logging + + logger = logging.getLogger(__name__) + + turn_count = 0 + max_turns = self._get_max_turns() + + # ARE are_simulation_main.py:270 + first_run = True + + while max_turns is None or turn_count < max_turns: + if not first_run: + if self.environment is not None: + # Between turns: drain notification queue + # ARE are_simulation_main.py:272-274 + user_messages, has_env, has_stop = self.environment.get_turn_notifications() + + if has_stop: + logger.warning("Environment stop message received in outer loop — stopping agent") + self._terminated = True + break + + if not user_messages and not has_env: + # No messages available, wait briefly + # ARE are_simulation_main.py:316-317 + time.sleep(1) + continue + + # Format user messages as [TASK] for the new turn + # ARE are_simulation_main.py:283, 361-365 + # ARE base_agent.py:96: "[TASK]: \n{content}\n" + task = "\n".join(user_messages) + self._messages.append({"role": "user", "content": f"[TASK]: \n{task}\n"}) + # else: no environment — skip notification handling and run + # inner loop directly (standalone/testing mode) + + first_run = False + + # Run inner step loop + # ARE: react_agent.run(task=task, reset=reset) → execute_agent_loop() + running_state = self._step_loop() + + if running_state == _RunningState.TERMINATED: + # ARE are_simulation_main.py:297-300: increment turn count + turn_count += 1 + if self._terminated: + return self._final_message or "" + elif running_state == _RunningState.PAUSED: + # ARE are_simulation_main.py:301-303: agent called + # wait_for_notification, continue outer loop + logger.debug("Agent paused (wait_for_notification), continuing outer loop") + + # Max turns reached + # ARE are_simulation_main.py:319-320 + if max_turns is not None and turn_count >= max_turns and not self._terminated: + logger.warning("Max turns (%d) reached — stopping agent", max_turns) - # Call LLM (text completion, no tools parameter) - response = self.model.chat(messages=messages, **self.llm_args) # type: ignore[arg-type] - content = response.content or "" + return self._final_message or "Agent terminated without final message." - if self.verbose >= 2: - print(f"[Iteration {self._iteration_count + 1}] LLM output:\n{content}\n") + def _get_max_turns(self) -> Optional[int]: + """Get max turns from the scenario, matching ARE's ``scenario.nb_turns``. - # Parse action from text - parsed = _parse_action_from_text(content) + Returns: + Number of turns, or None for unlimited + """ + if self.environment is None: + return None + try: + scenario = self.environment.get_scenario() + return getattr(scenario, "nb_turns", None) + except Exception: + return None - if parsed is None: - # Invalid format - retry - self._format_retry_count += 1 - if self._format_retry_count >= self.invalid_format_retries: - self._messages.append({"role": "assistant", "content": content}) - return f"Failed to parse action after {self.invalid_format_retries} retries. Last output: {content}" - - # Add error observation and retry - error_msg = ( - "Error: Invalid action format. Please use the correct format:\n" - "Thought: [your reasoning]\n\n" - "Action:\n" - '{"action": "tool_name", "action_input": {...}}' - ) - self._messages.append({"role": "assistant", "content": content}) - self._messages.append({"role": "user", "content": f"Observation: {error_msg}"}) - continue + def _step_loop(self) -> "_RunningState": + """Inner step loop matching ARE's ``execute_agent_loop()``. - thought, tool_name, tool_args = parsed - self._iteration_count += 1 - self._format_retry_count = 0 # Reset on successful parse + ARE ``base_agent.py:775-854``: iterates over steps within a single + turn. Terminates when the agent calls ``send_message_to_user`` + (TERMINATED) or ``wait_for_notification`` (PAUSED). - # Add assistant message (Thought + Action) - self._messages.append({"role": "assistant", "content": content}) + Key behavior matching ARE: - if self.verbose >= 1: - print(f"[Iteration {self._iteration_count}] Tool: {tool_name}") + - Iteration counter incremented on EVERY loop iteration (including errors) + - Format retries happen within a single iteration + - Max-iterations sends message to user via the actual tool + - Pre-step notification polling before each iteration + - Environment stop message checked for termination + - Simulated generation time: pause env before LLM, resume with offset after - # Check for termination tools - if tool_name in _TERMINATION_TOOLS: + Returns: + Running state: TERMINATED or PAUSED + """ + while self._iteration_count < self.max_iterations and not self._terminated: + # Check for environment stop BEFORE draining notifications. + # ARE's execute_agent_loop checks termination_condition (which peeks + # via has_environment_stop_message) in the while-condition, THEN runs + # pre_step (which drains via get_by_timestamp). Reversing this order + # causes the drain to consume ENVIRONMENT_STOP before the peek sees it. + # ARE agents/default_agent/termination_methods/are_simulation.py:98-99 + # ARE agents/default_agent/base_agent.py:776-799 + if self._check_environment_stop(): self._terminated = True + return _RunningState.TERMINATED + + # Pre-step: poll for notifications (matching ARE's pre-step) + # ARE agents/default_agent/steps/are_simulation.py:26-62 + self._pull_notifications() + + try: + # Build messages for LLM + messages = [{"role": "system", "content": self.system_prompt}] + self._messages + + # Pause environment before LLM generation + # ARE base_agent.py:623-627 + if self.simulated_generation_time_config is not None: + self._pause_env() + + # Call LLM with retry for invalid format + # ARE base_agent.py:629-666 + content, completion_duration = self._call_llm_with_format_retry(messages) + + # Resume environment after LLM generation with time offset + # ARE base_agent.py:680-689 + if self.simulated_generation_time_config is not None: + from are.simulation.agents.default_agent.base_agent import ( # type: ignore[import-not-found] + get_offset_from_time_config_mode, + ) + + offset = get_offset_from_time_config_mode( + time_config=self.simulated_generation_time_config, + completion_duration=completion_duration, + ) + self._resume_env(offset) + + if content is None: + # All format retries exhausted + continue + + # Parse action from text + parsed = _parse_action_from_text(content) + if parsed is None: + # This shouldn't happen after format retry, but handle it + self._messages.append({"role": "assistant", "content": content}) + error_msg = f"The LLM output was not formatted correctly: {content}" + # ARE base_agent.py:450-451: increment step for errors too + self._step_count += 1 + self._messages.append( + { + "role": "user", + "content": ( + f"[OUTPUT OF STEP {self._step_count}] ERROR:\n***\n{error_msg}\n***\n\n" + "Now let's retry: take care not to repeat previous errors! " + "If you have retried several times, try a completely different approach.\n" + ), + } + ) + continue + + thought, tool_name, tool_args = parsed + self._step_count += 1 + + # Add assistant message (Thought + Action) + self._messages.append({"role": "assistant", "content": content}) - # Execute the termination tool + if self.verbose >= 1: + print(f"[Iteration {self._iteration_count}] Tool: {tool_name}") + + # Check for termination tools + # ARE agents/default_agent/termination_methods/are_simulation.py:71-121 + if tool_name in _TERMINATION_TOOLS: + # send_message_to_user → TERMINATED + # ARE termination_methods/are_simulation.py:93-96 + self._terminated = True + observation = self._execute_tool(tool_name, tool_args) + self._final_message = tool_args.get("content", str(observation)) if isinstance(tool_args, dict) else str(observation) + return _RunningState.TERMINATED + + if tool_name == _PAUSE_TOOL: + # wait_for_notification → execute tool, add observation, PAUSED + # ARE termination_methods/are_simulation.py:34-36 + observation = self._execute_tool(tool_name, tool_args) + self._messages.append( + { + "role": "user", + "content": f"[OUTPUT OF STEP {self._step_count}] Observation:\n***\n{observation}\n***\n", + } + ) + return _RunningState.PAUSED + + # Execute regular tool observation = self._execute_tool(tool_name, tool_args) - # For send_message_to_user, capture the message - if tool_name == "AgentUserInterface__send_message_to_user": - self._final_message = tool_args.get("content", str(observation)) - return self._final_message + # Add observation in ARE's format + # ARE base_agent.py:102: "[OUTPUT OF STEP {i}] Observation:\n***\n{content}\n***\n" + self._messages.append( + { + "role": "user", + "content": f"[OUTPUT OF STEP {self._step_count}] Observation:\n***\n{observation}\n***\n", + } + ) + + except Exception as e: + # Match ARE error handling: log error, add to messages, continue + # ARE base_agent.py:839-840, base_agent.py:105-106 + # ARE base_agent.py:450-451: increment step for errors too + self._step_count += 1 + error_msg = str(e) + self._messages.append( + { + "role": "user", + "content": ( + f"[OUTPUT OF STEP {self._step_count}] ERROR:\n***\n{error_msg}\n***\n\n" + "Now let's retry: take care not to repeat previous errors! " + "If you have retried several times, try a completely different approach.\n" + ), + } + ) + finally: + # Safety resume: if environment is still paused due to an exception, + # resume without advancing time to prevent deadlock. + # ARE base_agent.py:841-848 + if self.simulated_generation_time_config is not None: + self._resume_env(0.0) + # ARE increments iterations on EVERY loop iteration, including errors + # ARE base_agent.py:849 + self._iteration_count += 1 + + # Max iterations reached: send message to user via tool + # ARE agents/default_agent/termination_methods/are_simulation.py:100-108 + if self._iteration_count >= self.max_iterations and not self._terminated: + max_iter_msg = f"Max iterations ({self.max_iterations}) reached. Stopping." + # Call the actual tool to record the event in the simulation + if "AgentUserInterface__send_message_to_user" in self.tools: + self._execute_tool("AgentUserInterface__send_message_to_user", {"content": max_iter_msg}) + self._terminated = True + self._final_message = max_iter_msg + return _RunningState.TERMINATED + + return _RunningState.TERMINATED + + def _call_llm_with_format_retry(self, messages: List[Dict[str, Any]]) -> Tuple[Optional[str], float]: + """Call LLM with retry for invalid format, matching ARE's behavior. + + ARE base_agent.py:629-666: retries until output contains Action: or Thought:, + up to ``invalid_format_retries`` times. + + Args: + messages: Messages to send to LLM + + Returns: + Tuple of (LLM output text or None, completion_duration in seconds). + completion_duration is the wall-clock time of the last successful LLM call. + """ + format_try_count = 0 + content: Optional[str] = None + completion_duration = 0.0 + + while content is None or ("Action:" not in content and "Thought:" not in content): + if content is not None: + # Invalid format - add error and retry + # ARE base_agent.py:642-650 + # ARE base_agent.py:450-451: increment step for errors too + self._step_count += 1 + error_msg = f"The LLM output was not formatted correctly: {content}" + self._messages.append( + { + "role": "user", + "content": ( + f"[OUTPUT OF STEP {self._step_count}] ERROR:\n***\n{error_msg}\n***\n\n" + "Now let's retry: take care not to repeat previous errors! " + "If you have retried several times, try a completely different approach.\n" + ), + } + ) + # Rebuild messages with the error + messages = [{"role": "system", "content": self.system_prompt}] + self._messages + + call_start = time.monotonic() + # Filter None values: allows users to disable params (e.g., stop=None for reasoning models) + active_args = {k: v for k, v in self.llm_args.items() if v is not None} + response = self.model.chat(messages=messages, **active_args) # type: ignore[arg-type] + completion_duration = time.monotonic() - call_start + content = response.content or "" + + # Boolean replacement (ARE litellm_engine.py:125, hf_engine.py:152). + # LLMs frequently output Python-style True/False in JSON blobs; + # ARE normalizes to JSON-valid true/false before any parsing. + content = content.replace("False", "false").replace("True", "true") - # For wait_for_notification, return the observation - return str(observation) + # Client-side stop-token truncation (ARE litellm_engine.py:126-127). + # Always applied as a universal fallback — works even when API-level + # stop sequences are disabled (stop=None) for reasoning models. + content = _apply_stop_truncation(content, _STOP_SEQUENCES) - # Execute tool - observation = self._execute_tool(tool_name, tool_args) + if self.verbose >= 2: + print(f"[Iteration {self._iteration_count}, format try {format_try_count}] LLM output:\n{content}\n") - # Add observation - self._messages.append({"role": "user", "content": f"Observation: {observation}"}) + format_try_count += 1 + # ARE base_agent.py:664-666: failsafe from infinite loop + if format_try_count > self.invalid_format_retries: + break - if self._iteration_count >= self.max_iterations: - return f"Max iterations ({self.max_iterations}) reached." + # ARE base_agent.py:705-708: raise error after retries exhausted + if content is None or ("Action:" not in content and "Thought:" not in content): + return None, completion_duration - return self._final_message or "Agent terminated without final message." + return content, completion_duration def _execute_tool(self, tool_name: str, tool_args: Dict[str, Any] | str) -> str: """Execute a tool call. + Raises on errors (matching ARE's json_action_executor.py:197-227). + Errors propagate to `_react_loop()` which formats them as ``ERROR:`` + messages, distinct from ``Observation:`` messages. + Args: tool_name: Name of the tool to call tool_args: Arguments for the tool (dict or string) Returns: Tool execution result as string + + Raises: + RuntimeError: If tool is not found or execution fails """ + # ARE json_action_executor.py:210-212: raises UnavailableToolAgentError if tool_name not in self.tools: - return f"Error: Tool '{tool_name}' not found. Available tools: {list(self.tools.keys())}" + raise RuntimeError(f"Error: unknown tool {tool_name}, should be instead one of {list(self.tools.keys())}.") try: # Match original ARE behavior: string args passed as positional argument @@ -654,7 +1221,16 @@ def _execute_tool(self, tool_name: str, tool_args: Dict[str, Any] | str) -> str: result = self.tools[tool_name](**tool_args) return str(result) except Exception as e: - return f"Error executing tool '{tool_name}': {e}" + # ARE json_action_executor.py:224-227: raises JsonExecutionAgentError + # with full tool description as a reminder + tool = self.tools[tool_name] + tool_desc = _get_tool_description_with_args(tool_name, tool) + raise RuntimeError( + f"Error in tool call execution: {e}\n" + f"You should only use this tool with a correct input.\n" + f"As a reminder, this tool's description is the following:\n" + f"{tool_desc}" + ) from e def get_messages(self) -> List[Dict[str, Any]]: """Get message history. @@ -754,6 +1330,8 @@ def __init__(self, agent_data: Optional[Dict[str, Any]] = None, **kwargs: Any): - llm_args: Optional model call arguments (temperature, max_tokens, etc.) - max_iterations: Max iterations per task (default: 80) - invalid_format_retries: Max retries for invalid format (default: 10) + - simulated_generation_time_config: Optional Gaia2SimulatedGenerationTimeConfig + for simulating LLM generation time in the simulation (default: None) - verbose: Verbosity level (default: 0) **kwargs: Additional Benchmark arguments """ @@ -809,6 +1387,7 @@ def setup_agents( # type: ignore[override] llm_args = merged_data.get("llm_args", {}) max_iterations = merged_data.get("max_iterations", _DEFAULT_MAX_ITERATIONS) invalid_format_retries = merged_data.get("invalid_format_retries", _DEFAULT_INVALID_FORMAT_RETRIES) + simulated_generation_time_config = merged_data.get("simulated_generation_time_config") verbose = merged_data.get("verbose", 0) # Derive seed for agent model (returns None if seeding disabled) @@ -818,11 +1397,13 @@ def setup_agents( # type: ignore[override] model = self.get_model_adapter(model_id, register_name="agent_model", seed=agent_seed) agent = DefaultGaia2Agent( - tools=tools, # type: ignore[arg-type] # AREToolWrapper is Callable + tools=tools, # type: ignore[arg-type] # Gaia2GenericTool has __call__ model=model, + environment=environment, llm_args=llm_args, max_iterations=max_iterations, invalid_format_retries=invalid_format_retries, + simulated_generation_time_config=simulated_generation_time_config, verbose=verbose, ) diff --git a/maseval/benchmark/gaia2/prompt_templates/agent_instructions.txt b/maseval/benchmark/gaia2/prompt_templates/agent_instructions.txt index 6a7d38a..8792dba 100644 --- a/maseval/benchmark/gaia2/prompt_templates/agent_instructions.txt +++ b/maseval/benchmark/gaia2/prompt_templates/agent_instructions.txt @@ -50,7 +50,3 @@ Action: Observation: The current temperature in Paris is 20 degrees Celsius and the weather is sunny. ============================ -EXECUTION GUIDELINES: -Take one action at a time and complete the thought/action/observation cycle before proceeding. Never generate the Observation field - it will be provided after each action. -If an action fails, analyze the error and try a different approach. Don't call tools unnecessarily - use your reasoning when you can solve something directly. -Continue iterating until the task is complete or you determine it's impossible with available tools. Pay attention to tool outputs and use them to inform subsequent actions. diff --git a/maseval/benchmark/gaia2/prompt_templates/environment_instructions.txt b/maseval/benchmark/gaia2/prompt_templates/environment_instructions.txt index b9503d4..41a7168 100644 --- a/maseval/benchmark/gaia2/prompt_templates/environment_instructions.txt +++ b/maseval/benchmark/gaia2/prompt_templates/environment_instructions.txt @@ -17,11 +17,10 @@ FUNDAMENTAL RULES FOR TASK EXECUTION: 5. INFORMATION: Use available tools to gather missing information before asking user. 6. AMBIGUITY: Execute all clear and unambiguous parts of a request immediately. When you encounter ambiguities, contradictions, or impossible elements, finish unambiguous subtasks and then stop and explicitly ask the user for clarification before proceeding with those specific parts. -NOTIFICATION SYSTEM: -The environment has a notification system. When you need to wait for something (e.g., a response from another user, a scheduled event), use the SystemApp__wait_for_notification tool. This will advance simulation time and return any notifications that occur. +{environment_hints} -TIME AWARENESS: -Use SystemApp__get_current_time to check the current time when needed for scheduling or time-sensitive tasks. +<> -TASK COMPLETION: -When you have completed the task, use AgentUserInterface__send_message_to_user to send your final response to the user. +<> + +<> diff --git a/maseval/benchmark/gaia2/tool_wrapper.py b/maseval/benchmark/gaia2/tool_wrapper.py index 0af98ae..39fd38f 100644 --- a/maseval/benchmark/gaia2/tool_wrapper.py +++ b/maseval/benchmark/gaia2/tool_wrapper.py @@ -1,6 +1,7 @@ """Gaia2 Benchmark - Tool Wrapper. -Wraps ARE AppTool instances for MASEval compatibility and tracing. +Framework-agnostic wrapper for ARE AppTool instances, following MACSGenericTool pattern. +Provides clean API with built-in tracing for MASEval compatibility. Reference Paper: "GAIA-2: A Controllable Multi-Turn Conversational Benchmark for Agents" """ @@ -16,14 +17,33 @@ from maseval.benchmark.gaia2.environment import Gaia2Environment -class AREToolWrapper(TraceableMixin, ConfigurableMixin): - """Wraps ARE AppTool for MASEval tracing and compatibility. +class Gaia2GenericTool(TraceableMixin, ConfigurableMixin): + """Framework-agnostic wrapper for ARE tools. - Records all tool invocations with inputs, outputs, timestamps, - and simulation time for post-hoc analysis. + Similar to MACSGenericTool - provides clean API with built-in tracing. + Developers wrap this for their framework using composition. + + Example for smolagents: + + class MySmolagentsTool(smolagents.Tool): + skip_forward_signature_validation = True + + def __init__(self, generic_tool: Gaia2GenericTool): + self.generic_tool = generic_tool + self.name = generic_tool.name + self.description = generic_tool.description + self.inputs = generic_tool.inputs + self.output_type = generic_tool.output_type + super().__init__() + + def forward(self, **kwargs) -> str: + return self.generic_tool(**kwargs) + + def gather_traces(self): + return self.generic_tool.gather_traces() This wrapper preserves ARE's native return types while adding - MASEval tracing capabilities. + MASEval tracing capabilities and providing a framework-agnostic interface. """ def __init__(self, are_tool: Any, environment: "Gaia2Environment"): @@ -34,17 +54,56 @@ def __init__(self, are_tool: Any, environment: "Gaia2Environment"): environment: The Gaia2Environment this tool belongs to """ super().__init__() - self.are_tool = are_tool - self.environment = environment + self._are_tool = are_tool + self._environment = environment + + # Delegate metadata extraction to ARE's AppToolAdapter (tool_utils.py:544-584). + # This is the source of truth for tool name, description, inputs, and output_type. + from are.simulation.tool_utils import AppToolAdapter # type: ignore[import-not-found] - # Extract tool metadata from ARE tool - self.name: str = getattr(are_tool, "name", str(are_tool)) - self.description: str = getattr(are_tool, "description", "") - self.inputs: Dict[str, Any] = self._extract_schema(are_tool) + adapter = AppToolAdapter(are_tool) + self.name: str = adapter.name + self.description: str = adapter.description + self.inputs: Dict[str, Any] = adapter.inputs + self.output_type: str = adapter.output_type + self.actual_return_type: Optional[str] = adapter.actual_return_type + self.input_schema: Dict[str, Any] = self._extract_schema(are_tool) # Initialize invocation history self.history = ToolInvocationHistory() + @staticmethod + def _extract_schema(are_tool: Any) -> Dict[str, Any]: + """Convert ARE's args list to JSON schema format for tracing/config. + + Args: + are_tool: ARE AppTool instance + + Returns: + JSON schema dictionary with properties and required fields + """ + args = getattr(are_tool, "args", None) + if not args: + return {} + + properties = {} + required = [] + + for arg in args: + param_name = getattr(arg, "name", None) + if not param_name: + continue + + properties[param_name] = { + "type": arg.arg_type, + "description": getattr(arg, "description", ""), + } + + if not arg.has_default: + required.append(param_name) + + return {"properties": properties, "required": required} + def __call__(self, **kwargs: Any) -> Any: """Execute tool and record invocation. @@ -63,7 +122,7 @@ def __call__(self, **kwargs: Any) -> Any: error_message = None try: - result = self.are_tool(**kwargs) + result = self._are_tool(**kwargs) except Exception as e: status = "error" error_message = str(e) @@ -71,7 +130,7 @@ def __call__(self, **kwargs: Any) -> Any: finally: sim_time_after = self._get_simulation_time() - # Record invocation with timing metadata + # Record invocation with timing metadata (same structure as before) self.history.add_invocation( inputs=kwargs, outputs=result if status == "success" else error_message, @@ -94,36 +153,10 @@ def _get_simulation_time(self) -> Optional[float]: Simulation time in seconds, or None if not available """ try: - return self.environment.get_simulation_time() + return self._environment.get_simulation_time() except Exception: return None - def _extract_schema(self, are_tool: Any) -> Dict[str, Any]: - """Convert ARE tool schema to MASEval input format. - - Args: - are_tool: ARE AppTool instance - - Returns: - Dictionary describing tool inputs - """ - schema: Dict[str, Any] = {} - - # Try to extract schema from ARE tool - # ARE tools typically have an 'inputs' or 'parameters' attribute - if hasattr(are_tool, "inputs"): - schema = dict(are_tool.inputs) if are_tool.inputs else {} - elif hasattr(are_tool, "parameters"): - schema = dict(are_tool.parameters) if are_tool.parameters else {} - elif hasattr(are_tool, "args_schema"): - # Pydantic schema format - try: - schema = are_tool.args_schema.model_json_schema() if are_tool.args_schema else {} - except Exception: - schema = {} - - return schema - def gather_traces(self) -> Dict[str, Any]: """Gather execution traces from this tool. @@ -148,14 +181,19 @@ def gather_config(self) -> Dict[str, Any]: **super().gather_config(), "name": self.name, "description": self.description, - "inputs_schema": self.inputs, + "input_schema": self.input_schema, } + def __repr__(self) -> str: + """String representation of the tool.""" + args = ", ".join(f"{k}: {v['type']}" for k, v in self.inputs.items()) + return f"{self.__class__.__name__}({self.name}({args}) -> {self.output_type})" + def wrap_are_tools( are_tools: List[Any], environment: "Gaia2Environment", -) -> Dict[str, AREToolWrapper]: +) -> Dict[str, Gaia2GenericTool]: """Wrap multiple ARE tools for MASEval. Args: @@ -165,10 +203,10 @@ def wrap_are_tools( Returns: Dictionary mapping tool names to wrapped tools """ - wrapped: Dict[str, AREToolWrapper] = {} + wrapped: Dict[str, Gaia2GenericTool] = {} for tool in are_tools: - wrapper = AREToolWrapper(tool, environment) + wrapper = Gaia2GenericTool(tool, environment) wrapped[wrapper.name] = wrapper return wrapped diff --git a/maseval/benchmark/multiagentbench/PROVENANCE.md b/maseval/benchmark/multiagentbench/PROVENANCE.md index 9fd516b..8f5c0dc 100644 --- a/maseval/benchmark/multiagentbench/PROVENANCE.md +++ b/maseval/benchmark/multiagentbench/PROVENANCE.md @@ -2,11 +2,21 @@ ## Source Information -- **Source Repository**: https://github.com/ulab-uiuc/MARBLE -- **Version**: Not yet pinned (clone latest and test) +- **Original Repository**: https://github.com/ulab-uiuc/MARBLE (where the original work was done) +- **Fork Used**: https://github.com/cemde/MARBLE (contains bug fixes) +- **Version**: Currently unpinned (tracking latest from fork while bug fixes are being added) - **License**: MIT (Copyright 2024 Haofei Yu) - **Vendoring**: Permitted by MIT license with attribution +**Note**: Once the fork is stable, we will pin to a specific commit hash for reproducibility. + +### Why We Use a Fork + +We vendor from https://github.com/cemde/MARBLE rather than the original repository because: +- The fork contains critical bug fixes needed for integration with MASEval +- All credit for the original work goes to the MARBLE team (Haofei Yu et al.) +- The fork maintains the same MIT license and contains no API changes, only bug fixes + ## Reference **Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" @@ -53,17 +63,31 @@ SOFTWARE. 3. **Environment constructor signature**: Some environments expect different constructor arguments. Check each environment's `__init__` signature before use. +## Architectural Differences from MARBLE + +### Result summarization before evaluation + +In MARBLE, agent results are summarized in the engine's coordination loop +(`Engine._summarize_results()` + `EnginePlanner.summarize_output()`) before +reaching the evaluator. MASEval does not use MARBLE's engine loop, so this +summarization logic has been moved into `MultiAgentBenchEvaluator` (see +`_summarize_results()` and `_summarize_output()` in `evaluator.py`). The +behaviour is identical: each agent result is truncated to 1000 characters, then +an LLM call condenses the truncated output into a compact summary. The +truncation length is configurable via `result_truncation_length`. + ## Local Patches Applied -None currently. Document any patches here if applied. +None currently. All bug fixes are maintained in the fork. ## Update Process -To update MARBLE to a newer version: +To update MARBLE to a newer version from the fork: 1. `cd maseval/benchmark/multiagentbench/marble` -2. `git fetch origin` -3. `git log --oneline origin/main` (review changes) -4. `git checkout ` -5. Run integration tests -6. Update this file with new version info +2. `git remote set-url origin https://github.com/cemde/MARBLE.git` (if needed) +3. `git fetch origin` +4. `git log --oneline origin/main` (review changes) +5. `git checkout ` +6. Run integration tests +7. Update this file with new version info diff --git a/maseval/benchmark/multiagentbench/README.md b/maseval/benchmark/multiagentbench/README.md index 1d5bba7..4a397a4 100644 --- a/maseval/benchmark/multiagentbench/README.md +++ b/maseval/benchmark/multiagentbench/README.md @@ -6,7 +6,11 @@ Framework-agnostic implementation of the MultiAgentBench benchmark suite from MA **Original Paper**: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) -**Original Repository**: https://github.com/ulab-uiuc/MARBLE +**Original Repository**: https://github.com/ulab-uiuc/MARBLE (where the original work was done) + +**Fork Used**: https://github.com/cemde/MARBLE (contains bug fixes for MASEval integration) + +> **Note**: We use a fork of MARBLE that contains critical bug fixes needed for integration. All credit for the original work goes to the MARBLE team (Haofei Yu et al.). ## Setup @@ -33,7 +37,7 @@ If you prefer to clone manually: ```bash cd maseval/benchmark/multiagentbench -git clone https://github.com/ulab-uiuc/MARBLE.git marble +git clone https://github.com/cemde/MARBLE.git marble cd marble # Pin to tested version (recommended) git checkout @@ -133,24 +137,24 @@ for result in results: ## Domains -MultiAgentBench includes 7 domains with different requirements: +MultiAgentBench includes 6 domains with different requirements: -| Domain | External Dependencies | Initial Support | -| --------------- | --------------------- | --------------- | -| Research | None | Yes | -| Bargaining | None | Yes | -| Coding | Filesystem access | Yes | -| Web | Network access | Yes | -| WorldSimulation | None | Yes | -| Database | Docker + PostgreSQL | Optional | -| Minecraft | External game server | Deferred | +| Domain | External Dependencies | Support | +| ---------- | ------------------------------------------ | ----------- | +| Research | None | Full | +| Bargaining | None | Full | +| Coding | Filesystem access | Full | +| Werewolf | None | Full | +| Database | Docker with PostgreSQL image | Full | +| Minecraft | Minecraft Server 1.19.2, Node.js, npm | Untested | ### Domain-Specific Notes - **Research/Bargaining**: Recommended for initial testing - no infrastructure required - **Coding**: Creates files in a workspace directory -- **Database**: Requires Docker with PostgreSQL image -- **Minecraft**: Not currently supported (requires external game server) +- **Werewolf**: Config-based social deduction game (no JSONL task data; uses MARBLE's WerewolfEnv engine) +- **Database**: Requires Docker with a PostgreSQL image. Ensure Docker is installed and running before using this domain. +- **Minecraft**: **Untested.** Requires a running Minecraft Server (version 1.19.2) on `localhost:25565`, plus Node.js and npm for the Mineflayer bot framework. MARBLE's MinecraftEnvironment connects real Minecraft bots to the server over the Minecraft protocol. You must set up and run your own Minecraft server before using this domain. See the MARBLE documentation for full server setup instructions. ## Known Limitations diff --git a/maseval/benchmark/multiagentbench/__init__.py b/maseval/benchmark/multiagentbench/__init__.py index cc7eb68..ca0308c 100644 --- a/maseval/benchmark/multiagentbench/__init__.py +++ b/maseval/benchmark/multiagentbench/__init__.py @@ -3,18 +3,20 @@ Framework-agnostic implementation of the MultiAgentBench benchmark suite for evaluating multi-agent collaboration and competition in LLM-based systems. -Original Repository: https://github.com/ulab-uiuc/MARBLE +Original Repository: https://github.com/ulab-uiuc/MARBLE (where the original work was done) +Fork Used: https://github.com/cemde/MARBLE (contains bug fixes for MASEval integration) Paper: "MultiAgentBench: Evaluating the Collaboration and Competition of LLM agents" (arXiv:2503.01935) +Note: We use a fork with bug fixes. All credit goes to the MARBLE team (Haofei Yu et al.). + Domains: - research: Research idea generation and collaboration - bargaining: Negotiation and bargaining scenarios - coding: Software development collaboration - database: Database manipulation and querying (requires Docker) - minecraft: Collaborative building (requires external server) - - web: Web-based task completion - - worldsimulation: World simulation and interaction + - werewolf: Adversarial social deduction game with roles Setup: This benchmark requires MARBLE source code. It will be automatically @@ -27,7 +29,7 @@ # Option 2: Manual clone # cd maseval/benchmark/multiagentbench - # git clone https://github.com/ulab-uiuc/MARBLE.git marble + # git clone https://github.com/cemde/MARBLE.git marble ``` See README.md in this directory for detailed setup instructions. diff --git a/maseval/benchmark/multiagentbench/_constants.py b/maseval/benchmark/multiagentbench/_constants.py index f852e5a..7d85574 100644 --- a/maseval/benchmark/multiagentbench/_constants.py +++ b/maseval/benchmark/multiagentbench/_constants.py @@ -4,5 +4,22 @@ environment.py, and adapters/marble_adapter.py. """ +import sys +from pathlib import Path + # Shared error message for MARBLE import failures MARBLE_IMPORT_ERROR = "MARBLE is not available. Clone MARBLE to maseval/benchmark/multiagentbench/marble/\nOriginal error: {error}" + +# Root of the vendored MARBLE clone (contains marble/ Python package) +_MARBLE_ROOT = str(Path(__file__).parent / "marble") + + +def ensure_marble_on_path() -> None: + """Add vendored MARBLE clone root to sys.path. + + MARBLE's internal code uses absolute imports like ``from marble.environments...``. + Since it's vendored (not installed), we add its clone root to sys.path so Python + can resolve these imports. + """ + if _MARBLE_ROOT not in sys.path: + sys.path.insert(0, _MARBLE_ROOT) diff --git a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py index 1da394a..b750c94 100644 --- a/maseval/benchmark/multiagentbench/adapters/marble_adapter.py +++ b/maseval/benchmark/multiagentbench/adapters/marble_adapter.py @@ -7,7 +7,7 @@ from typing import Any, Dict, List, Sequence, Tuple from maseval import AgentAdapter, AgentError, MessageHistory -from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR, ensure_marble_on_path class MarbleAgentAdapter(AgentAdapter): @@ -197,8 +197,9 @@ def create_marble_agents( Raises: ImportError: If MARBLE is not available """ + ensure_marble_on_path() try: - from ..marble.marble.agent.base_agent import BaseAgent # type: ignore[unresolved-import] + from marble.agent.base_agent import BaseAgent # type: ignore[import-untyped] except ImportError as e: raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e diff --git a/maseval/benchmark/multiagentbench/data_loader.py b/maseval/benchmark/multiagentbench/data_loader.py index 3a05aa9..0f903f3 100644 --- a/maseval/benchmark/multiagentbench/data_loader.py +++ b/maseval/benchmark/multiagentbench/data_loader.py @@ -7,17 +7,20 @@ import json import logging import os -import subprocess from pathlib import Path from typing import Any, Dict, FrozenSet, List, Optional +import git + from maseval import Task logger = logging.getLogger(__name__) # MARBLE repository configuration -MARBLE_REPO_URL = "https://github.com/ulab-uiuc/MARBLE.git" -MARBLE_DEFAULT_COMMIT = None # Set to a specific commit hash for reproducibility, or None for latest +# Original: https://github.com/ulab-uiuc/MARBLE (where the work was done) +# Using fork: https://github.com/cemde/MARBLE (contains bug fixes) +MARBLE_REPO_URL = "https://github.com/cemde/MARBLE.git" +MARBLE_DEFAULT_COMMIT = None # Unpinned while bug fixes are being added; will pin once stable # Valid domain names VALID_DOMAINS: FrozenSet[str] = frozenset( @@ -27,8 +30,7 @@ "minecraft", "research", "bargaining", - "web", - "worldsimulation", + "werewolf", } ) @@ -85,31 +87,18 @@ def download_marble( logger.info(f"Cloning MARBLE from {MARBLE_REPO_URL} to {target_dir}") try: - subprocess.run( - ["git", "clone", MARBLE_REPO_URL, str(target_dir)], - check=True, - capture_output=True, - text=True, - ) - except subprocess.CalledProcessError as e: - raise RuntimeError(f"Failed to clone MARBLE: {e.stderr}") from e - except FileNotFoundError: - raise RuntimeError("git is not installed or not in PATH. Please install git and try again.") + repo = git.Repo.clone_from(MARBLE_REPO_URL, str(target_dir)) + except (git.GitCommandError, git.exc.GitCommandNotFound) as e: + raise RuntimeError(f"Failed to clone MARBLE: {e}") from e # Checkout specific commit if requested checkout_commit = commit or MARBLE_DEFAULT_COMMIT if checkout_commit: logger.info(f"Checking out commit: {checkout_commit}") try: - subprocess.run( - ["git", "checkout", checkout_commit], - cwd=target_dir, - check=True, - capture_output=True, - text=True, - ) - except subprocess.CalledProcessError as e: - raise RuntimeError(f"Failed to checkout commit {checkout_commit}: {e.stderr}") from e + repo.git.checkout(checkout_commit) + except git.GitCommandError as e: + raise RuntimeError(f"Failed to checkout commit {checkout_commit}: {e}") from e # Create __init__.py at clone root so Python can traverse it as a package. # The actual MARBLE Python package lives at marble/marble/ inside the clone, @@ -146,13 +135,23 @@ def ensure_marble_exists(auto_download: bool = True) -> Path: # Check if MARBLE exists and has the expected structure if marble_dir.exists() and (marble_dir / "multiagentbench").exists(): + # Verify pinned commit if set + if MARBLE_DEFAULT_COMMIT: + try: + repo = git.Repo(marble_dir) + current_commit = repo.head.commit.hexsha + if current_commit != MARBLE_DEFAULT_COMMIT: + logger.info(f"MARBLE at {current_commit[:12]} but pinned to {MARBLE_DEFAULT_COMMIT[:12]}, checking out...") + repo.git.checkout(MARBLE_DEFAULT_COMMIT) + except (git.InvalidGitRepositoryError, git.GitCommandError): + logger.warning("Could not verify MARBLE commit (not a git repo or checkout failed)") return marble_dir if not auto_download: raise FileNotFoundError( f"MARBLE not found at {marble_dir}.\n" "Run `ensure_marble_exists(auto_download=True)` to download automatically,\n" - "or manually clone: git clone https://github.com/ulab-uiuc/MARBLE.git marble" + "or manually clone: git clone https://github.com/cemde/MARBLE.git marble" ) return download_marble(marble_dir) @@ -283,16 +282,165 @@ def _parse_task_entry(entry: Dict[str, Any], domain: str, idx: int) -> Task: ) +def _load_werewolf_tasks( + data_dir: Path, + limit: Optional[int] = None, +) -> List[Task]: + """Load werewolf tasks from MARBLE config files. + + Werewolf uses a config-based game engine rather than JSONL task data. + This function finds werewolf config YAMLs in the MARBLE configs directory + and constructs Task objects from them. + + Args: + data_dir: Resolved MARBLE data directory (typically marble/multiagentbench/) + limit: Maximum number of tasks to load (None for all) + + Returns: + List of Task objects for werewolf games + + Raises: + FileNotFoundError: If no werewolf configs found + """ + # Navigate from data_dir (marble/multiagentbench/) to MARBLE root (marble/) + marble_root = data_dir.parent + + # Search for werewolf config YAMLs + configs_dir = marble_root / "marble" / "configs" + if not configs_dir.exists(): + raise FileNotFoundError( + f"MARBLE configs directory not found: {configs_dir}\n" + "Ensure MARBLE is cloned to multiagentbench/marble/\n" + "See multiagentbench/README.md for setup instructions." + ) + + config_paths = sorted(configs_dir.glob("**/werewolf_config*.yaml")) + + if not config_paths: + raise FileNotFoundError(f"No werewolf config files found in {configs_dir}\nExpected files matching: **/werewolf_config*.yaml") + + tasks = [] + for idx, config_path in enumerate(config_paths): + if limit is not None and idx >= limit: + break + + # Parse the YAML config to extract game setup + try: + import yaml + + with config_path.open(encoding="utf-8") as f: + config = yaml.safe_load(f) + except ImportError: + # Fall back to basic parsing if PyYAML not available + config = _parse_werewolf_config_basic(config_path) + + roles = config.get("roles", []) + cooperation_mode = config.get("cooperation_mode", "cooperative") + + # Build agent specs from roles + agents = [] + for role_idx, role in enumerate(roles): + agents.append( + { + "agent_id": f"player_{role_idx}", + "profile": f"Werewolf game player with role: {role}", + "type": "WerewolfAgent", + "role": role, + } + ) + + task = Task( + id=f"werewolf_{idx}", + query="Play a Werewolf social deduction game", + environment_data={ + "scenario": "werewolf", + "coordinate_mode": cooperation_mode, + "relationships": [], + "environment": { + "type": "Werewolf", + "description": "Werewolf social deduction game", + }, + "task": { + "content": "Play a Werewolf social deduction game", + }, + "agents": agents, + "max_iterations": 20, + "engine_planner": {}, + "memory": {}, + "output": {}, + "werewolf_config_path": str(config_path), + "raw_marble_config": config, + }, + evaluation_data={ + "metrics": {}, + "output_format": "", + }, + metadata={ + "domain": "werewolf", + "task_id": idx, + "scenario": "werewolf", + "config_path": str(config_path), + }, + ) + tasks.append(task) + + return tasks + + +def _parse_werewolf_config_basic(config_path: Path) -> Dict[str, Any]: + """Parse a werewolf YAML config without PyYAML. + + Basic key-value and list parsing for werewolf configs. + + Args: + config_path: Path to the YAML config file + + Returns: + Parsed config dictionary + """ + config: Dict[str, Any] = {} + current_key = "" + current_list: List[str] = [] + + with config_path.open(encoding="utf-8") as f: + for line in f: + stripped = line.strip() + if not stripped or stripped.startswith("#"): + continue + + if stripped.startswith("- "): + # List item + current_list.append(stripped[2:].strip()) + config[current_key] = current_list + elif ":" in stripped: + # Key-value pair + key, _, value = stripped.partition(":") + key = key.strip() + value = value.strip().strip('"').strip("'") + + if value: + config[key] = value + else: + # Start of a list or nested object + current_key = key + current_list = [] + + return config + + def load_tasks( domain: str, data_dir: Optional[Path] = None, limit: Optional[int] = None, ) -> List[Task]: - """Load MultiAgentBench tasks from JSONL files. + """Load MultiAgentBench tasks for a domain. + + Most domains load from JSONL files. Werewolf uses config-based + task loading since it has no JSONL data (it uses a game engine). Args: domain: Domain name (one of: coding, database, minecraft, research, - bargaining, web, worldsimulation) + bargaining, werewolf) data_dir: Optional path to MARBLE data directory limit: Maximum number of tasks to load (None for all) @@ -317,6 +465,11 @@ def load_tasks( # Find data directory resolved_data_dir = _resolve_data_dir(data_dir) + + # Werewolf uses config-based loading (no JSONL data) + if domain_lower == "werewolf": + return _load_werewolf_tasks(resolved_data_dir, limit) + jsonl_path = resolved_data_dir / domain_lower / f"{domain_lower}_main.jsonl" if not jsonl_path.exists(): @@ -414,7 +567,7 @@ def get_domain_info(domain: str) -> Dict[str, Any]: }, "minecraft": { "requires_infrastructure": True, - "description": "Collaborative building in Minecraft (requires game server)", + "description": "Collaborative building in Minecraft (untested; requires Minecraft Server 1.19.2, Node.js, npm)", "coordination_mode": "cooperative", }, "research": { @@ -427,14 +580,9 @@ def get_domain_info(domain: str) -> Dict[str, Any]: "description": "Negotiation and bargaining scenarios", "coordination_mode": "cooperative", }, - "web": { - "requires_infrastructure": False, - "description": "Web-based task completion", - "coordination_mode": "star", - }, - "worldsimulation": { + "werewolf": { "requires_infrastructure": False, - "description": "World simulation and interaction tasks", + "description": "Adversarial social deduction game with roles (wolf, villager, seer, witch, guard)", "coordination_mode": "cooperative", }, } diff --git a/maseval/benchmark/multiagentbench/environment.py b/maseval/benchmark/multiagentbench/environment.py index 2c010d1..bfaff69 100644 --- a/maseval/benchmark/multiagentbench/environment.py +++ b/maseval/benchmark/multiagentbench/environment.py @@ -3,15 +3,16 @@ This module provides the MASEval Environment wrapper for MARBLE environments. """ +import importlib import shutil from typing import Any, Callable, Dict, Optional from maseval import Environment, EnvironmentError, ToolInvocationHistory -from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR, ensure_marble_on_path # Domains requiring external infrastructure -INFRASTRUCTURE_DOMAINS = frozenset({"database", "minecraft"}) +INFRASTRUCTURE_DOMAINS = frozenset({"database"}) class MultiAgentBenchEnvironment(Environment): @@ -37,9 +38,10 @@ def __init__( Raises: EnvironmentError: If required infrastructure is unavailable + ImportError: If MARBLE is not available """ self.domain = task_data.get("scenario", "") - self._marble_env: Optional[Any] = None + self._marble_env: Any = None self._tool_histories: Dict[str, ToolInvocationHistory] = {} super().__init__(task_data) @@ -75,18 +77,17 @@ def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: "max_iterations": env_config.get("max_iterations") or task_data.get("max_iterations", 10), } - # Try to create MARBLE environment (may fail if MARBLE not available) - try: - self._marble_env = self._create_marble_environment(domain, marble_config) - except ImportError: - # MARBLE not available - store config for later use - self._marble_env = None + # Pass werewolf config path for WerewolfEnv (different constructor) + if domain.lower() == "werewolf": + marble_config["werewolf_config_path"] = task_data.get("werewolf_config_path", "") + + self._marble_env = self._create_marble_environment(domain, marble_config) return { "domain": domain, "env_config": env_config, "task_config": task_config, - "marble_env_type": type(self._marble_env).__name__ if self._marble_env else "None", + "marble_env_type": type(self._marble_env).__name__, "max_iterations": marble_config["max_iterations"], } @@ -105,10 +106,6 @@ def _check_infrastructure(self, domain: str) -> bool: # Check Docker availability return shutil.which("docker") is not None - if domain_lower == "minecraft": - # Minecraft requires external server - always fail for now - return False - return True def _create_marble_environment( @@ -131,9 +128,11 @@ def _create_marble_environment( domain_lower = domain.lower() env_name = config.get("name", domain_lower) + ensure_marble_on_path() + # Import MARBLE environments try: - from .marble.marble.environments.base_env import BaseEnvironment # type: ignore[unresolved-import] + from marble.environments.base_env import BaseEnvironment # type: ignore[import-untyped] except ImportError as e: raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e @@ -142,10 +141,9 @@ def _create_marble_environment( "coding": "marble.environments.coding_env.CodingEnvironment", "database": "marble.environments.db_env.DBEnvironment", "research": "marble.environments.research_env.ResearchEnvironment", - "bargaining": "marble.environments.bargaining_env.BargainingEnvironment", - "web": "marble.environments.web_env.WebEnvironment", - "worldsimulation": "marble.environments.world_env.WorldSimulationEnvironment", + "bargaining": "marble.environments.world_env.WorldSimulationEnvironment", "minecraft": "marble.environments.minecraft_env.MinecraftEnvironment", + "werewolf": "marble.environments.werewolf_env.WerewolfEnv", } env_class_path = env_mapping.get(domain_lower) @@ -154,17 +152,18 @@ def _create_marble_environment( # Use base environment for unknown domains return BaseEnvironment(env_name, config) - try: - # Dynamic import of domain-specific environment - module_path, class_name = env_class_path.rsplit(".", 1) - # Adjust import path for vendored MARBLE - module_path = module_path.replace("marble.", ".marble.marble.", 1) - module = __import__(module_path, globals(), locals(), [class_name], 1) - env_class = getattr(module, class_name) - return env_class(env_name, config) - except (ImportError, AttributeError): - # Fall back to base environment - return BaseEnvironment(env_name, config) + # Dynamic import of domain-specific environment + module_path, class_name = env_class_path.rsplit(".", 1) + module = importlib.import_module(module_path) + env_class = getattr(module, class_name) + + # WerewolfEnv has a different constructor: (name, config_path, log_dir) + if domain_lower == "werewolf": + config_path = config.get("werewolf_config_path", "") + return env_class(env_name, config_path) + + # MARBLE domain envs take (config, name) while BaseEnvironment takes (name, config) + return env_class(config=config, name=env_name) def create_tools(self) -> Dict[str, Callable]: """Create tools from MARBLE environment for MASEval tracing. @@ -175,9 +174,6 @@ def create_tools(self) -> Dict[str, Callable]: Returns: Dict mapping tool names to wrapped callables """ - if self._marble_env is None: - return {} - tools: Dict[str, Callable] = {} # Get action handlers from MARBLE environment @@ -251,9 +247,6 @@ def get_tool_descriptions(self) -> Dict[str, Any]: Returns: Dict mapping tool names to their OpenAI-format descriptions """ - if self._marble_env is None: - return {} - return getattr(self._marble_env, "action_handler_descriptions", {}) def apply_action( @@ -272,15 +265,7 @@ def apply_action( Returns: Action result dictionary - Raises: - EnvironmentError: If MARBLE environment is not available """ - if self._marble_env is None: - raise EnvironmentError( - "MARBLE environment not available. Cannot execute actions.", - component="MultiAgentBenchEnvironment", - ) - return self._marble_env.apply_action(agent_id, action_name, arguments) def is_done(self) -> bool: @@ -289,8 +274,6 @@ def is_done(self) -> bool: Returns: True if done, False otherwise """ - if self._marble_env is None: - return False return self._marble_env.is_done() def is_task_completed(self) -> bool: @@ -299,8 +282,6 @@ def is_task_completed(self) -> bool: Returns: True if task completed, False otherwise """ - if self._marble_env is None: - return False return self._marble_env.is_task_completed() def get_marble_state(self) -> Dict[str, Any]: @@ -309,8 +290,6 @@ def get_marble_state(self) -> Dict[str, Any]: Returns: State dictionary from MARBLE environment """ - if self._marble_env is None: - return {} return self._marble_env.get_state() def gather_traces(self) -> Dict[str, Any]: @@ -323,13 +302,10 @@ def gather_traces(self) -> Dict[str, Any]: # Add domain-specific info traces["domain"] = self.domain - traces["marble_env_type"] = type(self._marble_env).__name__ if self._marble_env else "None" - - # Add MARBLE state if available - if self._marble_env is not None: - traces["marble_state"] = self.get_marble_state() - traces["is_done"] = self.is_done() - traces["is_task_completed"] = self.is_task_completed() + traces["marble_env_type"] = type(self._marble_env).__name__ + traces["marble_state"] = self.get_marble_state() + traces["is_done"] = self.is_done() + traces["is_task_completed"] = self.is_task_completed() # Collect tool invocation histories tool_traces = {} @@ -352,7 +328,7 @@ def gather_config(self) -> Dict[str, Any]: config = super().gather_config() config["domain"] = self.domain - config["marble_env_type"] = type(self._marble_env).__name__ if self._marble_env else "None" + config["marble_env_type"] = type(self._marble_env).__name__ # Add tool descriptions config["tool_descriptions"] = self.get_tool_descriptions() diff --git a/maseval/benchmark/multiagentbench/evaluator.py b/maseval/benchmark/multiagentbench/evaluator.py index 43ed8e3..ad5cd31 100644 --- a/maseval/benchmark/multiagentbench/evaluator.py +++ b/maseval/benchmark/multiagentbench/evaluator.py @@ -14,6 +14,11 @@ # Using None instead of -1 to avoid confusion with valid scores SCORE_NOT_EVALUATED: None = None +# Default per-agent result truncation length (in characters) before LLM summarization. +# Matches MARBLE's _summarize_results() in engine.py which truncates each formatted +# result line to 1000 characters. Set to None to disable truncation and LLM summarization. +DEFAULT_RESULT_TRUNCATION_LENGTH = 1000 + @dataclass class MultiAgentBenchMetrics: @@ -70,6 +75,7 @@ def __init__( model_adapter: ModelAdapter, metrics_config: Optional[Dict[str, Any]] = None, output_format: str = "", + result_truncation_length: Optional[int] = DEFAULT_RESULT_TRUNCATION_LENGTH, ): """Initialize the evaluator. @@ -78,11 +84,19 @@ def __init__( model_adapter: Model adapter for LLM evaluation metrics_config: Configuration for evaluation metrics output_format: Expected output format for task evaluation + result_truncation_length: Maximum characters per agent result before LLM + summarization. Matches MARBLE's ``_summarize_results()`` which truncates + each result to 1000 chars, then passes the truncated output through an + LLM summarization call (``planner.summarize_output()``). Set to ``None`` + to disable both truncation and LLM summarization, passing raw agent + results directly to the evaluator (not recommended for domains with + large outputs like research). """ self.domain = domain.lower() self.model_adapter = model_adapter self.metrics_config = metrics_config or {} self.output_format = output_format + self.result_truncation_length = result_truncation_length self._evaluation_prompts = self._load_evaluation_prompts() def _load_template(self, filename: str) -> str: @@ -119,6 +133,16 @@ def _load_evaluation_prompts(self) -> Dict[str, Any]: "prompt": self._load_template("coding.txt"), } }, + "werewolf": { + "task_evaluation": { + "prompt": self._load_template("werewolf.txt"), + } + }, + "minecraft": { + "task_evaluation": { + "prompt": self._load_template("minecraft.txt"), + } + }, } def filter_traces(self, traces: Dict[str, Any]) -> Dict[str, Any]: @@ -206,16 +230,30 @@ def __call__( # type: ignore[override] # Domain-specific evaluation task_desc = self._get_task_description(traces) - final_result = self._format_final_answer(final_answer) + + # MARBLE-compatible result summarization: truncate per-agent results then + # pass through an LLM summarization call before domain evaluation. This + # prevents token explosions for domains with large outputs (e.g. research). + # See MARBLE's Engine._summarize_results() and EnginePlanner.summarize_output(). + agent_results = self._extract_agent_results(final_answer) + if self.result_truncation_length is not None and agent_results: + truncated = self._summarize_results(agent_results) + final_result = self._summarize_output(truncated, task_desc, self.output_format) + else: + final_result = self._format_final_answer(final_answer) if self.domain == "research": metrics.task_evaluation = self._evaluate_research(task_desc, final_result) - elif self.domain in ("bargaining", "worldsimulation"): + elif self.domain == "bargaining": metrics.task_evaluation = self._evaluate_bargaining(task_desc, final_result) elif self.domain == "coding": metrics.task_evaluation = self._evaluate_coding(task_desc, final_result) elif self.domain == "database": metrics.task_evaluation = self._evaluate_database(task_desc, final_result) + elif self.domain == "werewolf": + metrics.task_evaluation = self._evaluate_werewolf(task_desc, final_result) + elif self.domain == "minecraft": + metrics.task_evaluation = self._evaluate_minecraft(task_desc, final_result) else: # Default: check if task has a completion marker metrics.task_completion = bool(final_result) @@ -235,6 +273,23 @@ def _get_task_description(self, traces: Dict[str, Any]) -> str: state = env_traces.get("marble_state", {}) return state.get("task_description", "") + def _extract_agent_results(self, final_answer: Any) -> List[Dict[str, Any]]: + """Extract the agent_results list from final_answer, if present. + + Args: + final_answer: Final output from agents (dict, list, str, or None). + + Returns: + List of agent result dicts, or empty list if not extractable. + """ + if isinstance(final_answer, dict): + results = final_answer.get("agent_results", []) + if results and isinstance(results, list): + return results + elif isinstance(final_answer, list): + return [r for r in final_answer if isinstance(r, dict)] + return [] + def _format_final_answer(self, final_answer: Any) -> str: """Format final answer for evaluation.""" if isinstance(final_answer, dict): @@ -247,6 +302,57 @@ def _format_final_answer(self, final_answer: Any) -> str: return "\n".join(f"[{r.get('agent_id', 'unknown')}]: {r.get('result', '')}" for r in final_answer if isinstance(r, dict)) return str(final_answer) if final_answer else "" + def _summarize_results(self, agent_results: List[Dict[str, Any]]) -> str: + """Truncate and concatenate agent results for LLM summarization. + + Matches MARBLE's ``Engine._summarize_results()`` (engine.py). Each agent's + result is formatted as ``"- {result}"`` and truncated to + ``self.result_truncation_length`` characters before concatenation. + + Args: + agent_results: List of dicts with ``agent_id`` and ``result`` keys, + as returned in ``final_answer["agent_results"]``. + + Returns: + Concatenated summary string with truncated per-agent results. + """ + summary = "Agents' Results Summary:\n" + for entry in agent_results: + result_str = str(entry.get("result", "")) + line = f"- {result_str}" + assert self.result_truncation_length is not None + summary += f"{line[: self.result_truncation_length]}\n" + return summary + + def _summarize_output(self, summary: str, task: str, output_format: str) -> str: + """Summarize truncated agent results via LLM call. + + Matches MARBLE's ``EnginePlanner.summarize_output()`` (engine_planner.py). + Uses temperature=0.0 and max_tokens=2048 for deterministic, compact output. + + Note: + The prompt text preserves MARBLE's original wording (including the + ``"thr"`` typo) to maintain reproduction fidelity with the reference + implementation. + + Args: + summary: Truncated results string from ``_summarize_results()``. + task: Task description string. + output_format: Expected JSON output format specification. + + Returns: + LLM-generated summary string. + """ + prompt = ( + f"Summarize the output of the agents for the task: {task}\n\n" + f"Now here is some result of thr agent: {summary}, please analyze it. " + f"Return the final output into a json following the format: {output_format}" + ) + return self.model_adapter.generate( + prompt, + generation_params={"temperature": 0.0, "max_tokens": 2048}, + ) + def _calculate_token_consumption(self, traces: Dict[str, Any]) -> int: """Calculate total token consumption.""" total = 0 @@ -259,27 +365,21 @@ def _calculate_token_consumption(self, traces: Dict[str, Any]) -> int: return total - def _evaluate_communication(self, task: str, communications: str) -> Optional[float]: + def _evaluate_communication(self, task: str, communications: str) -> float: """Evaluate communication quality using LLM.""" prompt_template = self._evaluation_prompts["communication"]["prompt"] prompt = prompt_template.format(task=task, communications=communications) - try: - response = self.model_adapter.generate(prompt) - return self._parse_score(response) - except Exception: - return None + response = self.model_adapter.generate(prompt) + return self._parse_score(response) def _evaluate_research(self, task: str, result: str) -> Dict[str, Any]: """Evaluate research task output.""" prompt_template = self._evaluation_prompts["research"]["task_evaluation"]["prompt"] prompt = prompt_template.format(task=task, result=result) - try: - response = self.model_adapter.generate(prompt) - return self._parse_research_ratings(response) - except Exception: - return {"innovation": None, "safety": None, "feasibility": None} + response = self.model_adapter.generate(prompt) + return self._parse_research_ratings(response) def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: """Evaluate bargaining/world simulation task output.""" @@ -287,29 +387,13 @@ def _evaluate_bargaining(self, task: str, result: str) -> Dict[str, Any]: buyer_prompt = self._evaluation_prompts["bargaining"]["task_evaluation"]["buyer_prompt"] seller_prompt = self._evaluation_prompts["bargaining"]["task_evaluation"]["seller_prompt"] - ratings = {"buyer": {}, "seller": {}} - - try: - buyer_response = self.model_adapter.generate(buyer_prompt.format(task=task, result=result)) - ratings["buyer"] = self._parse_bargaining_ratings(buyer_response) - except Exception: - ratings["buyer"] = { - "effectiveness_of_strategies": None, - "progress_and_outcome": None, - "interaction_dynamics": None, - } - - try: - seller_response = self.model_adapter.generate(seller_prompt.format(task=task, result=result)) - ratings["seller"] = self._parse_bargaining_ratings(seller_response) - except Exception: - ratings["seller"] = { - "effectiveness_of_strategies": None, - "progress_and_outcome": None, - "interaction_dynamics": None, - } - - return ratings + buyer_response = self.model_adapter.generate(buyer_prompt.format(task=task, result=result)) + seller_response = self.model_adapter.generate(seller_prompt.format(task=task, result=result)) + + return { + "buyer": self._parse_bargaining_ratings(buyer_response), + "seller": self._parse_bargaining_ratings(seller_response), + } def _evaluate_coding(self, task: str, result: str) -> Dict[str, Any]: """Evaluate coding task output.""" @@ -323,16 +407,8 @@ def _evaluate_coding(self, task: str, result: str) -> Dict[str, Any]: solution=result, ) - try: - response = self.model_adapter.generate(prompt) - return self._parse_coding_ratings(response) - except Exception: - return { - "instruction_following": None, - "executability": None, - "consistency": None, - "quality": None, - } + response = self.model_adapter.generate(prompt) + return self._parse_coding_ratings(response) def _evaluate_database(self, task: str, result: str) -> Dict[str, Any]: """Evaluate database task output. @@ -345,141 +421,224 @@ def _evaluate_database(self, task: str, result: str) -> Dict[str, Any]: "root_cause": [], # Would be filled from task data } - def _parse_score(self, response: str) -> Optional[float]: + def _evaluate_werewolf(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate werewolf game output.""" + prompt_template = self._evaluation_prompts["werewolf"]["task_evaluation"]["prompt"] + prompt = prompt_template.format(task=task, result=result) + + response = self.model_adapter.generate(prompt) + return self._parse_werewolf_ratings(response) + + def _evaluate_minecraft(self, task: str, result: str) -> Dict[str, Any]: + """Evaluate minecraft building task output. + + Warning: + Minecraft evaluation is untested. It requires a running Minecraft + Server (1.19.2) and Node.js/npm for Mineflayer bot dependencies. + """ + prompt_template = self._evaluation_prompts["minecraft"]["task_evaluation"]["prompt"] + prompt = prompt_template.format(task=task, result=result) + + response = self.model_adapter.generate(prompt) + return self._parse_minecraft_ratings(response) + + def _parse_score(self, response: str) -> float: """Parse a single score from LLM response. Returns: - Score as float (1-5), or None if parsing fails + Score as float (1-5) + + Raises: + ValueError: If the response cannot be parsed into a valid score """ - try: - content = response.strip() - - # Remove markdown code block markers - if content.startswith("```json"): - content = content[7:] - if content.startswith("```"): - content = content[3:] - if content.endswith("```"): - content = content[:-3] - content = content.strip() - - # Find JSON object - json_start = content.find("{") - json_end = content.rfind("}") + 1 - - if json_start >= 0 and json_end > json_start: - json_str = content[json_start:json_end] - rating_data = json.loads(json_str) - if isinstance(rating_data, dict) and "rating" in rating_data: - score = int(rating_data["rating"]) - if 1 <= score <= 5: - return float(score) - - return None - - except Exception: - return None - - def _parse_research_ratings(self, response: str) -> Dict[str, Optional[int]]: - """Parse research evaluation ratings.""" - try: - content = response.strip() - json_start = content.find("{") - json_end = content.rfind("}") + 1 - - if json_start >= 0 and json_end > json_start: - json_str = content[json_start:json_end] - ratings = json.loads(json_str) - return {k: int(v) for k, v in ratings.items()} - except Exception: - pass - - return {"innovation": None, "safety": None, "feasibility": None} - - def _parse_bargaining_ratings(self, response: str) -> Dict[str, Optional[int]]: - """Parse bargaining evaluation ratings.""" - try: - content = response.strip() - json_start = content.find("{") - json_end = content.rfind("}") + 1 - - if json_start >= 0 and json_end > json_start: - json_str = content[json_start:json_end] - ratings = json.loads(json_str) - return { - "effectiveness_of_strategies": int(ratings["effectiveness_of_strategies"]) - if "effectiveness_of_strategies" in ratings - else None, - "progress_and_outcome": int(ratings["progress_and_outcome"]) if "progress_and_outcome" in ratings else None, - "interaction_dynamics": int(ratings["interaction_dynamics"]) if "interaction_dynamics" in ratings else None, - } - except Exception: - pass + content = response.strip() + + # Remove markdown code block markers + if content.startswith("```json"): + content = content[7:] + if content.startswith("```"): + content = content[3:] + if content.endswith("```"): + content = content[:-3] + content = content.strip() + + # Find JSON object + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start < 0 or json_end <= json_start: + raise ValueError(f"No JSON object found in evaluator response: {response!r}") + + json_str = content[json_start:json_end] + rating_data = json.loads(json_str) + + if not isinstance(rating_data, dict) or "rating" not in rating_data: + raise ValueError(f"Expected {{'rating': ...}} in evaluator response, got: {rating_data!r}") + + score = int(rating_data["rating"]) + if not 1 <= score <= 5: + raise ValueError(f"Score {score} out of valid range 1-5 in evaluator response") + + return float(score) + def _parse_research_ratings(self, response: str) -> Dict[str, int]: + """Parse research evaluation ratings. + + Raises: + ValueError: If the response cannot be parsed into valid ratings + """ + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start < 0 or json_end <= json_start: + raise ValueError(f"No JSON object found in evaluator response: {response!r}") + + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return {k: int(v) for k, v in ratings.items()} + + def _parse_bargaining_ratings(self, response: str) -> Dict[str, int]: + """Parse bargaining evaluation ratings. + + Raises: + ValueError: If the response cannot be parsed into valid ratings + """ + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start < 0 or json_end <= json_start: + raise ValueError(f"No JSON object found in evaluator response: {response!r}") + + json_str = content[json_start:json_end] + ratings = json.loads(json_str) return { - "effectiveness_of_strategies": None, - "progress_and_outcome": None, - "interaction_dynamics": None, + "effectiveness_of_strategies": int(ratings["effectiveness_of_strategies"]), + "progress_and_outcome": int(ratings["progress_and_outcome"]), + "interaction_dynamics": int(ratings["interaction_dynamics"]), } - def _parse_coding_ratings(self, response: str) -> Dict[str, Optional[int]]: - """Parse coding evaluation ratings.""" - try: - content = response.strip() - json_start = content.find("{") - json_end = content.rfind("}") + 1 - - if json_start >= 0 and json_end > json_start: - json_str = content[json_start:json_end] - ratings = json.loads(json_str) - return { - "instruction_following": int(ratings["instruction_following"]) if "instruction_following" in ratings else None, - "executability": int(ratings["executability"]) if "executability" in ratings else None, - "consistency": int(ratings["consistency"]) if "consistency" in ratings else None, - "quality": int(ratings["quality"]) if "quality" in ratings else None, - } - except Exception: - pass + def _parse_coding_ratings(self, response: str) -> Dict[str, int]: + """Parse coding evaluation ratings. + + Raises: + ValueError: If the response cannot be parsed into valid ratings + """ + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + if json_start < 0 or json_end <= json_start: + raise ValueError(f"No JSON object found in evaluator response: {response!r}") + + json_str = content[json_start:json_end] + ratings = json.loads(json_str) return { - "instruction_following": None, - "executability": None, - "consistency": None, - "quality": None, + "instruction_following": int(ratings["instruction_following"]), + "executability": int(ratings["executability"]), + "consistency": int(ratings["consistency"]), + "quality": int(ratings["quality"]), } + def _parse_werewolf_ratings(self, response: str) -> Dict[str, int]: + """Parse werewolf evaluation ratings. + + Raises: + ValueError: If the response cannot be parsed into valid ratings + """ + keys = [ + "game_outcome", + "deception_detection", + "voting_strategy", + "role_fulfillment", + "information_usage", + "collaboration", + "survival_rate", + ] + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start < 0 or json_end <= json_start: + raise ValueError(f"No JSON object found in evaluator response: {response!r}") + + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return {k: int(ratings[k]) for k in keys} + + def _parse_minecraft_ratings(self, response: str) -> Dict[str, int]: + """Parse minecraft evaluation ratings. + + Raises: + ValueError: If the response cannot be parsed into valid ratings + """ + keys = [ + "structural_completeness", + "blueprint_accuracy", + "coordination", + "efficiency", + ] + content = response.strip() + json_start = content.find("{") + json_end = content.rfind("}") + 1 + + if json_start < 0 or json_end <= json_start: + raise ValueError(f"No JSON object found in evaluator response: {response!r}") + + json_str = content[json_start:json_end] + ratings = json.loads(json_str) + return {k: int(ratings[k]) for k in keys} + def _determine_completion(self, metrics: MultiAgentBenchMetrics) -> bool: """Determine if task was completed based on metrics. - A task is considered completed if all required scores are present (not None) - and positive (> 0). + For LLM-evaluated domains, scores are always positive ints if we reach this + point (parse failures raise before getting here), so this just checks > 0. """ eval_data = metrics.task_evaluation if not eval_data: return False - def _all_scores_valid(scores: List[Any]) -> bool: - """Check all scores are present and positive.""" + # Parse methods guarantee int scores or raise, so None checks are + # prob dead code — remove later if no edge cases surface. + def _all_scores_positive(scores: List[Any]) -> bool: return all(s is not None and s > 0 for s in scores) if self.domain == "research": - scores = [eval_data.get(k) for k in ["innovation", "safety", "feasibility"]] - return _all_scores_valid(scores) + return _all_scores_positive([eval_data[k] for k in ["innovation", "safety", "feasibility"]]) - elif self.domain in ("bargaining", "worldsimulation"): - buyer = eval_data.get("buyer", {}) - seller = eval_data.get("seller", {}) - buyer_scores = [buyer.get(k) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] - seller_scores = [seller.get(k) for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] - return _all_scores_valid(buyer_scores) and _all_scores_valid(seller_scores) + elif self.domain == "bargaining": + buyer = eval_data["buyer"] + seller = eval_data["seller"] + return _all_scores_positive( + [buyer[k] for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]] + ) and _all_scores_positive([seller[k] for k in ["effectiveness_of_strategies", "progress_and_outcome", "interaction_dynamics"]]) elif self.domain == "coding": - scores = [eval_data.get(k) for k in ["instruction_following", "executability", "consistency", "quality"]] - return _all_scores_valid(scores) + return _all_scores_positive([eval_data[k] for k in ["instruction_following", "executability", "consistency", "quality"]]) elif self.domain == "database": - # Database completion is determined by comparing prediction to labels return bool(eval_data.get("predicted")) + elif self.domain == "werewolf": + return _all_scores_positive( + [ + eval_data[k] + for k in [ + "game_outcome", + "deception_detection", + "voting_strategy", + "role_fulfillment", + "information_usage", + "collaboration", + "survival_rate", + ] + ] + ) + + elif self.domain == "minecraft": + return _all_scores_positive([eval_data[k] for k in ["structural_completeness", "blueprint_accuracy", "coordination", "efficiency"]]) + return False diff --git a/maseval/benchmark/multiagentbench/multiagentbench.py b/maseval/benchmark/multiagentbench/multiagentbench.py index 5441bb3..78f75bc 100644 --- a/maseval/benchmark/multiagentbench/multiagentbench.py +++ b/maseval/benchmark/multiagentbench/multiagentbench.py @@ -22,7 +22,7 @@ from maseval.core.callback import BenchmarkCallback from maseval.core.seeding import SeedGenerator -from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR +from maseval.benchmark.multiagentbench._constants import MARBLE_IMPORT_ERROR, ensure_marble_on_path from maseval.benchmark.multiagentbench.environment import MultiAgentBenchEnvironment from maseval.benchmark.multiagentbench.evaluator import ( MultiAgentBenchEvaluator, @@ -449,8 +449,9 @@ def _create_marble_env(self, task: Task) -> Any: Returns: MARBLE environment instance """ + ensure_marble_on_path() try: - from .marble.marble.environments.base_env import BaseEnvironment # type: ignore[unresolved-import] + from marble.environments.base_env import BaseEnvironment # type: ignore[import-untyped] except ImportError as e: raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e @@ -478,11 +479,11 @@ def _setup_agent_graph( task: Task with relationship data marble_env: MARBLE environment """ + ensure_marble_on_path() try: - from .marble.marble.graph.agent_graph import AgentGraph # type: ignore[unresolved-import] - except ImportError: - # MARBLE not available, skip graph setup - return + from marble.graph.agent_graph import AgentGraph # type: ignore[import-untyped] + except ImportError as e: + raise ImportError(MARBLE_IMPORT_ERROR.format(error=e)) from e # Extract MARBLE agents from adapters marble_agents = [adapter.marble_agent for adapter in agents_dict.values()] @@ -492,17 +493,12 @@ def _setup_agent_graph( coordination_mode = task.environment_data.get("coordinate_mode", "cooperative") config = SimpleNamespace(coordination_mode=coordination_mode, relationships=relationships) - try: - # Create agent graph - graph = AgentGraph(marble_agents, config) # type: ignore - - # Set graph on all agents - for agent in marble_agents: - agent.set_agent_graph(graph) + # Create agent graph + graph = AgentGraph(marble_agents, config) # type: ignore - except (ValueError, KeyError, AttributeError) as e: - # Graph creation failed, agents will work without inter-agent communication - logger.warning("AgentGraph setup failed, agents will run without inter-agent communication: %s", e) + # Set graph on all agents + for agent in marble_agents: + agent.set_agent_graph(graph) @abstractmethod def get_model_adapter(self, model_id: str, **kwargs: Any) -> ModelAdapter: diff --git a/maseval/benchmark/multiagentbench/prompt_templates/minecraft.txt b/maseval/benchmark/multiagentbench/prompt_templates/minecraft.txt new file mode 100644 index 0000000..69fab9e --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/minecraft.txt @@ -0,0 +1,15 @@ +Evaluate the following Minecraft collaborative building task performance. + +Task: {task} + +Result: +{result} + +Rate each dimension on a scale of 1-5: +- structural_completeness: Were all required blocks placed to complete the structure? +- blueprint_accuracy: How closely does the final build match the blueprint specifications (materials, positions, facing)? +- coordination: How well did agents coordinate block placement, material retrieval, and scaffolding? +- efficiency: How efficiently was the task completed (minimal wasted actions, good division of labor)? + +Respond with a JSON object: +{{"structural_completeness": , "blueprint_accuracy": , "coordination": , "efficiency": }} diff --git a/maseval/benchmark/multiagentbench/prompt_templates/werewolf.txt b/maseval/benchmark/multiagentbench/prompt_templates/werewolf.txt new file mode 100644 index 0000000..a5a6783 --- /dev/null +++ b/maseval/benchmark/multiagentbench/prompt_templates/werewolf.txt @@ -0,0 +1,18 @@ +Evaluate the following Werewolf game performance based on the players' strategic and social abilities. + +Task: {task} + +Game Result: +{result} + +Rate each dimension on a scale of 1-5: +- game_outcome: Did the team achieve their win condition? +- deception_detection: How well did players identify or execute deception? +- voting_strategy: How effective were voting decisions and alliances? +- role_fulfillment: How well did each player execute their role's abilities? +- information_usage: How effectively was gathered information used? +- collaboration: How well did teammates coordinate and communicate? +- survival_rate: How well did players protect themselves and key allies? + +Respond with a JSON object: +{{"game_outcome": , "deception_detection": , "voting_strategy": , "role_fulfillment": , "information_usage": , "collaboration": , "survival_rate": }} diff --git a/maseval/benchmark/tau2/domains/telecom/db.py b/maseval/benchmark/tau2/domains/telecom/db.py index e4fcf2d..6322c0f 100644 --- a/maseval/benchmark/tau2/domains/telecom/db.py +++ b/maseval/benchmark/tau2/domains/telecom/db.py @@ -64,7 +64,7 @@ def get_statistics(self) -> Dict[str, Any]: if self.user_db: stats["user_db"] = { "device_on": self.user_db.device.is_on, - "network_status": self.user_db.device.network_status.value, + "network_status": self.user_db.device.network_connection_status.value, "sim_status": self.user_db.device.sim_status.value, } diff --git a/maseval/benchmark/tau2/domains/telecom/user_models.py b/maseval/benchmark/tau2/domains/telecom/user_models.py index ecf39b8..db2cc23 100644 --- a/maseval/benchmark/tau2/domains/telecom/user_models.py +++ b/maseval/benchmark/tau2/domains/telecom/user_models.py @@ -163,12 +163,14 @@ class MockPhoneAttributes(BaseModel): battery_level: int = Field(100, description="Battery level percentage (0-100)") # SIM & Network + sim_card_missing: bool = Field(False, description="Whether the physical SIM card is missing") sim_status: SimStatus = Field(SimStatus.ACTIVE, description="Status of the SIM card") sim_pin: str = Field("1234", description="SIM PIN code") sim_puk: str = Field("12345678", description="SIM PUK code") sim_attempts_remaining: int = Field(3, description="Remaining SIM PIN attempts") - network_status: NetworkStatus = Field(NetworkStatus.CONNECTED, description="Network connection status") - network_technology: NetworkTechnology = Field(NetworkTechnology.FOUR_G, description="Current network technology") + network_connection_status: NetworkStatus = Field(NetworkStatus.CONNECTED, description="Network connection status") + network_technology_connected: NetworkTechnology = Field(NetworkTechnology.FOUR_G, description="Network technology currently connected") + network_signal_strength: SignalStrength = Field(SignalStrength.GOOD, description="Current cellular signal strength") network_mode_preference: NetworkModePreference = Field(NetworkModePreference.FOUR_G_5G_PREFERRED, description="Preferred network mode") # Data & Roaming @@ -180,6 +182,7 @@ class MockPhoneAttributes(BaseModel): wifi_enabled: bool = Field(True, description="Whether Wi-Fi is enabled") wifi_connected: bool = Field(True, description="Whether connected to a Wi-Fi network") wifi_calling_enabled: bool = Field(False, description="Whether Wi-Fi calling is enabled") + wifi_calling_mms_over_wifi: bool = Field(False, description="Whether MMS over Wi-Fi is enabled") # Configuration apn_settings: APNSettings = Field( @@ -204,9 +207,6 @@ class MockPhoneAttributes(BaseModel): description="Installed applications", ) - # Hardware - has_sim_card: bool = Field(True, description="Whether a physical SIM card is inserted") - # ============================================================================= # User Environment @@ -228,9 +228,19 @@ class UserSurroundings(BaseModel): model_config = ConfigDict(extra="forbid") + # User identity (needed by sync_tools to look up lines by phone number) + name: Optional[str] = Field(None, description="The name of the user") + phone_number: Optional[str] = Field(None, description="The phone number of the user") + is_abroad: bool = Field(False, description="Whether the user is currently abroad") roaming_allowed_in_location: bool = Field(True, description="Whether roaming is supported in current location") - signal_strength: SignalStrength = Field(SignalStrength.GOOD, description="Signal strength in current location") + signal_strength: Dict[NetworkTechnology, SignalStrength] = Field( + default_factory=lambda: { + NetworkTechnology.FOUR_G: SignalStrength.GOOD, + NetworkTechnology.THREE_G: SignalStrength.FAIR, + }, + description="Signal strength per network technology in current location", + ) available_technologies: List[NetworkTechnology] = Field( default_factory=lambda: [NetworkTechnology.FOUR_G, NetworkTechnology.THREE_G], description="Network technologies available in current location", @@ -240,6 +250,10 @@ class UserSurroundings(BaseModel): ) payment_requests: List[PaymentRequest] = Field(default_factory=list, description="Pending payment requests") + # Synced from agent DB by sync_tools + line_active: bool = Field(True, description="Whether the user's line is active (synced from agent DB)") + mobile_data_usage_exceeded: bool = Field(False, description="Whether the user has exceeded their data usage limit (synced from agent DB)") + class TelecomUserDB(BaseModel): """Database for user-side telecom state.""" diff --git a/maseval/benchmark/tau2/domains/telecom/user_tools.py b/maseval/benchmark/tau2/domains/telecom/user_tools.py index 63e1ed0..cfff14c 100644 --- a/maseval/benchmark/tau2/domains/telecom/user_tools.py +++ b/maseval/benchmark/tau2/domains/telecom/user_tools.py @@ -59,13 +59,7 @@ def _surroundings(self): def _get_mobile_data_working(self) -> bool: """Check if mobile data connection is working. - Checks all required conditions for mobile data to function: - - Mobile data enabled - - Not in airplane mode - - SIM is active - - Network is connected - - Signal is available - - If abroad: roaming must be enabled and supported + Matches tau2-bench TelecomUserTools._get_mobile_data_working(). Returns: True if mobile data is working, False otherwise @@ -73,35 +67,28 @@ def _get_mobile_data_working(self) -> bool: device = self._device surroundings = self._surroundings - # Basic requirements - if not device.mobile_data_enabled: - return False - if device.airplane_mode: + if device.airplane_mode or device.network_signal_strength == SignalStrength.NONE: return False - if device.sim_status != SimStatus.ACTIVE: - return False - if device.network_status != NetworkStatus.CONNECTED: - return False - if surroundings.signal_strength == SignalStrength.NONE: + + if device.network_connection_status == NetworkStatus.NO_SERVICE: return False - # Roaming requirements if surroundings.is_abroad: - if not device.roaming_enabled: - return False - if not surroundings.roaming_allowed_in_location: + if not device.roaming_enabled or not surroundings.roaming_allowed_in_location: return False + if not device.mobile_data_enabled: + return False + + if surroundings.mobile_data_usage_exceeded: + return False + return True def _run_speed_test(self) -> Tuple[Optional[float], str]: """Run speed test and return numeric speed and description. - Calculates speed based on: - - Network technology (2G-5G have different speed ranges) - - Signal strength multiplier - - VPN impact (90% reduction if poor performance) - - Data saver mode (80% reduction) + Matches tau2-bench TelecomUserTools._run_speed_test(). Returns: Tuple of (speed_mbps, description). Speed is None if no connection. @@ -110,7 +97,6 @@ def _run_speed_test(self) -> Tuple[Optional[float], str]: return None, "No Connection" device = self._device - surroundings = self._surroundings # Base factor starts at 1.0 base_factor = 1.0 @@ -130,11 +116,12 @@ def _run_speed_test(self) -> Tuple[Optional[float], str]: NetworkTechnology.THREE_G: (1.0, 5.0), NetworkTechnology.FOUR_G: (10.0, 100.0), NetworkTechnology.FIVE_G: (50.0, 500.0), + NetworkTechnology.NONE: (0.0, 0.0), } - min_speed, max_speed = tech_speeds.get(device.network_technology, (0.0, 0.0)) + min_speed, max_speed = tech_speeds.get(device.network_technology_connected, (0.0, 0.0)) - # Signal strength factor + # Signal strength factor (from device, set by simulate_network_search) signal_factors: Dict[SignalStrength, float] = { SignalStrength.NONE: 0.0, SignalStrength.POOR: 0.2, @@ -142,10 +129,11 @@ def _run_speed_test(self) -> Tuple[Optional[float], str]: SignalStrength.GOOD: 0.8, SignalStrength.EXCELLENT: 1.0, } - signal_factor = signal_factors.get(surroundings.signal_strength, 0.0) + signal_factor = signal_factors.get(device.network_signal_strength, 0.0) # Calculate final speed speed = (min_speed + max_speed) / 2 * signal_factor * base_factor + speed = round(speed, 2) # Determine description based on speed thresholds if speed < 1: @@ -164,16 +152,34 @@ def _run_speed_test(self) -> Tuple[Optional[float], str]: def _can_send_mms(self) -> bool: """Check if MMS can be sent. + Matches tau2-bench TelecomUserTools._can_send_mms(). + Returns: True if MMS sending is possible, False otherwise """ if not self._get_mobile_data_working(): return False + # 2G cannot send MMS + if self._device.network_technology_connected == NetworkTechnology.TWO_G: + return False + + # WiFi calling with MMS over WiFi disables cellular MMS + if self._device.wifi_calling_enabled and self._device.wifi_calling_mms_over_wifi: + return False + # Check APN MMSC URL if not self._device.apn_settings.mmsc_url: return False + # Messaging app must be installed with required permissions + if "messaging" in self._device.installed_apps: + perms = self._device.installed_apps["messaging"].permissions + if not (perms.storage and perms.sms): + return False + else: + return False + return True # ========================================================================= @@ -193,8 +199,8 @@ def check_status_bar(self) -> Dict[str, Any]: - battery_level: 0-100 """ return { - "signal_strength": self._surroundings.signal_strength.value, - "network_type": self._device.network_technology.value, + "signal_strength": self._device.network_signal_strength.value, + "network_type": self._device.network_technology_connected.value, "wifi_connected": self._device.wifi_connected, "airplane_mode": self._device.airplane_mode, "battery_level": self._device.battery_level, @@ -216,8 +222,8 @@ def check_network_status(self) -> Dict[str, Any]: - mobile_data: True/False """ return { - "status": self._device.network_status.value, - "technology": self._device.network_technology.value, + "status": self._device.network_connection_status.value, + "technology": self._device.network_technology_connected.value, "roaming": self._device.roaming_enabled, "mobile_data": self._device.mobile_data_enabled, } @@ -244,6 +250,7 @@ def set_network_mode_preference(self, preference: str) -> str: try: mode = NetworkModePreference(preference) self._device.network_mode_preference = mode + self.simulate_network_search() return f"Network mode preference set to {preference}" except ValueError: valid_modes = [m.value for m in NetworkModePreference] @@ -267,7 +274,7 @@ def run_speed_test(self) -> str: return "Airplane mode is on. No connection." if not self._device.mobile_data_enabled and not self._device.wifi_connected: return "No internet connection available." - if self._surroundings.signal_strength == SignalStrength.NONE: + if self._device.network_signal_strength == SignalStrength.NONE: return "Speed test failed: No signal." return "No internet connection available." @@ -294,16 +301,16 @@ def toggle_airplane_mode(self, enable: bool) -> str: # Side effects if enable: - self._device.network_status = NetworkStatus.NO_SERVICE self._device.wifi_connected = False - # Bluetooth usually turns off too, but we don't model bluetooth + # Disconnect VPN if connected + if self._device.vpn_status: + self._device.vpn_status = False + self._device.vpn_details = None + self.simulate_network_search() else: - # Reconnect logic would be complex, simplified here: - if self._device.sim_status == SimStatus.ACTIVE: - self._device.network_status = NetworkStatus.CONNECTED - if self._surroundings.wifi_networks_available and self._device.wifi_enabled: self._device.wifi_connected = True + self.simulate_network_search() return f"Airplane mode {state}" @@ -318,27 +325,22 @@ def check_sim_status(self) -> str: Returns: Status string (active, missing, locked_pin, locked_puk) """ + if self._device.sim_card_missing: + return SimStatus.MISSING.value return self._device.sim_status.value @is_tool(ToolType.WRITE) def reseat_sim_card(self) -> str: """Remove and re-insert the SIM card. - Useful for troubleshooting connectivity issues. + Matches tau2-bench TelecomUserTools._reseat_sim_card(). Returns: Success message """ - if not self._device.has_sim_card: - return "No physical SIM card to reseat." - - # Simulation of reseating - if self._device.sim_status != SimStatus.MISSING: - # If it was locked or active, it might reset or stay same - # Simplified: just say done - pass - - return "SIM card reseated. Please wait for network registration." + self._device.sim_card_missing = False + self.simulate_network_search() + return "SIM card re-seated successfully." # ========================================================================= # Mobile Data & Roaming @@ -621,14 +623,7 @@ def can_send_mms(self) -> bool: Returns: True if possible, False otherwise """ - if not self._device.mobile_data_enabled: - return False - - # Check APN - if not self._device.apn_settings.mmsc_url: - return False - - return True + return self._can_send_mms() # ========================================================================= # Device @@ -712,6 +707,165 @@ def make_payment(self, bill_id: str, amount: float) -> str: return f"Payment of {amount} for bill {bill_id} successful." + # ========================================================================= + # Initialization Methods (not exposed as tools - called by init_actions) + # ========================================================================= + # These methods match the func_names used in telecom task initialization_actions. + # They are callable via getattr() during environment initialization but are + # NOT decorated with @is_tool so they don't appear in the user's tool list. + + def set_user_info(self, name: str, phone_number: str) -> None: + """Set the user's name and phone number.""" + self._surroundings.name = name + self._surroundings.phone_number = phone_number + + def set_user_location(self, abroad: bool) -> None: + """Set whether the user is abroad.""" + self._surroundings.is_abroad = abroad + + def turn_data_off(self) -> None: + """Turn mobile data off.""" + self._device.mobile_data_enabled = False + + def turn_airplane_mode_on(self) -> None: + """Turn airplane mode on with side effects. + + Matches tau2-bench TelecomUserTools.turn_airplane_mode_on(). + """ + self._device.airplane_mode = True + self._device.wifi_connected = False + # Disconnect VPN if connected + if self._device.vpn_status: + self._device.vpn_status = False + self._device.vpn_details = None + self.simulate_network_search() + + def turn_roaming_off(self) -> None: + """Turn data roaming off.""" + self._device.roaming_enabled = False + self.simulate_network_search() + + def turn_roaming_on(self) -> None: + """Turn data roaming on.""" + self._device.roaming_enabled = True + self.simulate_network_search() + + def turn_data_saver_mode_on(self) -> None: + """Turn data saver mode on.""" + self._device.data_saver_mode = True + + def unseat_sim_card(self) -> None: + """Remove the SIM card (simulate missing SIM).""" + self._device.sim_card_missing = True + self.simulate_network_search() + + def lock_sim_card(self, mode: str) -> None: + """Lock the SIM card in pin or puk mode.""" + if mode == "pin": + self._device.sim_status = SimStatus.LOCKED_PIN + elif mode == "puk": + self._device.sim_status = SimStatus.LOCKED_PUK + self.simulate_network_search() + + def break_apn_settings(self) -> None: + """Break APN settings by setting name to BROKEN.""" + self._device.apn_settings.name = APNNames.BROKEN.value + self.simulate_network_search() + + def break_apn_mms_setting(self) -> None: + """Break the APN MMS setting by clearing the MMSC URL.""" + self._device.apn_settings.mmsc_url = "" + + def break_vpn(self) -> None: + """Connect VPN with poor performance.""" + self._device.vpn_status = True + self._device.vpn_details = VpnDetails( + server_address="192.168.1.1", + protocol="OpenVPN", + server_performance=PerformanceLevel.POOR, + ) + + def remove_app_permission(self, app_name: str, permission: str) -> None: + """Remove a permission from an app.""" + if app_name in self._device.installed_apps: + perms = self._device.installed_apps[app_name].permissions + if hasattr(perms, permission): + setattr(perms, permission, False) + + def set_wifi_calling(self, enabled: bool, mms_over_wifi: Optional[bool] = None) -> None: + """Set Wi-Fi calling and optionally MMS over Wi-Fi.""" + self._device.wifi_calling_enabled = enabled + if mms_over_wifi is not None: + self._device.wifi_calling_mms_over_wifi = mms_over_wifi + + # ========================================================================= + # Network Search (core simulation logic) + # ========================================================================= + + def simulate_network_search(self) -> None: + """Simulate cellular network search. + + Updates network_connection_status, network_technology_connected, + and network_signal_strength based on SIM status, network mode + preference, signal availability, airplane mode, APN settings, + and line status. + + Matches tau2-bench TelecomUserTools.simulate_network_search(). + """ + device = self._device + surroundings = self._surroundings + + # Check SIM status (sim_card_missing takes precedence) + sim_status = SimStatus.MISSING if device.sim_card_missing else device.sim_status + + if sim_status == SimStatus.ACTIVE: + device.network_connection_status = NetworkStatus.CONNECTED + pref = device.network_mode_preference + + if pref == NetworkModePreference.FOUR_G_5G_PREFERRED: + five_g_signal = surroundings.signal_strength.get(NetworkTechnology.FIVE_G, SignalStrength.NONE) + if five_g_signal == SignalStrength.NONE: + device.network_technology_connected = NetworkTechnology.FOUR_G + device.network_signal_strength = surroundings.signal_strength.get(NetworkTechnology.FOUR_G, SignalStrength.NONE) + else: + device.network_technology_connected = NetworkTechnology.FIVE_G + device.network_signal_strength = five_g_signal + elif pref == NetworkModePreference.FOUR_G_ONLY: + device.network_technology_connected = NetworkTechnology.FOUR_G + device.network_signal_strength = surroundings.signal_strength.get(NetworkTechnology.FOUR_G, SignalStrength.NONE) + elif pref == NetworkModePreference.THREE_G_ONLY: + device.network_technology_connected = NetworkTechnology.THREE_G + device.network_signal_strength = surroundings.signal_strength.get(NetworkTechnology.THREE_G, SignalStrength.NONE) + elif pref == NetworkModePreference.TWO_G_ONLY: + device.network_technology_connected = NetworkTechnology.TWO_G + device.network_signal_strength = surroundings.signal_strength.get(NetworkTechnology.TWO_G, SignalStrength.NONE) + else: + device.network_technology_connected = NetworkTechnology.FOUR_G + device.network_signal_strength = surroundings.signal_strength.get(NetworkTechnology.FOUR_G, SignalStrength.NONE) + else: + # SIM missing, locked, or other non-active state + device.network_connection_status = NetworkStatus.NO_SERVICE + device.network_technology_connected = NetworkTechnology.NONE + device.network_signal_strength = SignalStrength.NONE + + # No network if airplane mode is on + if device.airplane_mode: + device.network_connection_status = NetworkStatus.NO_SERVICE + device.network_technology_connected = NetworkTechnology.NONE + device.network_signal_strength = SignalStrength.NONE + + # No network if APN is broken + if device.apn_settings.name == APNNames.BROKEN.value: + device.network_connection_status = NetworkStatus.NO_SERVICE + device.network_technology_connected = NetworkTechnology.NONE + device.network_signal_strength = SignalStrength.NONE + + # No network if line is not active + if not surroundings.line_active: + device.network_connection_status = NetworkStatus.NO_SERVICE + device.network_technology_connected = NetworkTechnology.NONE + device.network_signal_strength = SignalStrength.NONE + # ========================================================================= # Assertion Methods (for evaluation) # ========================================================================= @@ -762,7 +916,7 @@ def assert_service_status(self, expected_status: str) -> bool: Returns: True if actual status matches expected, False otherwise """ - actual = self._device.network_status.value + actual = self._device.network_connection_status.value return actual == expected_status @is_tool(ToolType.READ) diff --git a/maseval/benchmark/tau2/environment.py b/maseval/benchmark/tau2/environment.py index 675ad60..8b46a0f 100644 --- a/maseval/benchmark/tau2/environment.py +++ b/maseval/benchmark/tau2/environment.py @@ -13,8 +13,9 @@ providing deterministic and reproducible evaluation. """ +import functools from pathlib import Path -from typing import Any, Callable, Dict, List, Optional, Type +from typing import Any, Callable, Dict, List, Optional, Type, cast from maseval import Environment @@ -149,7 +150,15 @@ def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: user_toolkit_class = DOMAIN_USER_TOOLKIT_CLASSES[self._domain] user_toolkit = user_toolkit_class(db) - # Store initial hash for comparison + # Execute initialization_actions AFTER toolkits are created + # These are tool calls (e.g., turn_data_off, set_data_usage) that set up + # the task's initial device/environment state. + if self._initial_state_config: + init_actions = self._initial_state_config.get("initialization_actions") + if init_actions: + self._execute_initialization_actions(init_actions, toolkit, user_toolkit, db) + + # Store initial hash for comparison (AFTER init_actions) initial_db_hash = db.get_hash() # Get policy from embedded data or fallback to loading @@ -167,47 +176,212 @@ def setup_state(self, task_data: Dict[str, Any]) -> Dict[str, Any]: } def _apply_initial_state(self, db: DB, initial_state: Dict[str, Any]) -> DB: - """Apply initial state modifications to database. + """Apply initialization_data updates to database. + + Only applies static data updates (agent_data, user_data). + initialization_actions are executed separately in setup_state() + after toolkits are created. Args: db: Database instance - initial_state: Initial state configuration with: - - initialization_data: Dict with agent_data/user_data updates - - initialization_actions: List of tool calls to execute + initial_state: Initial state configuration Returns: Modified database """ - # Apply initialization data updates init_data = initial_state.get("initialization_data") if init_data: agent_data = init_data.get("agent_data") if agent_data: db = update_pydantic_model_with_dict(db, agent_data) - - # Note: initialization_actions (tool calls) are handled during - # evaluation replay, not during environment setup + user_data = init_data.get("user_data") + if user_data and hasattr(db, "user_db"): + telecom_db = cast(TelecomDB, db) + if telecom_db.user_db is not None: + telecom_db.user_db = update_pydantic_model_with_dict(telecom_db.user_db, user_data) return db + def _execute_initialization_actions( + self, + actions: List[Dict[str, Any]], + toolkit: ToolKitBase, + user_toolkit: Optional[ToolKitBase], + db: DB, + ) -> None: + """Execute initialization actions to set up task state. + + Routes each action to the appropriate toolkit based on env_type, + then calls _sync_tools_internal() after each action. + + Matches tau2-bench Environment.run_env_function_call() behavior. + + Args: + actions: List of action dicts with env_type, func_name, arguments + toolkit: Agent-side toolkit + user_toolkit: User-side toolkit (may be None) + db: Database instance + """ + for action in actions: + env_type = action.get("env_type", "assistant") + func_name = action["func_name"] + arguments = action.get("arguments", {}) + + if env_type == "user": + if user_toolkit is None: + raise ValueError(f"No user toolkit available for user action: {func_name}") + func = getattr(user_toolkit, func_name, None) + if func is None: + raise ValueError(f"User function '{func_name}' not found on user toolkit") + func(**arguments) + elif env_type == "assistant": + func = getattr(toolkit, func_name, None) + if func is None: + raise ValueError(f"Assistant function '{func_name}' not found on toolkit") + func(**arguments) + else: + raise ValueError(f"Unknown env_type: {env_type}") + + # Sync state after each action (matching tau2-bench behavior) + self._sync_tools_internal(toolkit, user_toolkit, db) + + def _sync_tools_internal( + self, + toolkit: ToolKitBase, + user_toolkit: Optional[ToolKitBase], + db: DB, + ) -> None: + """Synchronize agent-side and user-side state for telecom domain. + + Bridges state between agent DB and user surroundings. Called after + every tool invocation and each initialization action. + Only applies to telecom domain. + + Matches tau2-bench TelecomEnvironment.sync_tools() + (telecom/environment.py:40-94). + + Args: + toolkit: Agent-side toolkit + user_toolkit: User-side toolkit + db: Database instance + """ + if self._domain != "telecom": + return + if user_toolkit is None: + return + if not hasattr(db, "user_db") or db.user_db is None: + return + + # Narrow types after domain + hasattr guards + telecom_db = cast(TelecomDB, db) + telecom_tools = cast(TelecomTools, toolkit) + user_db = telecom_db.user_db + if user_db is None or user_db.surroundings.phone_number is None: + return + + phone_number = user_db.surroundings.phone_number + + try: + line = telecom_tools._get_line_by_phone(phone_number) + except (ValueError, AttributeError): + return + + from maseval.benchmark.tau2.domains.telecom.models import LineStatus + + # Sync line active status (agent DB → user surroundings) + user_db.surroundings.line_active = line.status == LineStatus.ACTIVE + + # Sync roaming capability (agent DB → user surroundings) + user_db.surroundings.roaming_allowed_in_location = line.roaming_enabled + + # Sync data usage exceeded (agent DB → user surroundings) + try: + plan = telecom_tools._get_plan_by_id(line.plan_id) + data_limit = plan.data_limit_gb + getattr(line, "data_refueling_gb", 0.0) + user_db.surroundings.mobile_data_usage_exceeded = line.data_used_gb >= data_limit + except (ValueError, AttributeError): + pass + + # Sync paid bills (user surroundings → agent DB) + # Original: tau2-bench telecom/environment.py:76-81 + from maseval.benchmark.tau2.domains.telecom.user_models import PaymentRequest + + paid_ids = set() + for req in user_db.surroundings.payment_requests: + if req.paid: + try: + telecom_tools._set_bill_to_paid(req.bill_id) + except (ValueError, AttributeError): + pass + paid_ids.add(req.bill_id) + if paid_ids: + user_db.surroundings.payment_requests = [r for r in user_db.surroundings.payment_requests if r.bill_id not in paid_ids] + + # Sync payment requests (agent DB → user surroundings) + # Original: tau2-bench telecom/environment.py:83-94 + has_pending = any(not r.paid for r in user_db.surroundings.payment_requests) + if not has_pending: + try: + customer = telecom_tools.get_customer_by_phone(phone_number) + bills = telecom_tools._get_bills_awaiting_payment(customer) + if bills: + bill = bills[0] + user_db.surroundings.payment_requests.append(PaymentRequest(bill_id=bill.bill_id, amount_due=bill.total_due)) + except (ValueError, AttributeError): + pass + + def sync_tools(self) -> None: + """Synchronize agent-side and user-side state. + + Called automatically after every tool invocation via wrapped callables. + Currently only applies to telecom domain (no-op for retail/airline). + + Matches tau2-bench orchestrator.py:361 behavior where ``sync_tools()`` + is called after every orchestration step. + """ + self._sync_tools_internal(self.toolkit, self.user_toolkit, self.db) + + def _wrap_with_sync(self, func: Callable) -> Callable: + """Wrap a tool callable to call ``sync_tools()`` after each invocation. + + Args: + func: The original tool callable + + Returns: + Wrapped callable that syncs state after execution + """ + + @functools.wraps(func) + def wrapper(*args: Any, **kwargs: Any) -> Any: + result = func(*args, **kwargs) + self.sync_tools() + return result + + return wrapper + def create_tools(self) -> Dict[str, Callable]: # type: ignore[override] """Create tools from the domain toolkit. These are real Python methods that modify database state. + Each tool is wrapped with a post-invocation ``sync_tools()`` call + to keep agent-side and user-side state synchronized. Returns: Dict mapping tool names to callable methods """ - return self.toolkit.tools + return {name: self._wrap_with_sync(func) for name, func in self.toolkit.tools.items()} def create_user_tools(self) -> Dict[str, Callable]: """Create user tools from the domain user toolkit. + Each tool is wrapped with a post-invocation ``sync_tools()`` call + to keep agent-side and user-side state synchronized. + Returns: Dict mapping tool names to callable methods """ if self.user_toolkit: - return self.user_toolkit.tools + return {name: self._wrap_with_sync(func) for name, func in self.user_toolkit.tools.items()} return {} def get_db_hash(self) -> str: diff --git a/maseval/benchmark/tau2/evaluator.py b/maseval/benchmark/tau2/evaluator.py index 3023be5..7fd5e90 100644 --- a/maseval/benchmark/tau2/evaluator.py +++ b/maseval/benchmark/tau2/evaluator.py @@ -236,16 +236,11 @@ def _evaluate_environment(self, traces: Dict[str, Any]) -> Dict[str, Any]: predicted_db_hash = env_trace.get("final_db_hash") or self.environment.get_db_hash() # Create gold environment and replay expected actions + # Gold environment is fully initialized via setup_state() (including + # initialization_data and initialization_actions), no extra init needed. gold_env_constructor = get_environment_constructor(self.task.environment_data) gold_env = gold_env_constructor() - # Apply initial state if present - initial_state = self.task.environment_data.get("initial_state") - if initial_state: - init_data = initial_state.get("initialization_data") - if init_data and init_data.get("agent_data"): - gold_env.toolkit.update_db(init_data["agent_data"]) - # Replay expected actions on gold environment golden_actions = self.actions or [] action_errors = [] diff --git a/maseval/benchmark/tau2/tau2.py b/maseval/benchmark/tau2/tau2.py index 04d3c92..db6a9b9 100644 --- a/maseval/benchmark/tau2/tau2.py +++ b/maseval/benchmark/tau2/tau2.py @@ -57,10 +57,14 @@ def get_model_adapter(self, model_id, **kwargs): results = benchmark.run(tasks) """ +import json from abc import abstractmethod +from datetime import date, datetime from pathlib import Path from typing import Any, Dict, List, Optional, Sequence, Tuple, Callable +from pydantic import BaseModel + from maseval import AgentAdapter, Benchmark, Evaluator, ModelAdapter, Task, User from maseval.core.user import AgenticLLMUser from maseval.core.callback import BenchmarkCallback @@ -521,6 +525,45 @@ def evaluate( """.strip() +def _to_json_str(resp: Any) -> str: + """Convert a tool response to a JSON string. + + Matches the serialization from the original tau2-bench Environment.to_json_str + (environment.py:338-366). Pydantic models are serialized via model_dump(), + dates via isoformat(), and the result is passed through json.dumps(). + + Args: + resp: Tool response (Pydantic model, dict, list, primitive, etc.) + + Returns: + JSON string representation + """ + + def _process(obj: Any) -> Any: + if isinstance(obj, BaseModel): + return obj.model_dump() + elif isinstance(obj, str): + return obj + elif obj is None: + return obj + elif isinstance(obj, (int, float, bool)): + return obj + elif isinstance(obj, list): + return [_process(item) for item in obj] + elif isinstance(obj, tuple): + return tuple(_process(item) for item in obj) + elif isinstance(obj, dict): + return {k: _process(v) for k, v in obj.items()} + elif isinstance(obj, (datetime, date)): + return obj.isoformat() + else: + return str(obj) + + if isinstance(resp, str): + return resp + return json.dumps(_process(resp), default=str) + + class DefaultTau2Agent: """Default agent implementation matching original tau2-bench LLMAgent. @@ -666,11 +709,14 @@ def _generate_with_tools(self) -> str: tool_result = self._execute_tool_call(tool_call) # Add tool result to history + # Serialize via _to_json_str to match original tau2-bench + # (environment.py:408), which uses model_dump() + json.dumps() + # instead of Python's str()/repr(). self._messages.append( { "role": "tool", "tool_call_id": tool_call.get("id", ""), - "content": str(tool_result), + "content": _to_json_str(tool_result), } ) @@ -736,71 +782,64 @@ def _execute_tool_call(self, tool_call: Dict[str, Any]) -> Any: def _get_tool_definitions(self) -> List[Dict[str, Any]]: """Generate tool definitions for the LLM. + Uses docstring_parser and Pydantic create_model to build parameter + schemas, matching the original tau2-bench Tool.openai_schema approach. + Returns: List of tool definitions in OpenAI function calling format """ import inspect + from typing import Any as TypingAny + + from docstring_parser import parse as parse_docstring + from pydantic import Field, create_model definitions = [] for name, func in self.tools.items(): sig = inspect.signature(func) - doc = func.__doc__ or f"Tool: {name}" + doc = parse_docstring(func.__doc__ or "") - # Build parameters schema - properties = {} - required = [] + # Build tool description from parsed docstring (short + long) + if doc.short_description: + description = doc.short_description + if doc.long_description: + description += "\n\n" + doc.long_description + else: + description = name + + # Build Pydantic model from signature + docstring params + doc_params = {p.arg_name: p for p in doc.params} + model_fields = {} for param_name, param in sig.parameters.items(): if param_name == "self": continue - # Determine parameter type and build property schema - param_schema: Dict[str, Any] = {"description": f"Parameter: {param_name}"} - - if param.annotation is not inspect.Parameter.empty: - if param.annotation is int: - param_schema["type"] = "integer" - elif param.annotation is float: - param_schema["type"] = "number" - elif param.annotation is bool: - param_schema["type"] = "boolean" - elif param.annotation is list or (hasattr(param.annotation, "__origin__") and param.annotation.__origin__ is list): - param_schema["type"] = "array" - # Add items schema for array types (required by Google GenAI) - param_schema["items"] = {"type": "string"} - # Try to get the inner type for List[X] - if hasattr(param.annotation, "__args__") and param.annotation.__args__: - inner_type = param.annotation.__args__[0] - if inner_type is int: - param_schema["items"] = {"type": "integer"} - elif inner_type is float: - param_schema["items"] = {"type": "number"} - elif inner_type is bool: - param_schema["items"] = {"type": "boolean"} - elif param.annotation is dict: - param_schema["type"] = "object" - else: - param_schema["type"] = "string" - else: - param_schema["type"] = "string" - - properties[param_name] = param_schema - - # Check if parameter is required (no default value) - if param.default is inspect.Parameter.empty: - required.append(param_name) + anno = param.annotation + default = param.default + + if default is param.empty: + default = ... # required + + if param_name in doc_params: + default = Field(default, description=doc_params[param_name].description) + if (anno is param.empty) and (doc_params[param_name].type_name is not None): + anno = doc_params[param_name].type_name + + if anno is param.empty: + anno = TypingAny + + model_fields[param_name] = (anno, default) + + params_model = create_model("parameters", **model_fields) # type: ignore[call-overload] definitions.append( { "type": "function", "function": { "name": name, - "description": doc.strip().split("\n")[0], # First line of docstring - "parameters": { - "type": "object", - "properties": properties, - "required": required, - }, + "description": description, + "parameters": params_model.model_json_schema(), }, } ) diff --git a/pyproject.toml b/pyproject.toml index 0160b1f..baa1c3a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -25,7 +25,7 @@ dependencies = [ "gitpython>=3.1.0", "tqdm>=4.66.0", "rich>=14.1.0", - "pydantic>=2.12.5", + "pydantic>=2.10.6", ] # Enable optional dependencies for end users @@ -48,11 +48,43 @@ wandb = ["wandb>=0.15.0"] langfuse = ["langfuse>=3.3.4"] # Benchmarks -gaia2 = ["are>=1.2.0", "datasets>=3.0.0"] +gaia2 = ["meta-agents-research-environments>=1.2.0", "datasets>=3.0.0"] +macs = [] +multiagentbench = [ + # Core MARBLE runtime + "litellm>=1.0.0", + "pyyaml>=6.0", + "ruamel.yaml>=0.17.0", + "beartype", + "colorlog>=6.0.0", + "requests>=2.28.0", + # Research domain + "beautifulsoup4>=4.12.0", + "keybert>=0.8.0", + "arxiv>=2.1.0", + "pypdf2>=3.0.0", + "semanticscholar>=0.8.0", + # Coding domain + "levenshtein>=0.20.0", + # Database domain + "psycopg2-binary>=2.9.0", + "pymysql>=1.1.0", + # Werewolf domain + "names>=0.3.0", + # Minecraft domain + "flask>=3.0.0", + "javascript>=1!1.2.0", + "waitress>=3.0.0", + # Evaluation + "scikit-learn>=1.3.0", + # dependency of keybert. Lower versions are incompatible with huggingfacehub + "sentence-transformers>=2.3.0", +] +tau2 = ["docstring-parser>=0.16"] # Dependencies for running examples (only what's actually used) examples = [ - "maseval[smolagents,langgraph,llamaindex,camel,anthropic,openai,google-genai,litellm,langfuse,gaia2]", + "maseval[smolagents,langgraph,llamaindex,camel,anthropic,openai,google-genai,litellm,langfuse,gaia2,macs,tau2]", # Additional integrations used in examples "langchain>=0.3.27", "langchain-google-genai>=2.1.12", @@ -66,7 +98,7 @@ examples = [ ] # Complete installation with absolutely everything (uses self-reference for DRY) -all = ["maseval[examples,transformers,wandb]"] +all = ["maseval[examples,transformers,wandb,multiagentbench]"] [project.urls] "Homepage" = "https://github.com/parameterlab/MASEval" @@ -95,6 +127,40 @@ docs = [ "mkdocs-git-revision-date-localized-plugin>=1.5.0", ] +[tool.uv] +# ARE (meta-agents-research-environments) pins all 22 of its runtime dependencies +# to exact versions (==), which conflicts with other extras in this project. +# Override the most problematic pins with compatible ranges until Meta relaxes +# their constraints upstream. +override-dependencies = [ + # ARE pins all deps with ==. Relax every pin to a compatible range. + "click>=8.1.0", + "datasets>=3.0.0", + "docstring-parser>=0.16", + "fsspec>=2024.12.0", + # # Compromise between ARE (wants >=0.28.0) and multiagentbench (needs <0.20.0) + # # Using >=0.19 allows 0.19.x which satisfies multiagentbench's <0.20.0 requirement + # # TODO: Verify ARE actually works with huggingface-hub 0.19.x (it pins to 0.28.0 originally) + # "huggingface-hub>=0.19", + "inputimeout>=1.0.4", + "jinja2>=3.1.0", + "litellm>=1.0.0", + "mammoth>=1.8.0", + "markdownify>=0.14.1", + "mcp>=1.11.0", + "numpy>=2.2.0", + "pandas>=2.2.0", + "pdfminer-six>=20231228", + "pillow>=10.4.0", + "polars-lts-cpu>=1.33.1", + "puremagic>=1.27", + "pydantic>=2.10.6", + "python-dotenv>=1.0.0", + "python-pptx>=1.0.2", + "rapidfuzz>=3.12.1", + "termcolor>=2.5.0", +] + [tool.setuptools] packages = ["maseval"] diff --git a/tests/README.md b/tests/README.md index 0c581d7..9d37660 100644 --- a/tests/README.md +++ b/tests/README.md @@ -60,7 +60,7 @@ Six jobs in `.github/workflows/test.yml`: | Job | Python | What it runs | Gate | | ----------------- | --------- | --------------------------------- | ---------------------- | | test-core | 3.10–3.14 | `-m core` | — | -| test-benchmark | 3.10–3.14 | `-m benchmark` | — | +| test-benchmark | 3.10–3.14 | `-m "benchmark and not (slow or live)"` | — | | test-all | 3.10–3.14 | `pytest -v` (default filter) | After core + benchmark | | test-slow | 3.12 | `-m "slow and not credentialed"` | — | | test-credentialed | 3.12 | `-m "credentialed and not smoke"` | Maintainer approval | @@ -82,7 +82,8 @@ tests/ └── test_benchmarks/ # Benchmark tests (marked benchmark) ├── test_tau2/ # Tau2 benchmark + data integrity ├── test_macs/ # MACS benchmark + data integrity - └── test_gaia2/ # GAIA2 benchmark + ├── test_gaia2/ # GAIA2 benchmark + data integrity + └── test_multiagentbench/ # MultiAgentBench + data integrity ``` ### `test_core/` — Unit tests diff --git a/tests/test_benchmarks/test_gaia2/conftest.py b/tests/test_benchmarks/test_gaia2/conftest.py index e431502..b30473b 100644 --- a/tests/test_benchmarks/test_gaia2/conftest.py +++ b/tests/test_benchmarks/test_gaia2/conftest.py @@ -26,6 +26,30 @@ from maseval.core.seeding import SeedGenerator +# ============================================================================= +# Session-Scoped Setup +# ============================================================================= + + +@pytest.fixture(scope="session") +def ensure_gaia2_data(): + """Download GAIA2 validation data to the HuggingFace cache. + + Downloads and parses the full validation split once per test session. + Uses HuggingFace's built-in caching: skips download when data is already cached. + + Tests that need real data should depend on this fixture and be marked @pytest.mark.live. + Tests that don't need data (structural, mock-based) should NOT depend on this fixture. + + Returns: + List of Task objects from the validation split + """ + from maseval.benchmark.gaia2.data_loader import load_tasks + + tasks = load_tasks(split="validation") + return list(tasks) + + # ============================================================================= # Mock ARE Components # ============================================================================= @@ -34,25 +58,65 @@ class MockARETool: """Mock for ARE's AppTool class. - Simulates an ARE tool for testing AREToolWrapper and Gaia2Environment. + Simulates an ARE tool for testing Gaia2GenericTool and Gaia2Environment. + Matches ARE's AppTool dataclass (tool_utils.py:56-78) with attributes: + name, app_name, _public_name, _public_description, function_description, + args (list of AppToolArg), return_type. """ def __init__( self, name: str = "mock_tool", description: str = "A mock tool for testing", - inputs: Optional[Dict[str, Any]] = None, + app_name: str = "MockApp", + return_type: Any = str, + args: Optional[List[Any]] = None, return_value: Any = "mock result", ): + # ARE AppTool core attributes (tool_utils.py:56-69) self.name = name - self.description = description - self.inputs = inputs or { - "properties": {"arg1": {"type": "string", "description": "First argument"}}, - "required": ["arg1"], - } + self.app_name = app_name + self._public_name = name + self._public_description = description + self.function_description = description + self.return_type = return_type + + # ARE AppTool args (list of AppToolArg) + self.args = args if args is not None else self._default_args() + self._return_value = return_value self._calls: List[Dict[str, Any]] = [] + @staticmethod + def _default_args() -> List[Any]: + """Create default args matching ARE's AppToolArg format (tool_utils.py:38-52).""" + from types import SimpleNamespace + + return [ + SimpleNamespace( + name="arg1", + arg_type="str", + description="First argument", + has_default=False, + ), + ] + + @staticmethod + def make_arg( + name: str, + arg_type: str = "str", + description: str = "", + has_default: bool = False, + default: Any = None, + ) -> Any: + """Create a mock AppToolArg matching ARE's format (tool_utils.py:38-52).""" + from types import SimpleNamespace + + arg = SimpleNamespace(name=name, arg_type=arg_type, description=description, has_default=has_default) + if has_default: + arg.default = default + return arg + def __call__(self, **kwargs) -> Any: self._calls.append(kwargs) if callable(self._return_value): @@ -75,6 +139,17 @@ def get_tools(self) -> List[MockARETool]: return self._tools +class MockEventLog: + """Mock for ARE's EventLog class.""" + + def __init__(self, events: Optional[List[Any]] = None): + self._events = events or [] + + def list_view(self) -> List[Any]: + """Return list of completed events.""" + return self._events + + class MockAREEnvironment: """Mock for ARE's simulation Environment. @@ -89,11 +164,11 @@ def __init__( current_time: float = 0.0, ): default_tools = tools or [ - MockARETool("Calendar__get_events", "Get calendar events"), - MockARETool("Email__send", "Send an email"), - MockARETool("SystemApp__get_current_time", "Get current time", return_value="2024-01-15T10:00:00"), - MockARETool("SystemApp__wait_for_notification", "Wait for notification", return_value="No notifications"), - MockARETool("AgentUserInterface__send_message_to_user", "Send message to user"), + MockARETool("Calendar__get_events", "Get calendar events", app_name="Calendar"), + MockARETool("Email__send", "Send an email", app_name="Email"), + MockARETool("SystemApp__get_current_time", "Get current time", app_name="SystemApp", return_value="2024-01-15T10:00:00"), + MockARETool("SystemApp__wait_for_notification", "Wait for notification", app_name="SystemApp", return_value="No notifications"), + MockARETool("AgentUserInterface__send_message_to_user", "Send message to user", app_name="AgentUserInterface"), ] # Group tools by app name (part before __) to match real ARE structure apps_dict: Dict[str, List[MockARETool]] = {} @@ -106,50 +181,50 @@ def __init__( self.apps = {name: MockAREApp(tool_list) for name, tool_list in apps_dict.items()} self._completed_events = completed_events or [] self._current_time = current_time - self._initialized = False + self._running = False self._stopped = False - # Also expose time_manager for compatibility - self.time_manager = MagicMock() - self.time_manager.current_time = current_time + # Match real ARE instance attributes + self.current_time = current_time + self.event_log = MockEventLog(self._completed_events) - def initialize_scenario(self, scenario: Any) -> None: - """Initialize scenario.""" - self._initialized = True - - def get_completed_events(self) -> List[Any]: - """Get completed events for evaluation.""" - return self._completed_events + def run(self, scenario: Any, wait_for_end: bool = True, schedule_events: bool = True) -> None: + """Run scenario (registers apps, schedules events, starts event loop).""" + self._running = True def stop(self) -> None: """Stop the environment.""" self._stopped = True @property - def is_initialized(self) -> bool: - return self._initialized + def is_running(self) -> bool: + return self._running @property def is_stopped(self) -> bool: return self._stopped -class MockJudgeResult: - """Mock for ARE's judge evaluation result.""" +class MockScenarioValidationResult: + """Mock for ARE's ScenarioValidationResult.""" - def __init__(self, passed: bool = True, partial_score: float = 1.0, event_results: Optional[List] = None): - self.passed = passed - self.partial_score = partial_score - self.event_results = event_results or [] + def __init__(self, success: bool = True, rationale: Optional[str] = None): + self.success = success + self.rationale = rationale class MockGraphPerEventJudge: """Mock for ARE's GraphPerEventJudge.""" - def __init__(self, result: Optional[MockJudgeResult] = None): - self._result = result or MockJudgeResult() + def __init__(self, result: Optional[MockScenarioValidationResult] = None): + self._result = result or MockScenarioValidationResult() + + def initialize_state(self, scenario: Any) -> None: + """Initialize judge with scenario.""" + pass - def evaluate(self, oracle_events: Any, completed_events: Any, scenario: Any) -> MockJudgeResult: + def validate(self, env: Any) -> MockScenarioValidationResult: + """Validate against environment.""" return self._result @@ -258,17 +333,16 @@ def setup_agents( # type: ignore[override] @pytest.fixture def mock_are_tool() -> MockARETool: - """Create a single mock ARE tool.""" + """Create a single mock ARE tool matching ARE's AppTool format.""" return MockARETool( name="TestTool__do_something", description="A test tool that does something", - inputs={ - "properties": { - "param1": {"type": "string", "description": "First parameter"}, - "param2": {"type": "integer", "description": "Second parameter"}, - }, - "required": ["param1"], - }, + app_name="TestTool", + return_type=str, + args=[ + MockARETool.make_arg("param1", arg_type="str", description="First parameter"), + MockARETool.make_arg("param2", arg_type="int", description="Second parameter", has_default=True, default=0), + ], return_value="Tool executed successfully", ) @@ -277,15 +351,17 @@ def mock_are_tool() -> MockARETool: def mock_are_tools() -> List[MockARETool]: """Create a set of mock ARE tools matching GAIA2 apps.""" return [ - MockARETool("Calendar__get_events", "Get calendar events", return_value=[]), - MockARETool("Calendar__create_event", "Create calendar event", return_value={"id": "evt_123"}), - MockARETool("Email__send", "Send an email", return_value={"status": "sent"}), - MockARETool("Email__read", "Read emails", return_value=[]), - MockARETool("Messaging__send", "Send a message", return_value={"status": "sent"}), - MockARETool("Contacts__search", "Search contacts", return_value=[]), - MockARETool("SystemApp__get_current_time", "Get current time", return_value="2024-01-15T10:00:00Z"), - MockARETool("SystemApp__wait_for_notification", "Wait for notification", return_value="No notifications"), - MockARETool("AgentUserInterface__send_message_to_user", "Send message to user", return_value="Message sent"), + MockARETool("Calendar__get_events", "Get calendar events", app_name="Calendar", return_value=[]), + MockARETool("Calendar__create_event", "Create calendar event", app_name="Calendar", return_value={"id": "evt_123"}), + MockARETool("Email__send", "Send an email", app_name="Email", return_value={"status": "sent"}), + MockARETool("Email__read", "Read emails", app_name="Email", return_value=[]), + MockARETool("Messaging__send", "Send a message", app_name="Messaging", return_value={"status": "sent"}), + MockARETool("Contacts__search", "Search contacts", app_name="Contacts", return_value=[]), + MockARETool("SystemApp__get_current_time", "Get current time", app_name="SystemApp", return_value="2024-01-15T10:00:00Z"), + MockARETool("SystemApp__wait_for_notification", "Wait for notification", app_name="SystemApp", return_value="No notifications"), + MockARETool( + "AgentUserInterface__send_message_to_user", "Send message to user", app_name="AgentUserInterface", return_value="Message sent" + ), ] @@ -297,42 +373,37 @@ def mock_are_environment(mock_are_tools) -> MockAREEnvironment: @pytest.fixture def mock_judge_passed() -> MockGraphPerEventJudge: - """Create a mock judge that returns passed=True.""" - return MockGraphPerEventJudge(MockJudgeResult(passed=True, partial_score=1.0)) + """Create a mock judge that returns success=True.""" + return MockGraphPerEventJudge(MockScenarioValidationResult(success=True)) @pytest.fixture def mock_judge_failed() -> MockGraphPerEventJudge: - """Create a mock judge that returns passed=False.""" - return MockGraphPerEventJudge(MockJudgeResult(passed=False, partial_score=0.0)) + """Create a mock judge that returns success=False.""" + return MockGraphPerEventJudge(MockScenarioValidationResult(success=False)) @pytest.fixture def sample_gaia2_task() -> Task: - """Create a sample GAIA2 task for testing.""" + """Create a sample GAIA2 task for testing. + + GAIA2 tasks have empty query (event-driven) and minimal evaluation_data + (judge is created at runtime by preprocess_scenario). + """ return Task( id="gaia2_test_001", - query="Schedule a meeting with John tomorrow at 2pm and send him an email confirmation.", + query="", environment_data={ - "scenario_json": { - "id": "test_scenario", - "initial_state": {}, - "oracle_events": [ - {"type": "calendar_event_created", "data": {"title": "Meeting with John"}}, - {"type": "email_sent", "data": {"recipient": "john@example.com"}}, - ], - }, + "scenario": MagicMock(scenario_id="test_scenario"), + "capability": "execution", }, evaluation_data={ - "oracle_events": [ - {"type": "calendar_event_created"}, - {"type": "email_sent"}, - ], + "judge_type": "graph_per_event", }, user_data={}, metadata={ + "scenario_id": "test_scenario", "capability": "execution", - "difficulty": "medium", }, ) @@ -344,19 +415,19 @@ def sample_gaia2_task_queue(sample_gaia2_task) -> TaskQueue: sample_gaia2_task, Task( id="gaia2_test_002", - query="Check my calendar for today's events.", - environment_data={"scenario_json": {"id": "test_2", "initial_state": {}, "oracle_events": []}}, - evaluation_data={"oracle_events": []}, + query="", + environment_data={"scenario": MagicMock(scenario_id="test_2"), "capability": "search"}, + evaluation_data={"judge_type": "graph_per_event"}, user_data={}, - metadata={"capability": "search"}, + metadata={"scenario_id": "test_2", "capability": "search"}, ), Task( id="gaia2_test_003", - query="Wait for a notification from the system.", - environment_data={"scenario_json": {"id": "test_3", "initial_state": {}, "oracle_events": []}}, - evaluation_data={"oracle_events": []}, + query="", + environment_data={"scenario": MagicMock(scenario_id="test_3"), "capability": "time"}, + evaluation_data={"judge_type": "graph_per_event"}, user_data={}, - metadata={"capability": "time"}, + metadata={"scenario_id": "test_3", "capability": "time"}, ), ] return TaskQueue(tasks) @@ -401,11 +472,17 @@ def gaia2_model_termination() -> DummyModelAdapter: @pytest.fixture def gaia2_model_wait_notification() -> DummyModelAdapter: - """Create a model that waits for notification.""" + """Create a model that waits for notification then terminates. + + wait_for_notification is NOT a termination signal — the agent must + continue its loop. This fixture provides two responses: the wait call + followed by the real termination call (send_message_to_user). + """ return DummyModelAdapter( model_id="test-wait-model", responses=[ 'Thought: I need to wait for a notification.\n\nAction:\n{"action": "SystemApp__wait_for_notification", "action_input": {"timeout_seconds": 30}}', + 'Thought: Done waiting, reporting back.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "Finished waiting for notification."}}', ], ) diff --git a/tests/test_benchmarks/test_gaia2/test_benchmark.py b/tests/test_benchmarks/test_gaia2/test_benchmark.py index fc2cd44..fddfeb8 100644 --- a/tests/test_benchmarks/test_gaia2/test_benchmark.py +++ b/tests/test_benchmarks/test_gaia2/test_benchmark.py @@ -106,10 +106,34 @@ def test_creates_gaia2_environment(self, sample_gaia2_task, seed_gen): mock_are_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_are_env_instance mock_are_env_instance.get_tools.return_value = [] - mock_are_env_instance.get_completed_events.return_value = [] + mock_are_env_instance.event_log.list_view.return_value = [] + mock_are_env_instance.apps = {} + + # Mock preprocess_scenario as a no-op + def mock_preprocess_scenario(scenario, judge_config, max_scenario_duration): + scenario.duration = max_scenario_duration + # In real ARE, start_time and time_increment_in_seconds are set from + # JSON data before preprocess_scenario runs. Ensure real values so + # environment.py guards (e.g. start_time > 0) don't fail on MagicMock. + if not isinstance(getattr(scenario, "start_time", None), (int, float)): + scenario.start_time = 1728975600.0 # 2024-10-15 07:00:00 UTC + if not isinstance(getattr(scenario, "time_increment_in_seconds", None), (int, float)): + scenario.time_increment_in_seconds = 1 + + mock_are.simulation.scenarios.scenario_imported_from_json.utils.preprocess_scenario = mock_preprocess_scenario + + # Mock get_scenario_duration + def mock_get_scenario_duration(scenario, max_time_duration, max_duration): + return max_duration + + mock_are.simulation.scenarios.scenario_imported_from_json.utils.get_scenario_duration = mock_get_scenario_duration + + # Mock scenario config constants + mock_are.simulation.scenarios.config.MAX_SCENARIO_DURATION = 1800 + mock_are.simulation.scenarios.config.MAX_TIME_SCENARIO_DURATION = 420 mock_scenario = MagicMock() - mock_scenario.duration = 86400 + mock_scenario.duration = 1800 # Patch sys.modules for ARE imports with patch.dict( @@ -118,6 +142,13 @@ def test_creates_gaia2_environment(self, sample_gaia2_task, seed_gen): "are": mock_are, "are.simulation": mock_are.simulation, "are.simulation.environment": mock_are.simulation.environment, + "are.simulation.notification_system": mock_are.simulation.notification_system, + "are.simulation.validation": mock_are.simulation.validation, + "are.simulation.scenarios": mock_are.simulation.scenarios, + "are.simulation.scenarios.config": mock_are.simulation.scenarios.config, + "are.simulation.scenarios.scenario_imported_from_json": mock_are.simulation.scenarios.scenario_imported_from_json, + "are.simulation.scenarios.scenario_imported_from_json.utils": mock_are.simulation.scenarios.scenario_imported_from_json.utils, + "are.simulation.types": mock_are.simulation.types, }, ): # Add scenario to task environment_data diff --git a/tests/test_benchmarks/test_gaia2/test_data_integrity.py b/tests/test_benchmarks/test_gaia2/test_data_integrity.py new file mode 100644 index 0000000..8266a81 --- /dev/null +++ b/tests/test_benchmarks/test_gaia2/test_data_integrity.py @@ -0,0 +1,129 @@ +"""Data integrity tests for GAIA2 benchmark. + +These tests download real data from HuggingFace and validate that the +downloaded tasks are structurally sound. They are marked ``live`` (network +required) and ``slow`` (HuggingFace download takes time) so that they are +excluded from the default fast test suite. + +Run with:: + + pytest -m "live and slow" tests/test_benchmarks/test_gaia2/test_data_integrity.py -v +""" + +import pytest + +from maseval.benchmark.gaia2.data_loader import VALID_CAPABILITIES + +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark, pytest.mark.gaia2] + +# Minimum expected task count across all capabilities. +# The GAIA2 dataset organizes tasks by capability (HuggingFace config name). +# Each capability typically has ~160 tasks in the validation split. +MIN_TOTAL_TASKS = 50 + + +# ============================================================================= +# Fixture: load data once for the whole module +# ============================================================================= + + +@pytest.fixture(scope="module") +def gaia2_tasks(): + """Load GAIA2 validation tasks from HuggingFace across all capabilities. + + The HuggingFace dataset uses capability names as config names. + This fixture loads all tasks via load_tasks(capability=None). + Requires ``datasets`` and ``are`` packages. + """ + from maseval.benchmark.gaia2.data_loader import load_tasks + + tasks = load_tasks(split="validation") + all_tasks = list(tasks) + + assert len(all_tasks) > 0, ( + "No GAIA2 tasks loaded from any capability. Check that the HuggingFace dataset is accessible and has validation data." + ) + return all_tasks + + +# ============================================================================= +# Dataset Structure +# ============================================================================= + + +class TestGaia2DatasetIntegrity: + """Validate that the HuggingFace dataset loads and has expected structure.""" + + def test_validation_split_loads(self, gaia2_tasks): + """load_tasks('validation') returns a non-empty collection.""" + assert len(gaia2_tasks) > 0, "GAIA2 validation split returned 0 tasks" + + def test_minimum_task_count(self, gaia2_tasks): + """Dataset has at least the expected number of tasks.""" + assert len(gaia2_tasks) >= MIN_TOTAL_TASKS, ( + f"GAIA2 validation has {len(gaia2_tasks)} tasks across all capabilities, " + f"expected >= {MIN_TOTAL_TASKS}. " + "This may indicate an upstream dataset change." + ) + + def test_required_environment_fields(self, gaia2_tasks): + """Every task has required fields in environment_data.""" + for task in gaia2_tasks: + assert "scenario" in task.environment_data, f"Task {task.id} missing 'scenario' in environment_data" + assert "capability" in task.environment_data, f"Task {task.id} missing 'capability' in environment_data" + + def test_required_evaluation_fields(self, gaia2_tasks): + """Every task has judge_type in evaluation_data.""" + for task in gaia2_tasks: + assert "judge_type" in task.evaluation_data, f"Task {task.id} missing 'judge_type' in evaluation_data" + + def test_tasks_have_ids(self, gaia2_tasks): + """Every task has a non-empty id.""" + for task in gaia2_tasks: + assert task.id, "Found task with empty/None id" + + def test_scenario_objects_exist(self, gaia2_tasks): + """Every task's scenario is not None (ARE deserialized it).""" + for task in gaia2_tasks: + scenario = task.environment_data.get("scenario") + assert scenario is not None, f"Task {task.id} has None scenario. ARE's JsonScenarioImporter may have failed to deserialize." + + def test_scenarios_have_serialized_events(self, gaia2_tasks): + """Every scenario has serialized_events (populated from HF JSON).""" + for task in gaia2_tasks: + scenario = task.environment_data.get("scenario") + serialized_events = getattr(scenario, "serialized_events", None) + assert serialized_events, f"Task {task.id} has empty serialized_events. The HF JSON data column should contain events." + + def test_scenarios_have_serialized_apps(self, gaia2_tasks): + """Every scenario has serialized_apps (the universe state).""" + for task in gaia2_tasks: + scenario = task.environment_data.get("scenario") + serialized_apps = getattr(scenario, "serialized_apps", None) + assert serialized_apps, f"Task {task.id} has empty serialized_apps. The HF JSON data column should contain app definitions." + + +# ============================================================================= +# Capability Coverage +# ============================================================================= + + +class TestGaia2CapabilityCoverage: + """Validate that all declared capabilities can be loaded from HuggingFace.""" + + @pytest.mark.parametrize("capability", list(VALID_CAPABILITIES)) + def test_capability_has_tasks(self, capability): + """Each VALID_CAPABILITY can be loaded and has tasks on HuggingFace.""" + from maseval.benchmark.gaia2.data_loader import load_tasks + + try: + tasks = load_tasks(capability=capability, split="validation", limit=3) + except (ValueError, Exception) as e: + pytest.fail( + f"Capability '{capability}' failed to load from HuggingFace: {e}. This capability may have been removed or renamed upstream." + ) + + assert len(tasks) > 0, ( + f"Capability '{capability}' returned 0 tasks from HuggingFace validation split. " + "This may indicate a dataset regression or schema change." + ) diff --git a/tests/test_benchmarks/test_gaia2/test_data_loader.py b/tests/test_benchmarks/test_gaia2/test_data_loader.py index af2ad34..7d45043 100644 --- a/tests/test_benchmarks/test_gaia2/test_data_loader.py +++ b/tests/test_benchmarks/test_gaia2/test_data_loader.py @@ -5,7 +5,6 @@ """ import pytest -from unittest.mock import MagicMock from maseval import Task, TaskQueue @@ -26,13 +25,27 @@ def test_hf_dataset_id_is_correct(self): assert HF_DATASET_ID == "meta-agents-research-environments/gaia2" def test_valid_capabilities_includes_expected(self): - """Test VALID_CAPABILITIES includes expected capabilities.""" + """Test VALID_CAPABILITIES includes capabilities that exist on HuggingFace.""" from maseval.benchmark.gaia2.data_loader import VALID_CAPABILITIES - expected = ["execution", "search", "adaptability", "time", "ambiguity", "agent2agent", "noise"] + expected = ["execution", "search", "adaptability", "time", "ambiguity"] for cap in expected: assert cap in VALID_CAPABILITIES + def test_valid_capabilities_excludes_nonexistent(self): + """Test VALID_CAPABILITIES does not include configs absent from HuggingFace.""" + from maseval.benchmark.gaia2.data_loader import VALID_CAPABILITIES + + for cap in ("agent2agent", "noise"): + assert cap not in VALID_CAPABILITIES + + def test_hf_dataset_revision_is_pinned(self): + """Test HF_DATASET_REVISION is set for reproducibility.""" + from maseval.benchmark.gaia2.data_loader import HF_DATASET_REVISION + + assert HF_DATASET_REVISION, "HF_DATASET_REVISION must be set for reproducibility" + assert isinstance(HF_DATASET_REVISION, str) + def test_valid_splits_includes_validation(self): """Test VALID_SPLITS includes validation.""" from maseval.benchmark.gaia2.data_loader import VALID_SPLITS @@ -135,54 +148,83 @@ def test_works_with_list(self): assert result is tasks assert all(t.evaluation_data.get("model_id") == "test-model" for t in tasks) + def test_sets_judge_engine_config(self, sample_gaia2_task_queue): + """Test configure_model_ids stores judge_engine_config in evaluation_data.""" + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig, configure_model_ids -# ============================================================================= -# Test _get_scenario_metadata helper -# ============================================================================= - + config = Gaia2JudgeEngineConfig(provider="openrouter") + configure_model_ids(sample_gaia2_task_queue, judge_engine_config=config) -@pytest.mark.benchmark -class TestGetScenarioMetadata: - """Tests for _get_scenario_metadata helper function.""" + for task in sample_gaia2_task_queue: + assert task.evaluation_data.get("judge_engine_config") is config - def test_extracts_from_dict_metadata(self): - """Test extraction from dict-style metadata.""" - from maseval.benchmark.gaia2.data_loader import _get_scenario_metadata + def test_judge_engine_config_none_does_not_set(self, sample_gaia2_task_queue): + """Test configure_model_ids with None judge_engine_config does not modify evaluation_data.""" + from maseval.benchmark.gaia2.data_loader import configure_model_ids - scenario = MagicMock() - scenario.metadata = {"capability": "execution", "universe_id": "test"} + configure_model_ids(sample_gaia2_task_queue) - assert _get_scenario_metadata(scenario, "capability") == "execution" - assert _get_scenario_metadata(scenario, "universe_id") == "test" + for task in sample_gaia2_task_queue: + assert "judge_engine_config" not in task.evaluation_data - def test_returns_default_for_missing_key(self): - """Test returns default when key not found.""" - from maseval.benchmark.gaia2.data_loader import _get_scenario_metadata + def test_both_evaluator_and_judge_config(self, sample_gaia2_task_queue): + """Test configure_model_ids sets both evaluator model_id and judge_engine_config.""" + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig, configure_model_ids - scenario = MagicMock() - scenario.metadata = {"other": "value"} + config = Gaia2JudgeEngineConfig(model_name="gpt-4o", provider="openai") + configure_model_ids( + sample_gaia2_task_queue, + evaluator_model_id="gpt-4o", + judge_engine_config=config, + ) - assert _get_scenario_metadata(scenario, "missing") is None - assert _get_scenario_metadata(scenario, "missing", "default") == "default" + for task in sample_gaia2_task_queue: + assert task.evaluation_data.get("model_id") == "gpt-4o" + assert task.evaluation_data.get("judge_engine_config") is config - def test_handles_none_metadata(self): - """Test handles None metadata attribute.""" - from maseval.benchmark.gaia2.data_loader import _get_scenario_metadata - scenario = MagicMock() - scenario.metadata = None +# ============================================================================= +# Test Gaia2JudgeEngineConfig +# ============================================================================= - assert _get_scenario_metadata(scenario, "any_key") is None - assert _get_scenario_metadata(scenario, "any_key", "default") == "default" - def test_handles_object_metadata(self): - """Test handles object-style metadata with attributes.""" - from maseval.benchmark.gaia2.data_loader import _get_scenario_metadata +@pytest.mark.benchmark +class TestGaia2JudgeEngineConfig: + """Tests for Gaia2JudgeEngineConfig dataclass.""" + + def test_default_values_match_are(self): + """Test defaults match ARE's validation/configs.py:28-29.""" + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig + + config = Gaia2JudgeEngineConfig() + assert config.model_name == "meta-llama/Meta-Llama-3.3-70B-Instruct" + assert config.provider == "huggingface" + assert config.endpoint is None + + def test_custom_provider(self): + """Test custom provider can be set.""" + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig + + config = Gaia2JudgeEngineConfig(provider="openrouter") + assert config.provider == "openrouter" + assert config.model_name == "meta-llama/Meta-Llama-3.3-70B-Instruct" + + def test_custom_model_and_provider(self): + """Test custom model and provider can be set together.""" + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig + + config = Gaia2JudgeEngineConfig( + model_name="openai/gpt-4o", + provider="openrouter", + endpoint="https://openrouter.ai/api/v1", + ) + assert config.model_name == "openai/gpt-4o" + assert config.provider == "openrouter" + assert config.endpoint == "https://openrouter.ai/api/v1" - metadata = MagicMock() - metadata.capability = "search" - scenario = MagicMock() - scenario.metadata = metadata + def test_importable_from_package(self): + """Test Gaia2JudgeEngineConfig is importable from the gaia2 package.""" + from maseval.benchmark.gaia2 import Gaia2JudgeEngineConfig - # Note: dict access will fail, falls back to attribute - assert _get_scenario_metadata(scenario, "capability") == "search" + config = Gaia2JudgeEngineConfig() + assert config is not None diff --git a/tests/test_benchmarks/test_gaia2/test_default_agent.py b/tests/test_benchmarks/test_gaia2/test_default_agent.py index 16f411b..4317136 100644 --- a/tests/test_benchmarks/test_gaia2/test_default_agent.py +++ b/tests/test_benchmarks/test_gaia2/test_default_agent.py @@ -49,7 +49,11 @@ def test_parses_action_without_thought(self): assert tool_name == "Email__send" def test_parses_empty_action_input(self): - """Test parsing action with empty input.""" + """Test parsing action with empty input. + + ARE's ``parse_json_tool_call`` normalizes falsy action_input (including + empty dict ``{}``) to empty string ``""`` via ``action_input or ""``. + """ from maseval.benchmark.gaia2.gaia2 import _parse_action_from_text text = """Thought: Just getting the time. @@ -62,7 +66,8 @@ def test_parses_empty_action_input(self): assert result is not None _, tool_name, tool_args = result assert tool_name == "SystemApp__get_current_time" - assert tool_args == {} + # ARE normalizes empty/falsy action_input to "" (parse_json_tool_call: `action_input or ""`) + assert tool_args == "" def test_returns_none_for_invalid_format(self): """Test that invalid format returns None.""" @@ -79,8 +84,8 @@ def test_returns_none_for_invalid_format(self): result = _parse_action_from_text(text) assert result is None, f"Expected None for: {text[:50]}..." - def test_handles_trailing_comma_in_json(self): - """Test that trailing commas in JSON are handled.""" + def test_rejects_trailing_comma_in_json(self): + """Test that trailing commas in JSON are rejected (matching ARE).""" from maseval.benchmark.gaia2.gaia2 import _parse_action_from_text text = """Thought: Testing. @@ -88,12 +93,11 @@ def test_handles_trailing_comma_in_json(self): Action: {"action": "test_tool", "action_input": {"key": "value",}}""" + # ARE json_action_executor.py:33-57 does not fix trailing commas. + # Trailing commas are invalid JSON, so parsing should fail. result = _parse_action_from_text(text) - assert result is not None - _, tool_name, tool_args = result - assert tool_name == "test_tool" - assert tool_args == {"key": "value"} + assert result is None def test_handles_dict_action_input(self): """Test that nested dict action_input is handled.""" @@ -111,20 +115,77 @@ def test_handles_dict_action_input(self): assert tool_args == {"nested": "value"} +@pytest.mark.benchmark +class TestApplyStopTruncation: + """Tests for _apply_stop_truncation helper function. + + Matches ARE's LiteLLMEngine client-side truncation (litellm_engine.py:126-127). + """ + + def test_truncates_on_end_action(self): + """Test truncation at `` stop token.""" + from maseval.benchmark.gaia2.gaia2 import _apply_stop_truncation + + text = 'Thought: checking.\n\nAction:\n{"action": "test", "action_input": {}}extra stuff' + result = _apply_stop_truncation(text, ["", "Observation:"]) + + assert result == 'Thought: checking.\n\nAction:\n{"action": "test", "action_input": {}}' + assert "" not in result + assert "extra stuff" not in result + + def test_truncates_on_observation(self): + """Test truncation at ``Observation:`` stop token.""" + from maseval.benchmark.gaia2.gaia2 import _apply_stop_truncation + + text = 'Thought: checking.\n\nAction:\n{"action": "test", "action_input": {}}Observation: some output' + result = _apply_stop_truncation(text, ["", "Observation:"]) + + assert "Observation:" not in result + assert "some output" not in result + + def test_no_stop_token_returns_unchanged(self): + """Test that text without stop tokens passes through unchanged.""" + from maseval.benchmark.gaia2.gaia2 import _apply_stop_truncation + + text = 'Thought: checking.\n\nAction:\n{"action": "test", "action_input": {}}' + result = _apply_stop_truncation(text, ["", "Observation:"]) + + assert result == text + + def test_truncates_on_first_occurrence(self): + """Test that only content before the first stop token is kept.""" + from maseval.benchmark.gaia2.gaia2 import _apply_stop_truncation + + text = "beforemiddleafter" + result = _apply_stop_truncation(text, [""]) + + assert result == "before" + + def test_empty_stop_sequences(self): + """Test with empty stop sequences list.""" + from maseval.benchmark.gaia2.gaia2 import _apply_stop_truncation + + text = "some textmore" + result = _apply_stop_truncation(text, []) + + assert result == text + + @pytest.mark.benchmark class TestBuildToolDescriptions: """Tests for _build_tool_descriptions helper function.""" def test_builds_descriptions_from_tools(self, sample_tools_dict): - """Test building tool descriptions from a tools dict.""" + """Test building tool descriptions matching ARE's Jinja2 format.""" from maseval.benchmark.gaia2.gaia2 import _build_tool_descriptions # Create mock tools with proper attributes class MockTool: - def __init__(self, name, desc, inputs): + def __init__(self, name, desc, inputs, output_type="string"): self.name = name self.description = desc self.inputs = inputs + self.output_type = output_type def __call__(self, **kwargs): return "result" @@ -142,10 +203,11 @@ def __call__(self, **kwargs): result = _build_tool_descriptions(tools) - assert "TestTool" in result - assert "A test tool" in result + # ARE format: "- {name}: {desc}\n Takes inputs: {inputs}\n Returns an output of type: {output_type}" + assert "- TestTool: A test tool" in result + assert "Takes inputs:" in result assert "arg1" in result - assert "(required)" in result + assert "Returns an output of type: string" in result def test_handles_tools_without_attributes(self): """Test handling tools without description/inputs attributes.""" @@ -158,8 +220,10 @@ def plain_function(**kwargs): result = _build_tool_descriptions(tools) - assert "plain_tool" in result - assert "(no parameters)" in result + # ARE format: "- {name}: {desc}\n Takes inputs: {inputs}\n Returns an output of type: {output_type}" + assert "- plain_tool:" in result + assert "Takes inputs:" in result + assert "Returns an output of type: string" in result @pytest.mark.benchmark @@ -257,6 +321,27 @@ def test_allows_custom_parameters(self, sample_tools_dict, gaia2_model_react): assert agent.llm_args["temperature"] == 0.7 assert agent.verbose == 2 + def test_none_llm_args_override_defaults(self, sample_tools_dict, gaia2_model_react): + """Test that llm_args with None values override defaults. + + Reasoning models (o1, o3, GPT-5) don't support stop, temperature, etc. + Setting them to None omits them from the API call while client-side + stop-token truncation still works. + """ + from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent + + agent = DefaultGaia2Agent( + tools=sample_tools_dict, + model=gaia2_model_react, + llm_args={"stop": None, "temperature": None}, + ) + + # None values stored in llm_args (filtered out at call time) + assert agent.llm_args["stop"] is None + assert agent.llm_args["temperature"] is None + # max_tokens retains default + assert agent.llm_args["max_tokens"] == 16384 + def test_builds_system_prompt_with_tools(self, sample_tools_dict, gaia2_model_react): """Test that system prompt includes tool descriptions.""" from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent @@ -305,8 +390,13 @@ def test_terminates_on_send_message_to_user(self, sample_tools_dict, gaia2_model assert agent._terminated assert "ready to help" in result.lower() - def test_terminates_on_wait_for_notification(self, sample_tools_dict, gaia2_model_wait_notification): - """Test agent terminates when calling wait_for_notification.""" + def test_continues_after_wait_for_notification(self, sample_tools_dict, gaia2_model_wait_notification): + """Test agent pauses on wait_for_notification and resumes. + + wait_for_notification pauses the inner step loop (PAUSED state). The + outer turn loop continues — eventually the agent terminates via + send_message_to_user. Matches ARE's two-level loop architecture. + """ from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent agent = DefaultGaia2Agent( @@ -317,23 +407,27 @@ def test_terminates_on_wait_for_notification(self, sample_tools_dict, gaia2_mode result = agent.run("Wait for updates") assert agent._terminated - assert "notification" in result.lower() + # Agent continued past wait_for_notification and terminated via send_message_to_user + assert "finished waiting" in result.lower() + # At least 2 iterations: one for wait, one for send_message_to_user + assert agent.iteration_count >= 2 def test_retries_on_invalid_format(self, sample_tools_dict, gaia2_model_invalid_format): - """Test agent retries when format is invalid.""" + """Test agent retries when format is invalid then hits max iterations.""" from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent agent = DefaultGaia2Agent( tools=sample_tools_dict, model=gaia2_model_invalid_format, + max_iterations=3, invalid_format_retries=3, ) result = agent.run("Do something") - # Should fail after retries - assert "Failed to parse action" in result - assert agent._format_retry_count >= 3 + # Agent should exhaust iterations; ARE base_agent.py:849 increments on every iteration + assert "Max iterations (3) reached" in result + assert agent.iteration_count == 3 def test_respects_max_iterations(self, sample_tools_dict): """Test agent stops at max_iterations.""" @@ -356,7 +450,12 @@ def test_respects_max_iterations(self, sample_tools_dict): assert agent.iteration_count == 3 def test_handles_tool_not_found(self, sample_tools_dict, gaia2_model_react): - """Test agent handles unknown tool gracefully.""" + """Test agent handles unknown tool gracefully. + + ARE json_action_executor.py:210-212: raises UnavailableToolAgentError + with message "Error: unknown tool {name}, should be instead one of ...". + Error appears as ERROR: message (not Observation:) in agent context. + """ from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent model = DummyModelAdapter( @@ -373,13 +472,67 @@ def test_handles_tool_not_found(self, sample_tools_dict, gaia2_model_react): agent.run("Try unknown tool") - # Should have continued after error + # Should have continued after error and terminated + assert agent._terminated messages = agent.get_messages() - error_found = any("not found" in str(m.get("content", "")).lower() for m in messages) - assert error_found or agent._terminated + # Error is formatted as ERROR: (not Observation:) matching ARE + error_msgs = [m for m in messages if "ERROR:" in str(m.get("content", ""))] + assert len(error_msgs) >= 1 + # ARE error format: "Error: unknown tool {name}, should be instead one of ..." + error_content = str(error_msgs[0].get("content", "")) + assert "unknown tool" in error_content.lower() + assert "should be instead one of" in error_content.lower() def test_handles_tool_execution_error(self): - """Test agent handles tool execution errors gracefully.""" + """Test agent handles tool execution errors as ERROR: messages. + + ARE json_action_executor.py:224-227: raises JsonExecutionAgentError with + error details and tool description reminder. Errors appear as ERROR: + messages (not Observation:) in agent context. + """ + from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent + + class FailingTool: + description = "A tool that fails" + inputs = {"param": {"type": "string", "description": "A parameter"}} + output_type = "string" + + def __call__(self, **kwargs): + raise ValueError("Tool failed!") + + tools = { + "Failing__tool": FailingTool(), + "AgentUserInterface__send_message_to_user": lambda **kwargs: "sent", + } + + model = DummyModelAdapter( + responses=[ + 'Thought: Try failing.\n\nAction:\n{"action": "Failing__tool", "action_input": {}}', + 'Thought: Report error.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "Error occurred"}}', + ] + ) + + agent = DefaultGaia2Agent(tools=tools, model=model) + agent.run("Test error handling") + + # Error appears as ERROR: message (not Observation:) + messages = agent.get_messages() + error_msgs = [m for m in messages if "ERROR:" in str(m.get("content", ""))] + assert len(error_msgs) >= 1 + error_content = str(error_msgs[0].get("content", "")) + # ARE format: includes "Error in tool call execution:" and tool description reminder + assert "Error in tool call execution" in error_content + assert "As a reminder, this tool's description is the following" in error_content + # No Observation: message for the error + error_observations = [m for m in messages if "Observation:" in str(m.get("content", "")) and "Tool failed" in str(m.get("content", ""))] + assert len(error_observations) == 0 + + def test_step_counter_increments_on_errors(self): + """Test step counter increments for errors, not just observations. + + ARE base_agent.py:450-451: id_output_step incremented for BOTH + "observation" and "error" roles. Each output gets a unique step number. + """ from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent def failing_tool(**kwargs): @@ -387,23 +540,125 @@ def failing_tool(**kwargs): tools = { "Failing__tool": failing_tool, + "Calendar__get_events": lambda **kwargs: "[]", "AgentUserInterface__send_message_to_user": lambda **kwargs: "sent", } model = DummyModelAdapter( responses=[ + # Step 1: call failing tool -> ERROR at step 1 'Thought: Try failing.\n\nAction:\n{"action": "Failing__tool", "action_input": {}}', - 'Thought: Report error.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "Error occurred"}}', + # Step 2: call working tool -> Observation at step 2 + 'Thought: Try calendar.\n\nAction:\n{"action": "Calendar__get_events", "action_input": {}}', + # Step 3: terminate + 'Thought: Done.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "Done"}}', ] ) agent = DefaultGaia2Agent(tools=tools, model=model) - agent.run("Test error handling") + agent.run("Test step counting") - # Should have error in observation messages = agent.get_messages() - error_observed = any("error" in str(m.get("content", "")).lower() for m in messages) - assert error_observed + # Find all OUTPUT OF STEP messages + step_msgs = [m for m in messages if "[OUTPUT OF STEP" in str(m.get("content", ""))] + assert len(step_msgs) >= 2 + + # Extract step numbers + import re + + step_numbers = [] + for m in step_msgs: + match = re.search(r"\[OUTPUT OF STEP (\d+)\]", str(m.get("content", ""))) + if match: + step_numbers.append(int(match.group(1))) + + # All step numbers should be unique (no duplicates from error path) + assert len(step_numbers) == len(set(step_numbers)), f"Duplicate step numbers: {step_numbers}" + # Steps should be sequential + assert step_numbers == sorted(step_numbers), f"Steps not sequential: {step_numbers}" + + def test_unknown_tool_step_counter_increments(self): + """Test that calling an unknown tool increments the step counter. + + Ensures no duplicate step numbers when error is followed by success. + """ + from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent + + tools = { + "Calendar__get_events": lambda **kwargs: "[]", + "AgentUserInterface__send_message_to_user": lambda **kwargs: "sent", + } + + model = DummyModelAdapter( + responses=[ + # Step 1: unknown tool -> ERROR at step 1 + 'Thought: Try unknown.\n\nAction:\n{"action": "NonExistent__tool", "action_input": {}}', + # Step 2: valid tool -> Observation at step 2 + 'Thought: Try calendar.\n\nAction:\n{"action": "Calendar__get_events", "action_input": {}}', + # Step 3: terminate + 'Thought: Done.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "Done"}}', + ] + ) + + agent = DefaultGaia2Agent(tools=tools, model=model) + agent.run("Test unknown tool step counting") + + messages = agent.get_messages() + import re + + step_numbers = [] + for m in messages: + match = re.search(r"\[OUTPUT OF STEP (\d+)\]", str(m.get("content", ""))) + if match: + step_numbers.append(int(match.group(1))) + + assert len(step_numbers) == len(set(step_numbers)), f"Duplicate step numbers: {step_numbers}" + + def test_wait_for_notification_executes_tool_and_continues(self): + """Test wait_for_notification executes the tool and records observation. + + Verifies the multi-turn notification loop: the agent calls + wait_for_notification, the tool executes (advancing simulation + time), the observation is added to messages, and the agent + continues to call more tools before terminating. + """ + from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent + + wait_called = [] + + def mock_wait(**kwargs): + wait_called.append(kwargs) + return "No notifications" + + tools = { + "Calendar__get_events": lambda **kwargs: "[]", + "SystemApp__wait_for_notification": mock_wait, + "AgentUserInterface__send_message_to_user": lambda **kwargs: "sent", + } + + model = DummyModelAdapter( + responses=[ + # Step 1: wait for notification + 'Thought: Need to wait.\n\nAction:\n{"action": "SystemApp__wait_for_notification", "action_input": {"timeout": 240}}', + # Step 2: check calendar (agent continued!) + 'Thought: Check calendar after wait.\n\nAction:\n{"action": "Calendar__get_events", "action_input": {}}', + # Step 3: terminate + 'Thought: Done.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "All done."}}', + ] + ) + + agent = DefaultGaia2Agent(tools=tools, model=model) + result = agent.run("Multi-turn task") + + # wait_for_notification was actually executed as a tool + assert len(wait_called) == 1 + # Agent continued past wait and executed more tools + assert agent.iteration_count == 3 + assert result == "All done." + # Observation from wait is in messages + messages = agent.get_messages() + wait_observation = [m for m in messages if "No notifications" in str(m.get("content", ""))] + assert len(wait_observation) > 0 @pytest.mark.benchmark @@ -566,8 +821,54 @@ def test_stop_sequences_match_are(self): assert "Observation:" in _STOP_SEQUENCES def test_termination_tools_include_expected(self): - """Test TERMINATION_TOOLS include expected tool names.""" + """Test TERMINATION_TOOLS contains only send_message_to_user. + + wait_for_notification is NOT a termination tool — it pauses the + agent while ARE processes events, then the agent resumes. + """ from maseval.benchmark.gaia2.gaia2 import _TERMINATION_TOOLS assert "AgentUserInterface__send_message_to_user" in _TERMINATION_TOOLS - assert "SystemApp__wait_for_notification" in _TERMINATION_TOOLS + assert "SystemApp__wait_for_notification" not in _TERMINATION_TOOLS + + +# ============================================================================= +# Test ARE Import Delegation +# ============================================================================= + + +@pytest.mark.benchmark +class TestAREImportDelegation: + """Verify that functions delegate to ARE's implementations.""" + + def test_parse_action_uses_are_parse_json_tool_call(self): + """_parse_action_from_text delegates JSON parsing to ARE's parse_json_tool_call.""" + from are.simulation.agents.default_agent.tools.json_action_executor import parse_json_tool_call + + # If the import works, ARE is installed and the function is available. + # Verify _parse_action_from_text produces the same result as calling + # parse_json_tool_call on the action blob directly. + from maseval.benchmark.gaia2.gaia2 import _parse_action_from_text + + text = 'Thought: checking.\n\nAction:\n{"action": "Calendar__get_events", "action_input": {"date": "2024-01-15"}}' + result = _parse_action_from_text(text) + assert result is not None + _, tool_name, tool_args = result + + # Compare with ARE's direct output + are_name, are_args = parse_json_tool_call('{"action": "Calendar__get_events", "action_input": {"date": "2024-01-15"}}') + assert tool_name == are_name + assert tool_args == are_args + + def test_get_offset_uses_are_function(self): + """_react_loop uses ARE's get_offset_from_time_config_mode at runtime.""" + from are.simulation.agents.default_agent.base_agent import get_offset_from_time_config_mode + + # Verify the ARE function exists and works as expected + from maseval.benchmark.gaia2.gaia2 import Gaia2SimulatedGenerationTimeConfig + + config = Gaia2SimulatedGenerationTimeConfig(mode="measured") + assert get_offset_from_time_config_mode(config, 2.5) == 2.5 + + config_fixed = Gaia2SimulatedGenerationTimeConfig(mode="fixed", seconds=1.0) + assert get_offset_from_time_config_mode(config_fixed, 2.5) == 1.0 diff --git a/tests/test_benchmarks/test_gaia2/test_environment.py b/tests/test_benchmarks/test_gaia2/test_environment.py index fe8458f..c574e4b 100644 --- a/tests/test_benchmarks/test_gaia2/test_environment.py +++ b/tests/test_benchmarks/test_gaia2/test_environment.py @@ -8,6 +8,51 @@ from unittest.mock import MagicMock, patch +def _make_are_mock(): + """Create a fully-mocked ARE module structure for sys.modules patching. + + Returns a (mock_are, modules_dict) tuple where modules_dict can be + passed directly to ``patch.dict(sys.modules, modules_dict)``. + """ + mock_are = MagicMock() + + # Mock preprocess_scenario as a no-op + def _preprocess_scenario(scenario, judge_config, max_scenario_duration): + scenario.duration = max_scenario_duration + # In real ARE, start_time and time_increment_in_seconds are set from + # JSON data before preprocess_scenario runs. Ensure real values so + # environment.py guards (e.g. start_time > 0) don't fail on MagicMock. + if not isinstance(getattr(scenario, "start_time", None), (int, float)): + scenario.start_time = 1728975600.0 # 2024-10-15 07:00:00 UTC + if not isinstance(getattr(scenario, "time_increment_in_seconds", None), (int, float)): + scenario.time_increment_in_seconds = 1 + + mock_are.simulation.scenarios.scenario_imported_from_json.utils.preprocess_scenario = _preprocess_scenario + + # Mock get_scenario_duration to return a sensible default + def _get_scenario_duration(scenario, max_time_duration, max_duration): + return max_duration + + mock_are.simulation.scenarios.scenario_imported_from_json.utils.get_scenario_duration = _get_scenario_duration + + # Mock scenario config constants + mock_are.simulation.scenarios.config.MAX_SCENARIO_DURATION = 1800 + mock_are.simulation.scenarios.config.MAX_TIME_SCENARIO_DURATION = 420 + + modules = { + "are": mock_are, + "are.simulation": mock_are.simulation, + "are.simulation.environment": mock_are.simulation.environment, + "are.simulation.types": mock_are.simulation.types, + "are.simulation.validation": mock_are.simulation.validation, + "are.simulation.scenarios": mock_are.simulation.scenarios, + "are.simulation.scenarios.config": mock_are.simulation.scenarios.config, + "are.simulation.scenarios.scenario_imported_from_json": (mock_are.simulation.scenarios.scenario_imported_from_json), + "are.simulation.scenarios.scenario_imported_from_json.utils": (mock_are.simulation.scenarios.scenario_imported_from_json.utils), + } + return mock_are, modules + + # ============================================================================= # Test Gaia2Environment Class Structure # ============================================================================= @@ -49,24 +94,17 @@ def test_stores_scenario_from_task_data(self): """Test environment stores scenario from task data.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - # Create mock ARE modules - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario mock_scenario.scenario_id = "test_scenario" - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): task_data = {"scenario": mock_scenario} env = Gaia2Environment(task_data=task_data) @@ -76,16 +114,9 @@ def test_raises_without_scenario(self): """Test raises error when scenario is missing.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + _, modules = _make_are_mock() - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): with pytest.raises(ValueError, match="scenario"): Gaia2Environment(task_data={}) @@ -102,18 +133,24 @@ class TestGaia2EnvironmentCreateTools: def test_create_tools_wraps_are_tools(self): """Test create_tools returns wrapped ARE tools.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool + from types import SimpleNamespace - # Create mock ARE modules - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance - # Create mock app and tool - mock_tool = MagicMock() - mock_tool.name = "TestTool__do_something" - mock_tool.description = "Test tool" - mock_tool.inputs = {} + # Create mock tool matching ARE's AppTool interface (required by AppToolAdapter) + mock_tool = SimpleNamespace( + name="TestTool__do_something", + _public_name="TestTool__do_something", + _public_description="Test tool", + function_description="Test tool", + app_name="TestTool", + return_type=str, + args=[SimpleNamespace(name="arg1", arg_type="str", description="An argument", has_default=False)], + ) + mock_tool.__call__ = lambda **kw: "result" mock_app = MagicMock() mock_app.get_tools.return_value = [mock_tool] @@ -121,42 +158,121 @@ def test_create_tools_wraps_are_tools(self): mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario mock_scenario.scenario_id = "test" - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) tools = env.create_tools() assert "TestTool__do_something" in tools - assert isinstance(tools["TestTool__do_something"], AREToolWrapper) + assert isinstance(tools["TestTool__do_something"], Gaia2GenericTool) + + def test_create_tools_filters_aui_tools(self): + """Test create_tools filters out AUI message-retrieval tools.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + from types import SimpleNamespace + + mock_are, modules = _make_are_mock() + mock_env_instance = MagicMock() + mock_are.simulation.environment.Environment.return_value = mock_env_instance + + def _make_tool(name, app_name="TestApp"): + t = SimpleNamespace( + name=name, + _public_name=name, + _public_description=f"Desc for {name}", + function_description=f"Desc for {name}", + app_name=app_name, + return_type=str, + args=[], + ) + t.__call__ = lambda **kw: "result" + return t + + # Create tools including the 4 AUI tools that should be filtered + kept_tool = _make_tool("AgentUserInterface__send_message_to_user", "AgentUserInterface") + filtered_tools = [ + _make_tool("AgentUserInterface__get_last_message_from_user", "AgentUserInterface"), + _make_tool("AgentUserInterface__get_last_message_from_agent", "AgentUserInterface"), + _make_tool("AgentUserInterface__get_last_unread_messages", "AgentUserInterface"), + _make_tool("AgentUserInterface__get_all_messages", "AgentUserInterface"), + ] + other_tool = _make_tool("Calendar__create_event", "Calendar") + + mock_aui_app = MagicMock() + mock_aui_app.get_tools.return_value = [kept_tool] + filtered_tools + mock_calendar_app = MagicMock() + mock_calendar_app.get_tools.return_value = [other_tool] + mock_env_instance.apps = {"AgentUserInterface": mock_aui_app, "Calendar": mock_calendar_app} + + mock_scenario = MagicMock() + mock_scenario.duration = 86400 + mock_scenario.events = [] + mock_scenario.scenario_id = "test" + + with patch.dict(sys.modules, modules): + env = Gaia2Environment(task_data={"scenario": mock_scenario}) + tools = env.create_tools() + + # Kept tools should be present + assert "AgentUserInterface__send_message_to_user" in tools + assert "Calendar__create_event" in tools + + # Filtered tools should NOT be present + for ft in filtered_tools: + assert ft.name not in tools, f"{ft.name} should have been filtered out" + + def test_create_tools_sets_wait_for_user_response_false(self): + """Test create_tools sets wait_for_user_response=False on AUI app.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + from types import SimpleNamespace + + mock_are, modules = _make_are_mock() + mock_env_instance = MagicMock() + mock_are.simulation.environment.Environment.return_value = mock_env_instance + + mock_tool = SimpleNamespace( + name="AgentUserInterface__send_message_to_user", + _public_name="AgentUserInterface__send_message_to_user", + _public_description="Send message", + function_description="Send message", + app_name="AgentUserInterface", + return_type=str, + args=[], + ) + mock_tool.__call__ = lambda **kw: "result" + + mock_aui_app = MagicMock() + mock_aui_app.wait_for_user_response = True + mock_aui_app.get_tools.return_value = [mock_tool] + mock_env_instance.apps = {"AgentUserInterface": mock_aui_app} + + mock_scenario = MagicMock() + mock_scenario.duration = 86400 + mock_scenario.events = [] + mock_scenario.scenario_id = "test" + + with patch.dict(sys.modules, modules): + env = Gaia2Environment(task_data={"scenario": mock_scenario}) + env.create_tools() + + assert mock_aui_app.wait_for_user_response is False def test_create_tools_returns_empty_when_no_are_env(self): """Test create_tools returns empty dict when ARE env is None.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) # Manually set _are_env to None env._are_env = None @@ -178,22 +294,16 @@ def test_cleanup_stops_are_environment(self): """Test cleanup calls stop on ARE environment.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) env.cleanup() @@ -203,22 +313,16 @@ def test_cleanup_handles_no_are_environment(self): """Test cleanup handles case when no ARE environment.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) env._are_env = None @@ -229,7 +333,7 @@ def test_cleanup_handles_stop_error(self): """Test cleanup handles error during stop gracefully.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} @@ -237,15 +341,9 @@ def test_cleanup_handles_stop_error(self): mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) # Should not raise @@ -265,22 +363,16 @@ def test_get_are_environment_returns_are_env(self): """Test get_are_environment returns ARE environment.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) assert env.get_are_environment() is mock_env_instance @@ -289,22 +381,16 @@ def test_get_scenario_returns_scenario(self): """Test get_scenario returns scenario.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) assert env.get_scenario() is mock_scenario @@ -313,23 +399,17 @@ def test_get_simulation_time_returns_time(self): """Test get_simulation_time returns current time.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} - mock_env_instance.time_manager.current_time = 123.5 + mock_env_instance.current_time = 123.5 mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) assert env.get_simulation_time() == 123.5 @@ -338,22 +418,16 @@ def test_get_simulation_time_returns_zero_when_no_env(self): """Test get_simulation_time returns 0 when no environment.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) env._are_env = None @@ -373,23 +447,17 @@ def test_gather_traces_includes_type(self): """Test gather_traces includes type information.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} - mock_env_instance.time_manager.current_time = 0.0 + mock_env_instance.current_time = 0.0 mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment(task_data={"scenario": mock_scenario}) traces = env.gather_traces() @@ -400,23 +468,17 @@ def test_gather_config_includes_environment_info(self): """Test gather_config includes environment information.""" from maseval.benchmark.gaia2.environment import Gaia2Environment - mock_are = MagicMock() + mock_are, modules = _make_are_mock() mock_env_instance = MagicMock() mock_are.simulation.environment.Environment.return_value = mock_env_instance mock_env_instance.apps = {} mock_scenario = MagicMock() mock_scenario.duration = 86400 + mock_scenario.events = [] # No oracle events in mock scenario mock_scenario.scenario_id = "test_scenario" - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.environment": mock_are.simulation.environment, - }, - ): + with patch.dict(sys.modules, modules): env = Gaia2Environment( task_data={ "scenario": mock_scenario, @@ -427,3 +489,170 @@ def test_gather_config_includes_environment_info(self): assert "type" in config assert config["type"] == "Gaia2Environment" + + +# ============================================================================= +# Test Gaia2Environment Judge Engine Config +# ============================================================================= + + +@pytest.mark.benchmark +class TestGaia2EnvironmentJudgeEngineConfig: + """Tests for judge engine configuration in Gaia2Environment.""" + + def test_default_judge_config_when_no_engine_config(self): + """Test default GraphPerEventJudgeConfig is used when no judge_engine_config.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + mock_are, modules = _make_are_mock() + mock_env_instance = MagicMock() + mock_are.simulation.environment.Environment.return_value = mock_env_instance + mock_env_instance.apps = {} + + # Track calls to GraphPerEventJudgeConfig + mock_judge_config_cls = MagicMock() + mock_judge_config_instance = MagicMock() + mock_judge_config_cls.return_value = mock_judge_config_instance + mock_are.simulation.validation.GraphPerEventJudgeConfig = mock_judge_config_cls + + mock_scenario = MagicMock() + mock_scenario.duration = 86400 + mock_scenario.events = [] + + with patch.dict(sys.modules, modules): + Gaia2Environment(task_data={"scenario": mock_scenario}) + + # Default: GraphPerEventJudgeConfig() called with no arguments + mock_judge_config_cls.assert_called_once_with() + + def test_custom_judge_engine_config_creates_engine(self): + """Test custom judge_engine_config creates engine via create_judge_engine.""" + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig + from maseval.benchmark.gaia2.environment import Gaia2Environment + + mock_are, modules = _make_are_mock() + mock_env_instance = MagicMock() + mock_are.simulation.environment.Environment.return_value = mock_env_instance + mock_env_instance.apps = {} + + # Mock LLMEngineConfig + mock_llm_config_cls = MagicMock() + mock_llm_config_instance = MagicMock() + mock_llm_config_cls.return_value = mock_llm_config_instance + + # Mock create_judge_engine + mock_create_engine = MagicMock() + mock_engine = MagicMock() + mock_create_engine.return_value = mock_engine + + # Mock GraphPerEventJudgeConfig + mock_judge_config_cls = MagicMock() + mock_judge_config_instance = MagicMock() + mock_judge_config_cls.return_value = mock_judge_config_instance + + mock_are.simulation.validation.GraphPerEventJudgeConfig = mock_judge_config_cls + + # Add the extra ARE modules that get imported when judge_engine_config is set + modules["are.simulation.agents"] = mock_are.simulation.agents + modules["are.simulation.agents.are_simulation_agent_config"] = MagicMock(LLMEngineConfig=mock_llm_config_cls) + modules["are.simulation.validation.configs"] = MagicMock(create_judge_engine=mock_create_engine) + + mock_scenario = MagicMock() + mock_scenario.duration = 86400 + mock_scenario.events = [] + + judge_engine_config = Gaia2JudgeEngineConfig( + model_name="openai/gpt-4o", + provider="openrouter", + endpoint="https://openrouter.ai/api/v1", + ) + + with patch.dict(sys.modules, modules): + Gaia2Environment( + task_data={"scenario": mock_scenario}, + judge_engine_config=judge_engine_config, + ) + + # LLMEngineConfig should be created with the custom values + mock_llm_config_cls.assert_called_once_with( + model_name="openai/gpt-4o", + provider="openrouter", + endpoint="https://openrouter.ai/api/v1", + ) + + # create_judge_engine should be called with the LLMEngineConfig + mock_create_engine.assert_called_once_with(mock_llm_config_instance) + + # GraphPerEventJudgeConfig should be created with the custom engine + mock_judge_config_cls.assert_called_once_with(engine=mock_engine) + + +# ============================================================================= +# Test poll_notifications +# ============================================================================= + + +@pytest.mark.benchmark +class TestPollNotifications: + """Tests for Gaia2Environment.poll_notifications().""" + + def test_returns_empty_when_no_are_env(self): + """poll_notifications returns empty when ARE environment is not set up.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + # Create env without triggering setup_state (no ARE) + env = Gaia2Environment.__new__(Gaia2Environment) + env._are_env = None + env._scenario = None + env._judge_engine_config = None + env._tool_wrappers = {} + env.state = {} + + user_msgs, env_notifs, has_stop = env.poll_notifications() + + assert user_msgs == [] + assert env_notifs == [] + assert has_stop is False + + def test_returns_empty_when_no_notification_system(self): + """poll_notifications returns empty when notification_system is None.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + # Use a bare mock without notification_system attribute + mock_env = MagicMock(spec=[]) + + env = Gaia2Environment.__new__(Gaia2Environment) + env._are_env = mock_env + env._scenario = None + env._judge_engine_config = None + env._tool_wrappers = {} + env.state = {} + + user_msgs, env_notifs, has_stop = env.poll_notifications() + + assert user_msgs == [] + assert env_notifs == [] + assert has_stop is False + + def test_returns_empty_when_queue_empty(self): + """poll_notifications returns empty when queue has no messages.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + mock_ns = MagicMock() + mock_ns.message_queue.get_by_timestamp.return_value = [] + + mock_env = MagicMock() + mock_env.notification_system = mock_ns + + env = Gaia2Environment.__new__(Gaia2Environment) + env._are_env = mock_env + env._scenario = None + env._judge_engine_config = None + env._tool_wrappers = {} + env.state = {} + + user_msgs, env_notifs, has_stop = env.poll_notifications() + + assert user_msgs == [] + assert env_notifs == [] + assert has_stop is False diff --git a/tests/test_benchmarks/test_gaia2/test_evaluator.py b/tests/test_benchmarks/test_gaia2/test_evaluator.py index a7d3208..a3acdd1 100644 --- a/tests/test_benchmarks/test_gaia2/test_evaluator.py +++ b/tests/test_benchmarks/test_gaia2/test_evaluator.py @@ -3,9 +3,8 @@ Tests the evaluation layer that integrates with ARE's GraphPerEventJudge. """ -import sys import pytest -from unittest.mock import MagicMock, patch +from unittest.mock import MagicMock # ============================================================================= @@ -43,8 +42,8 @@ def test_stores_environment(self, sample_gaia2_task): assert evaluator.environment == mock_env - def test_extracts_oracle_events_from_task(self, sample_gaia2_task): - """Test evaluator extracts oracle events from task.""" + def test_extracts_judge_type_from_task(self, sample_gaia2_task): + """Test evaluator extracts judge_type from task evaluation_data.""" from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator mock_env = MagicMock() @@ -54,7 +53,7 @@ def test_extracts_oracle_events_from_task(self, sample_gaia2_task): environment=mock_env, ) - assert evaluator.oracle_events == sample_gaia2_task.evaluation_data.get("oracle_events", []) + assert evaluator.judge_type == sample_gaia2_task.evaluation_data.get("judge_type", "graph_per_event") def test_default_use_llm_judge_is_false(self, sample_gaia2_task): """Test use_llm_judge defaults to False.""" @@ -110,85 +109,257 @@ class TestGaia2EvaluatorCall: """Tests for Gaia2Evaluator.__call__().""" def test_returns_gsr_from_judge(self, sample_gaia2_task): - """Test evaluator returns GSR from ARE judge.""" + """Test evaluator returns GSR from ARE judge on single-turn scenario.""" + from types import SimpleNamespace + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + # Create scenario mock with an explicit judge (single-turn: nb_turns=1) + mock_scenario = MagicMock() + mock_judge = MagicMock() + mock_judge.state = SimpleNamespace(nb_turns=1, turn_idx=-1) + mock_result = MagicMock() + mock_result.success = True + mock_result.rationale = None + mock_judge.validate.return_value = mock_result + mock_scenario.judge = mock_judge + mock_env = MagicMock() mock_are_env = MagicMock() - mock_are_env.get_completed_events.return_value = [] mock_env.get_are_environment.return_value = mock_are_env - mock_env.get_scenario.return_value = MagicMock() + mock_env.get_scenario.return_value = mock_scenario evaluator = Gaia2Evaluator( task=sample_gaia2_task, environment=mock_env, ) - # Mock ARE imports via sys.modules - mock_are = MagicMock() - mock_judge = MagicMock() - mock_result = MagicMock() - mock_result.passed = True - mock_result.partial_score = 1.0 - mock_result.event_results = [] - mock_judge.evaluate.return_value = mock_result - mock_are.simulation.validation.JudgeFactory.create.return_value = mock_judge - - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.validation": mock_are.simulation.validation, - "are.simulation.validation.config": mock_are.simulation.validation.config, - }, - ): - result = evaluator({}, None) + result = evaluator({}, None) - assert result["gsr"] == 1.0 - assert result["passed"] is True + assert result["gsr"] == 1.0 + assert result["passed"] is True + # Single-turn: no intermediate judge calls, only validate + mock_judge.assert_not_called() + mock_judge.validate.assert_called_once_with(mock_are_env) def test_returns_zero_gsr_on_failure(self, sample_gaia2_task): """Test evaluator returns 0.0 GSR when judge fails.""" + from types import SimpleNamespace + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + # Create scenario mock with an explicit judge (single-turn) + mock_scenario = MagicMock() + mock_judge = MagicMock() + mock_judge.state = SimpleNamespace(nb_turns=1, turn_idx=-1) + mock_result = MagicMock() + mock_result.success = False + mock_result.rationale = "Failed" + mock_judge.validate.return_value = mock_result + mock_scenario.judge = mock_judge + mock_env = MagicMock() mock_are_env = MagicMock() - mock_are_env.get_completed_events.return_value = [] mock_env.get_are_environment.return_value = mock_are_env - mock_env.get_scenario.return_value = MagicMock() + mock_env.get_scenario.return_value = mock_scenario evaluator = Gaia2Evaluator( task=sample_gaia2_task, environment=mock_env, ) - # Mock ARE imports via sys.modules - mock_are = MagicMock() + result = evaluator({}, None) + + assert result["gsr"] == 0.0 + assert result["passed"] is False + + def test_multi_turn_calls_judge_for_intermediate_turns(self, sample_gaia2_task): + """Test evaluator calls judge(env) for intermediate turns before validate(). + + ARE's intended flow for nb_turns=N: call judge(env) for turns 0..N-2, + then judge.validate(env) for the final turn. This advances turn_idx so + the is_last_turn check in validate() passes. + ARE simulation/validation/base.py:104 + """ + from types import SimpleNamespace + + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + # Simulate judge state: turn_idx starts at -1 (no trigger conditions fired) + state = SimpleNamespace(nb_turns=3, turn_idx=-1) + intermediate_judgment = MagicMock() + intermediate_judgment.success = True + mock_judge = MagicMock() - mock_result = MagicMock() - mock_result.passed = False - mock_result.partial_score = 0.3 - mock_judge.evaluate.return_value = mock_result - mock_are.simulation.validation.JudgeFactory.create.return_value = mock_judge - - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.validation": mock_are.simulation.validation, - "are.simulation.validation.config": mock_are.simulation.validation.config, - }, - ): - result = evaluator({}, None) + mock_judge.state = state + + def judge_call(env): + state.turn_idx += 1 + return intermediate_judgment + + mock_judge.side_effect = judge_call + + mock_validate_result = MagicMock() + mock_validate_result.success = True + mock_validate_result.rationale = None + mock_judge.validate.return_value = mock_validate_result + + mock_scenario = MagicMock() + mock_scenario.judge = mock_judge - assert result["gsr"] == 0.0 - assert result["partial_gsr"] == 0.3 - assert result["passed"] is False + mock_env = MagicMock() + mock_are_env = MagicMock() + mock_env.get_are_environment.return_value = mock_are_env + mock_env.get_scenario.return_value = mock_scenario + + evaluator = Gaia2Evaluator(task=sample_gaia2_task, environment=mock_env) + result = evaluator({}, None) + + # Should call judge(env) twice for intermediate turns (0, 1) + assert mock_judge.call_count == 2 + mock_judge.assert_any_call(mock_are_env) + # Then validate for the final turn + mock_judge.validate.assert_called_once_with(mock_are_env) + assert result["gsr"] == 1.0 + assert result["passed"] is True + + def test_multi_turn_intermediate_failure_short_circuits(self, sample_gaia2_task): + """Test that a failed intermediate turn stops further judge calls. + + When an intermediate turn fails, the evaluator breaks early. The subsequent + validate() call returns failure via the last_turn_success check. + ARE simulation/validation/base.py:96-100 + """ + from types import SimpleNamespace + + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + state = SimpleNamespace(nb_turns=3, turn_idx=-1) + failed_judgment = MagicMock() + failed_judgment.success = False + failed_judgment.failure = "Turn 0 events did not match" + + mock_judge = MagicMock() + mock_judge.state = state + + def judge_call(env): + state.turn_idx += 1 + return failed_judgment + + mock_judge.side_effect = judge_call + + mock_validate_result = MagicMock() + mock_validate_result.success = False + mock_validate_result.rationale = "Last turn was already rejected" + mock_judge.validate.return_value = mock_validate_result + + mock_scenario = MagicMock() + mock_scenario.judge = mock_judge + + mock_env = MagicMock() + mock_are_env = MagicMock() + mock_env.get_are_environment.return_value = mock_are_env + mock_env.get_scenario.return_value = mock_scenario + + evaluator = Gaia2Evaluator(task=sample_gaia2_task, environment=mock_env) + result = evaluator({}, None) + + # Should only call judge(env) once (broke after first failure) + assert mock_judge.call_count == 1 + # validate() still called (returns failure via last_turn_success check) + mock_judge.validate.assert_called_once_with(mock_are_env) + assert result["gsr"] == 0.0 + assert result["passed"] is False + + def test_two_turn_scenario_calls_judge_once(self, sample_gaia2_task): + """Test 2-turn scenario calls judge(env) once then validate(env). + + This is the most common multi-turn case (adaptability scenarios). + nb_turns=2: one intermediate judge(env) call, then validate(env). + """ + from types import SimpleNamespace + + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + state = SimpleNamespace(nb_turns=2, turn_idx=-1) + intermediate_judgment = MagicMock() + intermediate_judgment.success = True + + mock_judge = MagicMock() + mock_judge.state = state + + def judge_call(env): + state.turn_idx += 1 + return intermediate_judgment + + mock_judge.side_effect = judge_call + + mock_validate_result = MagicMock() + mock_validate_result.success = True + mock_validate_result.rationale = None + mock_judge.validate.return_value = mock_validate_result + + mock_scenario = MagicMock() + mock_scenario.judge = mock_judge + + mock_env = MagicMock() + mock_are_env = MagicMock() + mock_env.get_are_environment.return_value = mock_are_env + mock_env.get_scenario.return_value = mock_scenario + + evaluator = Gaia2Evaluator(task=sample_gaia2_task, environment=mock_env) + result = evaluator({}, None) + + # One intermediate call + one validate + assert mock_judge.call_count == 1 + mock_judge.validate.assert_called_once_with(mock_are_env) + assert result["gsr"] == 1.0 + + def test_skips_intermediate_turns_if_already_judged(self, sample_gaia2_task): + """Test evaluator skips judge(env) calls if trigger conditions already fired. + + In online mode (default), ARE's ConditionCheckEvent trigger conditions + call judge(env) during the simulation, advancing turn_idx. The evaluator + checks turn_idx before calling judge(env) to avoid double-counting. + """ + from types import SimpleNamespace + + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + # turn_idx already at nb_turns-2 (trigger conditions fired for all intermediate turns) + state = SimpleNamespace(nb_turns=3, turn_idx=1) + + mock_judge = MagicMock() + mock_judge.state = state + + mock_validate_result = MagicMock() + mock_validate_result.success = True + mock_validate_result.rationale = None + mock_judge.validate.return_value = mock_validate_result + + mock_scenario = MagicMock() + mock_scenario.judge = mock_judge + + mock_env = MagicMock() + mock_are_env = MagicMock() + mock_env.get_are_environment.return_value = mock_are_env + mock_env.get_scenario.return_value = mock_scenario + + evaluator = Gaia2Evaluator(task=sample_gaia2_task, environment=mock_env) + result = evaluator({}, None) + + # No intermediate judge calls needed (already advanced) + mock_judge.assert_not_called() + mock_judge.validate.assert_called_once_with(mock_are_env) + assert result["gsr"] == 1.0 def test_handles_missing_are_environment(self, sample_gaia2_task): - """Test evaluator handles missing ARE environment.""" + """Test evaluator handles missing ARE environment. + + When ARE environment is not available, score is None (excluded from + scoring), matching ARE's behavior for no_validation results. + ARE benchmark/hf_upload_utils.py:47-48 + """ from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator mock_env = MagicMock() @@ -199,22 +370,96 @@ def test_handles_missing_are_environment(self, sample_gaia2_task): environment=mock_env, ) - # Mock ARE imports via sys.modules - mock_are = MagicMock() - with patch.dict( - sys.modules, - { - "are": mock_are, - "are.simulation": mock_are.simulation, - "are.simulation.validation": mock_are.simulation.validation, - "are.simulation.validation.config": mock_are.simulation.validation.config, + result = evaluator({}, None) + + assert result["gsr"] is None + assert result["passed"] is False + assert "error" in result + assert result["status"] == "no_validation" + + def test_fallback_judge_respects_judge_engine_config(self): + """Test evaluator fallback judge creation respects judge_engine_config.""" + import sys + from types import SimpleNamespace + from unittest.mock import patch + + from maseval import Task + from maseval.benchmark.gaia2.data_loader import Gaia2JudgeEngineConfig + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + judge_engine_config = Gaia2JudgeEngineConfig( + model_name="openai/gpt-4o", + provider="openrouter", + ) + + task = Task( + id="test_fallback", + query="", + environment_data={ + "scenario": MagicMock(scenario_id="test"), + "capability": "execution", + }, + evaluation_data={ + "judge_type": "graph_per_event", + "judge_engine_config": judge_engine_config, }, - ): - result = evaluator({}, None) + ) - assert result["gsr"] == 0.0 - assert result["passed"] is False - assert "error" in result + # Scenario without a judge (triggers fallback) + mock_scenario = MagicMock(spec=[]) + del mock_scenario.judge # Ensure getattr returns None + + mock_env = MagicMock() + mock_are_env = MagicMock() + mock_env.get_are_environment.return_value = mock_are_env + mock_env.get_scenario.return_value = mock_scenario + + # Mock ARE imports + mock_llm_config_cls = MagicMock() + mock_create_engine = MagicMock() + mock_engine = MagicMock() + mock_create_engine.return_value = mock_engine + + mock_judge_config_cls = MagicMock() + mock_judge = MagicMock() + mock_judge.state = SimpleNamespace(nb_turns=1, turn_idx=-1) + mock_validate_result = MagicMock() + mock_validate_result.success = True + mock_validate_result.rationale = None + mock_judge.validate.return_value = mock_validate_result + + mock_factory = MagicMock() + mock_factory.return_value.return_value = mock_judge + + mock_validation = MagicMock() + mock_validation.GraphPerEventJudgeConfig = mock_judge_config_cls + mock_validation.JudgeFactory = mock_factory + + mock_are = MagicMock() + modules = { + "are": mock_are, + "are.simulation": mock_are.simulation, + "are.simulation.validation": mock_validation, + "are.simulation.validation.configs": MagicMock(create_judge_engine=mock_create_engine), + "are.simulation.agents": mock_are.simulation.agents, + "are.simulation.agents.are_simulation_agent_config": MagicMock(LLMEngineConfig=mock_llm_config_cls), + } + + evaluator = Gaia2Evaluator(task=task, environment=mock_env) + + with patch.dict(sys.modules, modules): + evaluator({}, None) + + # Verify LLMEngineConfig was created with custom values + mock_llm_config_cls.assert_called_once_with( + model_name="openai/gpt-4o", + provider="openrouter", + endpoint=None, + ) + # Verify create_judge_engine was called + mock_create_engine.assert_called_once() + # Verify GraphPerEventJudgeConfig was created with the custom engine + mock_judge_config_cls.assert_called_once_with(engine=mock_engine) # ============================================================================= diff --git a/tests/test_benchmarks/test_gaia2/test_integration.py b/tests/test_benchmarks/test_gaia2/test_integration.py new file mode 100644 index 0000000..dd6aec5 --- /dev/null +++ b/tests/test_benchmarks/test_gaia2/test_integration.py @@ -0,0 +1,264 @@ +"""Integration tests for GAIA2 benchmark using real HuggingFace data. + +These tests validate that GAIA2 components work correctly with actual +downloaded data, not synthetic fixtures. They are marked ``live`` + ``slow`` ++ ``benchmark`` + ``gaia2`` because they download real data from HuggingFace +and exercise the ARE simulation stack. + +Run with:: + + pytest -m "live and slow" tests/test_benchmarks/test_gaia2/test_integration.py -v +""" + +import pytest + +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark, pytest.mark.gaia2] + + +# ============================================================================= +# Fixture: load a small set of real tasks +# ============================================================================= + + +@pytest.fixture(scope="module") +def real_gaia2_tasks(): + """Load a small set of GAIA2 tasks from HuggingFace. + + Loads 5 tasks to keep download and runtime manageable while still + exercising real data paths. + """ + from maseval.benchmark.gaia2.data_loader import load_tasks + + tasks = load_tasks(capability="execution", split="validation", limit=5) + return list(tasks) + + +@pytest.fixture(scope="module") +def first_real_task(real_gaia2_tasks): + """Return the first real GAIA2 task.""" + assert len(real_gaia2_tasks) > 0, "No GAIA2 tasks loaded" + return real_gaia2_tasks[0] + + +# ============================================================================= +# Environment Tests with Real Data +# ============================================================================= + + +class TestGaia2EnvironmentWithRealData: + """Test that real downloaded GAIA2 tasks work with Gaia2Environment.""" + + def test_environment_creates_from_real_task(self, first_real_task): + """Gaia2Environment can be created from a real task.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + env = Gaia2Environment(task_data=first_real_task.environment_data) + assert env is not None + + def test_environment_setup_state(self, first_real_task): + """setup_state() succeeds with a real ARE scenario. + + This exercises ARE's preprocess_scenario() with real scenario data. + """ + from maseval.benchmark.gaia2.environment import Gaia2Environment + + env = Gaia2Environment(task_data=first_real_task.environment_data) + try: + state = env.setup_state(first_real_task.environment_data) + + assert isinstance(state, dict) + assert "capability" in state + assert "duration" in state + assert state["duration"] > 0, "Scenario duration should be positive" + finally: + env.cleanup() + + def test_real_tools_are_created(self, first_real_task): + """Tools created from a real scenario are non-empty Gaia2GenericTool instances.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool + + env = Gaia2Environment(task_data=first_real_task.environment_data) + try: + env.setup_state(first_real_task.environment_data) + tools = env.create_tools() + + assert len(tools) > 0, "No tools created from real scenario. ARE environment should expose app tools (Calendar, Email, etc.)." + + for name, tool in tools.items(): + assert isinstance(tool, Gaia2GenericTool), f"Tool '{name}' is {type(tool).__name__}, expected Gaia2GenericTool" + assert tool.name, "Tool has empty name" + finally: + env.cleanup() + + def test_real_tools_have_descriptions(self, first_real_task): + """Tools from real scenarios have descriptions and inputs schema.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + env = Gaia2Environment(task_data=first_real_task.environment_data) + try: + env.setup_state(first_real_task.environment_data) + tools = env.create_tools() + + for name, tool in tools.items(): + # Every real ARE tool should have a description + assert tool.description, ( + f"Tool '{name}' has empty description. ARE tools should provide _public_description or function_description." + ) + # inputs should be a dict (possibly empty for tools with no args) + assert isinstance(tool.inputs, dict), f"Tool '{name}' inputs is {type(tool.inputs).__name__}, expected dict" + finally: + env.cleanup() + + def test_environment_traces(self, first_real_task): + """gather_traces() returns expected keys after real scenario setup.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + + env = Gaia2Environment(task_data=first_real_task.environment_data) + try: + env.setup_state(first_real_task.environment_data) + env.create_tools() + traces = env.gather_traces() + + assert isinstance(traces, dict) + assert "capability" in traces + assert "tool_count" in traces + assert traces["tool_count"] > 0 + finally: + env.cleanup() + + +# ============================================================================= +# Default Agent Tests with Real Tools +# ============================================================================= + + +class TestDefaultAgentWithRealTools: + """Test DefaultGaia2Agent construction with real ARE tools.""" + + def test_agent_builds_system_prompt_with_real_tools(self, first_real_task): + """DefaultGaia2Agent system prompt includes real tool names.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + from maseval.benchmark.gaia2.gaia2 import DefaultGaia2Agent + + from conftest import DummyModelAdapter + + env = Gaia2Environment(task_data=first_real_task.environment_data) + try: + env.setup_state(first_real_task.environment_data) + tools = env.create_tools() + + model = DummyModelAdapter( + model_id="test-model", + responses=[ + 'Thought: Done.\n\nAction:\n{"action": "AgentUserInterface__send_message_to_user", "action_input": {"content": "Done"}}' + ], + ) + + agent = DefaultGaia2Agent( + tools=tools, # type: ignore[arg-type] # Gaia2GenericTool has __call__ + model=model, + environment=env, + max_iterations=1, + ) + + # System prompt should mention real tool names + assert "AgentUserInterface__send_message_to_user" in agent.system_prompt, "System prompt should include the AgentUserInterface tool" + # Check at least one domain tool is mentioned + tool_names = list(tools.keys()) + mentioned = any(name in agent.system_prompt for name in tool_names) + assert mentioned, f"System prompt should mention at least one tool. Tool names: {tool_names[:5]}..." + finally: + env.cleanup() + + +# ============================================================================= +# Evaluator Tests with Real Oracle Events +# ============================================================================= + + +class TestGaia2EvaluatorWithRealData: + """Test Gaia2Evaluator with real oracle events.""" + + def test_evaluator_creates_from_real_task(self, first_real_task): + """Gaia2Evaluator can be created with real oracle events.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + env = Gaia2Environment(task_data=first_real_task.environment_data) + + evaluator = Gaia2Evaluator( + task=first_real_task, + environment=env, + use_llm_judge=False, + ) + assert evaluator is not None + + def test_evaluator_filter_traces_with_real_data(self, first_real_task): + """filter_traces() works with a synthetic trace structure.""" + from maseval.benchmark.gaia2.environment import Gaia2Environment + from maseval.benchmark.gaia2.evaluator import Gaia2Evaluator + + env = Gaia2Environment(task_data=first_real_task.environment_data) + + evaluator = Gaia2Evaluator( + task=first_real_task, + environment=env, + use_llm_judge=False, + ) + + # Provide a minimal synthetic trace + traces = { + "agents": { + "test_agent": { + "messages": [], + "iteration_count": 1, + } + }, + "tools": {}, + "environment": {"final_simulation_time": 0.0}, + } + + filtered = evaluator.filter_traces(traces) + assert isinstance(filtered, dict) + + +# ============================================================================= +# Pipeline Smoke Test +# ============================================================================= + + +class TestGaia2PipelineSmoke: + """Smoke test for the full GAIA2 pipeline with real data.""" + + def test_full_pipeline_single_task(self, first_real_task): + """Gaia2Benchmark.run() on one real task produces a result. + + Uses DummyModelAdapter (no API keys needed) and the ConcreteGaia2Benchmark + from conftest. The agent immediately sends a message to terminate. + """ + from maseval import TaskQueue + + from .conftest import ConcreteGaia2Benchmark + + benchmark = ConcreteGaia2Benchmark.create(progress_bar=False) + + task_queue = TaskQueue([first_real_task]) + results = benchmark.run(task_queue, agent_data={}) + + assert len(results) == 1, f"Expected 1 result, got {len(results)}. Check test_data_integrity tests first if this fails." + + result = results[0] + assert "status" in result, "Result missing 'status' key" + # All possible TaskExecutionStatus values + known_statuses = { + "success", + "agent_error", + "environment_error", + "user_error", + "task_timeout", + "unknown_execution_error", + "evaluation_failed", + "setup_failed", + } + assert result["status"] in known_statuses, f"Unexpected status '{result['status']}'. Known: {known_statuses}" diff --git a/tests/test_benchmarks/test_gaia2/test_tool_wrapper.py b/tests/test_benchmarks/test_gaia2/test_tool_wrapper.py index d926702..052e94a 100644 --- a/tests/test_benchmarks/test_gaia2/test_tool_wrapper.py +++ b/tests/test_benchmarks/test_gaia2/test_tool_wrapper.py @@ -1,6 +1,7 @@ -"""Tests for AREToolWrapper and wrap_are_tools. +"""Tests for Gaia2GenericTool and wrap_are_tools. -Tests the tool wrapper that provides MASEval tracing for ARE tools. +Tests the framework-agnostic tool wrapper that provides MASEval tracing +for ARE tools following the MACSGenericTool pattern. """ import pytest @@ -8,71 +9,79 @@ # ============================================================================= -# Test AREToolWrapper +# Test Gaia2GenericTool # ============================================================================= @pytest.mark.benchmark -class TestAREToolWrapperInit: - """Tests for AREToolWrapper initialization.""" +class TestGaia2GenericToolInit: + """Tests for Gaia2GenericTool initialization.""" def test_extracts_name_from_are_tool(self, mock_are_tool): - """Test wrapper extracts name from ARE tool.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + """Test wrapper extracts _public_name from ARE tool (tool_utils.py:546).""" + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) - assert wrapper.name == mock_are_tool.name + assert wrapper.name == mock_are_tool._public_name def test_extracts_description_from_are_tool(self, mock_are_tool): - """Test wrapper extracts description from ARE tool.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + """Test wrapper builds full description with app prefix and return type suffix.""" + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) - assert wrapper.description == mock_are_tool.description + # ARE tool_utils.py:550,581-582: "Acts on app {app_name}: {desc} Returns: {type}" + assert wrapper.description.startswith(f"Acts on app {mock_are_tool.app_name}: ") + assert mock_are_tool._public_description in wrapper.description - def test_extracts_inputs_schema(self, mock_are_tool): - """Test wrapper extracts inputs schema.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + def test_extracts_inputs_flat_dict(self, mock_are_tool): + """Test wrapper extracts inputs as flat dict matching ARE's AppToolAdapter format.""" + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) - assert "properties" in wrapper.inputs + # ARE tool_utils.py:572-578 — flat dict keyed by arg name + assert "param1" in wrapper.inputs + assert wrapper.inputs["param1"]["type"] == "string" + assert "param2" in wrapper.inputs + assert wrapper.inputs["param2"]["type"] == "integer" + # param2 has a default + assert wrapper.inputs["param2"]["default"] == 0 def test_initializes_empty_history(self, mock_are_tool): """Test wrapper initializes with empty history.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) assert len(wrapper.history.to_list()) == 0 @pytest.mark.benchmark -class TestAREToolWrapperCall: - """Tests for AREToolWrapper.__call__().""" +class TestGaia2GenericToolCall: + """Tests for Gaia2GenericTool.__call__().""" def test_delegates_to_are_tool(self, mock_are_tool): """Test wrapper delegates call to ARE tool.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) result = wrapper(arg1="test_value") @@ -82,12 +91,12 @@ def test_delegates_to_are_tool(self, mock_are_tool): def test_records_invocation_in_history(self, mock_are_tool): """Test wrapper records invocation in history.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 100.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) wrapper(arg1="value") @@ -99,12 +108,12 @@ def test_records_invocation_in_history(self, mock_are_tool): def test_records_simulation_time(self, mock_are_tool): """Test wrapper records simulation time before and after.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.side_effect = [10.0, 15.0] # before, after - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) wrapper(arg1="value") @@ -116,14 +125,14 @@ def test_records_simulation_time(self, mock_are_tool): def test_handles_tool_error(self, mock_are_tool): """Test wrapper handles tool execution error.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_are_tool._return_value = lambda **kw: (_ for _ in ()).throw(ValueError("Tool error")) mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) with pytest.raises(ValueError): wrapper(arg1="value") @@ -136,12 +145,12 @@ def test_handles_tool_error(self, mock_are_tool): def test_handles_missing_simulation_time(self, mock_are_tool): """Test wrapper handles missing simulation time gracefully.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.side_effect = Exception("Time not available") - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) # Should not raise result = wrapper(arg1="value") @@ -152,17 +161,17 @@ def test_handles_missing_simulation_time(self, mock_are_tool): @pytest.mark.benchmark -class TestAREToolWrapperTracing: - """Tests for AREToolWrapper tracing methods.""" +class TestGaia2GenericToolTracing: + """Tests for Gaia2GenericTool tracing methods.""" def test_gather_traces_includes_invocations(self, mock_are_tool): """Test gather_traces includes invocation history.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) wrapper(arg1="first") wrapper(arg1="second") @@ -175,69 +184,72 @@ def test_gather_traces_includes_invocations(self, mock_are_tool): def test_gather_config_includes_schema(self, mock_are_tool): """Test gather_config includes tool configuration.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_are_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) config = wrapper.gather_config() - assert config["name"] == mock_are_tool.name - assert config["description"] == mock_are_tool.description - assert "inputs_schema" in config + assert config["name"] == mock_are_tool._public_name + assert "Acts on app TestTool:" in config["description"] + assert "input_schema" in config @pytest.mark.benchmark -class TestAREToolWrapperSchemaExtraction: - """Tests for schema extraction from different ARE tool formats.""" +class TestGaia2GenericToolSchemaExtraction: + """Tests for schema extraction from ARE's args attribute.""" - def test_extracts_from_inputs_attribute(self): - """Test extraction from 'inputs' attribute.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper - - mock_tool = MagicMock() - mock_tool.name = "test_tool" - mock_tool.description = "Test" - mock_tool.inputs = {"key": "value"} + def test_extracts_from_are_args_attribute(self, mock_are_tool): + """Test extraction from ARE's 'args' attribute produces flat inputs dict.""" + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) - assert wrapper.inputs == {"key": "value"} + # inputs is flat dict (ARE tool_utils.py:572-578) + assert "param1" in wrapper.inputs + assert wrapper.inputs["param1"]["type"] == "string" + assert wrapper.inputs["param1"]["description"] == "First parameter" - def test_extracts_from_parameters_attribute(self): - """Test extraction from 'parameters' attribute when 'inputs' is absent.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + # input_schema keeps JSON schema format for tracing + assert "properties" in wrapper.input_schema + assert "param1" in wrapper.input_schema["properties"] - # Use spec to ensure tool only has specified attributes (no 'inputs') - mock_tool = MagicMock(spec=["name", "description", "parameters", "__call__"]) - mock_tool.name = "test_tool" - mock_tool.description = "Test" - mock_tool.parameters = {"param": "schema"} + def test_extracts_description_with_app_prefix(self, mock_are_tool): + """Test description includes app name prefix and return type suffix.""" + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_tool, mock_env) + wrapper = Gaia2GenericTool(mock_are_tool, mock_env) - assert wrapper.inputs == {"param": "schema"} + # ARE tool_utils.py:550 — prefix with app name + assert wrapper.description.startswith("Acts on app TestTool: ") + # ARE tool_utils.py:547-548 — includes public description + assert "A test tool that does something" in wrapper.description - def test_handles_missing_schema(self): - """Test handling when no schema is available.""" - from maseval.benchmark.gaia2.tool_wrapper import AREToolWrapper + def test_handles_empty_args(self): + """Test handling when args list is empty.""" + from maseval.benchmark.gaia2.tool_wrapper import Gaia2GenericTool - mock_tool = MagicMock(spec=[]) # No attributes + mock_tool = MagicMock(spec=["name", "_public_name", "_public_description", "app_name", "return_type", "args", "__call__"]) mock_tool.name = "test_tool" - mock_tool.description = "Test" + mock_tool._public_name = "test_tool" + mock_tool._public_description = "Test" + mock_tool.app_name = "TestApp" + mock_tool.return_type = None + mock_tool.args = [] mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 - wrapper = AREToolWrapper(mock_tool, mock_env) + wrapper = Gaia2GenericTool(mock_tool, mock_env) assert wrapper.inputs == {} @@ -253,7 +265,7 @@ class TestWrapAreTools: def test_wraps_multiple_tools(self, mock_are_tools): """Test wrapping multiple ARE tools.""" - from maseval.benchmark.gaia2.tool_wrapper import wrap_are_tools, AREToolWrapper + from maseval.benchmark.gaia2.tool_wrapper import wrap_are_tools, Gaia2GenericTool mock_env = MagicMock() mock_env.get_simulation_time.return_value = 0.0 @@ -263,7 +275,7 @@ def test_wraps_multiple_tools(self, mock_are_tools): assert len(wrapped) == len(mock_are_tools) for tool in mock_are_tools: assert tool.name in wrapped - assert isinstance(wrapped[tool.name], AREToolWrapper) + assert isinstance(wrapped[tool.name], Gaia2GenericTool) def test_returns_dict_keyed_by_name(self, mock_are_tools): """Test wrapped tools are keyed by name.""" diff --git a/tests/test_benchmarks/test_multiagentbench/conftest.py b/tests/test_benchmarks/test_multiagentbench/conftest.py index b9dc97c..cebb5ad 100644 --- a/tests/test_benchmarks/test_multiagentbench/conftest.py +++ b/tests/test_benchmarks/test_multiagentbench/conftest.py @@ -15,10 +15,61 @@ import pytest from typing import Any, Dict, List, Optional, Sequence, Tuple -from unittest.mock import MagicMock +from unittest.mock import MagicMock, patch from conftest import DummyModelAdapter from maseval import AgentAdapter, Task, MessageHistory +from maseval.benchmark.multiagentbench.environment import MultiAgentBenchEnvironment + + +# ============================================================================= +# Session-Scoped Setup +# ============================================================================= + + +@pytest.fixture(scope="session") +def ensure_marble_data(): + """Ensure MARBLE data is available, downloading if necessary. + + Uses ensure_marble_exists() which caches: skips clone when marble/ already exists. + + Tests that need real MARBLE data should depend on this fixture and be marked + @pytest.mark.live. + Tests that don't need data (structural, mock-based) should NOT depend on this fixture. + + Returns: + Path to the MARBLE directory + """ + from maseval.benchmark.multiagentbench.data_loader import ensure_marble_exists + + return ensure_marble_exists(auto_download=True) + + +@pytest.fixture(autouse=True) +def _mock_marble_environment(): + """Mock MARBLE environment creation for all Tier 1 (offline) tests. + + MultiAgentBenchEnvironment.setup_state() calls _create_marble_environment() + which imports marble — a vendored dependency not available in CI. This fixture + replaces that method with a MagicMock so structural tests can exercise the full + benchmark pipeline without marble installed. + + Tests that need the real _create_marble_environment can override this fixture + with a no-op at the class or file level (standard pytest fixture scoping). + """ + mock_env = MagicMock() + mock_env.is_done.return_value = False + mock_env.is_task_completed.return_value = False + mock_env.get_state.return_value = {} + mock_env._action_handlers = {} + mock_env.action_handler_descriptions = {} + + with patch.object( + MultiAgentBenchEnvironment, + "_create_marble_environment", + return_value=mock_env, + ): + yield # ============================================================================= @@ -226,7 +277,7 @@ def __init__( if model_factory is None: self._model_factory = lambda model_name: DummyModelAdapter( model_id=f"test-model-{model_name}", - responses=['{"rating": 4}'], + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], ) elif callable(model_factory): self._model_factory = model_factory diff --git a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py index 11670d4..7d60887 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_benchmark.py +++ b/tests/test_benchmarks/test_multiagentbench/test_benchmark.py @@ -1,7 +1,9 @@ """Tests for MultiAgentBench benchmark classes.""" +import sys + import pytest -from unittest.mock import MagicMock +from unittest.mock import MagicMock, patch from maseval import Task from maseval.benchmark.multiagentbench import ( @@ -10,6 +12,8 @@ MultiAgentBenchEvaluator, ) +pytestmark = pytest.mark.benchmark + class TestMultiAgentBenchBenchmark: """Tests for MultiAgentBenchBenchmark abstract class.""" @@ -450,10 +454,18 @@ def test_setup_agents_raises_import_error( benchmark = marble_benchmark_class(progress_bar=False) env = benchmark.setup_environment({}, sample_research_task, seed_gen) - with pytest.raises(ImportError, match="MARBLE is not available"): - benchmark.setup_agents({}, env, sample_research_task, None, seed_gen) + # Temporarily remove marble modules to simulate MARBLE not being available + marble_modules = {k: v for k, v in sys.modules.items() if "marble" in k} + for module_name in marble_modules: + sys.modules.pop(module_name, None) + + try: + with patch.dict("sys.modules", {"marble.agent.base_agent": None}): + with pytest.raises(ImportError, match="MARBLE is not available"): + benchmark.setup_agents({}, env, sample_research_task, None, seed_gen) + finally: + sys.modules.update(marble_modules) - @pytest.mark.xfail(reason="MARBLE is vendored and always available; test assumes it is not installed") def test_create_marble_env_raises_import_error( self, marble_benchmark_class, @@ -462,19 +474,36 @@ def test_create_marble_env_raises_import_error( """_create_marble_env should raise ImportError when MARBLE not available.""" benchmark = marble_benchmark_class(progress_bar=False) - with pytest.raises(ImportError, match="MARBLE is not available"): - benchmark._create_marble_env(sample_research_task) - - def test_setup_agent_graph_silently_fails( + # Mock MARBLE import to simulate it not being available + # Temporarily remove marble modules from sys.modules + marble_modules = {k: v for k, v in sys.modules.items() if "marble" in k} + for module_name in marble_modules: + sys.modules.pop(module_name, None) + + try: + # Patch the import to raise ImportError + with patch.dict("sys.modules", {"marble.environments.base_env": None}): + with pytest.raises(ImportError, match="MARBLE is not available"): + benchmark._create_marble_env(sample_research_task) + finally: + # Restore marble modules + sys.modules.update(marble_modules) + + def test_setup_agent_graph_with_missing_agents_raises( self, marble_benchmark_class, sample_research_task: Task, ): - """_setup_agent_graph should not raise when MARBLE not available.""" + """_setup_agent_graph should raise when agents referenced in relationships don't exist.""" benchmark = marble_benchmark_class(progress_bar=False) - # Should not raise, just return silently - benchmark._setup_agent_graph({}, sample_research_task, None) + # Mock AgentGraph so this test works without MARBLE installed. + # The real AgentGraph validates that relationship agents exist; simulate that. + mock_agent_graph_cls = MagicMock(side_effect=ValueError("Agent 'agent1' does not exist in the graph")) + + with patch.dict("sys.modules", {"marble.graph.agent_graph": MagicMock(AgentGraph=mock_agent_graph_cls)}): + with pytest.raises(ValueError, match="does not exist"): + benchmark._setup_agent_graph({}, sample_research_task, None) def test_run_agents_returns_structured_output( self, diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_integrity.py b/tests/test_benchmarks/test_multiagentbench/test_data_integrity.py new file mode 100644 index 0000000..56730bc --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_data_integrity.py @@ -0,0 +1,213 @@ +"""Data integrity tests for MultiAgentBench benchmark. + +These tests validate that MARBLE data (either locally cloned or freshly +downloaded) is structurally sound. They are marked ``live`` (network may +be required for initial clone) and ``slow`` (git clone takes time) so +that they are excluded from the default fast test suite. + +Run with:: + + pytest -m "live and slow" tests/test_benchmarks/test_multiagentbench/test_data_integrity.py -v +""" + +import json + +import pytest + +from maseval.benchmark.multiagentbench.data_loader import ( + VALID_DOMAINS, + load_tasks, +) + +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark] + +# JSONL domains that can be loaded with load_tasks(). +# Werewolf uses config-based loading (no JSONL data). +# Minecraft has JSONL data but entries lack 'scenario' and 'task_id' fields +# required by _parse_task_entry() — a pre-existing data loader limitation. +JSONL_DOMAINS = sorted(VALID_DOMAINS - {"werewolf", "minecraft"}) + +# Expected minimum task count per domain. MARBLE has 100 tasks per JSONL domain. +MIN_TASKS_PER_DOMAIN = 50 + + +# ============================================================================= +# Fixture: ensure MARBLE is available +# ============================================================================= + + +@pytest.fixture(scope="module") +def marble_dir(): + """Ensure MARBLE data is available. + + Uses ensure_marble_exists() which reuses an existing clone if present, + or downloads from GitHub if not. + """ + from maseval.benchmark.multiagentbench.data_loader import ensure_marble_exists + + return ensure_marble_exists(auto_download=True) + + +@pytest.fixture(scope="module") +def marble_data_dir(marble_dir): + """Resolve the MARBLE multiagentbench data directory.""" + data_dir = marble_dir / "multiagentbench" + assert data_dir.exists(), f"MARBLE multiagentbench directory not found at {data_dir}. The MARBLE clone may be incomplete." + return data_dir + + +# ============================================================================= +# MARBLE Data Presence +# ============================================================================= + + +class TestMarbleDataPresence: + """Validate that MARBLE data files exist for all domains.""" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_domain_directory_exists(self, domain, marble_data_dir): + """Domain directory exists in MARBLE multiagentbench/.""" + domain_dir = marble_data_dir / domain + assert domain_dir.exists(), ( + f"Domain directory missing: {domain_dir}. MARBLE clone may be incomplete or the domain was renamed upstream." + ) + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_domain_has_jsonl(self, domain, marble_data_dir): + """Domain has its {domain}_main.jsonl file.""" + jsonl_path = marble_data_dir / domain / f"{domain}_main.jsonl" + assert jsonl_path.exists(), f"JSONL file missing: {jsonl_path}. MARBLE data structure may have changed upstream." + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_jsonl_is_non_empty(self, domain, marble_data_dir): + """JSONL file has content.""" + jsonl_path = marble_data_dir / domain / f"{domain}_main.jsonl" + assert jsonl_path.stat().st_size > 100, f"JSONL file suspiciously small ({jsonl_path.stat().st_size} bytes): {jsonl_path}" + + +# ============================================================================= +# Task Structure +# ============================================================================= + + +class TestMarbleTaskStructure: + """Validate loaded tasks have expected structure.""" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_minimum_task_count(self, domain, marble_data_dir): + """Each JSONL domain has at least the expected number of tasks.""" + tasks = load_tasks(domain, data_dir=marble_data_dir) + assert len(tasks) >= MIN_TASKS_PER_DOMAIN, ( + f"Domain '{domain}' has {len(tasks)} tasks, expected >= {MIN_TASKS_PER_DOMAIN}. This may indicate upstream data loss." + ) + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_required_fields_in_environment_data(self, domain, marble_data_dir): + """Every task has required fields in environment_data.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=10) + for task in tasks: + assert "scenario" in task.environment_data, f"Task {task.id} missing 'scenario' in environment_data" + assert "agents" in task.environment_data, f"Task {task.id} missing 'agents' in environment_data" + assert "relationships" in task.environment_data, f"Task {task.id} missing 'relationships' in environment_data" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_agents_have_ids(self, domain, marble_data_dir): + """Every agent in every task has an agent_id.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=10) + for task in tasks: + agents = task.environment_data.get("agents", []) + assert len(agents) > 0, f"Task {task.id} in domain '{domain}' has no agents. MultiAgentBench tasks require at least one agent." + for agent in agents: + assert "agent_id" in agent, f"Agent in task {task.id} missing 'agent_id': {agent}" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_tasks_have_queries(self, domain, marble_data_dir): + """Every task has a non-empty query.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=10) + for task in tasks: + assert task.query, f"Task {task.id} in domain '{domain}' has empty query" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_tasks_have_metadata_domain(self, domain, marble_data_dir): + """Every task records its domain in metadata.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=5) + for task in tasks: + assert task.metadata.get("domain") == domain, ( + f"Task {task.id} metadata domain is '{task.metadata.get('domain')}', expected '{domain}'" + ) + + +# ============================================================================= +# JSONL Raw Data Validation +# ============================================================================= + + +class TestMarbleRawJsonl: + """Validate raw JSONL files parse correctly and have required schema.""" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_jsonl_entries_parse(self, domain, marble_data_dir): + """Every line in the JSONL file is valid JSON.""" + jsonl_path = marble_data_dir / domain / f"{domain}_main.jsonl" + with jsonl_path.open(encoding="utf-8") as f: + for idx, line in enumerate(f): + line = line.strip() + if not line: + continue + try: + entry = json.loads(line) + except json.JSONDecodeError as e: + pytest.fail(f"Line {idx} in {jsonl_path.name} is invalid JSON: {e}") + assert isinstance(entry, dict), f"Line {idx} in {jsonl_path.name} is not a JSON object" + + @pytest.mark.parametrize("domain", JSONL_DOMAINS) + def test_jsonl_required_fields(self, domain, marble_data_dir): + """Raw JSONL entries have the fields required by _parse_task_entry().""" + jsonl_path = marble_data_dir / domain / f"{domain}_main.jsonl" + required = {"scenario", "task_id", "task", "agents", "relationships"} + + with jsonl_path.open(encoding="utf-8") as f: + for idx, line in enumerate(f): + line = line.strip() + if not line: + continue + entry = json.loads(line) + missing = required - set(entry.keys()) + assert not missing, f"Line {idx} in {domain}_main.jsonl missing fields: {missing}" + if idx >= 9: # Check first 10 entries + break + + +# ============================================================================= +# Werewolf Config Files +# ============================================================================= + + +class TestMarbleWerewolfConfigs: + """Validate werewolf YAML config files.""" + + def test_werewolf_configs_directory_exists(self, marble_dir): + """MARBLE configs directory exists.""" + configs_dir = marble_dir / "marble" / "configs" + assert configs_dir.exists(), f"MARBLE configs directory not found at {configs_dir}. The MARBLE clone structure may have changed." + + def test_werewolf_config_files_exist(self, marble_dir): + """At least one werewolf config YAML exists.""" + configs_dir = marble_dir / "marble" / "configs" + werewolf_configs = list(configs_dir.glob("**/werewolf_config*.yaml")) + assert len(werewolf_configs) > 0, f"No werewolf config files found in {configs_dir}. Expected files matching: **/werewolf_config*.yaml" + + def test_werewolf_tasks_load(self, marble_data_dir): + """Werewolf tasks can be loaded from config files.""" + tasks = load_tasks("werewolf", data_dir=marble_data_dir) + assert len(tasks) > 0, "No werewolf tasks loaded. Check that werewolf config YAML files exist in marble/configs/." + + def test_werewolf_tasks_have_agents(self, marble_data_dir): + """Werewolf tasks have agent specifications with roles.""" + tasks = load_tasks("werewolf", data_dir=marble_data_dir) + for task in tasks: + agents = task.environment_data.get("agents", []) + assert len(agents) > 0, f"Werewolf task {task.id} has no agents" + for agent in agents: + assert "agent_id" in agent + assert "role" in agent, f"Werewolf agent {agent.get('agent_id')} missing 'role'" diff --git a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py index 555e9f0..959532b 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_data_loader.py +++ b/tests/test_benchmarks/test_multiagentbench/test_data_loader.py @@ -1,12 +1,12 @@ """Tests for MultiAgentBench data loading functionality.""" import json -import subprocess import tempfile from pathlib import Path from typing import Any, Dict from unittest.mock import MagicMock, patch +import git import pytest from maseval import Task @@ -20,15 +20,19 @@ VALID_DOMAINS, _parse_task_entry, _resolve_data_dir, + _load_werewolf_tasks, + _parse_werewolf_config_basic, ) +pytestmark = pytest.mark.benchmark + class TestValidDomains: """Tests for domain validation.""" def test_valid_domains_contains_expected(self): """VALID_DOMAINS should contain all expected domains.""" - expected = {"coding", "database", "minecraft", "research", "bargaining", "web", "worldsimulation"} + expected = {"coding", "database", "minecraft", "research", "bargaining", "werewolf"} assert expected == VALID_DOMAINS def test_valid_domains_is_frozen(self): @@ -364,54 +368,61 @@ def test_download_marble_force_removes_existing(self): with tempfile.TemporaryDirectory() as tmpdir: marble_dir = Path(tmpdir) / "marble" marble_dir.mkdir() - (marble_dir / "test_file.txt").write_text("test") + test_file = marble_dir / "test_file.txt" + test_file.write_text("test") - with patch("subprocess.run") as mock_run: - mock_run.return_value = MagicMock(returncode=0) + # Mock git operations + mock_repo = MagicMock() + with patch("git.Repo.clone_from", return_value=mock_repo) as mock_clone: download_marble(target_dir=marble_dir, force=True) - # Directory should have been removed and git clone called - mock_run.assert_called() + # Directory should have been removed (test file gone) + assert not test_file.exists() + # Git clone should have been called + mock_clone.assert_called_once() def test_download_marble_git_clone_called(self): """download_marble should call git clone.""" with tempfile.TemporaryDirectory() as tmpdir: marble_dir = Path(tmpdir) / "marble" - with patch("subprocess.run") as mock_run: - mock_run.return_value = MagicMock(returncode=0) + # Mock git operations + mock_repo = MagicMock() + with patch("git.Repo.clone_from", return_value=mock_repo) as mock_clone: download_marble(target_dir=marble_dir) - # Verify git clone was called - calls = mock_run.call_args_list - assert len(calls) >= 1 - clone_call = calls[0] - assert "git" in clone_call[0][0] - assert "clone" in clone_call[0][0] + # Verify git clone was called with correct arguments + mock_clone.assert_called_once() + args, kwargs = mock_clone.call_args + from maseval.benchmark.multiagentbench.data_loader import MARBLE_REPO_URL + + assert args[0] == MARBLE_REPO_URL + assert args[1] == str(marble_dir) def test_download_marble_with_commit(self): """download_marble should checkout specific commit if provided.""" with tempfile.TemporaryDirectory() as tmpdir: marble_dir = Path(tmpdir) / "marble" - with patch("subprocess.run") as mock_run: - mock_run.return_value = MagicMock(returncode=0) + # Mock git operations + mock_git = MagicMock() + mock_repo = MagicMock() + mock_repo.git = mock_git + + with patch("git.Repo.clone_from", return_value=mock_repo): download_marble(target_dir=marble_dir, commit="abc123") - # Verify git checkout was called - calls = mock_run.call_args_list - assert len(calls) >= 2 - checkout_call = calls[1] - assert "checkout" in checkout_call[0][0] - assert "abc123" in checkout_call[0][0] + # Verify git checkout was called with the commit + mock_git.checkout.assert_called_once_with("abc123") def test_download_marble_clone_fails(self): """download_marble should raise RuntimeError on clone failure.""" with tempfile.TemporaryDirectory() as tmpdir: marble_dir = Path(tmpdir) / "marble" - with patch("subprocess.run") as mock_run: - mock_run.side_effect = subprocess.CalledProcessError(1, "git clone", stderr="Clone failed") + with patch("git.Repo.clone_from") as mock_clone: + # Simulate git clone failure + mock_clone.side_effect = git.GitCommandError("clone", 1, stderr=b"Clone failed") with pytest.raises(RuntimeError, match="Failed to clone MARBLE"): download_marble(target_dir=marble_dir) @@ -421,10 +432,11 @@ def test_download_marble_git_not_found(self): with tempfile.TemporaryDirectory() as tmpdir: marble_dir = Path(tmpdir) / "marble" - with patch("subprocess.run") as mock_run: - mock_run.side_effect = FileNotFoundError() + with patch("git.Repo.clone_from") as mock_clone: + # Simulate git executable not found + mock_clone.side_effect = git.exc.GitCommandNotFound("git", "git: command not found") - with pytest.raises(RuntimeError, match="git is not installed"): + with pytest.raises(RuntimeError, match="Failed to clone MARBLE"): download_marble(target_dir=marble_dir) def test_download_marble_checkout_fails(self): @@ -432,13 +444,15 @@ def test_download_marble_checkout_fails(self): with tempfile.TemporaryDirectory() as tmpdir: marble_dir = Path(tmpdir) / "marble" - def mock_run_side_effect(*args, **kwargs): - cmd = args[0] - if "checkout" in cmd: - raise subprocess.CalledProcessError(1, "git checkout", stderr="Checkout failed") - return MagicMock(returncode=0) + # Mock git operations + mock_git = MagicMock() + mock_repo = MagicMock() + mock_repo.git = mock_git - with patch("subprocess.run", side_effect=mock_run_side_effect): + # Simulate checkout failure + mock_git.checkout.side_effect = git.GitCommandError("checkout", 1, stderr=b"Checkout failed") + + with patch("git.Repo.clone_from", return_value=mock_repo): with pytest.raises(RuntimeError, match="Failed to checkout commit"): download_marble(target_dir=marble_dir, commit="invalid") @@ -535,7 +549,7 @@ class TestGetDomainInfoAllDomains: @pytest.mark.parametrize( "domain", - ["coding", "database", "minecraft", "research", "bargaining", "web", "worldsimulation"], + ["coding", "database", "minecraft", "research", "bargaining", "werewolf"], ) def test_all_domains_have_info(self, domain): """All valid domains should return info.""" @@ -550,14 +564,158 @@ def test_coding_domain_info(self): assert info["coordination_mode"] == "tree" assert info["requires_infrastructure"] is False - def test_web_domain_info(self): - """Web domain should have star coordination.""" - info = get_domain_info("web") - assert info["coordination_mode"] == "star" - assert info["requires_infrastructure"] is False - - def test_worldsimulation_domain_info(self): - """WorldSimulation domain should have cooperative coordination.""" - info = get_domain_info("worldsimulation") + def test_werewolf_domain_info(self): + """Werewolf domain should have cooperative coordination.""" + info = get_domain_info("werewolf") assert info["coordination_mode"] == "cooperative" assert info["requires_infrastructure"] is False + + +class TestLoadWerewolfTasks: + """Tests for werewolf config-based task loading.""" + + def _create_werewolf_structure(self, tmpdir: Path) -> Path: + """Helper to create a mock MARBLE structure with werewolf config. + + Creates the expected structure where data_dir = tmpdir/multiagentbench/ + and data_dir.parent = tmpdir has marble/configs/. + + Returns: + data_dir (tmpdir/multiagentbench/) for passing to _load_werewolf_tasks + """ + # Create configs: tmpdir/marble/configs/test_config/werewolf_config/werewolf_config.yaml + configs_dir = tmpdir / "marble" / "configs" / "test_config" / "werewolf_config" + configs_dir.mkdir(parents=True) + + config_path = configs_dir / "werewolf_config.yaml" + config_content = ( + 'openai_api_key: "test"\nroles:\n - wolf\n - wolf\n - villager\n - villager\n - seer\ncooperation_mode: "cooperative"\n' + ) + config_path.write_text(config_content) + + # Create data_dir (simulating marble/multiagentbench/) + data_dir = tmpdir / "multiagentbench" + data_dir.mkdir() + + return data_dir + + def test_load_werewolf_tasks_finds_configs(self): + """_load_werewolf_tasks should find and parse werewolf configs.""" + with tempfile.TemporaryDirectory() as tmpdir: + data_dir = self._create_werewolf_structure(Path(tmpdir)) + + tasks = _load_werewolf_tasks(data_dir) + + assert len(tasks) == 1 + assert tasks[0].id == "werewolf_0" + assert tasks[0].metadata["domain"] == "werewolf" + assert tasks[0].environment_data["scenario"] == "werewolf" + + def test_load_werewolf_tasks_extracts_agents_from_roles(self): + """_load_werewolf_tasks should create agent specs from roles.""" + with tempfile.TemporaryDirectory() as tmpdir: + data_dir = self._create_werewolf_structure(Path(tmpdir)) + + tasks = _load_werewolf_tasks(data_dir) + + agents = tasks[0].environment_data["agents"] + assert len(agents) == 5 + assert agents[0]["role"] == "wolf" + assert agents[2]["role"] == "villager" + assert agents[4]["role"] == "seer" + + def test_load_werewolf_tasks_stores_config_path(self): + """_load_werewolf_tasks should store config path in task data.""" + with tempfile.TemporaryDirectory() as tmpdir: + data_dir = self._create_werewolf_structure(Path(tmpdir)) + + tasks = _load_werewolf_tasks(data_dir) + + config_path = tasks[0].environment_data["werewolf_config_path"] + assert "werewolf_config.yaml" in config_path + + def test_load_werewolf_tasks_with_limit(self): + """_load_werewolf_tasks should respect limit parameter.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_root = Path(tmpdir) + configs_dir = marble_root / "marble" / "configs" + + # Create two config files + for i in range(2): + cfg_dir = configs_dir / f"config_{i}" + cfg_dir.mkdir(parents=True) + cfg_path = cfg_dir / f"werewolf_config_{i}.yaml" + cfg_path.write_text("roles:\n - wolf\n - villager\n") + + # data_dir must be a subdir of marble_root + data_dir = marble_root / "multiagentbench" + data_dir.mkdir() + + tasks = _load_werewolf_tasks(data_dir, limit=1) + assert len(tasks) == 1 + + def test_load_werewolf_tasks_no_configs_raises(self): + """_load_werewolf_tasks should raise if no configs found.""" + with tempfile.TemporaryDirectory() as tmpdir: + marble_root = Path(tmpdir) + # Create configs dir but no werewolf configs + (marble_root / "marble" / "configs").mkdir(parents=True) + data_dir = marble_root / "multiagentbench" + data_dir.mkdir() + + with pytest.raises(FileNotFoundError, match="No werewolf config"): + _load_werewolf_tasks(data_dir) + + def test_load_werewolf_tasks_no_configs_dir_raises(self): + """_load_werewolf_tasks should raise if configs dir missing.""" + with tempfile.TemporaryDirectory() as tmpdir: + data_dir = Path(tmpdir) / "multiagentbench" + data_dir.mkdir() + + with pytest.raises(FileNotFoundError, match="configs directory not found"): + _load_werewolf_tasks(data_dir) + + def test_load_tasks_werewolf_domain_routes_to_config_loader(self): + """load_tasks('werewolf') should use config-based loading.""" + with tempfile.TemporaryDirectory() as tmpdir: + data_dir = self._create_werewolf_structure(Path(tmpdir)) + + tasks = load_tasks("werewolf", data_dir=data_dir) + assert len(tasks) == 1 + assert tasks[0].metadata["domain"] == "werewolf" + + +class TestParseWerewolfConfigBasic: + """Tests for _parse_werewolf_config_basic fallback parser.""" + + def test_parse_key_value_pairs(self): + """Should parse simple key-value pairs.""" + with tempfile.TemporaryDirectory() as tmpdir: + config_path = Path(tmpdir) / "test.yaml" + config_path.write_text('cooperation_mode: "cooperative"\nuse_random_names: True\n') + + config = _parse_werewolf_config_basic(config_path) + + assert config["cooperation_mode"] == "cooperative" + assert config["use_random_names"] == "True" + + def test_parse_list_values(self): + """Should parse list values.""" + with tempfile.TemporaryDirectory() as tmpdir: + config_path = Path(tmpdir) / "test.yaml" + config_path.write_text("roles:\n - wolf\n - villager\n - seer\n") + + config = _parse_werewolf_config_basic(config_path) + + assert config["roles"] == ["wolf", "villager", "seer"] + + def test_parse_skips_comments_and_empty_lines(self): + """Should skip comments and empty lines.""" + with tempfile.TemporaryDirectory() as tmpdir: + config_path = Path(tmpdir) / "test.yaml" + config_path.write_text("# Comment\n\nkey: value\n") + + config = _parse_werewolf_config_basic(config_path) + + assert config["key"] == "value" + assert len(config) == 1 diff --git a/tests/test_benchmarks/test_multiagentbench/test_environment.py b/tests/test_benchmarks/test_multiagentbench/test_environment.py index b89e2fc..b708fc6 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_environment.py +++ b/tests/test_benchmarks/test_multiagentbench/test_environment.py @@ -1,15 +1,19 @@ """Tests for MultiAgentBench environment.""" -import pytest +import sys from typing import Any, Dict from unittest.mock import patch, MagicMock +import pytest + from maseval.benchmark.multiagentbench.environment import ( MultiAgentBenchEnvironment, INFRASTRUCTURE_DOMAINS, ) from maseval import EnvironmentError +pytestmark = pytest.mark.benchmark + class TestInfrastructureDomains: """Tests for infrastructure domain constants.""" @@ -17,12 +21,12 @@ class TestInfrastructureDomains: def test_infrastructure_domains_contains_expected(self): """INFRASTRUCTURE_DOMAINS should contain expected domains.""" assert "database" in INFRASTRUCTURE_DOMAINS - assert "minecraft" in INFRASTRUCTURE_DOMAINS def test_infrastructure_domains_excludes_simple(self): """INFRASTRUCTURE_DOMAINS should not include simple domains.""" assert "research" not in INFRASTRUCTURE_DOMAINS assert "bargaining" not in INFRASTRUCTURE_DOMAINS + assert "minecraft" not in INFRASTRUCTURE_DOMAINS class TestMultiAgentBenchEnvironment: @@ -59,36 +63,14 @@ def test_is_done_initially_false(self, sample_research_task_data: Dict[str, Any] """is_done should return False initially.""" env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) - # Without MARBLE env, always returns False assert env.is_done() is False def test_is_task_completed_initially_false(self, sample_research_task_data: Dict[str, Any]): """is_task_completed should return False initially.""" env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) - # Without MARBLE env, always returns False assert env.is_task_completed() is False - @pytest.mark.xfail(reason="MARBLE is vendored and always available; test assumes it is not installed") - def test_get_marble_state_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): - """get_marble_state should return empty dict without MARBLE.""" - env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) - - assert env.get_marble_state() == {} - - def test_get_tool_descriptions_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): - """get_tool_descriptions should return empty dict without MARBLE.""" - env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) - - assert env.get_tool_descriptions() == {} - - def test_create_tools_empty_without_marble(self, sample_research_task_data: Dict[str, Any]): - """create_tools should return empty dict without MARBLE.""" - env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) - tools = env.create_tools() - - assert tools == {} - def test_gather_traces_includes_domain(self, sample_research_task_data: Dict[str, Any]): """gather_traces should include domain information.""" env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) @@ -96,6 +78,8 @@ def test_gather_traces_includes_domain(self, sample_research_task_data: Dict[str assert traces["domain"] == "research" assert "tool_invocations" in traces + assert "marble_state" in traces + assert "is_done" in traces def test_gather_config_includes_domain(self, sample_research_task_data: Dict[str, Any]): """gather_config should include domain information.""" @@ -106,6 +90,36 @@ def test_gather_config_includes_domain(self, sample_research_task_data: Dict[str assert "tool_descriptions" in config +class TestMultiAgentBenchEnvironmentRealMarble: + """Tests that need the real _create_marble_environment (no mock).""" + + @pytest.fixture(autouse=True) + def _mock_marble_environment(self): + """Override: let these tests use real marble imports.""" + yield + + def test_init_raises_without_marble(self, sample_research_task_data: Dict[str, Any]): + """Constructor should raise ImportError when MARBLE is not available.""" + marble_modules = {k: v for k, v in sys.modules.items() if "marble" in k} + for module_name in marble_modules: + sys.modules.pop(module_name, None) + + try: + with patch.dict("sys.modules", {"marble.environments.base_env": None}): + with pytest.raises(ImportError, match="MARBLE is not available"): + MultiAgentBenchEnvironment(task_data=sample_research_task_data) + finally: + sys.modules.update(marble_modules) + + @pytest.mark.live + @pytest.mark.slow + def test_marble_env_type_in_state(self, sample_research_task_data: Dict[str, Any]): + """setup_state should include MARBLE env type (needs real marble).""" + env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) + + assert env.state["marble_env_type"] == "ResearchEnvironment" + + class TestInfrastructureCheck: """Tests for infrastructure checking.""" @@ -122,8 +136,8 @@ def test_database_without_docker_raises(self): with pytest.raises(EnvironmentError, match="requires external infrastructure"): MultiAgentBenchEnvironment(task_data=task_data) - def test_database_with_docker_succeeds(self): - """Environment should succeed for database with Docker.""" + def test_database_with_docker_passes_infra_check(self): + """Database domain should pass infrastructure check when Docker is available.""" task_data = { "scenario": "database", "environment": {"type": "DB"}, @@ -132,16 +146,13 @@ def test_database_with_docker_succeeds(self): } with patch("shutil.which", return_value="/usr/bin/docker"): - # Should not raise, but MARBLE env creation may still fail - try: + # Mock _create_marble_environment since DBEnvironment.__init__ starts Docker + DB + with patch.object(MultiAgentBenchEnvironment, "_create_marble_environment", return_value=MagicMock()): env = MultiAgentBenchEnvironment(task_data=task_data) assert env.domain == "database" - except ImportError: - # Expected if MARBLE not available - pass - def test_minecraft_always_raises(self): - """Environment should raise for minecraft (not supported).""" + def test_minecraft_no_infrastructure_check(self): + """Minecraft should not require infrastructure check (fails at runtime instead).""" task_data = { "scenario": "minecraft", "environment": {"type": "Minecraft"}, @@ -149,20 +160,8 @@ def test_minecraft_always_raises(self): "agents": [{"agent_id": "agent1"}], } - with pytest.raises(EnvironmentError, match="requires external infrastructure"): - MultiAgentBenchEnvironment(task_data=task_data) - - -class TestApplyAction: - """Tests for apply_action method.""" - - @pytest.mark.xfail(reason="MARBLE is vendored and always available; test assumes it is not installed") - def test_apply_action_without_marble_raises(self, sample_research_task_data: Dict[str, Any]): - """apply_action should raise without MARBLE environment.""" - env = MultiAgentBenchEnvironment(task_data=sample_research_task_data) - - with pytest.raises(EnvironmentError, match="not available"): - env.apply_action("agent1", "some_action", {"arg": "value"}) + env = MultiAgentBenchEnvironment(task_data=task_data) + assert env.domain == "minecraft" class TestWithMockedMarbleEnv: diff --git a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py index a632771..179d95f 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_evaluator.py +++ b/tests/test_benchmarks/test_multiagentbench/test_evaluator.py @@ -4,10 +4,13 @@ from conftest import DummyModelAdapter from maseval.benchmark.multiagentbench.evaluator import ( + DEFAULT_RESULT_TRUNCATION_LENGTH, MultiAgentBenchEvaluator, MultiAgentBenchMetrics, ) +pytestmark = pytest.mark.benchmark + class TestMultiAgentBenchMetrics: """Tests for MultiAgentBenchMetrics dataclass.""" @@ -136,20 +139,17 @@ def test_parse_score_with_markdown(self, research_evaluator): assert score == 5.0 - def test_parse_score_no_json_returns_none(self, research_evaluator): - """_parse_score should return None when no valid JSON found.""" + def test_parse_score_no_json_raises(self, research_evaluator): + """_parse_score should raise when no valid JSON found.""" response = "The rating is 4 out of 5" - score = research_evaluator._parse_score(response) - - # No regex fallback - returns None for non-JSON responses - assert score is None - - def test_parse_score_default(self, research_evaluator): - """_parse_score should return None when parsing fails.""" - response = "No score here" - score = research_evaluator._parse_score(response) + with pytest.raises(ValueError, match="No JSON object found"): + research_evaluator._parse_score(response) - assert score is None + def test_parse_score_no_rating_key_raises(self, research_evaluator): + """_parse_score should raise when JSON has no rating key.""" + response = '{"score": 4}' + with pytest.raises(ValueError, match="Expected"): + research_evaluator._parse_score(response) def test_parse_research_ratings_valid(self, research_evaluator): """_parse_research_ratings should parse valid ratings.""" @@ -161,13 +161,9 @@ def test_parse_research_ratings_valid(self, research_evaluator): assert ratings["feasibility"] == 5 def test_parse_research_ratings_invalid(self, research_evaluator): - """_parse_research_ratings should return None for invalid.""" - response = "Invalid response" - ratings = research_evaluator._parse_research_ratings(response) - - assert ratings["innovation"] is None - assert ratings["safety"] is None - assert ratings["feasibility"] is None + """_parse_research_ratings should raise for invalid response.""" + with pytest.raises(ValueError, match="No JSON object found"): + research_evaluator._parse_research_ratings("Invalid response") def test_determine_completion_research_positive(self, research_evaluator): """_determine_completion should return True for positive research scores.""" @@ -187,10 +183,13 @@ def test_call_returns_expected_structure(self, research_evaluator): } final_answer = [{"agent_id": "agent1", "result": "Done"}] - # Mock model adapter to return research ratings + # Needs two responses: one for LLM summarization, one for research evaluation research_evaluator.model_adapter = DummyModelAdapter( model_id="test", - responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + responses=[ + '{"summary": "Agent completed the task"}', + '{"innovation": 4, "safety": 4, "feasibility": 4}', + ], ) result = research_evaluator(traces, final_answer) @@ -329,12 +328,9 @@ def test_determine_completion_coding_negative(self, coding_evaluator): assert coding_evaluator._determine_completion(metrics) is False def test_parse_coding_ratings_invalid(self, coding_evaluator): - """_parse_coding_ratings should return None for invalid response.""" - ratings = coding_evaluator._parse_coding_ratings("Invalid JSON") - assert ratings["instruction_following"] is None - assert ratings["executability"] is None - assert ratings["consistency"] is None - assert ratings["quality"] is None + """_parse_coding_ratings should raise for invalid response.""" + with pytest.raises(ValueError, match="No JSON object found"): + coding_evaluator._parse_coding_ratings("Invalid JSON") class TestDatabaseEvaluation: @@ -379,61 +375,181 @@ def test_call_database_domain(self, database_evaluator): } final_answer = [{"agent_id": "agent1", "result": "SELECT * FROM orders"}] + # Needs a response for the LLM summarization call + database_evaluator.model_adapter = DummyModelAdapter( + model_id="test", + responses=["SELECT * FROM orders"], + ) + result = database_evaluator(traces, final_answer) assert result["domain"] == "database" assert "passed" in result -class TestWorldSimulationEvaluation: - """Tests for worldsimulation domain (alias for bargaining).""" +class TestWerewolfEvaluation: + """Tests for werewolf-specific evaluation.""" @pytest.fixture - def worldsim_evaluator(self): - """Create evaluator for worldsimulation domain.""" + def werewolf_evaluator(self): + """Create evaluator with werewolf-specific responses.""" adapter = DummyModelAdapter( model_id="test", responses=[ - '{"effectiveness_of_strategies": 4, "progress_and_outcome": 4, "interaction_dynamics": 4}', - '{"effectiveness_of_strategies": 4, "progress_and_outcome": 4, "interaction_dynamics": 4}', + '{"game_outcome": 4, "deception_detection": 3, "voting_strategy": 4, ' + '"role_fulfillment": 5, "information_usage": 3, "collaboration": 4, "survival_rate": 3}', ], ) return MultiAgentBenchEvaluator( - domain="worldsimulation", + domain="werewolf", model_adapter=adapter, ) - def test_worldsimulation_uses_bargaining_eval(self, worldsim_evaluator): - """worldsimulation domain should use bargaining evaluation.""" + def test_evaluate_werewolf_returns_all_metrics(self, werewolf_evaluator): + """_evaluate_werewolf should return all werewolf metrics.""" + ratings = werewolf_evaluator._evaluate_werewolf("Task", "Result") + + assert "game_outcome" in ratings + assert "deception_detection" in ratings + assert "voting_strategy" in ratings + assert "role_fulfillment" in ratings + assert "information_usage" in ratings + assert "collaboration" in ratings + assert "survival_rate" in ratings + + def test_determine_completion_werewolf_positive(self, werewolf_evaluator): + """_determine_completion should work for werewolf domain.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "game_outcome": 4, + "deception_detection": 3, + "voting_strategy": 4, + "role_fulfillment": 5, + "information_usage": 3, + "collaboration": 4, + "survival_rate": 3, + } + ) + assert werewolf_evaluator._determine_completion(metrics) is True + + def test_determine_completion_werewolf_negative(self, werewolf_evaluator): + """_determine_completion should return False for None werewolf scores.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "game_outcome": 4, + "deception_detection": None, + "voting_strategy": 4, + "role_fulfillment": 5, + "information_usage": 3, + "collaboration": 4, + "survival_rate": 3, + } + ) + assert werewolf_evaluator._determine_completion(metrics) is False + + def test_call_werewolf_domain(self, werewolf_evaluator): + """__call__ should work for werewolf domain.""" traces = { "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, "environment": {}, } - final_answer = "Simulation result" + final_answer = "Villagers won the game" + + result = werewolf_evaluator(traces, final_answer) + + assert result["domain"] == "werewolf" + assert "passed" in result + assert "game_outcome" in result["metrics"]["task_evaluation"] + + def test_parse_werewolf_ratings_invalid(self, werewolf_evaluator): + """_parse_werewolf_ratings should raise for invalid response.""" + with pytest.raises(ValueError, match="No JSON object found"): + werewolf_evaluator._parse_werewolf_ratings("Invalid JSON") - result = worldsim_evaluator(traces, final_answer) - assert result["domain"] == "worldsimulation" - assert "buyer" in result["metrics"]["task_evaluation"] - assert "seller" in result["metrics"]["task_evaluation"] +class TestMinecraftEvaluation: + """Tests for minecraft-specific evaluation.""" - def test_determine_completion_worldsimulation(self, worldsim_evaluator): - """_determine_completion should work for worldsimulation domain.""" + @pytest.fixture + def minecraft_evaluator(self): + """Create evaluator with minecraft-specific responses.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"structural_completeness": 4, "blueprint_accuracy": 3, "coordination": 5, "efficiency": 4}', + ], + ) + return MultiAgentBenchEvaluator( + domain="minecraft", + model_adapter=adapter, + ) + + def test_evaluate_minecraft_returns_all_metrics(self, minecraft_evaluator): + """_evaluate_minecraft should return all minecraft metrics.""" + ratings = minecraft_evaluator._evaluate_minecraft("Task", "Result") + + assert "structural_completeness" in ratings + assert "blueprint_accuracy" in ratings + assert "coordination" in ratings + assert "efficiency" in ratings + + def test_determine_completion_minecraft_positive(self, minecraft_evaluator): + """_determine_completion should work for minecraft domain.""" metrics = MultiAgentBenchMetrics( task_evaluation={ - "buyer": { - "effectiveness_of_strategies": 4, - "progress_and_outcome": 4, - "interaction_dynamics": 4, - }, - "seller": { - "effectiveness_of_strategies": 4, - "progress_and_outcome": 4, - "interaction_dynamics": 4, - }, + "structural_completeness": 4, + "blueprint_accuracy": 3, + "coordination": 5, + "efficiency": 4, } ) - assert worldsim_evaluator._determine_completion(metrics) is True + assert minecraft_evaluator._determine_completion(metrics) is True + + def test_determine_completion_minecraft_negative(self, minecraft_evaluator): + """_determine_completion should return False for None minecraft scores.""" + metrics = MultiAgentBenchMetrics( + task_evaluation={ + "structural_completeness": 4, + "blueprint_accuracy": None, + "coordination": 5, + "efficiency": 4, + } + ) + assert minecraft_evaluator._determine_completion(metrics) is False + + def test_call_minecraft_domain(self, minecraft_evaluator): + """__call__ should work for minecraft domain.""" + traces = { + "agents": {"agent1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + final_answer = "Structure built successfully" + + result = minecraft_evaluator(traces, final_answer) + + assert result["domain"] == "minecraft" + assert "passed" in result + assert "structural_completeness" in result["metrics"]["task_evaluation"] + + def test_parse_minecraft_ratings_invalid(self, minecraft_evaluator): + """_parse_minecraft_ratings should raise for invalid response.""" + with pytest.raises(ValueError, match="No JSON object found"): + minecraft_evaluator._parse_minecraft_ratings("Invalid JSON") + + def test_evaluate_minecraft_on_error(self): + """_evaluate_minecraft should propagate LLM errors.""" + from unittest.mock import MagicMock + + mock_adapter = MagicMock() + mock_adapter.generate.side_effect = RuntimeError("Model failed") + + evaluator = MultiAgentBenchEvaluator( + domain="minecraft", + model_adapter=mock_adapter, + ) + + with pytest.raises(RuntimeError, match="Model failed"): + evaluator._evaluate_minecraft("Task", "Result") class TestUnknownDomainEvaluation: @@ -492,7 +608,7 @@ def test_evaluate_communication_returns_score(self, evaluator_with_comm_response assert score == 4.0 def test_evaluate_communication_on_error(self): - """_evaluate_communication should return None on error.""" + """_evaluate_communication should propagate LLM errors.""" from unittest.mock import MagicMock # Create mock that raises exception on generate @@ -504,8 +620,8 @@ def test_evaluate_communication_on_error(self): model_adapter=mock_adapter, ) - score = evaluator._evaluate_communication("Task", "Comms") - assert score is None + with pytest.raises(RuntimeError, match="Model failed"): + evaluator._evaluate_communication("Task", "Comms") def test_call_with_communications(self, evaluator_with_comm_response): """__call__ should evaluate communications when present.""" @@ -544,25 +660,21 @@ def failing_evaluator(self): ) def test_evaluate_research_on_error(self, failing_evaluator): - """_evaluate_research should return default values on error.""" - result = failing_evaluator._evaluate_research("Task", "Result") - assert result["innovation"] is None - assert result["safety"] is None - assert result["feasibility"] is None + """_evaluate_research should propagate LLM errors.""" + with pytest.raises(RuntimeError, match="Model failed"): + failing_evaluator._evaluate_research("Task", "Result") def test_evaluate_bargaining_on_error(self, failing_evaluator): - """_evaluate_bargaining should return default values on error.""" + """_evaluate_bargaining should propagate LLM errors.""" failing_evaluator.domain = "bargaining" - result = failing_evaluator._evaluate_bargaining("Task", "Result") - assert result["buyer"]["effectiveness_of_strategies"] is None - assert result["seller"]["effectiveness_of_strategies"] is None + with pytest.raises(RuntimeError, match="Model failed"): + failing_evaluator._evaluate_bargaining("Task", "Result") def test_evaluate_coding_on_error(self, failing_evaluator): - """_evaluate_coding should return default values on error.""" + """_evaluate_coding should propagate LLM errors.""" failing_evaluator.domain = "coding" - result = failing_evaluator._evaluate_coding("Task", "Result") - assert result["instruction_following"] is None - assert result["executability"] is None + with pytest.raises(RuntimeError, match="Model failed"): + failing_evaluator._evaluate_coding("Task", "Result") class TestParsingEdgeCases: @@ -578,11 +690,10 @@ def evaluator(self): ) def test_parse_score_out_of_range(self, evaluator): - """_parse_score should reject scores outside 1-5 range.""" + """_parse_score should raise for scores outside 1-5 range.""" response = '{"rating": 10}' - score = evaluator._parse_score(response) - # Should return None since score is out of range - assert score is None + with pytest.raises(ValueError, match="out of valid range"): + evaluator._parse_score(response) def test_parse_score_with_just_code_block(self, evaluator): """_parse_score should handle code block without json marker.""" @@ -596,17 +707,16 @@ def test_parse_score_with_text_before_json(self, evaluator): score = evaluator._parse_score(response) assert score == 5.0 - def test_parse_bargaining_ratings_partial(self, evaluator): - """_parse_bargaining_ratings should handle partial ratings.""" + def test_parse_bargaining_ratings_missing_keys(self, evaluator): + """_parse_bargaining_ratings should raise when required keys are missing.""" response = '{"effectiveness_of_strategies": 4}' # Missing other fields - ratings = evaluator._parse_bargaining_ratings(response) - assert ratings["effectiveness_of_strategies"] == 4 - assert ratings["progress_and_outcome"] is None + with pytest.raises(KeyError): + evaluator._parse_bargaining_ratings(response) def test_parse_bargaining_ratings_invalid(self, evaluator): - """_parse_bargaining_ratings should return None for invalid.""" - ratings = evaluator._parse_bargaining_ratings("Invalid") - assert ratings["effectiveness_of_strategies"] is None + """_parse_bargaining_ratings should raise for invalid response.""" + with pytest.raises(ValueError, match="No JSON object found"): + evaluator._parse_bargaining_ratings("Invalid") class TestFormatFinalAnswerEdgeCases: @@ -853,3 +963,155 @@ def test_completion_bargaining_partial_seller(self): } ) assert evaluator._determine_completion(metrics) is False + + +class TestResultSummarization: + """Tests for MARBLE-compatible result truncation and LLM summarization. + + MARBLE's evaluation pipeline truncates each agent result to 1000 chars + (_summarize_results) then summarizes via an LLM call (summarize_output) + before passing to the domain evaluator. + """ + + @pytest.fixture + def evaluator(self): + """Create evaluator with summarization enabled (default).""" + adapter = DummyModelAdapter( + model_id="test", + responses=["Summarized output"], + ) + return MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + output_format="Present the idea in 5Q format.", + ) + + def test_summarize_results_truncates_long_results(self, evaluator): + """Each agent result line should be truncated to result_truncation_length chars.""" + long_result = "x" * 2000 + agent_results = [{"agent_id": "a1", "result": long_result}] + summary = evaluator._summarize_results(agent_results) + + # Header line + one result line + lines = summary.strip().split("\n") + assert lines[0] == "Agents' Results Summary:" + # "- " prefix + content, truncated to DEFAULT_RESULT_TRUNCATION_LENGTH total + assert len(lines[1]) == DEFAULT_RESULT_TRUNCATION_LENGTH + + def test_summarize_results_short_results_unchanged(self, evaluator): + """Short results should not be truncated.""" + agent_results = [{"agent_id": "a1", "result": "short"}] + summary = evaluator._summarize_results(agent_results) + + lines = summary.strip().split("\n") + assert lines[1] == "- short" + + def test_summarize_results_multiple_agents(self, evaluator): + """Multiple agent results should all appear in the summary.""" + agent_results = [ + {"agent_id": "a1", "result": "Result 1"}, + {"agent_id": "a2", "result": "Result 2"}, + ] + summary = evaluator._summarize_results(agent_results) + + assert "Result 1" in summary + assert "Result 2" in summary + assert summary.startswith("Agents' Results Summary:\n") + + def test_summarize_results_empty_list(self, evaluator): + """Empty results should produce header-only summary.""" + summary = evaluator._summarize_results([]) + assert summary == "Agents' Results Summary:\n" + + def test_summarize_results_custom_truncation_length(self): + """Custom result_truncation_length should be respected.""" + adapter = DummyModelAdapter(model_id="test", responses=[]) + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + result_truncation_length=50, + ) + agent_results = [{"agent_id": "a1", "result": "x" * 200}] + summary = evaluator._summarize_results(agent_results) + + lines = summary.strip().split("\n") + assert len(lines[1]) == 50 + + def test_summarize_output_calls_model(self, evaluator): + """_summarize_output should call the model adapter and return its response.""" + result = evaluator._summarize_output("truncated summary", "research task", "5Q format") + assert isinstance(result, str) + assert result == "Summarized output" + + def test_call_with_summarization_uses_both_steps(self): + """__call__ with agent results should use truncation + LLM summarization.""" + adapter = DummyModelAdapter( + model_id="test", + responses=[ + '{"summary": "Agents collaborated on federated learning"}', + '{"innovation": 4, "safety": 4, "feasibility": 4}', + ], + ) + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + output_format="5Q format", + ) + traces = { + "agents": {"a1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {"marble_state": {"task_description": "Research task"}}, + } + final_answer = { + "agent_results": [{"agent_id": "a1", "result": "Raw paper search JSON " * 100}], + } + + result = evaluator(traces, final_answer) + + assert result["passed"] is True + assert result["metrics"]["task_evaluation"]["innovation"] == 4 + + def test_call_with_summarization_disabled(self): + """__call__ with result_truncation_length=None should skip summarization.""" + adapter = DummyModelAdapter( + model_id="test", + # Only one response needed: no summarization call, just domain eval + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + result_truncation_length=None, + ) + traces = { + "agents": {"a1": {"token_usage": 100, "action_log": [], "communication_log": []}}, + "environment": {}, + } + final_answer = { + "agent_results": [{"agent_id": "a1", "result": "Raw result"}], + } + + result = evaluator(traces, final_answer) + + assert result["domain"] == "research" + assert result["passed"] is True + + def test_call_string_final_answer_bypasses_summarization(self): + """String final_answer should bypass summarization (no agent_results to extract).""" + adapter = DummyModelAdapter( + model_id="test", + # Only one response needed: domain eval only + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + evaluator = MultiAgentBenchEvaluator( + domain="research", + model_adapter=adapter, + ) + traces = { + "agents": {}, + "environment": {}, + } + + result = evaluator(traces, "plain string result") + + assert result["domain"] == "research" + assert result["passed"] is True diff --git a/tests/test_benchmarks/test_multiagentbench/test_integration_real_data.py b/tests/test_benchmarks/test_multiagentbench/test_integration_real_data.py new file mode 100644 index 0000000..0264aec --- /dev/null +++ b/tests/test_benchmarks/test_multiagentbench/test_integration_real_data.py @@ -0,0 +1,245 @@ +"""Integration tests for MultiAgentBench using real MARBLE data. + +These tests validate that MultiAgentBench components work correctly with +actual MARBLE data, not synthetic fixtures. They are marked ``live`` + +``slow`` + ``benchmark`` because they require MARBLE to be cloned and +exercise the full pipeline. + +Run with:: + + pytest -m "live and slow" tests/test_benchmarks/test_multiagentbench/test_integration_real_data.py -v +""" + +import pytest + +from conftest import DummyModelAdapter +from maseval.benchmark.multiagentbench.data_loader import ( + VALID_DOMAINS, + configure_model_ids, + load_tasks, +) +from maseval.benchmark.multiagentbench.environment import ( + INFRASTRUCTURE_DOMAINS, + MultiAgentBenchEnvironment, +) +from maseval.benchmark.multiagentbench.evaluator import MultiAgentBenchEvaluator + +pytestmark = [pytest.mark.live, pytest.mark.slow, pytest.mark.benchmark] + + +@pytest.fixture(autouse=True) +def _mock_marble_environment(): + """Override: integration tests use real marble.""" + yield + + +# Domains that can be tested without external infrastructure (Docker, Minecraft Server) +NON_INFRA_DOMAINS = sorted(VALID_DOMAINS - INFRASTRUCTURE_DOMAINS - {"minecraft", "werewolf"}) + +# All domains except minecraft (untested upstream, requires game server) +EVALUATABLE_DOMAINS = sorted(VALID_DOMAINS - {"minecraft"}) + + +# ============================================================================= +# Fixture: ensure MARBLE is available +# ============================================================================= + + +@pytest.fixture(scope="module") +def marble_data_dir(): + """Ensure MARBLE is available and return the multiagentbench data dir.""" + from maseval.benchmark.multiagentbench.data_loader import ensure_marble_exists + + marble_dir = ensure_marble_exists(auto_download=True) + data_dir = marble_dir / "multiagentbench" + assert data_dir.exists(), f"MARBLE data dir not found: {data_dir}" + return data_dir + + +# ============================================================================= +# Data Loading with Real Data +# ============================================================================= + + +class TestMultiAgentBenchRealDataLoading: + """Test that load_tasks works with real MARBLE data for all domains.""" + + @pytest.mark.parametrize("domain", sorted(VALID_DOMAINS - {"werewolf", "minecraft"})) + def test_load_tasks_returns_tasks(self, domain, marble_data_dir): + """load_tasks(domain) returns a non-empty list of Tasks.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=5) + assert len(tasks) > 0, f"No tasks loaded for domain '{domain}'. Check test_data_integrity tests first." + + @pytest.mark.parametrize("domain", sorted(VALID_DOMAINS - {"werewolf", "minecraft"})) + def test_tasks_have_agents(self, domain, marble_data_dir): + """Every loaded task has at least one agent.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=3) + for task in tasks: + agents = task.environment_data.get("agents", []) + assert len(agents) >= 1, f"Task {task.id} in domain '{domain}' has {len(agents)} agents, expected at least 1." + + @pytest.mark.parametrize("domain", sorted(VALID_DOMAINS - {"werewolf", "minecraft"})) + def test_configure_model_ids_modifies_tasks(self, domain, marble_data_dir): + """configure_model_ids() sets llm and evaluator model_id.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=2) + configure_model_ids(tasks, agent_model_id="test-model") + + for task in tasks: + assert task.environment_data.get("llm") == "test-model", ( + f"Task {task.id}: environment_data['llm'] not set after configure_model_ids()" + ) + assert task.evaluation_data.get("model_id") == "test-model", ( + f"Task {task.id}: evaluation_data['model_id'] not set after configure_model_ids()" + ) + + def test_load_werewolf_tasks(self, marble_data_dir): + """Werewolf tasks load from config files.""" + tasks = load_tasks("werewolf", data_dir=marble_data_dir) + assert len(tasks) > 0, "No werewolf tasks loaded" + + +# ============================================================================= +# Environment Setup with Real Data +# ============================================================================= + + +class TestMultiAgentBenchRealEnvironment: + """Test MultiAgentBenchEnvironment with real MARBLE task data.""" + + @pytest.mark.parametrize("domain", NON_INFRA_DOMAINS) + def test_environment_creates(self, domain, marble_data_dir): + """MultiAgentBenchEnvironment can be created from a real task.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=1) + assert len(tasks) > 0 + + env = MultiAgentBenchEnvironment(task_data=tasks[0].environment_data) + assert env is not None + assert env.domain == domain + + @pytest.mark.parametrize("domain", NON_INFRA_DOMAINS) + def test_environment_setup_state(self, domain, marble_data_dir): + """setup_state() extracts domain and max_iterations from real data.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=1) + env = MultiAgentBenchEnvironment(task_data=tasks[0].environment_data) + state = env.setup_state(tasks[0].environment_data) + + assert isinstance(state, dict) + assert state.get("domain") == domain + assert "max_iterations" in state + assert state["max_iterations"] > 0 + + @pytest.mark.parametrize("domain", NON_INFRA_DOMAINS) + def test_environment_gather_traces(self, domain, marble_data_dir): + """gather_traces() returns dict with expected keys.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=1) + env = MultiAgentBenchEnvironment(task_data=tasks[0].environment_data) + env.setup_state(tasks[0].environment_data) + traces = env.gather_traces() + + assert isinstance(traces, dict) + assert "domain" in traces + assert traces["domain"] == domain + + @pytest.mark.parametrize("domain", NON_INFRA_DOMAINS) + def test_environment_gather_config(self, domain, marble_data_dir): + """gather_config() returns dict with domain and tool info.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=1) + env = MultiAgentBenchEnvironment(task_data=tasks[0].environment_data) + env.setup_state(tasks[0].environment_data) + config = env.gather_config() + + assert isinstance(config, dict) + assert config.get("domain") == domain + + +# ============================================================================= +# Evaluator with Real Data +# ============================================================================= + + +class TestMultiAgentBenchRealEvaluation: + """Test evaluator creation with real task data.""" + + @pytest.mark.parametrize("domain", EVALUATABLE_DOMAINS) + def test_evaluator_creates_from_real_domain(self, domain, marble_data_dir): + """MultiAgentBenchEvaluator can be created for each domain.""" + model = DummyModelAdapter( + model_id="test-eval-model", + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + evaluator = MultiAgentBenchEvaluator( + domain=domain, + model_adapter=model, + ) + assert evaluator is not None + + @pytest.mark.parametrize("domain", EVALUATABLE_DOMAINS) + def test_evaluator_filter_traces(self, domain, marble_data_dir): + """filter_traces() processes a synthetic trace structure.""" + model = DummyModelAdapter( + model_id="test-eval-model", + responses=['{"innovation": 4, "safety": 4, "feasibility": 4}'], + ) + evaluator = MultiAgentBenchEvaluator( + domain=domain, + model_adapter=model, + ) + + # Minimal synthetic traces + traces = { + "agents": { + "agent_1": { + "action_log": [], + "communication_log": [], + "token_usage": 100, + } + }, + "environment": { + "domain": domain, + }, + } + + filtered = evaluator.filter_traces(traces) + assert isinstance(filtered, dict) + + +# ============================================================================= +# Pipeline Smoke Test +# ============================================================================= + + +class TestMultiAgentBenchPipelineSmoke: + """Smoke test for the full pipeline with real data.""" + + @pytest.mark.parametrize("domain", NON_INFRA_DOMAINS) + def test_full_pipeline_single_task(self, domain, marble_data_dir, concrete_multiagentbench_benchmark): + """benchmark.run() on one real task produces results.""" + tasks = load_tasks(domain, data_dir=marble_data_dir, limit=1) + assert len(tasks) > 0, f"No tasks for domain '{domain}'. Check test_data_integrity tests first." + + # Configure model IDs (required for evaluation) + configure_model_ids(tasks, agent_model_id="test-model") + + benchmark = concrete_multiagentbench_benchmark( + progress_bar=False, + max_invocations=1, + ) + + results = benchmark.run(tasks, agent_data={}) + + assert len(results) == 1, f"Expected 1 result for domain '{domain}', got {len(results)}." + + result = results[0] + assert "status" in result, "Result missing 'status' key" + # All possible TaskExecutionStatus values + known_statuses = { + "success", + "agent_error", + "environment_error", + "user_error", + "task_timeout", + "unknown_execution_error", + "evaluation_failed", + "setup_failed", + } + assert result["status"] in known_statuses, f"Unexpected status '{result['status']}' for domain '{domain}'. Known: {known_statuses}" diff --git a/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py index 93ef253..d7c78b9 100644 --- a/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py +++ b/tests/test_benchmarks/test_multiagentbench/test_marble_adapter.py @@ -8,6 +8,8 @@ ) from maseval import AgentError +pytestmark = pytest.mark.benchmark + class TestMarbleAgentAdapter: """Tests for MarbleAgentAdapter class.""" @@ -194,14 +196,25 @@ class TestCreateMarbleAgentsImportError: def test_create_marble_agents_import_error(self): """create_marble_agents should raise ImportError when MARBLE not available.""" + import sys + from unittest.mock import patch + from maseval.benchmark.multiagentbench.adapters.marble_adapter import ( create_marble_agents, ) - # This will fail because MARBLE is not installed - with pytest.raises(ImportError, match="MARBLE is not available"): - create_marble_agents( - agent_configs=[{"agent_id": "test"}], - marble_env=MagicMock(), - model="gpt-4o", - ) + # Temporarily remove marble modules to simulate MARBLE not being available + marble_modules = {k: v for k, v in sys.modules.items() if "marble" in k} + for module_name in marble_modules: + sys.modules.pop(module_name, None) + + try: + with patch.dict("sys.modules", {"marble.agent.base_agent": None}): + with pytest.raises(ImportError, match="MARBLE is not available"): + create_marble_agents( + agent_configs=[{"agent_id": "test"}], + marble_env=MagicMock(), + model="gpt-4o", + ) + finally: + sys.modules.update(marble_modules) diff --git a/tests/test_benchmarks/test_tau2/test_domains/test_telecom_user_tools.py b/tests/test_benchmarks/test_tau2/test_domains/test_telecom_user_tools.py index a753c97..875cc92 100644 --- a/tests/test_benchmarks/test_tau2/test_domains/test_telecom_user_tools.py +++ b/tests/test_benchmarks/test_tau2/test_domains/test_telecom_user_tools.py @@ -98,14 +98,14 @@ def test_toggle_airplane_mode(self, telecom_user_toolkit): # Enable telecom_user_toolkit.use_tool("toggle_airplane_mode", enable=True) assert telecom_user_toolkit.db.user_db.device.airplane_mode is True - assert telecom_user_toolkit.db.user_db.device.network_status == NetworkStatus.NO_SERVICE + assert telecom_user_toolkit.db.user_db.device.network_connection_status == NetworkStatus.NO_SERVICE # Disable telecom_user_toolkit.use_tool("toggle_airplane_mode", enable=False) assert telecom_user_toolkit.db.user_db.device.airplane_mode is False # Should reconnect if SIM is active if telecom_user_toolkit.db.user_db.device.sim_status == SimStatus.ACTIVE: - assert telecom_user_toolkit.db.user_db.device.network_status == NetworkStatus.CONNECTED + assert telecom_user_toolkit.db.user_db.device.network_connection_status == NetworkStatus.CONNECTED def test_toggle_data(self, telecom_user_toolkit): """toggle_data changes state.""" @@ -204,7 +204,7 @@ def test_check_sim_status(self, telecom_user_toolkit): def test_reseat_sim_card(self, telecom_user_toolkit): """Reseats SIM card.""" result = telecom_user_toolkit.use_tool("reseat_sim_card") - assert "reseat" in result.lower() or "no" in result.lower() + assert "re-seated" in result.lower() # ============================================================================= diff --git a/tests/test_benchmarks/test_tau2/test_initialization.py b/tests/test_benchmarks/test_tau2/test_initialization.py new file mode 100644 index 0000000..fdcd5f7 --- /dev/null +++ b/tests/test_benchmarks/test_tau2/test_initialization.py @@ -0,0 +1,297 @@ +"""Tests for tau2 telecom initialization_actions execution. + +Verifies that initialization_actions from telecom tasks are properly executed +during environment setup, matching tau2-bench behavior. +""" + +import pytest + +from maseval.benchmark.tau2.domains.telecom.user_models import ( + APNNames, + NetworkModePreference, + NetworkStatus, + NetworkTechnology, + PerformanceLevel, + SignalStrength, + SimStatus, +) + +pytestmark = [pytest.mark.live, pytest.mark.benchmark] + + +# ============================================================================= +# Initialization Method Tests +# ============================================================================= + + +class TestInitializationMethods: + """Tests for individual initialization methods on TelecomUserTools.""" + + def test_set_user_info(self, telecom_user_toolkit): + """set_user_info sets name and phone_number on surroundings.""" + telecom_user_toolkit.set_user_info(name="John Smith", phone_number="555-123-2002") + + assert telecom_user_toolkit.db.user_db.surroundings.name == "John Smith" + assert telecom_user_toolkit.db.user_db.surroundings.phone_number == "555-123-2002" + + def test_set_user_location(self, telecom_user_toolkit): + """set_user_location sets is_abroad.""" + telecom_user_toolkit.set_user_location(abroad=True) + assert telecom_user_toolkit.db.user_db.surroundings.is_abroad is True + + telecom_user_toolkit.set_user_location(abroad=False) + assert telecom_user_toolkit.db.user_db.surroundings.is_abroad is False + + def test_turn_data_off(self, telecom_user_toolkit): + """turn_data_off disables mobile data.""" + assert telecom_user_toolkit.db.user_db.device.mobile_data_enabled is True + telecom_user_toolkit.turn_data_off() + assert telecom_user_toolkit.db.user_db.device.mobile_data_enabled is False + + def test_turn_airplane_mode_on(self, telecom_user_toolkit): + """turn_airplane_mode_on enables airplane mode with side effects.""" + telecom_user_toolkit.turn_airplane_mode_on() + + device = telecom_user_toolkit.db.user_db.device + assert device.airplane_mode is True + assert device.wifi_connected is False + assert device.network_connection_status == NetworkStatus.NO_SERVICE + assert device.network_technology_connected == NetworkTechnology.NONE + assert device.network_signal_strength == SignalStrength.NONE + + def test_turn_roaming_off(self, telecom_user_toolkit): + """turn_roaming_off disables roaming.""" + telecom_user_toolkit.db.user_db.device.roaming_enabled = True + telecom_user_toolkit.turn_roaming_off() + assert telecom_user_toolkit.db.user_db.device.roaming_enabled is False + + def test_turn_roaming_on(self, telecom_user_toolkit): + """turn_roaming_on enables roaming.""" + telecom_user_toolkit.turn_roaming_on() + assert telecom_user_toolkit.db.user_db.device.roaming_enabled is True + + def test_turn_data_saver_mode_on(self, telecom_user_toolkit): + """turn_data_saver_mode_on enables data saver.""" + telecom_user_toolkit.turn_data_saver_mode_on() + assert telecom_user_toolkit.db.user_db.device.data_saver_mode is True + + def test_unseat_sim_card(self, telecom_user_toolkit): + """unseat_sim_card marks SIM as missing and drops network.""" + telecom_user_toolkit.unseat_sim_card() + + device = telecom_user_toolkit.db.user_db.device + assert device.sim_card_missing is True + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_lock_sim_card_pin(self, telecom_user_toolkit): + """lock_sim_card with pin locks SIM and drops network.""" + telecom_user_toolkit.lock_sim_card(mode="pin") + + device = telecom_user_toolkit.db.user_db.device + assert device.sim_status == SimStatus.LOCKED_PIN + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_lock_sim_card_puk(self, telecom_user_toolkit): + """lock_sim_card with puk locks SIM and drops network.""" + telecom_user_toolkit.lock_sim_card(mode="puk") + + device = telecom_user_toolkit.db.user_db.device + assert device.sim_status == SimStatus.LOCKED_PUK + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_break_apn_settings(self, telecom_user_toolkit): + """break_apn_settings sets APN to broken and drops network.""" + telecom_user_toolkit.break_apn_settings() + + device = telecom_user_toolkit.db.user_db.device + assert device.apn_settings.name == APNNames.BROKEN.value + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_break_apn_mms_setting(self, telecom_user_toolkit): + """break_apn_mms_setting clears MMSC URL.""" + telecom_user_toolkit.break_apn_mms_setting() + assert telecom_user_toolkit.db.user_db.device.apn_settings.mmsc_url == "" + + def test_break_vpn(self, telecom_user_toolkit): + """break_vpn connects VPN with poor performance.""" + telecom_user_toolkit.break_vpn() + + device = telecom_user_toolkit.db.user_db.device + assert device.vpn_status is True + assert device.vpn_details is not None + assert device.vpn_details.server_performance == PerformanceLevel.POOR + + def test_remove_app_permission(self, telecom_user_toolkit): + """remove_app_permission revokes a permission.""" + # Default messaging app has sms=True + assert telecom_user_toolkit.db.user_db.device.installed_apps["messaging"].permissions.sms is True + + telecom_user_toolkit.remove_app_permission("messaging", "sms") + assert telecom_user_toolkit.db.user_db.device.installed_apps["messaging"].permissions.sms is False + + def test_set_wifi_calling(self, telecom_user_toolkit): + """set_wifi_calling sets wifi calling and mms_over_wifi.""" + telecom_user_toolkit.set_wifi_calling(enabled=True, mms_over_wifi=True) + + device = telecom_user_toolkit.db.user_db.device + assert device.wifi_calling_enabled is True + assert device.wifi_calling_mms_over_wifi is True + + +# ============================================================================= +# simulate_network_search Tests +# ============================================================================= + + +class TestSimulateNetworkSearch: + """Tests for simulate_network_search behavior.""" + + def test_active_sim_default_preference(self, telecom_user_toolkit): + """With active SIM and default 4G/5G preference, connects to best available.""" + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.CONNECTED + # Default signal_strength only has 4G and 3G, no 5G → should connect to 4G + assert device.network_technology_connected == NetworkTechnology.FOUR_G + assert device.network_signal_strength == SignalStrength.GOOD + + def test_3g_only_preference(self, telecom_user_toolkit): + """With 3G only preference, connects to 3G.""" + telecom_user_toolkit.db.user_db.device.network_mode_preference = NetworkModePreference.THREE_G_ONLY + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.CONNECTED + assert device.network_technology_connected == NetworkTechnology.THREE_G + assert device.network_signal_strength == SignalStrength.FAIR + + def test_2g_only_no_signal(self, telecom_user_toolkit): + """With 2G only preference but no 2G signal, gets no signal.""" + telecom_user_toolkit.db.user_db.device.network_mode_preference = NetworkModePreference.TWO_G_ONLY + # Default signal_strength doesn't include 2G + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_signal_strength == SignalStrength.NONE + + def test_airplane_mode_no_service(self, telecom_user_toolkit): + """Airplane mode results in NO_SERVICE.""" + telecom_user_toolkit.db.user_db.device.airplane_mode = True + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.NO_SERVICE + assert device.network_technology_connected == NetworkTechnology.NONE + + def test_missing_sim_no_service(self, telecom_user_toolkit): + """Missing SIM card results in NO_SERVICE.""" + telecom_user_toolkit.db.user_db.device.sim_card_missing = True + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_locked_sim_no_service(self, telecom_user_toolkit): + """Locked SIM card results in NO_SERVICE.""" + telecom_user_toolkit.db.user_db.device.sim_status = SimStatus.LOCKED_PIN + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_broken_apn_no_service(self, telecom_user_toolkit): + """Broken APN results in NO_SERVICE.""" + telecom_user_toolkit.db.user_db.device.apn_settings.name = APNNames.BROKEN.value + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + def test_inactive_line_no_service(self, telecom_user_toolkit): + """Inactive line results in NO_SERVICE.""" + telecom_user_toolkit.db.user_db.surroundings.line_active = False + telecom_user_toolkit.simulate_network_search() + + device = telecom_user_toolkit.db.user_db.device + assert device.network_connection_status == NetworkStatus.NO_SERVICE + + +# ============================================================================= +# Speed Test After Initialization Tests +# ============================================================================= + + +class TestSpeedTestWithInitialization: + """Tests verifying speed test behavior matches tau2-bench after init.""" + + def test_speed_test_after_data_off(self, telecom_user_toolkit): + """Speed test returns No Connection after turn_data_off.""" + telecom_user_toolkit.turn_data_off() + result = telecom_user_toolkit.use_tool("run_speed_test") + assert "no" in result.lower() + + def test_speed_test_after_airplane_mode(self, telecom_user_toolkit): + """Speed test returns no connection after turn_airplane_mode_on.""" + telecom_user_toolkit.turn_airplane_mode_on() + result = telecom_user_toolkit.use_tool("run_speed_test") + assert "airplane" in result.lower() or "no" in result.lower() + + def test_speed_test_after_break_vpn(self, telecom_user_toolkit): + """Speed test returns reduced speed after break_vpn (poor VPN).""" + telecom_user_toolkit.break_vpn() + speed, desc = telecom_user_toolkit._run_speed_test() + # With poor VPN (0.1x factor), 4G Good signal: + # (10+100)/2 * 0.8 * 0.1 = 4.4 Mbps → "Poor" + assert speed is not None + assert speed < 5.0 + assert desc == "Poor" + + +# ============================================================================= +# All Init Func Names Callable Test +# ============================================================================= + + +class TestAllFuncNamesCallable: + """Verify all func_names used in task JSON are callable on the toolkits.""" + + USER_FUNC_NAMES = [ + "set_user_info", + "set_user_location", + "turn_data_off", + "turn_airplane_mode_on", + "turn_roaming_off", + "turn_roaming_on", + "turn_data_saver_mode_on", + "unseat_sim_card", + "lock_sim_card", + "break_apn_settings", + "break_apn_mms_setting", + "break_vpn", + "remove_app_permission", + "set_network_mode_preference", + "set_wifi_calling", + "simulate_network_search", + ] + + ASSISTANT_FUNC_NAMES = [ + "set_data_usage", + "enable_roaming", + "disable_roaming", + "suspend_line_for_overdue_bill", + ] + + @pytest.mark.parametrize("func_name", USER_FUNC_NAMES) + def test_user_func_callable(self, telecom_user_toolkit, func_name): + """Each user-side init func_name is callable on user toolkit.""" + func = getattr(telecom_user_toolkit, func_name, None) + assert func is not None, f"User function '{func_name}' not found" + assert callable(func), f"User function '{func_name}' not callable" + + @pytest.mark.parametrize("func_name", ASSISTANT_FUNC_NAMES) + def test_assistant_func_callable(self, telecom_toolkit, func_name): + """Each assistant-side init func_name is callable on agent toolkit.""" + func = getattr(telecom_toolkit, func_name, None) + assert func is not None, f"Assistant function '{func_name}' not found" + assert callable(func), f"Assistant function '{func_name}' not callable" diff --git a/uv.lock b/uv.lock index 03f392f..7570f6f 100644 --- a/uv.lock +++ b/uv.lock @@ -3,15 +3,40 @@ revision = 3 requires-python = ">=3.10" resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'linux'", - "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'linux'", "python_full_version == '3.11.*' and sys_platform == 'linux'", "python_full_version < '3.11' and sys_platform == 'linux'", + "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'linux'", "python_full_version == '3.11.*' and sys_platform != 'linux'", "python_full_version < '3.11' and sys_platform != 'linux'", ] +[manifest] +overrides = [ + { name = "click", specifier = ">=8.1.0" }, + { name = "datasets", specifier = ">=3.0.0" }, + { name = "docstring-parser", specifier = ">=0.16" }, + { name = "fsspec", specifier = ">=2024.12.0" }, + { name = "inputimeout", specifier = ">=1.0.4" }, + { name = "jinja2", specifier = ">=3.1.0" }, + { name = "litellm", specifier = ">=1.0.0" }, + { name = "mammoth", specifier = ">=1.8.0" }, + { name = "markdownify", specifier = ">=0.14.1" }, + { name = "mcp", specifier = ">=1.11.0" }, + { name = "numpy", specifier = ">=2.2.0" }, + { name = "pandas", specifier = ">=2.2.0" }, + { name = "pdfminer-six", specifier = ">=20231228" }, + { name = "pillow", specifier = ">=10.4.0" }, + { name = "polars-lts-cpu", specifier = ">=1.33.1" }, + { name = "puremagic", specifier = ">=1.27" }, + { name = "pydantic", specifier = ">=2.10.6" }, + { name = "python-dotenv", specifier = ">=1.0.0" }, + { name = "python-pptx", specifier = ">=1.0.2" }, + { name = "rapidfuzz", specifier = ">=3.12.1" }, + { name = "termcolor", specifier = ">=2.5.0" }, +] + [[package]] name = "aiohappyeyeballs" version = "2.6.1" @@ -214,19 +239,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/81/29/5ecc3a15d5a33e31b26c11426c45c501e439cb865d0bff96315d86443b78/appnope-0.1.4-py2.py3-none-any.whl", hash = "sha256:502575ee11cd7a28c0205f379b525beefebab9d161b7c964670864014ed7213c", size = 4321, upload-time = "2024-02-06T09:43:09.663Z" }, ] -[[package]] -name = "are" -version = "2.1.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "nfa" }, - { name = "reiter" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/91/38/ea2a46ed9f9f39b427f331f1d48a42e3f3fd317b19c9cb6207c970d8beee/are-2.1.0.tar.gz", hash = "sha256:3d7287dd84a51d659b396d9937054bd7655ae145f7984fa3c8d8ea9c466d6bc2", size = 13359, upload-time = "2022-08-04T06:14:11.78Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/96/18/866e93ff1b1ee083171455188533baa130b0d3792280c248c074b1e92e7e/are-2.1.0-py3-none-any.whl", hash = "sha256:1f7706e7c7cc0ed4959b7abce94852a697649780760e63c2600245f151272f6c", size = 9805, upload-time = "2022-08-04T06:14:10.345Z" }, -] - [[package]] name = "argon2-cffi" version = "25.1.0" @@ -288,6 +300,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/ed/c9/d7977eaacb9df673210491da99e6a247e93df98c715fc43fd136ce1d3d33/arrow-1.4.0-py3-none-any.whl", hash = "sha256:749f0769958ebdc79c173ff0b0670d59051a535fa26e8eba02953dc19eb43205", size = 68797, upload-time = "2025-10-18T17:46:45.663Z" }, ] +[[package]] +name = "arxiv" +version = "2.4.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "feedparser" }, + { name = "requests" }, + { name = "typing-extensions", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8d/aa/dc1c6c633f63fce090e7c067af8c528a5e61218a61c266ff615d46cbde0a/arxiv-2.4.0.tar.gz", hash = "sha256:cabe5470d031aa3f22d2744a7600391c62c3489653f0c62bec9019e62bb0554b", size = 74546, upload-time = "2026-01-05T02:43:16.823Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/11/63/9e71153b2d48c98f8079c90d7211bc65515cc1ad18c3328c3c0472e68f44/arxiv-2.4.0-py3-none-any.whl", hash = "sha256:c02ccb09a777aaadd75d3bc1d2627894ef9c987c651d0dacd864b9f69fb0569f", size = 12065, upload-time = "2026-01-05T02:43:12.542Z" }, +] + [[package]] name = "astor" version = "0.8.1" @@ -384,6 +410,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b4/d6/f9168956276934162ec8d48232f9920f2985ee45aa7602e3c6b4bc203613/banks-2.2.0-py3-none-any.whl", hash = "sha256:963cd5c85a587b122abde4f4064078def35c50c688c1b9d36f43c92503854e7d", size = 29244, upload-time = "2025-07-18T16:28:27.835Z" }, ] +[[package]] +name = "beartype" +version = "0.22.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/94/1009e248bbfbab11397abca7193bea6626806be9a327d399810d523a07cb/beartype-0.22.9.tar.gz", hash = "sha256:8f82b54aa723a2848a56008d18875f91c1db02c32ef6a62319a002e3e25a975f", size = 1608866, upload-time = "2025-12-13T06:50:30.72Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/71/cc/18245721fa7747065ab478316c7fea7c74777d07f37ae60db2e84f8172e8/beartype-0.22.9-py3-none-any.whl", hash = "sha256:d16c9bbc61ea14637596c5f6fbff2ee99cbe3573e46a716401734ef50c3060c2", size = 1333658, upload-time = "2025-12-13T06:50:28.266Z" }, +] + [[package]] name = "beautifulsoup4" version = "4.14.3" @@ -414,6 +449,15 @@ css = [ { name = "tinycss2" }, ] +[[package]] +name = "blinker" +version = "1.9.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/21/28/9b3f50ce0e048515135495f198351908d99540d69bfdc8c1d15b73dc55ce/blinker-1.9.0.tar.gz", hash = "sha256:b4ce2265a7abece45e7cc896e98dbebe6cead56bcf805a3d23136d145f5445bf", size = 22460, upload-time = "2024-11-08T17:25:47.436Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/10/cb/f2ad4230dc2eb1a74edf38f1a38b9b52277f75bef262d8908e60d957e13c/blinker-1.9.0-py3-none-any.whl", hash = "sha256:ba0efaa9080b619ff2f3459d1d500c57bddea4a6b424b60a91141db6fd2f08bc", size = 8458, upload-time = "2024-11-08T17:25:46.184Z" }, +] + [[package]] name = "cachetools" version = "6.2.4" @@ -647,6 +691,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/98/78/01c019cdb5d6498122777c1a43056ebb3ebfeef2076d9d026bfe15583b2b/click-8.3.1-py3-none-any.whl", hash = "sha256:981153a64e25f12d547d3426c367a4857371575ee7ad18df2a6183ab0545b2a6", size = 108274, upload-time = "2025-11-15T20:45:41.139Z" }, ] +[[package]] +name = "cobble" +version = "0.1.4" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/54/7a/a507c709be2c96e1bb6102eb7b7f4026c5e5e223ef7d745a17d239e9d844/cobble-0.1.4.tar.gz", hash = "sha256:de38be1539992c8a06e569630717c485a5f91be2192c461ea2b220607dfa78aa", size = 3805, upload-time = "2024-06-01T18:11:09.528Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d5/e1/3714a2f371985215c219c2a70953d38e3eed81ef165aed061d21de0e998b/cobble-0.1.4-py3-none-any.whl", hash = "sha256:36c91b1655e599fd428e2b95fdd5f0da1ca2e9f1abb0bc871dec21a0e78a2b44", size = 3984, upload-time = "2024-06-01T18:11:07.911Z" }, +] + [[package]] name = "colorama" version = "0.4.6" @@ -656,6 +709,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d1/d6/3965ed04c63042e047cb6a3e6ed1a63a35087b6a609aa3a15ed8ac56c221/colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6", size = 25335, upload-time = "2022-10-25T02:36:20.889Z" }, ] +[[package]] +name = "colorlog" +version = "6.10.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "colorama", marker = "sys_platform == 'win32'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a2/61/f083b5ac52e505dfc1c624eafbf8c7589a0d7f32daa398d2e7590efa5fda/colorlog-6.10.1.tar.gz", hash = "sha256:eb4ae5cb65fe7fec7773c2306061a8e63e02efc2c72eba9d27b0fa23c94f1321", size = 17162, upload-time = "2025-10-16T16:14:11.978Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6d/c1/e419ef3723a074172b68aaa89c9f3de486ed4c2399e2dbd8113a4fdcaf9e/colorlog-6.10.1-py3-none-any.whl", hash = "sha256:2d7e8348291948af66122cff006c9f8da6255d224e7cf8e37d8de2df3bad8c9c", size = 11743, upload-time = "2025-10-16T16:14:10.512Z" }, +] + [[package]] name = "comm" version = "0.2.3" @@ -834,6 +899,31 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/0d/c3/e90f4a4feae6410f914f8ebac129b9ae7a8c92eb60a638012dde42030a9d/cryptography-46.0.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:6b5063083824e5509fdba180721d55909ffacccc8adbec85268b48439423d78c", size = 3438528, upload-time = "2025-10-15T23:18:26.227Z" }, ] +[[package]] +name = "cuda-bindings" +version = "12.9.4" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cuda-pathfinder", marker = "sys_platform == 'linux'" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/7a/d8/b546104b8da3f562c1ff8ab36d130c8fe1dd6a045ced80b4f6ad74f7d4e1/cuda_bindings-12.9.4-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4d3c842c2a4303b2a580fe955018e31aea30278be19795ae05226235268032e5", size = 12148218, upload-time = "2025-10-21T14:51:28.855Z" }, + { url = "https://files.pythonhosted.org/packages/45/e7/b47792cc2d01c7e1d37c32402182524774dadd2d26339bd224e0e913832e/cuda_bindings-12.9.4-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c912a3d9e6b6651853eed8eed96d6800d69c08e94052c292fec3f282c5a817c9", size = 12210593, upload-time = "2025-10-21T14:51:36.574Z" }, + { url = "https://files.pythonhosted.org/packages/a9/c1/dabe88f52c3e3760d861401bb994df08f672ec893b8f7592dc91626adcf3/cuda_bindings-12.9.4-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fda147a344e8eaeca0c6ff113d2851ffca8f7dfc0a6c932374ee5c47caa649c8", size = 12151019, upload-time = "2025-10-21T14:51:43.167Z" }, + { url = "https://files.pythonhosted.org/packages/63/56/e465c31dc9111be3441a9ba7df1941fe98f4aa6e71e8788a3fb4534ce24d/cuda_bindings-12.9.4-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:32bdc5a76906be4c61eb98f546a6786c5773a881f3b166486449b5d141e4a39f", size = 11906628, upload-time = "2025-10-21T14:51:49.905Z" }, + { url = "https://files.pythonhosted.org/packages/a3/84/1e6be415e37478070aeeee5884c2022713c1ecc735e6d82d744de0252eee/cuda_bindings-12.9.4-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:56e0043c457a99ac473ddc926fe0dc4046694d99caef633e92601ab52cbe17eb", size = 11925991, upload-time = "2025-10-21T14:51:56.535Z" }, + { url = "https://files.pythonhosted.org/packages/d1/af/6dfd8f2ed90b1d4719bc053ff8940e494640fe4212dc3dd72f383e4992da/cuda_bindings-12.9.4-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8b72ee72a9cc1b531db31eebaaee5c69a8ec3500e32c6933f2d3b15297b53686", size = 11922703, upload-time = "2025-10-21T14:52:03.585Z" }, + { url = "https://files.pythonhosted.org/packages/6c/19/90ac264acc00f6df8a49378eedec9fd2db3061bf9263bf9f39fd3d8377c3/cuda_bindings-12.9.4-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d80bffc357df9988dca279734bc9674c3934a654cab10cadeed27ce17d8635ee", size = 11924658, upload-time = "2025-10-21T14:52:10.411Z" }, +] + +[[package]] +name = "cuda-pathfinder" +version = "1.3.3" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0b/02/4dbe7568a42e46582248942f54dc64ad094769532adbe21e525e4edf7bc4/cuda_pathfinder-1.3.3-py3-none-any.whl", hash = "sha256:9984b664e404f7c134954a771be8775dfd6180ea1e1aef4a5a37d4be05d9bbb1", size = 27154, upload-time = "2025-12-04T22:35:08.996Z" }, +] + [[package]] name = "dataclasses-json" version = "0.6.7" @@ -849,13 +939,12 @@ wheels = [ [[package]] name = "datasets" -version = "4.5.0" +version = "4.0.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "dill" }, { name = "filelock" }, - { name = "fsspec", extra = ["http"] }, - { name = "httpx" }, + { name = "fsspec" }, { name = "huggingface-hub" }, { name = "multiprocess" }, { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, @@ -868,9 +957,9 @@ dependencies = [ { name = "tqdm" }, { name = "xxhash" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/55/bf/bb927bde63d649296c83e883171ae77074717c1b80fe2868b328bd0dbcbb/datasets-4.5.0.tar.gz", hash = "sha256:00c698ce1c2452e646cc5fad47fef39d3fe78dd650a8a6eb205bb45eb63cd500", size = 588384, upload-time = "2026-01-14T18:27:54.297Z" } +sdist = { url = "https://files.pythonhosted.org/packages/e3/9d/348ed92110ba5f9b70b51ca1078d4809767a835aa2b7ce7e74ad2b98323d/datasets-4.0.0.tar.gz", hash = "sha256:9657e7140a9050db13443ba21cb5de185af8af944479b00e7ff1e00a61c8dbf1", size = 569566, upload-time = "2025-07-09T14:35:52.431Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/fc/d5/0d563ea3c205eee226dc8053cf7682a8ac588db8acecd0eda2b587987a0b/datasets-4.5.0-py3-none-any.whl", hash = "sha256:b5d7e08096ffa407dd69e58b1c0271c9b2506140839b8d99af07375ad31b6726", size = 515196, upload-time = "2026-01-14T18:27:52.419Z" }, + { url = "https://files.pythonhosted.org/packages/eb/62/eb8157afb21bd229c864521c1ab4fa8e9b4f1b06bafdd8c4668a7a31b5dd/datasets-4.0.0-py3-none-any.whl", hash = "sha256:7ef95e62025fd122882dbce6cb904c8cd3fbc829de6669a5eb939c77d50e203d", size = 494825, upload-time = "2025-07-09T14:35:50.658Z" }, ] [[package]] @@ -934,11 +1023,11 @@ wheels = [ [[package]] name = "dill" -version = "0.4.0" +version = "0.3.8" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/12/80/630b4b88364e9a8c8c5797f4602d0f76ef820909ee32f0bacb9f90654042/dill-0.4.0.tar.gz", hash = "sha256:0633f1d2df477324f53a895b02c901fb961bdbf65a17122586ea7019292cbcf0", size = 186976, upload-time = "2025-04-16T00:41:48.867Z" } +sdist = { url = "https://files.pythonhosted.org/packages/17/4d/ac7ffa80c69ea1df30a8aa11b3578692a5118e7cd1aa157e3ef73b092d15/dill-0.3.8.tar.gz", hash = "sha256:3ebe3c479ad625c4553aca177444d89b486b1d84982eeacded644afc0cf797ca", size = 184847, upload-time = "2024-01-27T23:42:16.145Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/50/3d/9373ad9c56321fdab5b41197068e1d8c25883b3fea29dd361f9b55116869/dill-0.4.0-py3-none-any.whl", hash = "sha256:44f54bf6412c2c8464c14e8243eb163690a9800dbe2c367330883b19c7561049", size = 119668, upload-time = "2025-04-16T00:41:47.671Z" }, + { url = "https://files.pythonhosted.org/packages/c9/7a/cef76fd8438a42f96db64ddaa85280485a9c395e7df3db8158cfec1eee34/dill-0.3.8-py3-none-any.whl", hash = "sha256:c36ca9ffb54365bdd2f8eb3eff7d2a21237f8452b57ace88b1ac615b7e815bd7", size = 116252, upload-time = "2024-01-27T23:42:14.239Z" }, ] [[package]] @@ -1070,6 +1159,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1f/f6/da4db31001e854025ffd26bc9ba0740a9cbba2c3259695f7c5834908b336/fastuuid-0.14.0-cp314-cp314-win_amd64.whl", hash = "sha256:df61342889d0f5e7a32f7284e55ef95103f2110fee433c2ae7c2c0956d76ac8a", size = 156457, upload-time = "2025-10-19T22:33:44.579Z" }, ] +[[package]] +name = "feedparser" +version = "6.0.12" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "sgmllib3k" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/dc/79/db7edb5e77d6dfbc54d7d9df72828be4318275b2e580549ff45a962f6461/feedparser-6.0.12.tar.gz", hash = "sha256:64f76ce90ae3e8ef5d1ede0f8d3b50ce26bcce71dd8ae5e82b1cd2d4a5f94228", size = 286579, upload-time = "2025-09-10T13:33:59.486Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/4e/eb/c96d64137e29ae17d83ad2552470bafe3a7a915e85434d9942077d7fd011/feedparser-6.0.12-py3-none-any.whl", hash = "sha256:6bbff10f5a52662c00a2e3f86a38928c37c48f77b3c511aedcd51de933549324", size = 81480, upload-time = "2025-09-10T13:33:58.022Z" }, +] + [[package]] name = "filelock" version = "3.20.1" @@ -1088,6 +1189,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/18/79/1b8fa1bb3568781e84c9200f951c735f3f157429f44be0495da55894d620/filetype-1.2.0-py2.py3-none-any.whl", hash = "sha256:7ce71b6880181241cf7ac8697a2f1eb6a8bd9b429f7ad6d27b8db9ba5f1c2d25", size = 19970, upload-time = "2022-11-02T17:34:01.425Z" }, ] +[[package]] +name = "flask" +version = "3.1.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "blinker" }, + { name = "click" }, + { name = "itsdangerous" }, + { name = "jinja2" }, + { name = "markupsafe" }, + { name = "werkzeug" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/dc/6d/cfe3c0fcc5e477df242b98bfe186a4c34357b4847e87ecaef04507332dab/flask-3.1.2.tar.gz", hash = "sha256:bf656c15c80190ed628ad08cdfd3aaa35beb087855e2f494910aa3774cc4fd87", size = 720160, upload-time = "2025-08-19T21:03:21.205Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ec/f9/7f9263c5695f4bd0023734af91bedb2ff8209e8de6ead162f35d8dc762fd/flask-3.1.2-py3-none-any.whl", hash = "sha256:ca1d8112ec8a6158cc29ea4858963350011b5c846a414cdb7a954aa9e967d03c", size = 103308, upload-time = "2025-08-19T21:03:19.499Z" }, +] + [[package]] name = "fqdn" version = "1.5.1" @@ -1227,11 +1345,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/eb/02/a6b21098b1d5d6249b7c5ab69dde30108a71e4e819d4a9778f1de1d5b70d/fsspec-2025.10.0-py3-none-any.whl", hash = "sha256:7c7712353ae7d875407f97715f0e1ffcc21e33d5b24556cb1e090ae9409ec61d", size = 200966, upload-time = "2025-10-30T14:58:42.53Z" }, ] -[package.optional-dependencies] -http = [ - { name = "aiohttp" }, -] - [[package]] name = "ghp-import" version = "2.1.0" @@ -1464,7 +1577,7 @@ wheels = [ [[package]] name = "huggingface-hub" -version = "0.36.0" +version = "0.33.4" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "filelock" }, @@ -1476,9 +1589,9 @@ dependencies = [ { name = "tqdm" }, { name = "typing-extensions" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/98/63/4910c5fa9128fdadf6a9c5ac138e8b1b6cee4ca44bf7915bbfbce4e355ee/huggingface_hub-0.36.0.tar.gz", hash = "sha256:47b3f0e2539c39bf5cde015d63b72ec49baff67b6931c3d97f3f84532e2b8d25", size = 463358, upload-time = "2025-10-23T12:12:01.413Z" } +sdist = { url = "https://files.pythonhosted.org/packages/4b/9e/9366b7349fc125dd68b9d384a0fea84d67b7497753fe92c71b67e13f47c4/huggingface_hub-0.33.4.tar.gz", hash = "sha256:6af13478deae120e765bfd92adad0ae1aec1ad8c439b46f23058ad5956cbca0a", size = 426674, upload-time = "2025-07-11T12:32:48.694Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/cb/bd/1a875e0d592d447cbc02805fd3fe0f497714d6a2583f59d14fa9ebad96eb/huggingface_hub-0.36.0-py3-none-any.whl", hash = "sha256:7bcc9ad17d5b3f07b57c78e79d527102d08313caa278a641993acddcb894548d", size = 566094, upload-time = "2025-10-23T12:11:59.557Z" }, + { url = "https://files.pythonhosted.org/packages/46/7b/98daa50a2db034cab6cd23a3de04fa2358cb691593d28e9130203eb7a805/huggingface_hub-0.33.4-py3-none-any.whl", hash = "sha256:09f9f4e7ca62547c70f8b82767eefadd2667f4e116acba2e3e62a5a81815a7bb", size = 515339, upload-time = "2025-07-11T12:32:46.346Z" }, ] [[package]] @@ -1520,6 +1633,14 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12", size = 7484, upload-time = "2025-10-18T21:55:41.639Z" }, ] +[[package]] +name = "inputimeout" +version = "1.0.4" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/97/9c/1646ca469bc2dc299ac393c8d31136c6c22a35ca1e373fa462ac01100d37/inputimeout-1.0.4-py3-none-any.whl", hash = "sha256:f4e23d27753cfc25268eefc8d52a3edc46280ad831d226617c51882423475a43", size = 4639, upload-time = "2018-03-02T14:28:06.903Z" }, +] + [[package]] name = "ipykernel" version = "6.31.0" @@ -1577,9 +1698,9 @@ version = "9.8.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'linux'", - "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'linux'", "python_full_version == '3.11.*' and sys_platform == 'linux'", + "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'linux'", "python_full_version == '3.11.*' and sys_platform != 'linux'", ] @@ -1642,6 +1763,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/7b/55/e5326141505c5d5e34c5e0935d2908a74e4561eca44108fbfb9c13d2911a/isoduration-20.11.0-py3-none-any.whl", hash = "sha256:b2904c2a4228c3d44f409c8ae8e2370eb21a26f7ac2ec5446df141dde3452042", size = 11321, upload-time = "2020-11-01T10:59:58.02Z" }, ] +[[package]] +name = "itsdangerous" +version = "2.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9c/cb/8ac0172223afbccb63986cc25049b154ecfb5e85932587206f42317be31d/itsdangerous-2.2.0.tar.gz", hash = "sha256:e0050c0b7da1eea53ffaf149c0cfbb5c6e2e2b69c4bef22c81fa6eb73e5f6173", size = 54410, upload-time = "2024-04-16T21:28:15.614Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/04/96/92447566d16df59b2a776c0fb82dbc4d9e07cd95062562af01e408583fc4/itsdangerous-2.2.0-py3-none-any.whl", hash = "sha256:c6242fc49e35958c8b15141343aa660db5fc54d4f13a1db01a3f5891b98700ef", size = 16234, upload-time = "2024-04-16T21:28:14.499Z" }, +] + +[[package]] +name = "javascript" +version = "1!1.2.6" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/1d/e5/782b7cfba2491e96ff463e24fadb4486ce2bc226f2071e493a9caa07f345/javascript-1!1.2.6.tar.gz", hash = "sha256:442e885b54dd9a6afe797dd6d5c3c575ec38da02a7d16749bf315aad0fa620c9", size = 38508, upload-time = "2025-09-25T11:15:44.411Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/95/4f/43e4b0bd6b76930e921cf5d9357cefb8ace9a2615bf53c05ff2e314ec434/javascript-1!1.2.6-py3-none-any.whl", hash = "sha256:0c68af196d450715bb74e9a25f11db67435070d91ceaff5ef28c4b4c95235ebf", size = 34802, upload-time = "2025-09-25T11:15:42.142Z" }, +] + [[package]] name = "jedi" version = "0.19.2" @@ -2060,6 +2199,23 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/bd/0d/2d240e7098e0cafba4d25e9530e7596b1bb1bd4476e41b10346bcaaa36d6/jupytext-1.18.1-py3-none-any.whl", hash = "sha256:24f999400726a1c658beae55e15fdd2a6255ab1a418697864cd779874e6011ab", size = 167143, upload-time = "2025-10-19T15:06:28.975Z" }, ] +[[package]] +name = "keybert" +version = "0.9.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "2.4.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "rich" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "sentence-transformers" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/b7/a8/a73003cc0f6a606eb30c9e19a85b5a89bef076e9adcb642aed29e47c693c/keybert-0.9.0.tar.gz", hash = "sha256:eec8ebd37d3d9fa238d7075e727adb993de4f3ed67c94112840e4cc0e55d7657", size = 32235, upload-time = "2025-02-07T08:45:13.187Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/82/07/e2f42a8ec3ff1935debbf2a5255570d22033fca3fe3180d5af99a6c9ee8c/keybert-0.9.0-py3-none-any.whl", hash = "sha256:afa2f300a72f69d279e4482bc85d8b34493b119876dc0818cb4f260466285b36", size = 41364, upload-time = "2025-02-07T08:45:08.093Z" }, +] + [[package]] name = "langchain" version = "1.2.0" @@ -2213,6 +2369,99 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/82/3d/14ce75ef66813643812f3093ab17e46d3a206942ce7376d31ec2d36229e7/lark-1.3.1-py3-none-any.whl", hash = "sha256:c629b661023a014c37da873b4ff58a817398d12635d3bbb2c5a03be7fe5d1e12", size = 113151, upload-time = "2025-10-27T18:25:54.882Z" }, ] +[[package]] +name = "levenshtein" +version = "0.27.3" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "rapidfuzz" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/82/56/dcf68853b062e3b94bdc3d011cc4198779abc5b9dc134146a062920ce2e2/levenshtein-0.27.3.tar.gz", hash = "sha256:1ac326b2c84215795163d8a5af471188918b8797b4953ec87aaba22c9c1f9fc0", size = 393269, upload-time = "2025-11-01T12:14:31.04Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ab/07/e8d04ec84fae72f0a75a2c46f897fe2abb82a657707a902a414faa5f8a72/levenshtein-0.27.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d61eff70799fd5e710625da8a13e5adabd62bfd9f70abb9c531af6cad458cd27", size = 171954, upload-time = "2025-11-01T12:12:40.151Z" }, + { url = "https://files.pythonhosted.org/packages/8d/13/606682ad2a7f0c01178cbc1f8de1b53d86e5dd8a03983c8feb8a6f403e76/levenshtein-0.27.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:477efed87edf72ad0d3870038479ed2f63020a42e69c6a38a32a550e51f8e70e", size = 158414, upload-time = "2025-11-01T12:12:42.169Z" }, + { url = "https://files.pythonhosted.org/packages/ce/c5/9627e1fc5cbfaff7fbf2e95aaf29340929ff2e92ae2d185b967a36942262/levenshtein-0.27.3-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8ef99b9827d7d1100fc4398ac5522bd56766b894561c0cbdea0a01b93f24e642", size = 133822, upload-time = "2025-11-01T12:12:43.243Z" }, + { url = "https://files.pythonhosted.org/packages/32/88/9e24a51b99b3dd6b3706a94bd258b2254edab5392e92c2e6d9b0773eba8f/levenshtein-0.27.3-cp310-cp310-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9091e8ca9fff6088836abf372f8871fb480e44603defa526e1c3ae2f1d70acc5", size = 114383, upload-time = "2025-11-01T12:12:44.4Z" }, + { url = "https://files.pythonhosted.org/packages/4c/95/9a11eb769bad0583712e2772e90ef92929d4ff4931fbb34efe79a0bff493/levenshtein-0.27.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6ffdb2329712c5595eda3532a4f701f87f6c73a0f7aaac240681bf0b54310d63", size = 153061, upload-time = "2025-11-01T12:12:46.215Z" }, + { url = "https://files.pythonhosted.org/packages/b3/86/47387ed38df23ed3a6640032cdca97367eacb2a2d2075d97d6e88f43b40e/levenshtein-0.27.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:35856330eac1b968b45a5abbc4a3d14279bd9d1224be727cb1aac9ac4928a419", size = 1115566, upload-time = "2025-11-01T12:12:47.965Z" }, + { url = "https://files.pythonhosted.org/packages/dc/17/ed94dadabdf7e86940f6179238312a6750688f44565a4eb19ae5a87ce8a8/levenshtein-0.27.3-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:5377e237f6a13f5b0618621cca7992848993470c011716c3ad09cdf19c3b13ab", size = 1007140, upload-time = "2025-11-01T12:12:49.283Z" }, + { url = "https://files.pythonhosted.org/packages/52/25/c971c043aec0994c5600789d2bf4c183e2f389ee21559bb46a06c6f46ec2/levenshtein-0.27.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:e30614186eb5c43833b62ae7d893a116b88373eec8cf3f3d62ba51aa5962d8ea", size = 1185316, upload-time = "2025-11-01T12:12:50.849Z" }, + { url = "https://files.pythonhosted.org/packages/3c/54/2a1a1af73470cd6ca0d709efb1786fe4651eee9a3cb5b767903defb4fe9c/levenshtein-0.27.3-cp310-cp310-win32.whl", hash = "sha256:5499342fd6b003bd5abc28790c7b333884838f7fd8c50570a6520bbaf5e2a35b", size = 84312, upload-time = "2025-11-01T12:12:52.366Z" }, + { url = "https://files.pythonhosted.org/packages/10/15/50f508790a7b7e0d6258ec85add62c257ab27ca70e5e8a1bae8350305932/levenshtein-0.27.3-cp310-cp310-win_amd64.whl", hash = "sha256:9e2792730388bec6a85d4d3e3a9b53b8a4b509722bea1a78a39a1a0a7d8f0e13", size = 94376, upload-time = "2025-11-01T12:12:53.361Z" }, + { url = "https://files.pythonhosted.org/packages/9a/3f/ca3e54e5144695cc8a34601d275fabfc97c2ab9b824cbe0b49a0173a0575/levenshtein-0.27.3-cp310-cp310-win_arm64.whl", hash = "sha256:8a2a274b55562a49c6e9dadb16d05f6c27ffa98906b55d5c122893457ca6e464", size = 87216, upload-time = "2025-11-01T12:12:54.674Z" }, + { url = "https://files.pythonhosted.org/packages/0e/fd/42e28a86e2f04a2e064faa1eab7d81a35fb111212b508ce7e450f839943d/levenshtein-0.27.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:245b6ffb6e1b0828cafbce35c500cb3265d0962c121d090669f177968c5a2980", size = 172216, upload-time = "2025-11-01T12:12:55.727Z" }, + { url = "https://files.pythonhosted.org/packages/1d/f4/fe665c8e5d8ebe4266807e43af72db9d4f84d4f513ea86eacca3aaf5f77b/levenshtein-0.27.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8f44c98fa23f489eb7b2ad87d5dd24b6a784434bb5edb73f6b0513309c949690", size = 158616, upload-time = "2025-11-01T12:12:56.99Z" }, + { url = "https://files.pythonhosted.org/packages/22/46/9998bc56729444e350c083635b94c3eae97218b8a618cdc89f6825eec08c/levenshtein-0.27.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1f5f85a1fc96dfc147bba82b4c67d6346ea26c27ef77a6a9de689118e26dddbe", size = 134222, upload-time = "2025-11-01T12:12:58.437Z" }, + { url = "https://files.pythonhosted.org/packages/19/09/914b3fc22c083728904f8dc7876a2a90a602b4769f27f5320176cbd6f781/levenshtein-0.27.3-cp311-cp311-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:18ceddd38d0e990d2c1c9b72f3e191dace87e2f8f0446207ce9e9cd2bfdfc8a1", size = 114902, upload-time = "2025-11-01T12:12:59.645Z" }, + { url = "https://files.pythonhosted.org/packages/d2/ee/f361bfa5afe24698fb07ae7811e00c2984131023c7688299dea4fd3f2f4c/levenshtein-0.27.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:222b81adca29ee4128183328c6e1b25a48c817d14a008ab49e74be9df963b293", size = 153562, upload-time = "2025-11-01T12:13:00.745Z" }, + { url = "https://files.pythonhosted.org/packages/a3/4f/614d0ab9777ebb91895ce1c9390ec2f244f53f7ddf7e29f36b0ca33f3841/levenshtein-0.27.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ee3769ab6e89c24f901e6b7004100630e86721464d7d0384860a322d7953d3a5", size = 1115732, upload-time = "2025-11-01T12:13:02.219Z" }, + { url = "https://files.pythonhosted.org/packages/24/d9/f33c4e35399349ec2eb7be53ed49459bf6e59c31668868c89cf6f7964029/levenshtein-0.27.3-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:03eba8fda9f3f2b4b0760263fa20b20a90ab00cbeeab4d0d9d899b4f77912b0a", size = 1009023, upload-time = "2025-11-01T12:13:03.954Z" }, + { url = "https://files.pythonhosted.org/packages/2e/63/e8803a6d71488334c100afc79a98efc8cf0086ad29ee7f1d083f7f2c584d/levenshtein-0.27.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c870b19e2d5c7bc7f16213cc10312b82d873a4d46e1c6d51857a12ef39a76552", size = 1185850, upload-time = "2025-11-01T12:13:05.341Z" }, + { url = "https://files.pythonhosted.org/packages/09/55/a6a815ef76a6d5f7a2ee4e1edc8e8f1f935b9fa278634cc687af19b86de9/levenshtein-0.27.3-cp311-cp311-win32.whl", hash = "sha256:1987622e9b8ba2ae47dc27469291da1f58462660fa34f4358e9d9c1830fb1355", size = 84375, upload-time = "2025-11-01T12:13:06.647Z" }, + { url = "https://files.pythonhosted.org/packages/e5/36/cf4c36ffe91994e772b682ff4c3cb721bd50ac05d4a887baa35f4d3b2268/levenshtein-0.27.3-cp311-cp311-win_amd64.whl", hash = "sha256:a2b2aa81851e01bb09667b07e80c3fbf0f5a7c6ee9cd80caf43cce705e65832a", size = 94598, upload-time = "2025-11-01T12:13:07.68Z" }, + { url = "https://files.pythonhosted.org/packages/92/4b/43e820c3a13033908925eae8614ad7c0be1e5868836770565174012158c0/levenshtein-0.27.3-cp311-cp311-win_arm64.whl", hash = "sha256:a084b335c54def1aef9a594b7163faa44dd00056323808bab783f43d8e4c1395", size = 87133, upload-time = "2025-11-01T12:13:08.701Z" }, + { url = "https://files.pythonhosted.org/packages/7c/8e/3be9d8e0245704e3af5258fb6cb157c3d59902e1351e95edf6ed8a8c0434/levenshtein-0.27.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2de7f095b0ca8e44de9de986ccba661cd0dec3511c751b499e76b60da46805e9", size = 169622, upload-time = "2025-11-01T12:13:10.026Z" }, + { url = "https://files.pythonhosted.org/packages/a6/42/a2b2fda5e8caf6ecd5aac142f946a77574a3961e65da62c12fd7e48e5cb1/levenshtein-0.27.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:d9b8b29e5d5145a3c958664c85151b1bb4b26e4ca764380b947e6a96a321217c", size = 159183, upload-time = "2025-11-01T12:13:11.197Z" }, + { url = "https://files.pythonhosted.org/packages/eb/c4/f083fabbd61c449752df1746533538f4a8629e8811931b52f66e6c4290ad/levenshtein-0.27.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fc975465a51b1c5889eadee1a583b81fba46372b4b22df28973e49e8ddb8f54a", size = 133120, upload-time = "2025-11-01T12:13:12.363Z" }, + { url = "https://files.pythonhosted.org/packages/4e/e5/b6421e04cb0629615b8efd6d4d167dd2b1afb5097b87bb83cd992004dcca/levenshtein-0.27.3-cp312-cp312-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:57573ed885118554770979fdee584071b66103f6d50beddeabb54607a1213d81", size = 114988, upload-time = "2025-11-01T12:13:13.486Z" }, + { url = "https://files.pythonhosted.org/packages/e5/77/39ee0e8d3028e90178e1031530ccc98563f8f2f0d905ec784669dcf0fa90/levenshtein-0.27.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:23aff800a6dd5d91bb3754a6092085aa7ad46b28e497682c155c74f681cfaa2d", size = 153346, upload-time = "2025-11-01T12:13:14.744Z" }, + { url = "https://files.pythonhosted.org/packages/3c/0d/c0f367bbd260dbd7a4e134fd21f459e0f5eac43deac507952b46a1d8a93a/levenshtein-0.27.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:c08a952432b8ad9dccb145f812176db94c52cda732311ddc08d29fd3bf185b0a", size = 1114538, upload-time = "2025-11-01T12:13:15.851Z" }, + { url = "https://files.pythonhosted.org/packages/d8/ef/ae71433f7b4db0bd2af7974785e36cdec899919203fb82e647c5a6109c07/levenshtein-0.27.3-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:3bfcb2d78ab9cc06a1e75da8fcfb7a430fe513d66cfe54c07e50f32805e5e6db", size = 1009734, upload-time = "2025-11-01T12:13:17.212Z" }, + { url = "https://files.pythonhosted.org/packages/27/dc/62c28b812dcb0953fc32ab7adf3d0e814e43c8560bb28d9269a44d874adf/levenshtein-0.27.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:ba7235f6dcb31a217247468295e2dd4c6c1d3ac81629dc5d355d93e1a5f4c185", size = 1185581, upload-time = "2025-11-01T12:13:18.661Z" }, + { url = "https://files.pythonhosted.org/packages/56/e8/2e7ab9c565793220edb8e5432f9a846386a157075bdd032a90e9585bce38/levenshtein-0.27.3-cp312-cp312-win32.whl", hash = "sha256:ea80d70f1d18c161a209be556b9094968627cbaae620e102459ef9c320a98cbb", size = 84660, upload-time = "2025-11-01T12:13:19.87Z" }, + { url = "https://files.pythonhosted.org/packages/2c/a6/907a1fc8587dc91c40156973e09d106ab064c06eb28dc4700ba0fe54d654/levenshtein-0.27.3-cp312-cp312-win_amd64.whl", hash = "sha256:fbaa1219d9b2d955339a37e684256a861e9274a3fe3a6ee1b8ea8724c3231ed9", size = 94909, upload-time = "2025-11-01T12:13:21.323Z" }, + { url = "https://files.pythonhosted.org/packages/d5/d6/e04f0ddf6a71df3cdd1817b71703490ac874601ed460b2af172d3752c321/levenshtein-0.27.3-cp312-cp312-win_arm64.whl", hash = "sha256:2edbaa84f887ea1d9d8e4440af3fdda44769a7855d581c6248d7ee51518402a8", size = 87358, upload-time = "2025-11-01T12:13:22.393Z" }, + { url = "https://files.pythonhosted.org/packages/3e/f2/162e9ea7490b36bbf05776c8e3a8114c75aa78546ddda8e8f36731db3da6/levenshtein-0.27.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:e55aa9f9453fd89d4a9ff1f3c4a650b307d5f61a7eed0568a52fbd2ff2eba107", size = 169230, upload-time = "2025-11-01T12:13:23.735Z" }, + { url = "https://files.pythonhosted.org/packages/01/2d/7316ba7f94e3d60e89bd120526bc71e4812866bb7162767a2a10f73f72c5/levenshtein-0.27.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:ae4d484453c48939ecd01c5c213530c68dd5cd6e5090f0091ef69799ec7a8a9f", size = 158643, upload-time = "2025-11-01T12:13:25.549Z" }, + { url = "https://files.pythonhosted.org/packages/5e/87/85433cb1e51c45016f061d96fea3106b6969f700e2cbb56c15de82d0deeb/levenshtein-0.27.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:d18659832567ee387b266be390da0de356a3aa6cf0e8bc009b6042d8188e131f", size = 132881, upload-time = "2025-11-01T12:13:26.822Z" }, + { url = "https://files.pythonhosted.org/packages/40/1c/3ce66c9a7da169a43dd89146d69df9dec935e6f86c70c6404f48d1291d2c/levenshtein-0.27.3-cp313-cp313-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:027b3d142cc8ea2ab4e60444d7175f65a94dde22a54382b2f7b47cc24936eb53", size = 114650, upload-time = "2025-11-01T12:13:28.382Z" }, + { url = "https://files.pythonhosted.org/packages/73/60/7138e98884ca105c76ef192f5b43165d6eac6f32b432853ebe9f09ee50c9/levenshtein-0.27.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ffdca6989368cc64f347f0423c528520f12775b812e170a0eb0c10e4c9b0f3ff", size = 153127, upload-time = "2025-11-01T12:13:29.781Z" }, + { url = "https://files.pythonhosted.org/packages/df/8f/664ac8b83026d7d1382866b68babae17e92b7b6ff8dc3c6205c0066b8ce1/levenshtein-0.27.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:fa00ab389386032b02a1c9050ec3c6aa824d2bbcc692548fdc44a46b71c058c6", size = 1114602, upload-time = "2025-11-01T12:13:31.651Z" }, + { url = "https://files.pythonhosted.org/packages/2c/c8/8905d96cf2d7ed6af7eb39a8be0925ef335729473c1e9d1f56230ecaffc5/levenshtein-0.27.3-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:691c9003c6c481b899a5c2f72e8ce05a6d956a9668dc75f2a3ce9f4381a76dc6", size = 1008036, upload-time = "2025-11-01T12:13:33.006Z" }, + { url = "https://files.pythonhosted.org/packages/c7/57/01c37608121380a6357a297625562adad1c1fc8058d4f62279b735108927/levenshtein-0.27.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:12f7fc8bf0c24492fe97905348e020b55b9fc6dbaab7cd452566d1a466cb5e15", size = 1185338, upload-time = "2025-11-01T12:13:34.452Z" }, + { url = "https://files.pythonhosted.org/packages/dd/57/bceab41d40b58dee7927a8d1d18ed3bff7c95c5e530fb60093ce741a8c26/levenshtein-0.27.3-cp313-cp313-win32.whl", hash = "sha256:9f4872e4e19ee48eed39f214eea4eca42e5ef303f8a4a488d8312370674dbf3a", size = 84562, upload-time = "2025-11-01T12:13:35.858Z" }, + { url = "https://files.pythonhosted.org/packages/42/1d/74f1ff589bb687d0cad2bbdceef208dc070f56d1e38a3831da8c00bf13bb/levenshtein-0.27.3-cp313-cp313-win_amd64.whl", hash = "sha256:83aa2422e9a9af2c9d3e56a53e3e8de6bae58d1793628cae48c4282577c5c2c6", size = 94658, upload-time = "2025-11-01T12:13:36.963Z" }, + { url = "https://files.pythonhosted.org/packages/21/3c/22c86d3c8f254141096fd6089d2e9fdf98b1472c7a5d79d36d3557ec2d83/levenshtein-0.27.3-cp313-cp313-win_arm64.whl", hash = "sha256:d4adaf1edbcf38c3f2e290b52f4dcb5c6deff20308c26ef1127a106bc2d23e9f", size = 86929, upload-time = "2025-11-01T12:13:37.997Z" }, + { url = "https://files.pythonhosted.org/packages/0e/bc/9b7cf1b5fa098b86844d42de22549304699deff309c5c9e28b9a3fc4076a/levenshtein-0.27.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:272e24764b8210337b65a1cfd69ce40df5d2de1a3baf1234e7f06d2826ba2e7a", size = 170360, upload-time = "2025-11-01T12:13:39.019Z" }, + { url = "https://files.pythonhosted.org/packages/dc/95/997f2c83bd4712426bf0de8143b5e4403c7ebbafb5d1271983e774de3ae7/levenshtein-0.27.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:329a8e748a4e14d56daaa11f07bce3fde53385d05bad6b3f6dd9ee7802cdc915", size = 159098, upload-time = "2025-11-01T12:13:40.17Z" }, + { url = "https://files.pythonhosted.org/packages/fc/96/123c3316ae2f72c73be4fba9756924af015da4c0e5b12804f5753c0ee511/levenshtein-0.27.3-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a5fea1a9c6b9cc8729e467e2174b4359ff6bac27356bb5f31898e596b4ce133a", size = 136655, upload-time = "2025-11-01T12:13:41.262Z" }, + { url = "https://files.pythonhosted.org/packages/45/72/a3180d437736b1b9eacc3100be655a756deafb91de47c762d40eb45a9d91/levenshtein-0.27.3-cp313-cp313t-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:3a61aa825819b6356555091d8a575d1235bd9c3753a68316a261af4856c3b487", size = 117511, upload-time = "2025-11-01T12:13:42.647Z" }, + { url = "https://files.pythonhosted.org/packages/61/f9/ba7c546a4b99347938e6661104064ab6a3651c601d59f241ffdc37510ecc/levenshtein-0.27.3-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a51de7a514e8183f0a82f2947d01b014d2391426543b1c076bf5a26328cec4e4", size = 155656, upload-time = "2025-11-01T12:13:44.208Z" }, + { url = "https://files.pythonhosted.org/packages/42/cd/5edd6e1e02c3e47c8121761756dd0f85f816b636f25509118b687e6b0f96/levenshtein-0.27.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:53cbf726d6e92040c9be7e594d959d496bd62597ea48eba9d96105898acbeafe", size = 1116689, upload-time = "2025-11-01T12:13:45.485Z" }, + { url = "https://files.pythonhosted.org/packages/95/67/25ca0119e0c6ec17226c72638f48ef8887124597ac48ad5da111c0b3a825/levenshtein-0.27.3-cp313-cp313t-musllinux_1_2_armv7l.whl", hash = "sha256:191b358afead8561c4fcfed22f83c13bb6c8da5f5789e277f0c5aa1c45ca612f", size = 1003166, upload-time = "2025-11-01T12:13:47.126Z" }, + { url = "https://files.pythonhosted.org/packages/45/64/ab216f3fb3cef1ee7e222665537f9340d828ef84c99409ba31f2ef2a3947/levenshtein-0.27.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:ba1318d0635b834b8f0397014a7c43f007e65fce396a47614780c881bdff828b", size = 1189362, upload-time = "2025-11-01T12:13:48.627Z" }, + { url = "https://files.pythonhosted.org/packages/31/58/b150034858de0899a5a222974b6710618ebc0779a0695df070f7ab559a0b/levenshtein-0.27.3-cp313-cp313t-win32.whl", hash = "sha256:8dd9e1db6c3b35567043e155a686e4827c4aa28a594bd81e3eea84d3a1bd5875", size = 86149, upload-time = "2025-11-01T12:13:50.588Z" }, + { url = "https://files.pythonhosted.org/packages/0a/c4/bbe46a11073641450200e6a604b3b62d311166e8061c492612a40e560e85/levenshtein-0.27.3-cp313-cp313t-win_amd64.whl", hash = "sha256:7813ecdac7a6223264ebfea0c8d69959c43d21a99694ef28018d22c4265c2af6", size = 96685, upload-time = "2025-11-01T12:13:51.641Z" }, + { url = "https://files.pythonhosted.org/packages/23/65/30b362ad9bfc1085741776a08b6ddee3f434e9daac2920daaee2e26271bf/levenshtein-0.27.3-cp313-cp313t-win_arm64.whl", hash = "sha256:8f05a0d23d13a6f802c7af595d0e43f5b9b98b6ed390cec7a35cb5d6693b882b", size = 88538, upload-time = "2025-11-01T12:13:52.757Z" }, + { url = "https://files.pythonhosted.org/packages/f3/e1/2f705da403f865a5fa3449b155738dc9c53021698fd6926253a9af03180b/levenshtein-0.27.3-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a6728bfae9a86002f0223576675fc7e2a6e7735da47185a1d13d1eaaa73dd4be", size = 169457, upload-time = "2025-11-01T12:13:53.778Z" }, + { url = "https://files.pythonhosted.org/packages/76/2c/bb6ef359e007fe7b6b3195b68a94f4dd3ecd1885ee337ee8fbd4df55996f/levenshtein-0.27.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:8e5037c4a6f97a238e24aad6f98a1e984348b7931b1b04b6bd02bd4f8238150d", size = 158680, upload-time = "2025-11-01T12:13:55.005Z" }, + { url = "https://files.pythonhosted.org/packages/51/7b/de1999f4cf1cfebc3fbbf03a6d58498952d6560d9798af4b0a566e6b6f30/levenshtein-0.27.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c6cf5ecf9026bf24cf66ad019c6583f50058fae3e1b3c20e8812455b55d597f1", size = 133167, upload-time = "2025-11-01T12:13:56.426Z" }, + { url = "https://files.pythonhosted.org/packages/c7/da/aaa7f3a0a8ae8744b284043653652db3d7d93595517f9ed8158c03287692/levenshtein-0.27.3-cp314-cp314-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:9285084bd2fc19adb47dab54ed4a71f57f78fe0d754e4a01e3c75409a25aed24", size = 114530, upload-time = "2025-11-01T12:13:57.883Z" }, + { url = "https://files.pythonhosted.org/packages/29/ce/ed422816fb30ffa3bc11597b30d5deca06b4a1388707a04215da73c65b53/levenshtein-0.27.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ce3bbbe92172a08b599d79956182c6b7ab6ec8d4adbe7237417a363b968ad87b", size = 153325, upload-time = "2025-11-01T12:13:59.318Z" }, + { url = "https://files.pythonhosted.org/packages/d9/5a/a225477a0bda154f19f1c07a5e35500d631ae25dfd620b479027d79f0d4c/levenshtein-0.27.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9dac48fab9d166ca90e12fb6cf6c7c8eb9c41aacf7136584411e20f7f136f745", size = 1114956, upload-time = "2025-11-01T12:14:00.543Z" }, + { url = "https://files.pythonhosted.org/packages/ca/c4/a1be1040f3cce516a5e2be68453fd0c32ac63b2e9d31f476723fd8002c09/levenshtein-0.27.3-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:d37a83722dc5326c93d17078e926c4732dc4f3488dc017c6839e34cd16af92b7", size = 1007610, upload-time = "2025-11-01T12:14:02.036Z" }, + { url = "https://files.pythonhosted.org/packages/86/d7/6f50e8a307e0c2befd819b481eb3a4c2eacab3dd8101982423003fac8ea3/levenshtein-0.27.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:3466cb8294ce586e49dd467560a153ab8d296015c538223f149f9aefd3d9f955", size = 1185379, upload-time = "2025-11-01T12:14:03.385Z" }, + { url = "https://files.pythonhosted.org/packages/6b/e5/5d8fb1b3ebd5735f53221bf95c923066bcfc132234925820128f7eee5b47/levenshtein-0.27.3-cp314-cp314-win32.whl", hash = "sha256:c848bf2457b268672b7e9e73b44f18f49856420ac50b2564cf115a6e4ef82688", size = 86328, upload-time = "2025-11-01T12:14:04.74Z" }, + { url = "https://files.pythonhosted.org/packages/30/82/8a9ccbdb4e38bd4d516f2804999dccb8cb4bcb4e33f52851735da0c73ea7/levenshtein-0.27.3-cp314-cp314-win_amd64.whl", hash = "sha256:742633f024362a4ed6ef9d7e75d68f74b041ae738985fcf55a0e6d1d4cade438", size = 96640, upload-time = "2025-11-01T12:14:06.24Z" }, + { url = "https://files.pythonhosted.org/packages/14/86/f9d15919f59f5d92c6baa500315e1fa0143a39d811427b83c54f038267ca/levenshtein-0.27.3-cp314-cp314-win_arm64.whl", hash = "sha256:9eed6851224b19e8d588ddb8eb8a4ae3c2dcabf3d1213985f0b94a67e517b1df", size = 89689, upload-time = "2025-11-01T12:14:07.379Z" }, + { url = "https://files.pythonhosted.org/packages/ed/f6/10f44975ae6dc3047b2cd260e3d4c3a5258b8d10690a42904115de24fc51/levenshtein-0.27.3-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:77de69a345c76227b51a4521cd85442eb3da54c7eb6a06663a20c058fc49e683", size = 170518, upload-time = "2025-11-01T12:14:09.196Z" }, + { url = "https://files.pythonhosted.org/packages/08/07/fa294a145a0c99a814a9a807614962c1ee0f5749ca691645980462027d5d/levenshtein-0.27.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:eba2756dc1f5b962b0ff80e49abb2153d5e809cc5e7fa5e85be9410ce474795d", size = 159097, upload-time = "2025-11-01T12:14:10.404Z" }, + { url = "https://files.pythonhosted.org/packages/ae/50/24bdf37813fc30f293e53b46022b091144f4737a6a66663d2235b311bb98/levenshtein-0.27.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2c8fcb498287e971d84260f67808ff1a06b3f6212d80fea75cf5155db80606ff", size = 136650, upload-time = "2025-11-01T12:14:11.579Z" }, + { url = "https://files.pythonhosted.org/packages/d0/a9/0399c7a190b277cdea3acc801129d9d30da57c3fa79519e7b8c3f080d86c/levenshtein-0.27.3-cp314-cp314t-manylinux_2_24_armv7l.manylinux_2_31_armv7l.whl", hash = "sha256:f067092c67464faab13e00a5c1a80da93baca8955d4d49579861400762e35591", size = 117515, upload-time = "2025-11-01T12:14:12.877Z" }, + { url = "https://files.pythonhosted.org/packages/bf/a4/1c27533e97578b385a4b8079abe8d1ce2e514717c761efbe4bf7bbd0ac2e/levenshtein-0.27.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:92415f32c68491203f2855d05eef3277d376182d014cf0859c013c89f277fbbf", size = 155711, upload-time = "2025-11-01T12:14:13.985Z" }, + { url = "https://files.pythonhosted.org/packages/50/35/bbc26638394a72b1e31a685ec251c995ee66a630c7e5c86f98770928b632/levenshtein-0.27.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:ef61eeaf1e0a42d7d947978d981fe4b9426b98b3dd8c1582c535f10dee044c3f", size = 1116692, upload-time = "2025-11-01T12:14:15.359Z" }, + { url = "https://files.pythonhosted.org/packages/cd/83/32fcf28b388f8dc6c36b54552b9bae289dab07d43df104893158c834cbcc/levenshtein-0.27.3-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:103bb2e9049d1aa0d1216dd09c1c9106ecfe7541bbdc1a0490b9357d42eec8f2", size = 1003167, upload-time = "2025-11-01T12:14:17.469Z" }, + { url = "https://files.pythonhosted.org/packages/d1/79/1fbf2877ec4b819f373a32ebe3c48a61ee810693593a6015108b0be97b78/levenshtein-0.27.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:6a64ddd1986b2a4c468b09544382287315c53585eb067f6e200c337741e057ee", size = 1189417, upload-time = "2025-11-01T12:14:19.081Z" }, + { url = "https://files.pythonhosted.org/packages/d3/ac/dad4e09f1f7459c64172e48e40ed2baf3aa92d38205bcbd1b4ff00853701/levenshtein-0.27.3-cp314-cp314t-win32.whl", hash = "sha256:957244f27dc284ccb030a8b77b8a00deb7eefdcd70052a4b1d96f375780ae9dc", size = 88144, upload-time = "2025-11-01T12:14:20.667Z" }, + { url = "https://files.pythonhosted.org/packages/c0/61/cd51dc8b8a382e17c559a9812734c3a9afc2dab7d36253516335ee16ae50/levenshtein-0.27.3-cp314-cp314t-win_amd64.whl", hash = "sha256:ccd7eaa6d8048c3ec07c93cfbcdefd4a3ae8c6aca3a370f2023ee69341e5f076", size = 98516, upload-time = "2025-11-01T12:14:21.786Z" }, + { url = "https://files.pythonhosted.org/packages/27/5e/3fb67e882c1fee01ebb7abc1c0a6669e5ff8acd060e93bfe7229e9ce6e4f/levenshtein-0.27.3-cp314-cp314t-win_arm64.whl", hash = "sha256:1d8520b89b7a27bb5aadbcc156715619bcbf556a8ac46ad932470945dca6e1bd", size = 91020, upload-time = "2025-11-01T12:14:22.944Z" }, + { url = "https://files.pythonhosted.org/packages/b4/bc/21983893d3f40c6990e2e51c02dd48cfca350a36214be90d7c58f5f85896/levenshtein-0.27.3-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:d2d7d22b6117a143f0cf101fe18a3ca90bd949fc33716a42d6165b9768d4a78c", size = 166073, upload-time = "2025-11-01T12:14:24.436Z" }, + { url = "https://files.pythonhosted.org/packages/ef/bb/52deb821ebf0cfc61baf7c9ebc5601649cfbfdaaaf156867786d1c5332d5/levenshtein-0.27.3-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:a55e7a2f317abd28576636e1f840fd268261f447c496a8481a9997a5ce889c59", size = 153629, upload-time = "2025-11-01T12:14:25.623Z" }, + { url = "https://files.pythonhosted.org/packages/60/0c/b72e6e2d16efd57c143785a30370ca50c2e355a9d0d678edb1c024865447/levenshtein-0.27.3-pp311-pypy311_pp73-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:55fa5f11952c38186bd4719e936eb4595b3d519218634924928787c36840256c", size = 130242, upload-time = "2025-11-01T12:14:26.926Z" }, + { url = "https://files.pythonhosted.org/packages/b5/b0/0aafad0dab03a58fd507773d3ff94ec13efdd3772ba217f85366213ab7ae/levenshtein-0.27.3-pp311-pypy311_pp73-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:559d3588e6766134d95f84f830cf40166360e1769d253f5f83474bff10a24341", size = 150655, upload-time = "2025-11-01T12:14:28.034Z" }, + { url = "https://files.pythonhosted.org/packages/b7/77/42dbcbafe9e0b0eb14cb6b08378c8c3bdc563ee34ee58f62e708e7f8956e/levenshtein-0.27.3-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:82d40da143c1b9e27adcd34a33dfcc4a0761aa717c5f618b9c6f57dec5d7a958", size = 92370, upload-time = "2025-11-01T12:14:29.143Z" }, +] + [[package]] name = "litellm" version = "1.80.0" @@ -2303,6 +2552,142 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/90/ac/e911594a2f10445717ea45b61b3a93f3bb91594320745fe1bb796c2dc87a/llama_index_workflows-2.11.5-py3-none-any.whl", hash = "sha256:3c5a419129114bb0b1bd83b88aa5f653f84181b2e39e33473e8747ec6e88538e", size = 91982, upload-time = "2025-11-24T18:37:58.265Z" }, ] +[[package]] +name = "lxml" +version = "6.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/aa/88/262177de60548e5a2bfc46ad28232c9e9cbde697bd94132aeb80364675cb/lxml-6.0.2.tar.gz", hash = "sha256:cd79f3367bd74b317dda655dc8fcfa304d9eb6e4fb06b7168c5cf27f96e0cd62", size = 4073426, upload-time = "2025-09-22T04:04:59.287Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/db/8a/f8192a08237ef2fb1b19733f709db88a4c43bc8ab8357f01cb41a27e7f6a/lxml-6.0.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e77dd455b9a16bbd2a5036a63ddbd479c19572af81b624e79ef422f929eef388", size = 8590589, upload-time = "2025-09-22T04:00:10.51Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/27bcd07ae17ff5e5536e8d88f4c7d581b48963817a13de11f3ac3329bfa2/lxml-6.0.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:5d444858b9f07cefff6455b983aea9a67f7462ba1f6cbe4a21e8bf6791bf2153", size = 4629671, upload-time = "2025-09-22T04:00:15.411Z" }, + { url = "https://files.pythonhosted.org/packages/02/5a/a7d53b3291c324e0b6e48f3c797be63836cc52156ddf8f33cd72aac78866/lxml-6.0.2-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f952dacaa552f3bb8834908dddd500ba7d508e6ea6eb8c52eb2d28f48ca06a31", size = 4999961, upload-time = "2025-09-22T04:00:17.619Z" }, + { url = "https://files.pythonhosted.org/packages/f5/55/d465e9b89df1761674d8672bb3e4ae2c47033b01ec243964b6e334c6743f/lxml-6.0.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:71695772df6acea9f3c0e59e44ba8ac50c4f125217e84aab21074a1a55e7e5c9", size = 5157087, upload-time = "2025-09-22T04:00:19.868Z" }, + { url = "https://files.pythonhosted.org/packages/62/38/3073cd7e3e8dfc3ba3c3a139e33bee3a82de2bfb0925714351ad3d255c13/lxml-6.0.2-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:17f68764f35fd78d7c4cc4ef209a184c38b65440378013d24b8aecd327c3e0c8", size = 5067620, upload-time = "2025-09-22T04:00:21.877Z" }, + { url = "https://files.pythonhosted.org/packages/4a/d3/1e001588c5e2205637b08985597827d3827dbaaece16348c8822bfe61c29/lxml-6.0.2-cp310-cp310-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:058027e261afed589eddcfe530fcc6f3402d7fd7e89bfd0532df82ebc1563dba", size = 5406664, upload-time = "2025-09-22T04:00:23.714Z" }, + { url = "https://files.pythonhosted.org/packages/20/cf/cab09478699b003857ed6ebfe95e9fb9fa3d3c25f1353b905c9b73cfb624/lxml-6.0.2-cp310-cp310-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a8ffaeec5dfea5881d4c9d8913a32d10cfe3923495386106e4a24d45300ef79c", size = 5289397, upload-time = "2025-09-22T04:00:25.544Z" }, + { url = "https://files.pythonhosted.org/packages/a3/84/02a2d0c38ac9a8b9f9e5e1bbd3f24b3f426044ad618b552e9549ee91bd63/lxml-6.0.2-cp310-cp310-manylinux_2_31_armv7l.whl", hash = "sha256:f2e3b1a6bb38de0bc713edd4d612969dd250ca8b724be8d460001a387507021c", size = 4772178, upload-time = "2025-09-22T04:00:27.602Z" }, + { url = "https://files.pythonhosted.org/packages/56/87/e1ceadcc031ec4aa605fe95476892d0b0ba3b7f8c7dcdf88fdeff59a9c86/lxml-6.0.2-cp310-cp310-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d6690ec5ec1cce0385cb20896b16be35247ac8c2046e493d03232f1c2414d321", size = 5358148, upload-time = "2025-09-22T04:00:29.323Z" }, + { url = "https://files.pythonhosted.org/packages/fe/13/5bb6cf42bb228353fd4ac5f162c6a84fd68a4d6f67c1031c8cf97e131fc6/lxml-6.0.2-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f2a50c3c1d11cad0ebebbac357a97b26aa79d2bcaf46f256551152aa85d3a4d1", size = 5112035, upload-time = "2025-09-22T04:00:31.061Z" }, + { url = "https://files.pythonhosted.org/packages/e4/e2/ea0498552102e59834e297c5c6dff8d8ded3db72ed5e8aad77871476f073/lxml-6.0.2-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:3efe1b21c7801ffa29a1112fab3b0f643628c30472d507f39544fd48e9549e34", size = 4799111, upload-time = "2025-09-22T04:00:33.11Z" }, + { url = "https://files.pythonhosted.org/packages/6a/9e/8de42b52a73abb8af86c66c969b3b4c2a96567b6ac74637c037d2e3baa60/lxml-6.0.2-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:59c45e125140b2c4b33920d21d83681940ca29f0b83f8629ea1a2196dc8cfe6a", size = 5351662, upload-time = "2025-09-22T04:00:35.237Z" }, + { url = "https://files.pythonhosted.org/packages/28/a2/de776a573dfb15114509a37351937c367530865edb10a90189d0b4b9b70a/lxml-6.0.2-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:452b899faa64f1805943ec1c0c9ebeaece01a1af83e130b69cdefeda180bb42c", size = 5314973, upload-time = "2025-09-22T04:00:37.086Z" }, + { url = "https://files.pythonhosted.org/packages/50/a0/3ae1b1f8964c271b5eec91db2043cf8c6c0bce101ebb2a633b51b044db6c/lxml-6.0.2-cp310-cp310-win32.whl", hash = "sha256:1e786a464c191ca43b133906c6903a7e4d56bef376b75d97ccbb8ec5cf1f0a4b", size = 3611953, upload-time = "2025-09-22T04:00:39.224Z" }, + { url = "https://files.pythonhosted.org/packages/d1/70/bd42491f0634aad41bdfc1e46f5cff98825fb6185688dc82baa35d509f1a/lxml-6.0.2-cp310-cp310-win_amd64.whl", hash = "sha256:dacf3c64ef3f7440e3167aa4b49aa9e0fb99e0aa4f9ff03795640bf94531bcb0", size = 4032695, upload-time = "2025-09-22T04:00:41.402Z" }, + { url = "https://files.pythonhosted.org/packages/d2/d0/05c6a72299f54c2c561a6c6cbb2f512e047fca20ea97a05e57931f194ac4/lxml-6.0.2-cp310-cp310-win_arm64.whl", hash = "sha256:45f93e6f75123f88d7f0cfd90f2d05f441b808562bf0bc01070a00f53f5028b5", size = 3680051, upload-time = "2025-09-22T04:00:43.525Z" }, + { url = "https://files.pythonhosted.org/packages/77/d5/becbe1e2569b474a23f0c672ead8a29ac50b2dc1d5b9de184831bda8d14c/lxml-6.0.2-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:13e35cbc684aadf05d8711a5d1b5857c92e5e580efa9a0d2be197199c8def607", size = 8634365, upload-time = "2025-09-22T04:00:45.672Z" }, + { url = "https://files.pythonhosted.org/packages/28/66/1ced58f12e804644426b85d0bb8a4478ca77bc1761455da310505f1a3526/lxml-6.0.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:3b1675e096e17c6fe9c0e8c81434f5736c0739ff9ac6123c87c2d452f48fc938", size = 4650793, upload-time = "2025-09-22T04:00:47.783Z" }, + { url = "https://files.pythonhosted.org/packages/11/84/549098ffea39dfd167e3f174b4ce983d0eed61f9d8d25b7bf2a57c3247fc/lxml-6.0.2-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8ac6e5811ae2870953390452e3476694196f98d447573234592d30488147404d", size = 4944362, upload-time = "2025-09-22T04:00:49.845Z" }, + { url = "https://files.pythonhosted.org/packages/ac/bd/f207f16abf9749d2037453d56b643a7471d8fde855a231a12d1e095c4f01/lxml-6.0.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5aa0fc67ae19d7a64c3fe725dc9a1bb11f80e01f78289d05c6f62545affec438", size = 5083152, upload-time = "2025-09-22T04:00:51.709Z" }, + { url = "https://files.pythonhosted.org/packages/15/ae/bd813e87d8941d52ad5b65071b1affb48da01c4ed3c9c99e40abb266fbff/lxml-6.0.2-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:de496365750cc472b4e7902a485d3f152ecf57bd3ba03ddd5578ed8ceb4c5964", size = 5023539, upload-time = "2025-09-22T04:00:53.593Z" }, + { url = "https://files.pythonhosted.org/packages/02/cd/9bfef16bd1d874fbe0cb51afb00329540f30a3283beb9f0780adbb7eec03/lxml-6.0.2-cp311-cp311-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:200069a593c5e40b8f6fc0d84d86d970ba43138c3e68619ffa234bc9bb806a4d", size = 5344853, upload-time = "2025-09-22T04:00:55.524Z" }, + { url = "https://files.pythonhosted.org/packages/b8/89/ea8f91594bc5dbb879734d35a6f2b0ad50605d7fb419de2b63d4211765cc/lxml-6.0.2-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d2de809c2ee3b888b59f995625385f74629707c9355e0ff856445cdcae682b7", size = 5225133, upload-time = "2025-09-22T04:00:57.269Z" }, + { url = "https://files.pythonhosted.org/packages/b9/37/9c735274f5dbec726b2db99b98a43950395ba3d4a1043083dba2ad814170/lxml-6.0.2-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:b2c3da8d93cf5db60e8858c17684c47d01fee6405e554fb55018dd85fc23b178", size = 4677944, upload-time = "2025-09-22T04:00:59.052Z" }, + { url = "https://files.pythonhosted.org/packages/20/28/7dfe1ba3475d8bfca3878365075abe002e05d40dfaaeb7ec01b4c587d533/lxml-6.0.2-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:442de7530296ef5e188373a1ea5789a46ce90c4847e597856570439621d9c553", size = 5284535, upload-time = "2025-09-22T04:01:01.335Z" }, + { url = "https://files.pythonhosted.org/packages/e7/cf/5f14bc0de763498fc29510e3532bf2b4b3a1c1d5d0dff2e900c16ba021ef/lxml-6.0.2-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2593c77efde7bfea7f6389f1ab249b15ed4aa5bc5cb5131faa3b843c429fbedb", size = 5067343, upload-time = "2025-09-22T04:01:03.13Z" }, + { url = "https://files.pythonhosted.org/packages/1c/b0/bb8275ab5472f32b28cfbbcc6db7c9d092482d3439ca279d8d6fa02f7025/lxml-6.0.2-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:3e3cb08855967a20f553ff32d147e14329b3ae70ced6edc2f282b94afbc74b2a", size = 4725419, upload-time = "2025-09-22T04:01:05.013Z" }, + { url = "https://files.pythonhosted.org/packages/25/4c/7c222753bc72edca3b99dbadba1b064209bc8ed4ad448af990e60dcce462/lxml-6.0.2-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:2ed6c667fcbb8c19c6791bbf40b7268ef8ddf5a96940ba9404b9f9a304832f6c", size = 5275008, upload-time = "2025-09-22T04:01:07.327Z" }, + { url = "https://files.pythonhosted.org/packages/6c/8c/478a0dc6b6ed661451379447cdbec77c05741a75736d97e5b2b729687828/lxml-6.0.2-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b8f18914faec94132e5b91e69d76a5c1d7b0c73e2489ea8929c4aaa10b76bbf7", size = 5248906, upload-time = "2025-09-22T04:01:09.452Z" }, + { url = "https://files.pythonhosted.org/packages/2d/d9/5be3a6ab2784cdf9accb0703b65e1b64fcdd9311c9f007630c7db0cfcce1/lxml-6.0.2-cp311-cp311-win32.whl", hash = "sha256:6605c604e6daa9e0d7f0a2137bdc47a2e93b59c60a65466353e37f8272f47c46", size = 3610357, upload-time = "2025-09-22T04:01:11.102Z" }, + { url = "https://files.pythonhosted.org/packages/e2/7d/ca6fb13349b473d5732fb0ee3eec8f6c80fc0688e76b7d79c1008481bf1f/lxml-6.0.2-cp311-cp311-win_amd64.whl", hash = "sha256:e5867f2651016a3afd8dd2c8238baa66f1e2802f44bc17e236f547ace6647078", size = 4036583, upload-time = "2025-09-22T04:01:12.766Z" }, + { url = "https://files.pythonhosted.org/packages/ab/a2/51363b5ecd3eab46563645f3a2c3836a2fc67d01a1b87c5017040f39f567/lxml-6.0.2-cp311-cp311-win_arm64.whl", hash = "sha256:4197fb2534ee05fd3e7afaab5d8bfd6c2e186f65ea7f9cd6a82809c887bd1285", size = 3680591, upload-time = "2025-09-22T04:01:14.874Z" }, + { url = "https://files.pythonhosted.org/packages/f3/c8/8ff2bc6b920c84355146cd1ab7d181bc543b89241cfb1ebee824a7c81457/lxml-6.0.2-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:a59f5448ba2ceccd06995c95ea59a7674a10de0810f2ce90c9006f3cbc044456", size = 8661887, upload-time = "2025-09-22T04:01:17.265Z" }, + { url = "https://files.pythonhosted.org/packages/37/6f/9aae1008083bb501ef63284220ce81638332f9ccbfa53765b2b7502203cf/lxml-6.0.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:e8113639f3296706fbac34a30813929e29247718e88173ad849f57ca59754924", size = 4667818, upload-time = "2025-09-22T04:01:19.688Z" }, + { url = "https://files.pythonhosted.org/packages/f1/ca/31fb37f99f37f1536c133476674c10b577e409c0a624384147653e38baf2/lxml-6.0.2-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a8bef9b9825fa8bc816a6e641bb67219489229ebc648be422af695f6e7a4fa7f", size = 4950807, upload-time = "2025-09-22T04:01:21.487Z" }, + { url = "https://files.pythonhosted.org/packages/da/87/f6cb9442e4bada8aab5ae7e1046264f62fdbeaa6e3f6211b93f4c0dd97f1/lxml-6.0.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:65ea18d710fd14e0186c2f973dc60bb52039a275f82d3c44a0e42b43440ea534", size = 5109179, upload-time = "2025-09-22T04:01:23.32Z" }, + { url = "https://files.pythonhosted.org/packages/c8/20/a7760713e65888db79bbae4f6146a6ae5c04e4a204a3c48896c408cd6ed2/lxml-6.0.2-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c371aa98126a0d4c739ca93ceffa0fd7a5d732e3ac66a46e74339acd4d334564", size = 5023044, upload-time = "2025-09-22T04:01:25.118Z" }, + { url = "https://files.pythonhosted.org/packages/a2/b0/7e64e0460fcb36471899f75831509098f3fd7cd02a3833ac517433cb4f8f/lxml-6.0.2-cp312-cp312-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:700efd30c0fa1a3581d80a748157397559396090a51d306ea59a70020223d16f", size = 5359685, upload-time = "2025-09-22T04:01:27.398Z" }, + { url = "https://files.pythonhosted.org/packages/b9/e1/e5df362e9ca4e2f48ed6411bd4b3a0ae737cc842e96877f5bf9428055ab4/lxml-6.0.2-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:c33e66d44fe60e72397b487ee92e01da0d09ba2d66df8eae42d77b6d06e5eba0", size = 5654127, upload-time = "2025-09-22T04:01:29.629Z" }, + { url = "https://files.pythonhosted.org/packages/c6/d1/232b3309a02d60f11e71857778bfcd4acbdb86c07db8260caf7d008b08f8/lxml-6.0.2-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:90a345bbeaf9d0587a3aaffb7006aa39ccb6ff0e96a57286c0cb2fd1520ea192", size = 5253958, upload-time = "2025-09-22T04:01:31.535Z" }, + { url = "https://files.pythonhosted.org/packages/35/35/d955a070994725c4f7d80583a96cab9c107c57a125b20bb5f708fe941011/lxml-6.0.2-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:064fdadaf7a21af3ed1dcaa106b854077fbeada827c18f72aec9346847cd65d0", size = 4711541, upload-time = "2025-09-22T04:01:33.801Z" }, + { url = "https://files.pythonhosted.org/packages/1e/be/667d17363b38a78c4bd63cfd4b4632029fd68d2c2dc81f25ce9eb5224dd5/lxml-6.0.2-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:fbc74f42c3525ac4ffa4b89cbdd00057b6196bcefe8bce794abd42d33a018092", size = 5267426, upload-time = "2025-09-22T04:01:35.639Z" }, + { url = "https://files.pythonhosted.org/packages/ea/47/62c70aa4a1c26569bc958c9ca86af2bb4e1f614e8c04fb2989833874f7ae/lxml-6.0.2-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6ddff43f702905a4e32bc24f3f2e2edfe0f8fde3277d481bffb709a4cced7a1f", size = 5064917, upload-time = "2025-09-22T04:01:37.448Z" }, + { url = "https://files.pythonhosted.org/packages/bd/55/6ceddaca353ebd0f1908ef712c597f8570cc9c58130dbb89903198e441fd/lxml-6.0.2-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:6da5185951d72e6f5352166e3da7b0dc27aa70bd1090b0eb3f7f7212b53f1bb8", size = 4788795, upload-time = "2025-09-22T04:01:39.165Z" }, + { url = "https://files.pythonhosted.org/packages/cf/e8/fd63e15da5e3fd4c2146f8bbb3c14e94ab850589beab88e547b2dbce22e1/lxml-6.0.2-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:57a86e1ebb4020a38d295c04fc79603c7899e0df71588043eb218722dabc087f", size = 5676759, upload-time = "2025-09-22T04:01:41.506Z" }, + { url = "https://files.pythonhosted.org/packages/76/47/b3ec58dc5c374697f5ba37412cd2728f427d056315d124dd4b61da381877/lxml-6.0.2-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:2047d8234fe735ab77802ce5f2297e410ff40f5238aec569ad7c8e163d7b19a6", size = 5255666, upload-time = "2025-09-22T04:01:43.363Z" }, + { url = "https://files.pythonhosted.org/packages/19/93/03ba725df4c3d72afd9596eef4a37a837ce8e4806010569bedfcd2cb68fd/lxml-6.0.2-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6f91fd2b2ea15a6800c8e24418c0775a1694eefc011392da73bc6cef2623b322", size = 5277989, upload-time = "2025-09-22T04:01:45.215Z" }, + { url = "https://files.pythonhosted.org/packages/c6/80/c06de80bfce881d0ad738576f243911fccf992687ae09fd80b734712b39c/lxml-6.0.2-cp312-cp312-win32.whl", hash = "sha256:3ae2ce7d6fedfb3414a2b6c5e20b249c4c607f72cb8d2bb7cc9c6ec7c6f4e849", size = 3611456, upload-time = "2025-09-22T04:01:48.243Z" }, + { url = "https://files.pythonhosted.org/packages/f7/d7/0cdfb6c3e30893463fb3d1e52bc5f5f99684a03c29a0b6b605cfae879cd5/lxml-6.0.2-cp312-cp312-win_amd64.whl", hash = "sha256:72c87e5ee4e58a8354fb9c7c84cbf95a1c8236c127a5d1b7683f04bed8361e1f", size = 4011793, upload-time = "2025-09-22T04:01:50.042Z" }, + { url = "https://files.pythonhosted.org/packages/ea/7b/93c73c67db235931527301ed3785f849c78991e2e34f3fd9a6663ffda4c5/lxml-6.0.2-cp312-cp312-win_arm64.whl", hash = "sha256:61cb10eeb95570153e0c0e554f58df92ecf5109f75eacad4a95baa709e26c3d6", size = 3672836, upload-time = "2025-09-22T04:01:52.145Z" }, + { url = "https://files.pythonhosted.org/packages/53/fd/4e8f0540608977aea078bf6d79f128e0e2c2bba8af1acf775c30baa70460/lxml-6.0.2-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:9b33d21594afab46f37ae58dfadd06636f154923c4e8a4d754b0127554eb2e77", size = 8648494, upload-time = "2025-09-22T04:01:54.242Z" }, + { url = "https://files.pythonhosted.org/packages/5d/f4/2a94a3d3dfd6c6b433501b8d470a1960a20ecce93245cf2db1706adf6c19/lxml-6.0.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:6c8963287d7a4c5c9a432ff487c52e9c5618667179c18a204bdedb27310f022f", size = 4661146, upload-time = "2025-09-22T04:01:56.282Z" }, + { url = "https://files.pythonhosted.org/packages/25/2e/4efa677fa6b322013035d38016f6ae859d06cac67437ca7dc708a6af7028/lxml-6.0.2-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:1941354d92699fb5ffe6ed7b32f9649e43c2feb4b97205f75866f7d21aa91452", size = 4946932, upload-time = "2025-09-22T04:01:58.989Z" }, + { url = "https://files.pythonhosted.org/packages/ce/0f/526e78a6d38d109fdbaa5049c62e1d32fdd70c75fb61c4eadf3045d3d124/lxml-6.0.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bb2f6ca0ae2d983ded09357b84af659c954722bbf04dea98030064996d156048", size = 5100060, upload-time = "2025-09-22T04:02:00.812Z" }, + { url = "https://files.pythonhosted.org/packages/81/76/99de58d81fa702cc0ea7edae4f4640416c2062813a00ff24bd70ac1d9c9b/lxml-6.0.2-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eb2a12d704f180a902d7fa778c6d71f36ceb7b0d317f34cdc76a5d05aa1dd1df", size = 5019000, upload-time = "2025-09-22T04:02:02.671Z" }, + { url = "https://files.pythonhosted.org/packages/b5/35/9e57d25482bc9a9882cb0037fdb9cc18f4b79d85df94fa9d2a89562f1d25/lxml-6.0.2-cp313-cp313-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:6ec0e3f745021bfed19c456647f0298d60a24c9ff86d9d051f52b509663feeb1", size = 5348496, upload-time = "2025-09-22T04:02:04.904Z" }, + { url = "https://files.pythonhosted.org/packages/a6/8e/cb99bd0b83ccc3e8f0f528e9aa1f7a9965dfec08c617070c5db8d63a87ce/lxml-6.0.2-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:846ae9a12d54e368933b9759052d6206a9e8b250291109c48e350c1f1f49d916", size = 5643779, upload-time = "2025-09-22T04:02:06.689Z" }, + { url = "https://files.pythonhosted.org/packages/d0/34/9e591954939276bb679b73773836c6684c22e56d05980e31d52a9a8deb18/lxml-6.0.2-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ef9266d2aa545d7374938fb5c484531ef5a2ec7f2d573e62f8ce722c735685fd", size = 5244072, upload-time = "2025-09-22T04:02:08.587Z" }, + { url = "https://files.pythonhosted.org/packages/8d/27/b29ff065f9aaca443ee377aff699714fcbffb371b4fce5ac4ca759e436d5/lxml-6.0.2-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:4077b7c79f31755df33b795dc12119cb557a0106bfdab0d2c2d97bd3cf3dffa6", size = 4718675, upload-time = "2025-09-22T04:02:10.783Z" }, + { url = "https://files.pythonhosted.org/packages/2b/9f/f756f9c2cd27caa1a6ef8c32ae47aadea697f5c2c6d07b0dae133c244fbe/lxml-6.0.2-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a7c5d5e5f1081955358533be077166ee97ed2571d6a66bdba6ec2f609a715d1a", size = 5255171, upload-time = "2025-09-22T04:02:12.631Z" }, + { url = "https://files.pythonhosted.org/packages/61/46/bb85ea42d2cb1bd8395484fd72f38e3389611aa496ac7772da9205bbda0e/lxml-6.0.2-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8f8d0cbd0674ee89863a523e6994ac25fd5be9c8486acfc3e5ccea679bad2679", size = 5057175, upload-time = "2025-09-22T04:02:14.718Z" }, + { url = "https://files.pythonhosted.org/packages/95/0c/443fc476dcc8e41577f0af70458c50fe299a97bb6b7505bb1ae09aa7f9ac/lxml-6.0.2-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:2cbcbf6d6e924c28f04a43f3b6f6e272312a090f269eff68a2982e13e5d57659", size = 4785688, upload-time = "2025-09-22T04:02:16.957Z" }, + { url = "https://files.pythonhosted.org/packages/48/78/6ef0b359d45bb9697bc5a626e1992fa5d27aa3f8004b137b2314793b50a0/lxml-6.0.2-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:dfb874cfa53340009af6bdd7e54ebc0d21012a60a4e65d927c2e477112e63484", size = 5660655, upload-time = "2025-09-22T04:02:18.815Z" }, + { url = "https://files.pythonhosted.org/packages/ff/ea/e1d33808f386bc1339d08c0dcada6e4712d4ed8e93fcad5f057070b7988a/lxml-6.0.2-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:fb8dae0b6b8b7f9e96c26fdd8121522ce5de9bb5538010870bd538683d30e9a2", size = 5247695, upload-time = "2025-09-22T04:02:20.593Z" }, + { url = "https://files.pythonhosted.org/packages/4f/47/eba75dfd8183673725255247a603b4ad606f4ae657b60c6c145b381697da/lxml-6.0.2-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:358d9adae670b63e95bc59747c72f4dc97c9ec58881d4627fe0120da0f90d314", size = 5269841, upload-time = "2025-09-22T04:02:22.489Z" }, + { url = "https://files.pythonhosted.org/packages/76/04/5c5e2b8577bc936e219becb2e98cdb1aca14a4921a12995b9d0c523502ae/lxml-6.0.2-cp313-cp313-win32.whl", hash = "sha256:e8cd2415f372e7e5a789d743d133ae474290a90b9023197fd78f32e2dc6873e2", size = 3610700, upload-time = "2025-09-22T04:02:24.465Z" }, + { url = "https://files.pythonhosted.org/packages/fe/0a/4643ccc6bb8b143e9f9640aa54e38255f9d3b45feb2cbe7ae2ca47e8782e/lxml-6.0.2-cp313-cp313-win_amd64.whl", hash = "sha256:b30d46379644fbfc3ab81f8f82ae4de55179414651f110a1514f0b1f8f6cb2d7", size = 4010347, upload-time = "2025-09-22T04:02:26.286Z" }, + { url = "https://files.pythonhosted.org/packages/31/ef/dcf1d29c3f530577f61e5fe2f1bd72929acf779953668a8a47a479ae6f26/lxml-6.0.2-cp313-cp313-win_arm64.whl", hash = "sha256:13dcecc9946dca97b11b7c40d29fba63b55ab4170d3c0cf8c0c164343b9bfdcf", size = 3671248, upload-time = "2025-09-22T04:02:27.918Z" }, + { url = "https://files.pythonhosted.org/packages/03/15/d4a377b385ab693ce97b472fe0c77c2b16ec79590e688b3ccc71fba19884/lxml-6.0.2-cp314-cp314-macosx_10_13_universal2.whl", hash = "sha256:b0c732aa23de8f8aec23f4b580d1e52905ef468afb4abeafd3fec77042abb6fe", size = 8659801, upload-time = "2025-09-22T04:02:30.113Z" }, + { url = "https://files.pythonhosted.org/packages/c8/e8/c128e37589463668794d503afaeb003987373c5f94d667124ffd8078bbd9/lxml-6.0.2-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:4468e3b83e10e0317a89a33d28f7aeba1caa4d1a6fd457d115dd4ffe90c5931d", size = 4659403, upload-time = "2025-09-22T04:02:32.119Z" }, + { url = "https://files.pythonhosted.org/packages/00/ce/74903904339decdf7da7847bb5741fc98a5451b42fc419a86c0c13d26fe2/lxml-6.0.2-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:abd44571493973bad4598a3be7e1d807ed45aa2adaf7ab92ab7c62609569b17d", size = 4966974, upload-time = "2025-09-22T04:02:34.155Z" }, + { url = "https://files.pythonhosted.org/packages/1f/d3/131dec79ce61c5567fecf82515bd9bc36395df42501b50f7f7f3bd065df0/lxml-6.0.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:370cd78d5855cfbffd57c422851f7d3864e6ae72d0da615fca4dad8c45d375a5", size = 5102953, upload-time = "2025-09-22T04:02:36.054Z" }, + { url = "https://files.pythonhosted.org/packages/3a/ea/a43ba9bb750d4ffdd885f2cd333572f5bb900cd2408b67fdda07e85978a0/lxml-6.0.2-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:901e3b4219fa04ef766885fb40fa516a71662a4c61b80c94d25336b4934b71c0", size = 5055054, upload-time = "2025-09-22T04:02:38.154Z" }, + { url = "https://files.pythonhosted.org/packages/60/23/6885b451636ae286c34628f70a7ed1fcc759f8d9ad382d132e1c8d3d9bfd/lxml-6.0.2-cp314-cp314-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:a4bf42d2e4cf52c28cc1812d62426b9503cdb0c87a6de81442626aa7d69707ba", size = 5352421, upload-time = "2025-09-22T04:02:40.413Z" }, + { url = "https://files.pythonhosted.org/packages/48/5b/fc2ddfc94ddbe3eebb8e9af6e3fd65e2feba4967f6a4e9683875c394c2d8/lxml-6.0.2-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:b2c7fdaa4d7c3d886a42534adec7cfac73860b89b4e5298752f60aa5984641a0", size = 5673684, upload-time = "2025-09-22T04:02:42.288Z" }, + { url = "https://files.pythonhosted.org/packages/29/9c/47293c58cc91769130fbf85531280e8cc7868f7fbb6d92f4670071b9cb3e/lxml-6.0.2-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:98a5e1660dc7de2200b00d53fa00bcd3c35a3608c305d45a7bbcaf29fa16e83d", size = 5252463, upload-time = "2025-09-22T04:02:44.165Z" }, + { url = "https://files.pythonhosted.org/packages/9b/da/ba6eceb830c762b48e711ded880d7e3e89fc6c7323e587c36540b6b23c6b/lxml-6.0.2-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:dc051506c30b609238d79eda75ee9cab3e520570ec8219844a72a46020901e37", size = 4698437, upload-time = "2025-09-22T04:02:46.524Z" }, + { url = "https://files.pythonhosted.org/packages/a5/24/7be3f82cb7990b89118d944b619e53c656c97dc89c28cfb143fdb7cd6f4d/lxml-6.0.2-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:8799481bbdd212470d17513a54d568f44416db01250f49449647b5ab5b5dccb9", size = 5269890, upload-time = "2025-09-22T04:02:48.812Z" }, + { url = "https://files.pythonhosted.org/packages/1b/bd/dcfb9ea1e16c665efd7538fc5d5c34071276ce9220e234217682e7d2c4a5/lxml-6.0.2-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9261bb77c2dab42f3ecd9103951aeca2c40277701eb7e912c545c1b16e0e4917", size = 5097185, upload-time = "2025-09-22T04:02:50.746Z" }, + { url = "https://files.pythonhosted.org/packages/21/04/a60b0ff9314736316f28316b694bccbbabe100f8483ad83852d77fc7468e/lxml-6.0.2-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:65ac4a01aba353cfa6d5725b95d7aed6356ddc0a3cd734de00124d285b04b64f", size = 4745895, upload-time = "2025-09-22T04:02:52.968Z" }, + { url = "https://files.pythonhosted.org/packages/d6/bd/7d54bd1846e5a310d9c715921c5faa71cf5c0853372adf78aee70c8d7aa2/lxml-6.0.2-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:b22a07cbb82fea98f8a2fd814f3d1811ff9ed76d0fc6abc84eb21527596e7cc8", size = 5695246, upload-time = "2025-09-22T04:02:54.798Z" }, + { url = "https://files.pythonhosted.org/packages/fd/32/5643d6ab947bc371da21323acb2a6e603cedbe71cb4c99c8254289ab6f4e/lxml-6.0.2-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:d759cdd7f3e055d6bc8d9bec3ad905227b2e4c785dc16c372eb5b5e83123f48a", size = 5260797, upload-time = "2025-09-22T04:02:57.058Z" }, + { url = "https://files.pythonhosted.org/packages/33/da/34c1ec4cff1eea7d0b4cd44af8411806ed943141804ac9c5d565302afb78/lxml-6.0.2-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:945da35a48d193d27c188037a05fec5492937f66fb1958c24fc761fb9d40d43c", size = 5277404, upload-time = "2025-09-22T04:02:58.966Z" }, + { url = "https://files.pythonhosted.org/packages/82/57/4eca3e31e54dc89e2c3507e1cd411074a17565fa5ffc437c4ae0a00d439e/lxml-6.0.2-cp314-cp314-win32.whl", hash = "sha256:be3aaa60da67e6153eb15715cc2e19091af5dc75faef8b8a585aea372507384b", size = 3670072, upload-time = "2025-09-22T04:03:38.05Z" }, + { url = "https://files.pythonhosted.org/packages/e3/e0/c96cf13eccd20c9421ba910304dae0f619724dcf1702864fd59dd386404d/lxml-6.0.2-cp314-cp314-win_amd64.whl", hash = "sha256:fa25afbadead523f7001caf0c2382afd272c315a033a7b06336da2637d92d6ed", size = 4080617, upload-time = "2025-09-22T04:03:39.835Z" }, + { url = "https://files.pythonhosted.org/packages/d5/5d/b3f03e22b3d38d6f188ef044900a9b29b2fe0aebb94625ce9fe244011d34/lxml-6.0.2-cp314-cp314-win_arm64.whl", hash = "sha256:063eccf89df5b24e361b123e257e437f9e9878f425ee9aae3144c77faf6da6d8", size = 3754930, upload-time = "2025-09-22T04:03:41.565Z" }, + { url = "https://files.pythonhosted.org/packages/5e/5c/42c2c4c03554580708fc738d13414801f340c04c3eff90d8d2d227145275/lxml-6.0.2-cp314-cp314t-macosx_10_13_universal2.whl", hash = "sha256:6162a86d86893d63084faaf4ff937b3daea233e3682fb4474db07395794fa80d", size = 8910380, upload-time = "2025-09-22T04:03:01.645Z" }, + { url = "https://files.pythonhosted.org/packages/bf/4f/12df843e3e10d18d468a7557058f8d3733e8b6e12401f30b1ef29360740f/lxml-6.0.2-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:414aaa94e974e23a3e92e7ca5b97d10c0cf37b6481f50911032c69eeb3991bba", size = 4775632, upload-time = "2025-09-22T04:03:03.814Z" }, + { url = "https://files.pythonhosted.org/packages/e4/0c/9dc31e6c2d0d418483cbcb469d1f5a582a1cd00a1f4081953d44051f3c50/lxml-6.0.2-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:48461bd21625458dd01e14e2c38dd0aea69addc3c4f960c30d9f59d7f93be601", size = 4975171, upload-time = "2025-09-22T04:03:05.651Z" }, + { url = "https://files.pythonhosted.org/packages/e7/2b/9b870c6ca24c841bdd887504808f0417aa9d8d564114689266f19ddf29c8/lxml-6.0.2-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:25fcc59afc57d527cfc78a58f40ab4c9b8fd096a9a3f964d2781ffb6eb33f4ed", size = 5110109, upload-time = "2025-09-22T04:03:07.452Z" }, + { url = "https://files.pythonhosted.org/packages/bf/0c/4f5f2a4dd319a178912751564471355d9019e220c20d7db3fb8307ed8582/lxml-6.0.2-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5179c60288204e6ddde3f774a93350177e08876eaf3ab78aa3a3649d43eb7d37", size = 5041061, upload-time = "2025-09-22T04:03:09.297Z" }, + { url = "https://files.pythonhosted.org/packages/12/64/554eed290365267671fe001a20d72d14f468ae4e6acef1e179b039436967/lxml-6.0.2-cp314-cp314t-manylinux_2_26_i686.manylinux_2_28_i686.whl", hash = "sha256:967aab75434de148ec80597b75062d8123cadf2943fb4281f385141e18b21338", size = 5306233, upload-time = "2025-09-22T04:03:11.651Z" }, + { url = "https://files.pythonhosted.org/packages/7a/31/1d748aa275e71802ad9722df32a7a35034246b42c0ecdd8235412c3396ef/lxml-6.0.2-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:d100fcc8930d697c6561156c6810ab4a508fb264c8b6779e6e61e2ed5e7558f9", size = 5604739, upload-time = "2025-09-22T04:03:13.592Z" }, + { url = "https://files.pythonhosted.org/packages/8f/41/2c11916bcac09ed561adccacceaedd2bf0e0b25b297ea92aab99fd03d0fa/lxml-6.0.2-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2ca59e7e13e5981175b8b3e4ab84d7da57993eeff53c07764dcebda0d0e64ecd", size = 5225119, upload-time = "2025-09-22T04:03:15.408Z" }, + { url = "https://files.pythonhosted.org/packages/99/05/4e5c2873d8f17aa018e6afde417c80cc5d0c33be4854cce3ef5670c49367/lxml-6.0.2-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:957448ac63a42e2e49531b9d6c0fa449a1970dbc32467aaad46f11545be9af1d", size = 4633665, upload-time = "2025-09-22T04:03:17.262Z" }, + { url = "https://files.pythonhosted.org/packages/0f/c9/dcc2da1bebd6275cdc723b515f93edf548b82f36a5458cca3578bc899332/lxml-6.0.2-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b7fc49c37f1786284b12af63152fe1d0990722497e2d5817acfe7a877522f9a9", size = 5234997, upload-time = "2025-09-22T04:03:19.14Z" }, + { url = "https://files.pythonhosted.org/packages/9c/e2/5172e4e7468afca64a37b81dba152fc5d90e30f9c83c7c3213d6a02a5ce4/lxml-6.0.2-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:e19e0643cc936a22e837f79d01a550678da8377d7d801a14487c10c34ee49c7e", size = 5090957, upload-time = "2025-09-22T04:03:21.436Z" }, + { url = "https://files.pythonhosted.org/packages/a5/b3/15461fd3e5cd4ddcb7938b87fc20b14ab113b92312fc97afe65cd7c85de1/lxml-6.0.2-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:1db01e5cf14345628e0cbe71067204db658e2fb8e51e7f33631f5f4735fefd8d", size = 4764372, upload-time = "2025-09-22T04:03:23.27Z" }, + { url = "https://files.pythonhosted.org/packages/05/33/f310b987c8bf9e61c4dd8e8035c416bd3230098f5e3cfa69fc4232de7059/lxml-6.0.2-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:875c6b5ab39ad5291588aed6925fac99d0097af0dd62f33c7b43736043d4a2ec", size = 5634653, upload-time = "2025-09-22T04:03:25.767Z" }, + { url = "https://files.pythonhosted.org/packages/70/ff/51c80e75e0bc9382158133bdcf4e339b5886c6ee2418b5199b3f1a61ed6d/lxml-6.0.2-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:cdcbed9ad19da81c480dfd6dd161886db6096083c9938ead313d94b30aadf272", size = 5233795, upload-time = "2025-09-22T04:03:27.62Z" }, + { url = "https://files.pythonhosted.org/packages/56/4d/4856e897df0d588789dd844dbed9d91782c4ef0b327f96ce53c807e13128/lxml-6.0.2-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:80dadc234ebc532e09be1975ff538d154a7fa61ea5031c03d25178855544728f", size = 5257023, upload-time = "2025-09-22T04:03:30.056Z" }, + { url = "https://files.pythonhosted.org/packages/0f/85/86766dfebfa87bea0ab78e9ff7a4b4b45225df4b4d3b8cc3c03c5cd68464/lxml-6.0.2-cp314-cp314t-win32.whl", hash = "sha256:da08e7bb297b04e893d91087df19638dc7a6bb858a954b0cc2b9f5053c922312", size = 3911420, upload-time = "2025-09-22T04:03:32.198Z" }, + { url = "https://files.pythonhosted.org/packages/fe/1a/b248b355834c8e32614650b8008c69ffeb0ceb149c793961dd8c0b991bb3/lxml-6.0.2-cp314-cp314t-win_amd64.whl", hash = "sha256:252a22982dca42f6155125ac76d3432e548a7625d56f5a273ee78a5057216eca", size = 4406837, upload-time = "2025-09-22T04:03:34.027Z" }, + { url = "https://files.pythonhosted.org/packages/92/aa/df863bcc39c5e0946263454aba394de8a9084dbaff8ad143846b0d844739/lxml-6.0.2-cp314-cp314t-win_arm64.whl", hash = "sha256:bb4c1847b303835d89d785a18801a883436cdfd5dc3d62947f9c49e24f0f5a2c", size = 3822205, upload-time = "2025-09-22T04:03:36.249Z" }, + { url = "https://files.pythonhosted.org/packages/e7/9c/780c9a8fce3f04690b374f72f41306866b0400b9d0fdf3e17aaa37887eed/lxml-6.0.2-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:e748d4cf8fef2526bb2a589a417eba0c8674e29ffcb570ce2ceca44f1e567bf6", size = 3939264, upload-time = "2025-09-22T04:04:32.892Z" }, + { url = "https://files.pythonhosted.org/packages/f5/5a/1ab260c00adf645d8bf7dec7f920f744b032f69130c681302821d5debea6/lxml-6.0.2-pp310-pypy310_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4ddb1049fa0579d0cbd00503ad8c58b9ab34d1254c77bc6a5576d96ec7853dba", size = 4216435, upload-time = "2025-09-22T04:04:34.907Z" }, + { url = "https://files.pythonhosted.org/packages/f2/37/565f3b3d7ffede22874b6d86be1a1763d00f4ea9fc5b9b6ccb11e4ec8612/lxml-6.0.2-pp310-pypy310_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cb233f9c95f83707dae461b12b720c1af9c28c2d19208e1be03387222151daf5", size = 4325913, upload-time = "2025-09-22T04:04:37.205Z" }, + { url = "https://files.pythonhosted.org/packages/22/ec/f3a1b169b2fb9d03467e2e3c0c752ea30e993be440a068b125fc7dd248b0/lxml-6.0.2-pp310-pypy310_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bc456d04db0515ce3320d714a1eac7a97774ff0849e7718b492d957da4631dd4", size = 4269357, upload-time = "2025-09-22T04:04:39.322Z" }, + { url = "https://files.pythonhosted.org/packages/77/a2/585a28fe3e67daa1cf2f06f34490d556d121c25d500b10082a7db96e3bcd/lxml-6.0.2-pp310-pypy310_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2613e67de13d619fd283d58bda40bff0ee07739f624ffee8b13b631abf33083d", size = 4412295, upload-time = "2025-09-22T04:04:41.647Z" }, + { url = "https://files.pythonhosted.org/packages/7b/d9/a57dd8bcebd7c69386c20263830d4fa72d27e6b72a229ef7a48e88952d9a/lxml-6.0.2-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:24a8e756c982c001ca8d59e87c80c4d9dcd4d9b44a4cbeb8d9be4482c514d41d", size = 3516913, upload-time = "2025-09-22T04:04:43.602Z" }, + { url = "https://files.pythonhosted.org/packages/0b/11/29d08bc103a62c0eba8016e7ed5aeebbf1e4312e83b0b1648dd203b0e87d/lxml-6.0.2-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:1c06035eafa8404b5cf475bb37a9f6088b0aca288d4ccc9d69389750d5543700", size = 3949829, upload-time = "2025-09-22T04:04:45.608Z" }, + { url = "https://files.pythonhosted.org/packages/12/b3/52ab9a3b31e5ab8238da241baa19eec44d2ab426532441ee607165aebb52/lxml-6.0.2-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c7d13103045de1bdd6fe5d61802565f1a3537d70cd3abf596aa0af62761921ee", size = 4226277, upload-time = "2025-09-22T04:04:47.754Z" }, + { url = "https://files.pythonhosted.org/packages/a0/33/1eaf780c1baad88224611df13b1c2a9dfa460b526cacfe769103ff50d845/lxml-6.0.2-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:0a3c150a95fbe5ac91de323aa756219ef9cf7fde5a3f00e2281e30f33fa5fa4f", size = 4330433, upload-time = "2025-09-22T04:04:49.907Z" }, + { url = "https://files.pythonhosted.org/packages/7a/c1/27428a2ff348e994ab4f8777d3a0ad510b6b92d37718e5887d2da99952a2/lxml-6.0.2-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:60fa43be34f78bebb27812ed90f1925ec99560b0fa1decdb7d12b84d857d31e9", size = 4272119, upload-time = "2025-09-22T04:04:51.801Z" }, + { url = "https://files.pythonhosted.org/packages/f0/d0/3020fa12bcec4ab62f97aab026d57c2f0cfd480a558758d9ca233bb6a79d/lxml-6.0.2-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:21c73b476d3cfe836be731225ec3421fa2f048d84f6df6a8e70433dff1376d5a", size = 4417314, upload-time = "2025-09-22T04:04:55.024Z" }, + { url = "https://files.pythonhosted.org/packages/6c/77/d7f491cbc05303ac6801651aabeb262d43f319288c1ea96c66b1d2692ff3/lxml-6.0.2-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:27220da5be049e936c3aca06f174e8827ca6445a4353a1995584311487fc4e3e", size = 3518768, upload-time = "2025-09-22T04:04:57.097Z" }, +] + +[[package]] +name = "mammoth" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cobble" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0b/03/54ad55930465d571b4d74cfca4446c8353bf3410d2c5dc5c52a18dc6d039/mammoth-1.8.0.tar.gz", hash = "sha256:7e8aa7db53f4aa7e9620b22bf8b716f1a16c84e969de1a0b1f920c756184e3d8", size = 50336, upload-time = "2024-06-16T17:06:55.657Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b9/19/79f55df27d0ac645588bb556aedcabc2ffa946a562d3a772b17dbeea8254/mammoth-1.8.0-py2.py3-none-any.whl", hash = "sha256:b2abf2340809b13a903c8b65a27846466290b869f0dd56e4a1e3072c4be1ea86", size = 52313, upload-time = "2024-06-16T17:06:52.766Z" }, +] + [[package]] name = "markdown" version = "3.10" @@ -2324,6 +2709,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/94/54/e7d793b573f298e1c9013b8c4dade17d481164aa517d1d7148619c2cedbf/markdown_it_py-4.0.0-py3-none-any.whl", hash = "sha256:87327c59b172c5011896038353a81343b6754500a08cd7a4973bb48c6d578147", size = 87321, upload-time = "2025-08-11T12:57:51.923Z" }, ] +[[package]] +name = "markdownify" +version = "0.14.1" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "beautifulsoup4" }, + { name = "six" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/1b/75/483a4bcca436fe88d02dc7686c372631d833848951b368700bdc0c770bb7/markdownify-0.14.1.tar.gz", hash = "sha256:a62a7a216947ed0b8dafb95b99b2ef4a0edd1e18d5653c656f68f03db2bfb2f1", size = 14332, upload-time = "2024-11-24T22:08:30.775Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/65/0b/74cec93a7b05edf4fc3ea1c899fe8a37f041d7b9d303c75abf7a162924e0/markdownify-0.14.1-py3-none-any.whl", hash = "sha256:4c46a6c0c12c6005ddcd49b45a5a890398b002ef51380cd319db62df5e09bc2a", size = 11530, upload-time = "2024-11-24T22:08:29.005Z" }, +] + [[package]] name = "markupsafe" version = "3.0.3" @@ -2435,25 +2833,46 @@ dependencies = [ [package.optional-dependencies] all = [ { name = "anthropic" }, - { name = "are" }, + { name = "arxiv" }, + { name = "beartype" }, + { name = "beautifulsoup4" }, { name = "camel-ai" }, + { name = "colorlog" }, { name = "datasets" }, + { name = "docstring-parser" }, + { name = "flask" }, { name = "google-genai" }, { name = "ipykernel" }, { name = "ipywidgets" }, + { name = "javascript" }, { name = "jupyter" }, + { name = "keybert" }, { name = "langchain" }, { name = "langchain-google-genai" }, { name = "langfuse" }, { name = "langgraph" }, + { name = "levenshtein" }, { name = "litellm" }, { name = "llama-index-core" }, { name = "mcp" }, + { name = "meta-agents-research-environments" }, + { name = "names" }, { name = "openai" }, + { name = "psycopg2-binary" }, + { name = "pymysql" }, + { name = "pypdf2" }, { name = "python-dotenv" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "ruamel-yaml" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "semanticscholar" }, + { name = "sentence-transformers" }, { name = "smolagents" }, { name = "transformers" }, { name = "typing-extensions" }, + { name = "waitress" }, { name = "wandb" }, ] anthropic = [ @@ -2464,9 +2883,9 @@ camel = [ ] examples = [ { name = "anthropic" }, - { name = "are" }, { name = "camel-ai" }, { name = "datasets" }, + { name = "docstring-parser" }, { name = "google-genai" }, { name = "ipykernel" }, { name = "ipywidgets" }, @@ -2478,14 +2897,15 @@ examples = [ { name = "litellm" }, { name = "llama-index-core" }, { name = "mcp" }, + { name = "meta-agents-research-environments" }, { name = "openai" }, { name = "python-dotenv" }, { name = "smolagents" }, { name = "typing-extensions" }, ] gaia2 = [ - { name = "are" }, { name = "datasets" }, + { name = "meta-agents-research-environments" }, ] google-genai = [ { name = "google-genai" }, @@ -2502,12 +2922,38 @@ litellm = [ llamaindex = [ { name = "llama-index-core" }, ] +multiagentbench = [ + { name = "arxiv" }, + { name = "beartype" }, + { name = "beautifulsoup4" }, + { name = "colorlog" }, + { name = "flask" }, + { name = "javascript" }, + { name = "keybert" }, + { name = "levenshtein" }, + { name = "litellm" }, + { name = "names" }, + { name = "psycopg2-binary" }, + { name = "pymysql" }, + { name = "pypdf2" }, + { name = "pyyaml" }, + { name = "requests" }, + { name = "ruamel-yaml" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "semanticscholar" }, + { name = "sentence-transformers" }, + { name = "waitress" }, +] openai = [ { name = "openai" }, ] smolagents = [ { name = "smolagents" }, ] +tau2 = [ + { name = "docstring-parser" }, +] transformers = [ { name = "transformers" }, ] @@ -2536,34 +2982,55 @@ docs = [ [package.metadata] requires-dist = [ { name = "anthropic", marker = "extra == 'anthropic'", specifier = ">=0.40.0" }, - { name = "are", marker = "extra == 'gaia2'", specifier = ">=1.2.0" }, + { name = "arxiv", marker = "extra == 'multiagentbench'", specifier = ">=2.1.0" }, + { name = "beartype", marker = "extra == 'multiagentbench'" }, + { name = "beautifulsoup4", marker = "extra == 'multiagentbench'", specifier = ">=4.12.0" }, { name = "camel-ai", marker = "extra == 'camel'", specifier = ">=0.2.0" }, + { name = "colorlog", marker = "extra == 'multiagentbench'", specifier = ">=6.0.0" }, { name = "datasets", marker = "extra == 'gaia2'", specifier = ">=3.0.0" }, + { name = "docstring-parser", marker = "extra == 'tau2'", specifier = ">=0.16" }, + { name = "flask", marker = "extra == 'multiagentbench'", specifier = ">=3.0.0" }, { name = "gitpython", specifier = ">=3.1.0" }, { name = "google-genai", marker = "extra == 'google-genai'", specifier = ">=1.37.0" }, { name = "ipykernel", marker = "extra == 'examples'", specifier = ">=6.0.0" }, { name = "ipywidgets", marker = "extra == 'examples'", specifier = ">=8.0.0" }, + { name = "javascript", marker = "extra == 'multiagentbench'", specifier = ">=1!1.2.0" }, { name = "jupyter", marker = "extra == 'examples'", specifier = ">=1.0.0" }, + { name = "keybert", marker = "extra == 'multiagentbench'", specifier = ">=0.8.0" }, { name = "langchain", marker = "extra == 'examples'", specifier = ">=0.3.27" }, { name = "langchain-google-genai", marker = "extra == 'examples'", specifier = ">=2.1.12" }, { name = "langfuse", marker = "extra == 'langfuse'", specifier = ">=3.3.4" }, { name = "langgraph", marker = "extra == 'langgraph'", specifier = ">=0.6.0" }, + { name = "levenshtein", marker = "extra == 'multiagentbench'", specifier = ">=0.20.0" }, { name = "litellm", marker = "extra == 'litellm'", specifier = ">=1.0.0" }, + { name = "litellm", marker = "extra == 'multiagentbench'", specifier = ">=1.0.0" }, { name = "llama-index-core", marker = "extra == 'llamaindex'", specifier = ">=0.12.0" }, - { name = "maseval", extras = ["examples", "transformers", "wandb"], marker = "extra == 'all'" }, - { name = "maseval", extras = ["smolagents", "langgraph", "llamaindex", "camel", "anthropic", "openai", "google-genai", "litellm", "langfuse", "gaia2"], marker = "extra == 'examples'" }, + { name = "maseval", extras = ["examples", "transformers", "wandb", "multiagentbench"], marker = "extra == 'all'" }, + { name = "maseval", extras = ["smolagents", "langgraph", "llamaindex", "camel", "anthropic", "openai", "google-genai", "litellm", "langfuse", "gaia2", "macs", "tau2"], marker = "extra == 'examples'" }, { name = "mcp", marker = "extra == 'examples'", specifier = ">=1.22.0" }, + { name = "meta-agents-research-environments", marker = "extra == 'gaia2'", specifier = ">=1.2.0" }, + { name = "names", marker = "extra == 'multiagentbench'", specifier = ">=0.3.0" }, { name = "openai", marker = "extra == 'openai'", specifier = ">=1.107.2" }, - { name = "pydantic", specifier = ">=2.12.5" }, + { name = "psycopg2-binary", marker = "extra == 'multiagentbench'", specifier = ">=2.9.0" }, + { name = "pydantic", specifier = ">=2.10.6" }, + { name = "pymysql", marker = "extra == 'multiagentbench'", specifier = ">=1.1.0" }, + { name = "pypdf2", marker = "extra == 'multiagentbench'", specifier = ">=3.0.0" }, { name = "python-dotenv", marker = "extra == 'examples'", specifier = ">=1.0.0" }, + { name = "pyyaml", marker = "extra == 'multiagentbench'", specifier = ">=6.0" }, + { name = "requests", marker = "extra == 'multiagentbench'", specifier = ">=2.28.0" }, { name = "rich", specifier = ">=14.1.0" }, + { name = "ruamel-yaml", marker = "extra == 'multiagentbench'", specifier = ">=0.17.0" }, + { name = "scikit-learn", marker = "extra == 'multiagentbench'", specifier = ">=1.3.0" }, + { name = "semanticscholar", marker = "extra == 'multiagentbench'", specifier = ">=0.8.0" }, + { name = "sentence-transformers", marker = "extra == 'multiagentbench'", specifier = ">=2.3.0" }, { name = "smolagents", marker = "extra == 'smolagents'", specifier = ">=1.21.3" }, { name = "tqdm", specifier = ">=4.66.0" }, { name = "transformers", marker = "extra == 'transformers'", specifier = ">=4.37.0" }, { name = "typing-extensions", marker = "extra == 'examples'", specifier = ">=4.0.0" }, + { name = "waitress", marker = "extra == 'multiagentbench'", specifier = ">=3.0.0" }, { name = "wandb", marker = "extra == 'wandb'", specifier = ">=0.15.0" }, ] -provides-extras = ["smolagents", "langgraph", "llamaindex", "camel", "anthropic", "openai", "google-genai", "transformers", "litellm", "wandb", "langfuse", "gaia2", "examples", "all"] +provides-extras = ["smolagents", "langgraph", "llamaindex", "camel", "anthropic", "openai", "google-genai", "transformers", "litellm", "wandb", "langfuse", "gaia2", "macs", "multiagentbench", "tau2", "examples", "all"] [package.metadata.requires-dev] dev = [ @@ -2650,6 +3117,40 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/2c/19/04f9b178c2d8a15b076c8b5140708fa6ffc5601fb6f1e975537072df5b2a/mergedeep-1.3.4-py3-none-any.whl", hash = "sha256:70775750742b25c0d8f36c55aed03d24c3384d17c951b3175d898bd778ef0307", size = 6354, upload-time = "2021-02-05T18:55:29.583Z" }, ] +[[package]] +name = "meta-agents-research-environments" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "click" }, + { name = "datasets" }, + { name = "docstring-parser" }, + { name = "fsspec" }, + { name = "huggingface-hub" }, + { name = "inputimeout" }, + { name = "jinja2" }, + { name = "litellm" }, + { name = "mammoth" }, + { name = "markdownify" }, + { name = "mcp" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "2.4.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "pandas" }, + { name = "pdfminer-six" }, + { name = "pillow" }, + { name = "polars-lts-cpu" }, + { name = "puremagic" }, + { name = "pydantic" }, + { name = "python-dotenv" }, + { name = "python-pptx" }, + { name = "rapidfuzz" }, + { name = "termcolor" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/34/17/19cb42484e2faab08ad0896ed4a1696028f2788a71069c1836a3535d75e4/meta_agents_research_environments-1.2.0.tar.gz", hash = "sha256:ad99047aef6d597ed9c4fd74fd634b7a648fd20c104212a5686958f7a9a4bcd4", size = 21966877, upload-time = "2025-11-11T10:54:14.375Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ed/dc/6c8741faf4144227b7d21cede49999f35c163c9ebd7a47f025f0e8df7faa/meta_agents_research_environments-1.2.0-py3-none-any.whl", hash = "sha256:c16caa85abb36f42172ee4cb9e1df478d953d610f05442120f9b4066db85e5b2", size = 1390109, upload-time = "2025-11-11T10:54:09.651Z" }, +] + [[package]] name = "mistune" version = "3.1.4" @@ -2814,6 +3315,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/81/06/c5f8deba7d2cbdfa7967a716ae801aa9ca5f734b8f54fd473ef77a088dbe/mkdocstrings_python-2.0.1-py3-none-any.whl", hash = "sha256:66ecff45c5f8b71bf174e11d49afc845c2dfc7fc0ab17a86b6b337e0f24d8d90", size = 105055, upload-time = "2025-12-03T14:26:10.184Z" }, ] +[[package]] +name = "mpmath" +version = "1.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f", size = 508106, upload-time = "2023-03-07T16:47:11.061Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c", size = 536198, upload-time = "2023-03-07T16:47:09.197Z" }, +] + [[package]] name = "multidict" version = "6.7.0" @@ -2954,25 +3464,20 @@ wheels = [ [[package]] name = "multiprocess" -version = "0.70.18" +version = "0.70.16" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "dill" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/72/fd/2ae3826f5be24c6ed87266bc4e59c46ea5b059a103f3d7e7eb76a52aeecb/multiprocess-0.70.18.tar.gz", hash = "sha256:f9597128e6b3e67b23956da07cf3d2e5cba79e2f4e0fba8d7903636663ec6d0d", size = 1798503, upload-time = "2025-04-17T03:11:27.742Z" } +sdist = { url = "https://files.pythonhosted.org/packages/b5/ae/04f39c5d0d0def03247c2893d6f2b83c136bf3320a2154d7b8858f2ba72d/multiprocess-0.70.16.tar.gz", hash = "sha256:161af703d4652a0e1410be6abccecde4a7ddffd19341be0a7011b94aeb171ac1", size = 1772603, upload-time = "2024-01-28T18:52:34.85Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/c8/f8/7f9a8f08bf98cea1dfaa181e05cc8bbcb59cecf044b5a9ac3cce39f9c449/multiprocess-0.70.18-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:25d4012dcaaf66b9e8e955f58482b42910c2ee526d532844d8bcf661bbc604df", size = 135083, upload-time = "2025-04-17T03:11:04.223Z" }, - { url = "https://files.pythonhosted.org/packages/e5/03/b7b10dbfc17b2b3ce07d4d30b3ba8367d0ed32d6d46cd166e298f161dd46/multiprocess-0.70.18-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:06b19433de0d02afe5869aec8931dd5c01d99074664f806c73896b0d9e527213", size = 135128, upload-time = "2025-04-17T03:11:06.045Z" }, - { url = "https://files.pythonhosted.org/packages/c1/a3/5f8d3b9690ea5580bee5868ab7d7e2cfca74b7e826b28192b40aa3881cdc/multiprocess-0.70.18-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:6fa1366f994373aaf2d4738b0f56e707caeaa05486e97a7f71ee0853823180c2", size = 135132, upload-time = "2025-04-17T03:11:07.533Z" }, - { url = "https://files.pythonhosted.org/packages/55/4d/9af0d1279c84618bcd35bf5fd7e371657358c7b0a523e54a9cffb87461f8/multiprocess-0.70.18-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:8b8940ae30139e04b076da6c5b83e9398585ebdf0f2ad3250673fef5b2ff06d6", size = 144695, upload-time = "2025-04-17T03:11:09.161Z" }, - { url = "https://files.pythonhosted.org/packages/17/bf/87323e79dd0562474fad3373c21c66bc6c3c9963b68eb2a209deb4c8575e/multiprocess-0.70.18-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:0929ba95831adb938edbd5fb801ac45e705ecad9d100b3e653946b7716cb6bd3", size = 144742, upload-time = "2025-04-17T03:11:10.072Z" }, - { url = "https://files.pythonhosted.org/packages/dd/74/cb8c831e58dc6d5cf450b17c7db87f14294a1df52eb391da948b5e0a0b94/multiprocess-0.70.18-pp311-pypy311_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:4d77f8e4bfe6c6e2e661925bbf9aed4d5ade9a1c6502d5dfc10129b9d1141797", size = 144745, upload-time = "2025-04-17T03:11:11.453Z" }, - { url = "https://files.pythonhosted.org/packages/ba/d8/0cba6cf51a1a31f20471fbc823a716170c73012ddc4fb85d706630ed6e8f/multiprocess-0.70.18-py310-none-any.whl", hash = "sha256:60c194974c31784019c1f459d984e8f33ee48f10fcf42c309ba97b30d9bd53ea", size = 134948, upload-time = "2025-04-17T03:11:20.223Z" }, - { url = "https://files.pythonhosted.org/packages/4b/88/9039f2fed1012ef584751d4ceff9ab4a51e5ae264898f0b7cbf44340a859/multiprocess-0.70.18-py311-none-any.whl", hash = "sha256:5aa6eef98e691281b3ad923be2832bf1c55dd2c859acd73e5ec53a66aae06a1d", size = 144462, upload-time = "2025-04-17T03:11:21.657Z" }, - { url = "https://files.pythonhosted.org/packages/bf/b6/5f922792be93b82ec6b5f270bbb1ef031fd0622847070bbcf9da816502cc/multiprocess-0.70.18-py312-none-any.whl", hash = "sha256:9b78f8e5024b573730bfb654783a13800c2c0f2dfc0c25e70b40d184d64adaa2", size = 150287, upload-time = "2025-04-17T03:11:22.69Z" }, - { url = "https://files.pythonhosted.org/packages/ee/25/7d7e78e750bc1aecfaf0efbf826c69a791d2eeaf29cf20cba93ff4cced78/multiprocess-0.70.18-py313-none-any.whl", hash = "sha256:871743755f43ef57d7910a38433cfe41319e72be1bbd90b79c7a5ac523eb9334", size = 151917, upload-time = "2025-04-17T03:11:24.044Z" }, - { url = "https://files.pythonhosted.org/packages/3b/c3/ca84c19bd14cdfc21c388fdcebf08b86a7a470ebc9f5c3c084fc2dbc50f7/multiprocess-0.70.18-py38-none-any.whl", hash = "sha256:dbf705e52a154fe5e90fb17b38f02556169557c2dd8bb084f2e06c2784d8279b", size = 132636, upload-time = "2025-04-17T03:11:24.936Z" }, - { url = "https://files.pythonhosted.org/packages/6c/28/dd72947e59a6a8c856448a5e74da6201cb5502ddff644fbc790e4bd40b9a/multiprocess-0.70.18-py39-none-any.whl", hash = "sha256:e78ca805a72b1b810c690b6b4cc32579eba34f403094bbbae962b7b5bf9dfcb8", size = 133478, upload-time = "2025-04-17T03:11:26.253Z" }, + { url = "https://files.pythonhosted.org/packages/ef/76/6e712a2623d146d314f17598df5de7224c85c0060ef63fd95cc15a25b3fa/multiprocess-0.70.16-pp310-pypy310_pp73-macosx_10_13_x86_64.whl", hash = "sha256:476887be10e2f59ff183c006af746cb6f1fd0eadcfd4ef49e605cbe2659920ee", size = 134980, upload-time = "2024-01-28T18:52:15.731Z" }, + { url = "https://files.pythonhosted.org/packages/0f/ab/1e6e8009e380e22254ff539ebe117861e5bdb3bff1fc977920972237c6c7/multiprocess-0.70.16-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:d951bed82c8f73929ac82c61f01a7b5ce8f3e5ef40f5b52553b4f547ce2b08ec", size = 134982, upload-time = "2024-01-28T18:52:17.783Z" }, + { url = "https://files.pythonhosted.org/packages/bc/f7/7ec7fddc92e50714ea3745631f79bd9c96424cb2702632521028e57d3a36/multiprocess-0.70.16-py310-none-any.whl", hash = "sha256:c4a9944c67bd49f823687463660a2d6daae94c289adff97e0f9d696ba6371d02", size = 134824, upload-time = "2024-01-28T18:52:26.062Z" }, + { url = "https://files.pythonhosted.org/packages/50/15/b56e50e8debaf439f44befec5b2af11db85f6e0f344c3113ae0be0593a91/multiprocess-0.70.16-py311-none-any.whl", hash = "sha256:af4cabb0dac72abfb1e794fa7855c325fd2b55a10a44628a3c1ad3311c04127a", size = 143519, upload-time = "2024-01-28T18:52:28.115Z" }, + { url = "https://files.pythonhosted.org/packages/0a/7d/a988f258104dcd2ccf1ed40fdc97e26c4ac351eeaf81d76e266c52d84e2f/multiprocess-0.70.16-py312-none-any.whl", hash = "sha256:fc0544c531920dde3b00c29863377f87e1632601092ea2daca74e4beb40faa2e", size = 146741, upload-time = "2024-01-28T18:52:29.395Z" }, + { url = "https://files.pythonhosted.org/packages/ea/89/38df130f2c799090c978b366cfdf5b96d08de5b29a4a293df7f7429fa50b/multiprocess-0.70.16-py38-none-any.whl", hash = "sha256:a71d82033454891091a226dfc319d0cfa8019a4e888ef9ca910372a446de4435", size = 132628, upload-time = "2024-01-28T18:52:30.853Z" }, + { url = "https://files.pythonhosted.org/packages/da/d9/f7f9379981e39b8c2511c9e0326d212accacb82f12fbfdc1aa2ce2a7b2b6/multiprocess-0.70.16-py39-none-any.whl", hash = "sha256:a0bafd3ae1b732eac64be2e72038231c1ba97724b60b09400d68f229fcc2fbf3", size = 133351, upload-time = "2024-01-28T18:52:31.981Z" }, ] [[package]] @@ -2984,6 +3489,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl", hash = "sha256:1be4cccdb0f2482337c4743e60421de3a356cd97508abadd57d47403e94f5505", size = 4963, upload-time = "2025-04-22T14:54:22.983Z" }, ] +[[package]] +name = "names" +version = "0.3.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/44/4e/f9cb7ef2df0250f4ba3334fbdabaa94f9c88097089763d8e85ada8092f84/names-0.3.0.tar.gz", hash = "sha256:726e46254f2ed03f1ffb5d941dae3bc67c35123941c29becd02d48d0caa2a671", size = 789099, upload-time = "2013-05-14T14:55:49.073Z" } + [[package]] name = "nbclient" version = "0.10.3" @@ -3067,9 +3578,9 @@ version = "3.6.1" source = { registry = "https://pypi.org/simple" } resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'linux'", - "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'linux'", "python_full_version == '3.11.*' and sys_platform == 'linux'", + "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'linux'", "python_full_version == '3.11.*' and sys_platform != 'linux'", ] @@ -3078,18 +3589,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762", size = 2068504, upload-time = "2025-12-08T17:02:38.159Z" }, ] -[[package]] -name = "nfa" -version = "3.1.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "reiter" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/25/e4/66d6b0145d3564fe918b20003ddae8d72eb80f15bca47461f7ceca167169/nfa-3.1.0.tar.gz", hash = "sha256:6629a787664469458020a3f714066b2f0ce32dd1a99ce048ef6149b9b80aa243", size = 18272, upload-time = "2022-08-04T06:04:29.394Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/20/7d/39d438c838cebce0dcf33aa7d5949422cc1f73cd2a42073a42e724ed4863/nfa-3.1.0-py3-none-any.whl", hash = "sha256:732391b0122e38a36fc33e07ce475fa4169c00325e3997d989b544e2a661c78f", size = 13966, upload-time = "2022-08-04T06:04:27.851Z" }, -] - [[package]] name = "nltk" version = "3.9.2" @@ -3214,9 +3713,9 @@ version = "2.4.0" source = { registry = "https://pypi.org/simple" } resolution-markers = [ "python_full_version >= '3.14' and sys_platform == 'linux'", - "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'linux'", "python_full_version == '3.11.*' and sys_platform == 'linux'", + "python_full_version >= '3.14' and sys_platform != 'linux'", "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'linux'", "python_full_version == '3.11.*' and sys_platform != 'linux'", ] @@ -3295,6 +3794,140 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/11/73/edeacba3167b1ca66d51b1a5a14697c2c40098b5ffa01811c67b1785a5ab/numpy-2.4.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:a39fb973a726e63223287adc6dafe444ce75af952d711e400f3bf2b36ef55a7b", size = 12489376, upload-time = "2025-12-20T16:18:16.524Z" }, ] +[[package]] +name = "nvidia-cublas-cu12" +version = "12.8.4.1" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/dc/61/e24b560ab2e2eaeb3c839129175fb330dfcfc29e5203196e5541a4c44682/nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:8ac4e771d5a348c551b2a426eda6193c19aa630236b418086020df5ba9667142", size = 594346921, upload-time = "2025-03-07T01:44:31.254Z" }, +] + +[[package]] +name = "nvidia-cuda-cupti-cu12" +version = "12.8.90" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f8/02/2adcaa145158bf1a8295d83591d22e4103dbfd821bcaf6f3f53151ca4ffa/nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ea0cb07ebda26bb9b29ba82cda34849e73c166c18162d3913575b0c9db9a6182", size = 10248621, upload-time = "2025-03-07T01:40:21.213Z" }, +] + +[[package]] +name = "nvidia-cuda-nvrtc-cu12" +version = "12.8.93" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/05/6b/32f747947df2da6994e999492ab306a903659555dddc0fbdeb9d71f75e52/nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:a7756528852ef889772a84c6cd89d41dfa74667e24cca16bb31f8f061e3e9994", size = 88040029, upload-time = "2025-03-07T01:42:13.562Z" }, +] + +[[package]] +name = "nvidia-cuda-runtime-cu12" +version = "12.8.90" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/0d/9b/a997b638fcd068ad6e4d53b8551a7d30fe8b404d6f1804abf1df69838932/nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:adade8dcbd0edf427b7204d480d6066d33902cab2a4707dcfc48a2d0fd44ab90", size = 954765, upload-time = "2025-03-07T01:40:01.615Z" }, +] + +[[package]] +name = "nvidia-cudnn-cu12" +version = "9.10.2.21" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas-cu12", marker = "sys_platform == 'linux'" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/ba/51/e123d997aa098c61d029f76663dedbfb9bc8dcf8c60cbd6adbe42f76d049/nvidia_cudnn_cu12-9.10.2.21-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:949452be657fa16687d0930933f032835951ef0892b37d2d53824d1a84dc97a8", size = 706758467, upload-time = "2025-06-06T21:54:08.597Z" }, +] + +[[package]] +name = "nvidia-cufft-cu12" +version = "11.3.3.83" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink-cu12", marker = "sys_platform == 'linux'" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/1f/13/ee4e00f30e676b66ae65b4f08cb5bcbb8392c03f54f2d5413ea99a5d1c80/nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4d2dd21ec0b88cf61b62e6b43564355e5222e4a3fb394cac0db101f2dd0d4f74", size = 193118695, upload-time = "2025-03-07T01:45:27.821Z" }, +] + +[[package]] +name = "nvidia-cufile-cu12" +version = "1.13.1.3" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/bb/fe/1bcba1dfbfb8d01be8d93f07bfc502c93fa23afa6fd5ab3fc7c1df71038a/nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1d069003be650e131b21c932ec3d8969c1715379251f8d23a1860554b1cb24fc", size = 1197834, upload-time = "2025-03-07T01:45:50.723Z" }, +] + +[[package]] +name = "nvidia-curand-cu12" +version = "10.3.9.90" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/fb/aa/6584b56dc84ebe9cf93226a5cde4d99080c8e90ab40f0c27bda7a0f29aa1/nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:b32331d4f4df5d6eefa0554c565b626c7216f87a06a4f56fab27c3b68a830ec9", size = 63619976, upload-time = "2025-03-07T01:46:23.323Z" }, +] + +[[package]] +name = "nvidia-cusolver-cu12" +version = "11.7.3.90" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-cublas-cu12", marker = "sys_platform == 'linux'" }, + { name = "nvidia-cusparse-cu12", marker = "sys_platform == 'linux'" }, + { name = "nvidia-nvjitlink-cu12", marker = "sys_platform == 'linux'" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/85/48/9a13d2975803e8cf2777d5ed57b87a0b6ca2cc795f9a4f59796a910bfb80/nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:4376c11ad263152bd50ea295c05370360776f8c3427b30991df774f9fb26c450", size = 267506905, upload-time = "2025-03-07T01:47:16.273Z" }, +] + +[[package]] +name = "nvidia-cusparse-cu12" +version = "12.5.8.93" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "nvidia-nvjitlink-cu12", marker = "sys_platform == 'linux'" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/c2/f5/e1854cb2f2bcd4280c44736c93550cc300ff4b8c95ebe370d0aa7d2b473d/nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1ec05d76bbbd8b61b06a80e1eaf8cf4959c3d4ce8e711b65ebd0443bb0ebb13b", size = 288216466, upload-time = "2025-03-07T01:48:13.779Z" }, +] + +[[package]] +name = "nvidia-cusparselt-cu12" +version = "0.7.1" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/56/79/12978b96bd44274fe38b5dde5cfb660b1d114f70a65ef962bcbbed99b549/nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl", hash = "sha256:f1bb701d6b930d5a7cea44c19ceb973311500847f81b634d802b7b539dc55623", size = 287193691, upload-time = "2025-02-26T00:15:44.104Z" }, +] + +[[package]] +name = "nvidia-nccl-cu12" +version = "2.27.5" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6e/89/f7a07dc961b60645dbbf42e80f2bc85ade7feb9a491b11a1e973aa00071f/nvidia_nccl_cu12-2.27.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ad730cf15cb5d25fe849c6e6ca9eb5b76db16a80f13f425ac68d8e2e55624457", size = 322348229, upload-time = "2025-06-26T04:11:28.385Z" }, +] + +[[package]] +name = "nvidia-nvjitlink-cu12" +version = "12.8.93" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f6/74/86a07f1d0f42998ca31312f998bd3b9a7eff7f52378f4f270c8679c77fb9/nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:81ff63371a7ebd6e6451970684f916be2eab07321b73c9d244dc2b4da7f73b88", size = 39254836, upload-time = "2025-03-07T01:49:55.661Z" }, +] + +[[package]] +name = "nvidia-nvshmem-cu12" +version = "3.4.5" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b5/09/6ea3ea725f82e1e76684f0708bbedd871fc96da89945adeba65c3835a64c/nvidia_nvshmem_cu12-3.4.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:042f2500f24c021db8a06c5eec2539027d57460e1c1a762055a6554f72c369bd", size = 139103095, upload-time = "2025-09-06T00:32:31.266Z" }, +] + +[[package]] +name = "nvidia-nvtx-cu12" +version = "12.8.90" +source = { registry = "https://pypi.org/simple" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/eb/86626c1bbc2edb86323022371c39aa48df6fd8b0a1647bc274577f72e90b/nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5b17e2001cc0d751a5bc2c6ec6d26ad95913324a4adb86788c944f8ce9ba441f", size = 89954, upload-time = "2025-03-07T01:42:44.131Z" }, +] + [[package]] name = "openai" version = "1.109.1" @@ -3561,7 +4194,7 @@ wheels = [ [[package]] name = "pandas" -version = "2.3.3" +version = "2.2.3" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, @@ -3570,55 +4203,42 @@ dependencies = [ { name = "pytz" }, { name = "tzdata" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/33/01/d40b85317f86cf08d853a4f495195c73815fdf205eef3993821720274518/pandas-2.3.3.tar.gz", hash = "sha256:e05e1af93b977f7eafa636d043f9f94c7ee3ac81af99c13508215942e64c993b", size = 4495223, upload-time = "2025-09-29T23:34:51.853Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/3d/f7/f425a00df4fcc22b292c6895c6831c0c8ae1d9fac1e024d16f98a9ce8749/pandas-2.3.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:376c6446ae31770764215a6c937f72d917f214b43560603cd60da6408f183b6c", size = 11555763, upload-time = "2025-09-29T23:16:53.287Z" }, - { url = "https://files.pythonhosted.org/packages/13/4f/66d99628ff8ce7857aca52fed8f0066ce209f96be2fede6cef9f84e8d04f/pandas-2.3.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:e19d192383eab2f4ceb30b412b22ea30690c9e618f78870357ae1d682912015a", size = 10801217, upload-time = "2025-09-29T23:17:04.522Z" }, - { url = "https://files.pythonhosted.org/packages/1d/03/3fc4a529a7710f890a239cc496fc6d50ad4a0995657dccc1d64695adb9f4/pandas-2.3.3-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5caf26f64126b6c7aec964f74266f435afef1c1b13da3b0636c7518a1fa3e2b1", size = 12148791, upload-time = "2025-09-29T23:17:18.444Z" }, - { url = "https://files.pythonhosted.org/packages/40/a8/4dac1f8f8235e5d25b9955d02ff6f29396191d4e665d71122c3722ca83c5/pandas-2.3.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dd7478f1463441ae4ca7308a70e90b33470fa593429f9d4c578dd00d1fa78838", size = 12769373, upload-time = "2025-09-29T23:17:35.846Z" }, - { url = "https://files.pythonhosted.org/packages/df/91/82cc5169b6b25440a7fc0ef3a694582418d875c8e3ebf796a6d6470aa578/pandas-2.3.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:4793891684806ae50d1288c9bae9330293ab4e083ccd1c5e383c34549c6e4250", size = 13200444, upload-time = "2025-09-29T23:17:49.341Z" }, - { url = "https://files.pythonhosted.org/packages/10/ae/89b3283800ab58f7af2952704078555fa60c807fff764395bb57ea0b0dbd/pandas-2.3.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:28083c648d9a99a5dd035ec125d42439c6c1c525098c58af0fc38dd1a7a1b3d4", size = 13858459, upload-time = "2025-09-29T23:18:03.722Z" }, - { url = "https://files.pythonhosted.org/packages/85/72/530900610650f54a35a19476eca5104f38555afccda1aa11a92ee14cb21d/pandas-2.3.3-cp310-cp310-win_amd64.whl", hash = "sha256:503cf027cf9940d2ceaa1a93cfb5f8c8c7e6e90720a2850378f0b3f3b1e06826", size = 11346086, upload-time = "2025-09-29T23:18:18.505Z" }, - { url = "https://files.pythonhosted.org/packages/c1/fa/7ac648108144a095b4fb6aa3de1954689f7af60a14cf25583f4960ecb878/pandas-2.3.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:602b8615ebcc4a0c1751e71840428ddebeb142ec02c786e8ad6b1ce3c8dec523", size = 11578790, upload-time = "2025-09-29T23:18:30.065Z" }, - { url = "https://files.pythonhosted.org/packages/9b/35/74442388c6cf008882d4d4bdfc4109be87e9b8b7ccd097ad1e7f006e2e95/pandas-2.3.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:8fe25fc7b623b0ef6b5009149627e34d2a4657e880948ec3c840e9402e5c1b45", size = 10833831, upload-time = "2025-09-29T23:38:56.071Z" }, - { url = "https://files.pythonhosted.org/packages/fe/e4/de154cbfeee13383ad58d23017da99390b91d73f8c11856f2095e813201b/pandas-2.3.3-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b468d3dad6ff947df92dcb32ede5b7bd41a9b3cceef0a30ed925f6d01fb8fa66", size = 12199267, upload-time = "2025-09-29T23:18:41.627Z" }, - { url = "https://files.pythonhosted.org/packages/bf/c9/63f8d545568d9ab91476b1818b4741f521646cbdd151c6efebf40d6de6f7/pandas-2.3.3-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b98560e98cb334799c0b07ca7967ac361a47326e9b4e5a7dfb5ab2b1c9d35a1b", size = 12789281, upload-time = "2025-09-29T23:18:56.834Z" }, - { url = "https://files.pythonhosted.org/packages/f2/00/a5ac8c7a0e67fd1a6059e40aa08fa1c52cc00709077d2300e210c3ce0322/pandas-2.3.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:1d37b5848ba49824e5c30bedb9c830ab9b7751fd049bc7914533e01c65f79791", size = 13240453, upload-time = "2025-09-29T23:19:09.247Z" }, - { url = "https://files.pythonhosted.org/packages/27/4d/5c23a5bc7bd209231618dd9e606ce076272c9bc4f12023a70e03a86b4067/pandas-2.3.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:db4301b2d1f926ae677a751eb2bd0e8c5f5319c9cb3f88b0becbbb0b07b34151", size = 13890361, upload-time = "2025-09-29T23:19:25.342Z" }, - { url = "https://files.pythonhosted.org/packages/8e/59/712db1d7040520de7a4965df15b774348980e6df45c129b8c64d0dbe74ef/pandas-2.3.3-cp311-cp311-win_amd64.whl", hash = "sha256:f086f6fe114e19d92014a1966f43a3e62285109afe874f067f5abbdcbb10e59c", size = 11348702, upload-time = "2025-09-29T23:19:38.296Z" }, - { url = "https://files.pythonhosted.org/packages/9c/fb/231d89e8637c808b997d172b18e9d4a4bc7bf31296196c260526055d1ea0/pandas-2.3.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6d21f6d74eb1725c2efaa71a2bfc661a0689579b58e9c0ca58a739ff0b002b53", size = 11597846, upload-time = "2025-09-29T23:19:48.856Z" }, - { url = "https://files.pythonhosted.org/packages/5c/bd/bf8064d9cfa214294356c2d6702b716d3cf3bb24be59287a6a21e24cae6b/pandas-2.3.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:3fd2f887589c7aa868e02632612ba39acb0b8948faf5cc58f0850e165bd46f35", size = 10729618, upload-time = "2025-09-29T23:39:08.659Z" }, - { url = "https://files.pythonhosted.org/packages/57/56/cf2dbe1a3f5271370669475ead12ce77c61726ffd19a35546e31aa8edf4e/pandas-2.3.3-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ecaf1e12bdc03c86ad4a7ea848d66c685cb6851d807a26aa245ca3d2017a1908", size = 11737212, upload-time = "2025-09-29T23:19:59.765Z" }, - { url = "https://files.pythonhosted.org/packages/e5/63/cd7d615331b328e287d8233ba9fdf191a9c2d11b6af0c7a59cfcec23de68/pandas-2.3.3-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b3d11d2fda7eb164ef27ffc14b4fcab16a80e1ce67e9f57e19ec0afaf715ba89", size = 12362693, upload-time = "2025-09-29T23:20:14.098Z" }, - { url = "https://files.pythonhosted.org/packages/a6/de/8b1895b107277d52f2b42d3a6806e69cfef0d5cf1d0ba343470b9d8e0a04/pandas-2.3.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:a68e15f780eddf2b07d242e17a04aa187a7ee12b40b930bfdd78070556550e98", size = 12771002, upload-time = "2025-09-29T23:20:26.76Z" }, - { url = "https://files.pythonhosted.org/packages/87/21/84072af3187a677c5893b170ba2c8fbe450a6ff911234916da889b698220/pandas-2.3.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:371a4ab48e950033bcf52b6527eccb564f52dc826c02afd9a1bc0ab731bba084", size = 13450971, upload-time = "2025-09-29T23:20:41.344Z" }, - { url = "https://files.pythonhosted.org/packages/86/41/585a168330ff063014880a80d744219dbf1dd7a1c706e75ab3425a987384/pandas-2.3.3-cp312-cp312-win_amd64.whl", hash = "sha256:a16dcec078a01eeef8ee61bf64074b4e524a2a3f4b3be9326420cabe59c4778b", size = 10992722, upload-time = "2025-09-29T23:20:54.139Z" }, - { url = "https://files.pythonhosted.org/packages/cd/4b/18b035ee18f97c1040d94debd8f2e737000ad70ccc8f5513f4eefad75f4b/pandas-2.3.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:56851a737e3470de7fa88e6131f41281ed440d29a9268dcbf0002da5ac366713", size = 11544671, upload-time = "2025-09-29T23:21:05.024Z" }, - { url = "https://files.pythonhosted.org/packages/31/94/72fac03573102779920099bcac1c3b05975c2cb5f01eac609faf34bed1ca/pandas-2.3.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:bdcd9d1167f4885211e401b3036c0c8d9e274eee67ea8d0758a256d60704cfe8", size = 10680807, upload-time = "2025-09-29T23:21:15.979Z" }, - { url = "https://files.pythonhosted.org/packages/16/87/9472cf4a487d848476865321de18cc8c920b8cab98453ab79dbbc98db63a/pandas-2.3.3-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e32e7cc9af0f1cc15548288a51a3b681cc2a219faa838e995f7dc53dbab1062d", size = 11709872, upload-time = "2025-09-29T23:21:27.165Z" }, - { url = "https://files.pythonhosted.org/packages/15/07/284f757f63f8a8d69ed4472bfd85122bd086e637bf4ed09de572d575a693/pandas-2.3.3-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:318d77e0e42a628c04dc56bcef4b40de67918f7041c2b061af1da41dcff670ac", size = 12306371, upload-time = "2025-09-29T23:21:40.532Z" }, - { url = "https://files.pythonhosted.org/packages/33/81/a3afc88fca4aa925804a27d2676d22dcd2031c2ebe08aabd0ae55b9ff282/pandas-2.3.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4e0a175408804d566144e170d0476b15d78458795bb18f1304fb94160cabf40c", size = 12765333, upload-time = "2025-09-29T23:21:55.77Z" }, - { url = "https://files.pythonhosted.org/packages/8d/0f/b4d4ae743a83742f1153464cf1a8ecfafc3ac59722a0b5c8602310cb7158/pandas-2.3.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:93c2d9ab0fc11822b5eece72ec9587e172f63cff87c00b062f6e37448ced4493", size = 13418120, upload-time = "2025-09-29T23:22:10.109Z" }, - { url = "https://files.pythonhosted.org/packages/4f/c7/e54682c96a895d0c808453269e0b5928a07a127a15704fedb643e9b0a4c8/pandas-2.3.3-cp313-cp313-win_amd64.whl", hash = "sha256:f8bfc0e12dc78f777f323f55c58649591b2cd0c43534e8355c51d3fede5f4dee", size = 10993991, upload-time = "2025-09-29T23:25:04.889Z" }, - { url = "https://files.pythonhosted.org/packages/f9/ca/3f8d4f49740799189e1395812f3bf23b5e8fc7c190827d55a610da72ce55/pandas-2.3.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:75ea25f9529fdec2d2e93a42c523962261e567d250b0013b16210e1d40d7c2e5", size = 12048227, upload-time = "2025-09-29T23:22:24.343Z" }, - { url = "https://files.pythonhosted.org/packages/0e/5a/f43efec3e8c0cc92c4663ccad372dbdff72b60bdb56b2749f04aa1d07d7e/pandas-2.3.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:74ecdf1d301e812db96a465a525952f4dde225fdb6d8e5a521d47e1f42041e21", size = 11411056, upload-time = "2025-09-29T23:22:37.762Z" }, - { url = "https://files.pythonhosted.org/packages/46/b1/85331edfc591208c9d1a63a06baa67b21d332e63b7a591a5ba42a10bb507/pandas-2.3.3-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6435cb949cb34ec11cc9860246ccb2fdc9ecd742c12d3304989017d53f039a78", size = 11645189, upload-time = "2025-09-29T23:22:51.688Z" }, - { url = "https://files.pythonhosted.org/packages/44/23/78d645adc35d94d1ac4f2a3c4112ab6f5b8999f4898b8cdf01252f8df4a9/pandas-2.3.3-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:900f47d8f20860de523a1ac881c4c36d65efcb2eb850e6948140fa781736e110", size = 12121912, upload-time = "2025-09-29T23:23:05.042Z" }, - { url = "https://files.pythonhosted.org/packages/53/da/d10013df5e6aaef6b425aa0c32e1fc1f3e431e4bcabd420517dceadce354/pandas-2.3.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:a45c765238e2ed7d7c608fc5bc4a6f88b642f2f01e70c0c23d2224dd21829d86", size = 12712160, upload-time = "2025-09-29T23:23:28.57Z" }, - { url = "https://files.pythonhosted.org/packages/bd/17/e756653095a083d8a37cbd816cb87148debcfcd920129b25f99dd8d04271/pandas-2.3.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c4fc4c21971a1a9f4bdb4c73978c7f7256caa3e62b323f70d6cb80db583350bc", size = 13199233, upload-time = "2025-09-29T23:24:24.876Z" }, - { url = "https://files.pythonhosted.org/packages/04/fd/74903979833db8390b73b3a8a7d30d146d710bd32703724dd9083950386f/pandas-2.3.3-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:ee15f284898e7b246df8087fc82b87b01686f98ee67d85a17b7ab44143a3a9a0", size = 11540635, upload-time = "2025-09-29T23:25:52.486Z" }, - { url = "https://files.pythonhosted.org/packages/21/00/266d6b357ad5e6d3ad55093a7e8efc7dd245f5a842b584db9f30b0f0a287/pandas-2.3.3-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:1611aedd912e1ff81ff41c745822980c49ce4a7907537be8692c8dbc31924593", size = 10759079, upload-time = "2025-09-29T23:26:33.204Z" }, - { url = "https://files.pythonhosted.org/packages/ca/05/d01ef80a7a3a12b2f8bbf16daba1e17c98a2f039cbc8e2f77a2c5a63d382/pandas-2.3.3-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6d2cefc361461662ac48810cb14365a365ce864afe85ef1f447ff5a1e99ea81c", size = 11814049, upload-time = "2025-09-29T23:27:15.384Z" }, - { url = "https://files.pythonhosted.org/packages/15/b2/0e62f78c0c5ba7e3d2c5945a82456f4fac76c480940f805e0b97fcbc2f65/pandas-2.3.3-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ee67acbbf05014ea6c763beb097e03cd629961c8a632075eeb34247120abcb4b", size = 12332638, upload-time = "2025-09-29T23:27:51.625Z" }, - { url = "https://files.pythonhosted.org/packages/c5/33/dd70400631b62b9b29c3c93d2feee1d0964dc2bae2e5ad7a6c73a7f25325/pandas-2.3.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:c46467899aaa4da076d5abc11084634e2d197e9460643dd455ac3db5856b24d6", size = 12886834, upload-time = "2025-09-29T23:28:21.289Z" }, - { url = "https://files.pythonhosted.org/packages/d3/18/b5d48f55821228d0d2692b34fd5034bb185e854bdb592e9c640f6290e012/pandas-2.3.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6253c72c6a1d990a410bc7de641d34053364ef8bcd3126f7e7450125887dffe3", size = 13409925, upload-time = "2025-09-29T23:28:58.261Z" }, - { url = "https://files.pythonhosted.org/packages/a6/3d/124ac75fcd0ecc09b8fdccb0246ef65e35b012030defb0e0eba2cbbbe948/pandas-2.3.3-cp314-cp314-win_amd64.whl", hash = "sha256:1b07204a219b3b7350abaae088f451860223a52cfb8a6c53358e7948735158e5", size = 11109071, upload-time = "2025-09-29T23:32:27.484Z" }, - { url = "https://files.pythonhosted.org/packages/89/9c/0e21c895c38a157e0faa1fb64587a9226d6dd46452cac4532d80c3c4a244/pandas-2.3.3-cp314-cp314t-macosx_10_13_x86_64.whl", hash = "sha256:2462b1a365b6109d275250baaae7b760fd25c726aaca0054649286bcfbb3e8ec", size = 12048504, upload-time = "2025-09-29T23:29:31.47Z" }, - { url = "https://files.pythonhosted.org/packages/d7/82/b69a1c95df796858777b68fbe6a81d37443a33319761d7c652ce77797475/pandas-2.3.3-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:0242fe9a49aa8b4d78a4fa03acb397a58833ef6199e9aa40a95f027bb3a1b6e7", size = 11410702, upload-time = "2025-09-29T23:29:54.591Z" }, - { url = "https://files.pythonhosted.org/packages/f9/88/702bde3ba0a94b8c73a0181e05144b10f13f29ebfc2150c3a79062a8195d/pandas-2.3.3-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a21d830e78df0a515db2b3d2f5570610f5e6bd2e27749770e8bb7b524b89b450", size = 11634535, upload-time = "2025-09-29T23:30:21.003Z" }, - { url = "https://files.pythonhosted.org/packages/a4/1e/1bac1a839d12e6a82ec6cb40cda2edde64a2013a66963293696bbf31fbbb/pandas-2.3.3-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2e3ebdb170b5ef78f19bfb71b0dc5dc58775032361fa188e814959b74d726dd5", size = 12121582, upload-time = "2025-09-29T23:30:43.391Z" }, - { url = "https://files.pythonhosted.org/packages/44/91/483de934193e12a3b1d6ae7c8645d083ff88dec75f46e827562f1e4b4da6/pandas-2.3.3-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:d051c0e065b94b7a3cea50eb1ec32e912cd96dba41647eb24104b6c6c14c5788", size = 12699963, upload-time = "2025-09-29T23:31:10.009Z" }, - { url = "https://files.pythonhosted.org/packages/70/44/5191d2e4026f86a2a109053e194d3ba7a31a2d10a9c2348368c63ed4e85a/pandas-2.3.3-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:3869faf4bd07b3b66a9f462417d0ca3a9df29a9f6abd5d0d0dbab15dac7abe87", size = 13202175, upload-time = "2025-09-29T23:31:59.173Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/9c/d6/9f8431bacc2e19dca897724cd097b1bb224a6ad5433784a44b587c7c13af/pandas-2.2.3.tar.gz", hash = "sha256:4f18ba62b61d7e192368b84517265a99b4d7ee8912f8708660fb4a366cc82667", size = 4399213, upload-time = "2024-09-20T13:10:04.827Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/aa/70/c853aec59839bceed032d52010ff5f1b8d87dc3114b762e4ba2727661a3b/pandas-2.2.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1948ddde24197a0f7add2bdc4ca83bf2b1ef84a1bc8ccffd95eda17fd836ecb5", size = 12580827, upload-time = "2024-09-20T13:08:42.347Z" }, + { url = "https://files.pythonhosted.org/packages/99/f2/c4527768739ffa4469b2b4fff05aa3768a478aed89a2f271a79a40eee984/pandas-2.2.3-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:381175499d3802cde0eabbaf6324cce0c4f5d52ca6f8c377c29ad442f50f6348", size = 11303897, upload-time = "2024-09-20T13:08:45.807Z" }, + { url = "https://files.pythonhosted.org/packages/ed/12/86c1747ea27989d7a4064f806ce2bae2c6d575b950be087837bdfcabacc9/pandas-2.2.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d9c45366def9a3dd85a6454c0e7908f2b3b8e9c138f5dc38fed7ce720d8453ed", size = 66480908, upload-time = "2024-09-20T18:37:13.513Z" }, + { url = "https://files.pythonhosted.org/packages/44/50/7db2cd5e6373ae796f0ddad3675268c8d59fb6076e66f0c339d61cea886b/pandas-2.2.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:86976a1c5b25ae3f8ccae3a5306e443569ee3c3faf444dfd0f41cda24667ad57", size = 13064210, upload-time = "2024-09-20T13:08:48.325Z" }, + { url = "https://files.pythonhosted.org/packages/61/61/a89015a6d5536cb0d6c3ba02cebed51a95538cf83472975275e28ebf7d0c/pandas-2.2.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:b8661b0238a69d7aafe156b7fa86c44b881387509653fdf857bebc5e4008ad42", size = 16754292, upload-time = "2024-09-20T19:01:54.443Z" }, + { url = "https://files.pythonhosted.org/packages/ce/0d/4cc7b69ce37fac07645a94e1d4b0880b15999494372c1523508511b09e40/pandas-2.2.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:37e0aced3e8f539eccf2e099f65cdb9c8aa85109b0be6e93e2baff94264bdc6f", size = 14416379, upload-time = "2024-09-20T13:08:50.882Z" }, + { url = "https://files.pythonhosted.org/packages/31/9e/6ebb433de864a6cd45716af52a4d7a8c3c9aaf3a98368e61db9e69e69a9c/pandas-2.2.3-cp310-cp310-win_amd64.whl", hash = "sha256:56534ce0746a58afaf7942ba4863e0ef81c9c50d3f0ae93e9497d6a41a057645", size = 11598471, upload-time = "2024-09-20T13:08:53.332Z" }, + { url = "https://files.pythonhosted.org/packages/a8/44/d9502bf0ed197ba9bf1103c9867d5904ddcaf869e52329787fc54ed70cc8/pandas-2.2.3-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:66108071e1b935240e74525006034333f98bcdb87ea116de573a6a0dccb6c039", size = 12602222, upload-time = "2024-09-20T13:08:56.254Z" }, + { url = "https://files.pythonhosted.org/packages/52/11/9eac327a38834f162b8250aab32a6781339c69afe7574368fffe46387edf/pandas-2.2.3-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7c2875855b0ff77b2a64a0365e24455d9990730d6431b9e0ee18ad8acee13dbd", size = 11321274, upload-time = "2024-09-20T13:08:58.645Z" }, + { url = "https://files.pythonhosted.org/packages/45/fb/c4beeb084718598ba19aa9f5abbc8aed8b42f90930da861fcb1acdb54c3a/pandas-2.2.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cd8d0c3be0515c12fed0bdbae072551c8b54b7192c7b1fda0ba56059a0179698", size = 15579836, upload-time = "2024-09-20T19:01:57.571Z" }, + { url = "https://files.pythonhosted.org/packages/cd/5f/4dba1d39bb9c38d574a9a22548c540177f78ea47b32f99c0ff2ec499fac5/pandas-2.2.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c124333816c3a9b03fbeef3a9f230ba9a737e9e5bb4060aa2107a86cc0a497fc", size = 13058505, upload-time = "2024-09-20T13:09:01.501Z" }, + { url = "https://files.pythonhosted.org/packages/b9/57/708135b90391995361636634df1f1130d03ba456e95bcf576fada459115a/pandas-2.2.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:63cc132e40a2e084cf01adf0775b15ac515ba905d7dcca47e9a251819c575ef3", size = 16744420, upload-time = "2024-09-20T19:02:00.678Z" }, + { url = "https://files.pythonhosted.org/packages/86/4a/03ed6b7ee323cf30404265c284cee9c65c56a212e0a08d9ee06984ba2240/pandas-2.2.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:29401dbfa9ad77319367d36940cd8a0b3a11aba16063e39632d98b0e931ddf32", size = 14440457, upload-time = "2024-09-20T13:09:04.105Z" }, + { url = "https://files.pythonhosted.org/packages/ed/8c/87ddf1fcb55d11f9f847e3c69bb1c6f8e46e2f40ab1a2d2abadb2401b007/pandas-2.2.3-cp311-cp311-win_amd64.whl", hash = "sha256:3fc6873a41186404dad67245896a6e440baacc92f5b716ccd1bc9ed2995ab2c5", size = 11617166, upload-time = "2024-09-20T13:09:06.917Z" }, + { url = "https://files.pythonhosted.org/packages/17/a3/fb2734118db0af37ea7433f57f722c0a56687e14b14690edff0cdb4b7e58/pandas-2.2.3-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:b1d432e8d08679a40e2a6d8b2f9770a5c21793a6f9f47fdd52c5ce1948a5a8a9", size = 12529893, upload-time = "2024-09-20T13:09:09.655Z" }, + { url = "https://files.pythonhosted.org/packages/e1/0c/ad295fd74bfac85358fd579e271cded3ac969de81f62dd0142c426b9da91/pandas-2.2.3-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a5a1595fe639f5988ba6a8e5bc9649af3baf26df3998a0abe56c02609392e0a4", size = 11363475, upload-time = "2024-09-20T13:09:14.718Z" }, + { url = "https://files.pythonhosted.org/packages/c6/2a/4bba3f03f7d07207481fed47f5b35f556c7441acddc368ec43d6643c5777/pandas-2.2.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5de54125a92bb4d1c051c0659e6fcb75256bf799a732a87184e5ea503965bce3", size = 15188645, upload-time = "2024-09-20T19:02:03.88Z" }, + { url = "https://files.pythonhosted.org/packages/38/f8/d8fddee9ed0d0c0f4a2132c1dfcf0e3e53265055da8df952a53e7eaf178c/pandas-2.2.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:fffb8ae78d8af97f849404f21411c95062db1496aeb3e56f146f0355c9989319", size = 12739445, upload-time = "2024-09-20T13:09:17.621Z" }, + { url = "https://files.pythonhosted.org/packages/20/e8/45a05d9c39d2cea61ab175dbe6a2de1d05b679e8de2011da4ee190d7e748/pandas-2.2.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6dfcb5ee8d4d50c06a51c2fffa6cff6272098ad6540aed1a76d15fb9318194d8", size = 16359235, upload-time = "2024-09-20T19:02:07.094Z" }, + { url = "https://files.pythonhosted.org/packages/1d/99/617d07a6a5e429ff90c90da64d428516605a1ec7d7bea494235e1c3882de/pandas-2.2.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:062309c1b9ea12a50e8ce661145c6aab431b1e99530d3cd60640e255778bd43a", size = 14056756, upload-time = "2024-09-20T13:09:20.474Z" }, + { url = "https://files.pythonhosted.org/packages/29/d4/1244ab8edf173a10fd601f7e13b9566c1b525c4f365d6bee918e68381889/pandas-2.2.3-cp312-cp312-win_amd64.whl", hash = "sha256:59ef3764d0fe818125a5097d2ae867ca3fa64df032331b7e0917cf5d7bf66b13", size = 11504248, upload-time = "2024-09-20T13:09:23.137Z" }, + { url = "https://files.pythonhosted.org/packages/64/22/3b8f4e0ed70644e85cfdcd57454686b9057c6c38d2f74fe4b8bc2527214a/pandas-2.2.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:f00d1345d84d8c86a63e476bb4955e46458b304b9575dcf71102b5c705320015", size = 12477643, upload-time = "2024-09-20T13:09:25.522Z" }, + { url = "https://files.pythonhosted.org/packages/e4/93/b3f5d1838500e22c8d793625da672f3eec046b1a99257666c94446969282/pandas-2.2.3-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:3508d914817e153ad359d7e069d752cdd736a247c322d932eb89e6bc84217f28", size = 11281573, upload-time = "2024-09-20T13:09:28.012Z" }, + { url = "https://files.pythonhosted.org/packages/f5/94/6c79b07f0e5aab1dcfa35a75f4817f5c4f677931d4234afcd75f0e6a66ca/pandas-2.2.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:22a9d949bfc9a502d320aa04e5d02feab689d61da4e7764b62c30b991c42c5f0", size = 15196085, upload-time = "2024-09-20T19:02:10.451Z" }, + { url = "https://files.pythonhosted.org/packages/e8/31/aa8da88ca0eadbabd0a639788a6da13bb2ff6edbbb9f29aa786450a30a91/pandas-2.2.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f3a255b2c19987fbbe62a9dfd6cff7ff2aa9ccab3fc75218fd4b7530f01efa24", size = 12711809, upload-time = "2024-09-20T13:09:30.814Z" }, + { url = "https://files.pythonhosted.org/packages/ee/7c/c6dbdb0cb2a4344cacfb8de1c5808ca885b2e4dcfde8008266608f9372af/pandas-2.2.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:800250ecdadb6d9c78eae4990da62743b857b470883fa27f652db8bdde7f6659", size = 16356316, upload-time = "2024-09-20T19:02:13.825Z" }, + { url = "https://files.pythonhosted.org/packages/57/b7/8b757e7d92023b832869fa8881a992696a0bfe2e26f72c9ae9f255988d42/pandas-2.2.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:6374c452ff3ec675a8f46fd9ab25c4ad0ba590b71cf0656f8b6daa5202bca3fb", size = 14022055, upload-time = "2024-09-20T13:09:33.462Z" }, + { url = "https://files.pythonhosted.org/packages/3b/bc/4b18e2b8c002572c5a441a64826252ce5da2aa738855747247a971988043/pandas-2.2.3-cp313-cp313-win_amd64.whl", hash = "sha256:61c5ad4043f791b61dd4752191d9f07f0ae412515d59ba8f005832a532f8736d", size = 11481175, upload-time = "2024-09-20T13:09:35.871Z" }, + { url = "https://files.pythonhosted.org/packages/76/a3/a5d88146815e972d40d19247b2c162e88213ef51c7c25993942c39dbf41d/pandas-2.2.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3b71f27954685ee685317063bf13c7709a7ba74fc996b84fc6821c59b0f06468", size = 12615650, upload-time = "2024-09-20T13:09:38.685Z" }, + { url = "https://files.pythonhosted.org/packages/9c/8c/f0fd18f6140ddafc0c24122c8a964e48294acc579d47def376fef12bcb4a/pandas-2.2.3-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:38cf8125c40dae9d5acc10fa66af8ea6fdf760b2714ee482ca691fc66e6fcb18", size = 11290177, upload-time = "2024-09-20T13:09:41.141Z" }, + { url = "https://files.pythonhosted.org/packages/ed/f9/e995754eab9c0f14c6777401f7eece0943840b7a9fc932221c19d1abee9f/pandas-2.2.3-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ba96630bc17c875161df3818780af30e43be9b166ce51c9a18c1feae342906c2", size = 14651526, upload-time = "2024-09-20T19:02:16.905Z" }, + { url = "https://files.pythonhosted.org/packages/25/b0/98d6ae2e1abac4f35230aa756005e8654649d305df9a28b16b9ae4353bff/pandas-2.2.3-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1db71525a1538b30142094edb9adc10be3f3e176748cd7acc2240c2f2e5aa3a4", size = 11871013, upload-time = "2024-09-20T13:09:44.39Z" }, + { url = "https://files.pythonhosted.org/packages/cc/57/0f72a10f9db6a4628744c8e8f0df4e6e21de01212c7c981d31e50ffc8328/pandas-2.2.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:15c0e1e02e93116177d29ff83e8b1619c93ddc9c49083f237d4312337a61165d", size = 15711620, upload-time = "2024-09-20T19:02:20.639Z" }, + { url = "https://files.pythonhosted.org/packages/ab/5f/b38085618b950b79d2d9164a711c52b10aefc0ae6833b96f626b7021b2ed/pandas-2.2.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:ad5b65698ab28ed8d7f18790a0dc58005c7629f227be9ecc1072aa74c0c1d43a", size = 13098436, upload-time = "2024-09-20T13:09:48.112Z" }, ] [[package]] @@ -3648,6 +4268,19 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/cc/20/ff623b09d963f88bfde16306a54e12ee5ea43e9b597108672ff3a408aad6/pathspec-0.12.1-py3-none-any.whl", hash = "sha256:a0d503e138a4c123b27490a4f7beda6a01c6f288df0e4a8b79c7eb0dc7b4cc08", size = 31191, upload-time = "2023-12-10T22:30:43.14Z" }, ] +[[package]] +name = "pdfminer-six" +version = "20231228" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "charset-normalizer" }, + { name = "cryptography" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/31/b1/a43e3bd872ded4deea4f8efc7aff1703fca8c5455d0c06e20506a06a44ff/pdfminer.six-20231228.tar.gz", hash = "sha256:6004da3ad1a7a4d45930cb950393df89b068e73be365a6ff64a838d37bcb08c4", size = 7362505, upload-time = "2023-12-28T21:25:32.863Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/eb/9c/e46fe7502b32d7db6af6e36a9105abb93301fa1ec475b5ddcba8b35ae23a/pdfminer.six-20231228-py3-none-any.whl", hash = "sha256:e8d3c3310e6fbc1fe414090123ab01351634b4ecb021232206c4c9a8ca3e3b8f", size = 5614515, upload-time = "2023-12-28T21:25:30.329Z" }, +] + [[package]] name = "pexpect" version = "4.9.0" @@ -3662,61 +4295,69 @@ wheels = [ [[package]] name = "pillow" -version = "10.4.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/cd/74/ad3d526f3bf7b6d3f408b73fde271ec69dfac8b81341a318ce825f2b3812/pillow-10.4.0.tar.gz", hash = "sha256:166c1cd4d24309b30d61f79f4a9114b7b2313d7450912277855ff5dfd7cd4a06", size = 46555059, upload-time = "2024-07-01T09:48:43.583Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/0e/69/a31cccd538ca0b5272be2a38347f8839b97a14be104ea08b0db92f749c74/pillow-10.4.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:4d9667937cfa347525b319ae34375c37b9ee6b525440f3ef48542fcf66f2731e", size = 3509271, upload-time = "2024-07-01T09:45:22.07Z" }, - { url = "https://files.pythonhosted.org/packages/9a/9e/4143b907be8ea0bce215f2ae4f7480027473f8b61fcedfda9d851082a5d2/pillow-10.4.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:543f3dc61c18dafb755773efc89aae60d06b6596a63914107f75459cf984164d", size = 3375658, upload-time = "2024-07-01T09:45:25.292Z" }, - { url = "https://files.pythonhosted.org/packages/8a/25/1fc45761955f9359b1169aa75e241551e74ac01a09f487adaaf4c3472d11/pillow-10.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:7928ecbf1ece13956b95d9cbcfc77137652b02763ba384d9ab508099a2eca856", size = 4332075, upload-time = "2024-07-01T09:45:27.94Z" }, - { url = "https://files.pythonhosted.org/packages/5e/dd/425b95d0151e1d6c951f45051112394f130df3da67363b6bc75dc4c27aba/pillow-10.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e4d49b85c4348ea0b31ea63bc75a9f3857869174e2bf17e7aba02945cd218e6f", size = 4444808, upload-time = "2024-07-01T09:45:30.305Z" }, - { url = "https://files.pythonhosted.org/packages/b1/84/9a15cc5726cbbfe7f9f90bfb11f5d028586595907cd093815ca6644932e3/pillow-10.4.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:6c762a5b0997f5659a5ef2266abc1d8851ad7749ad9a6a5506eb23d314e4f46b", size = 4356290, upload-time = "2024-07-01T09:45:32.868Z" }, - { url = "https://files.pythonhosted.org/packages/b5/5b/6651c288b08df3b8c1e2f8c1152201e0b25d240e22ddade0f1e242fc9fa0/pillow-10.4.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:a985e028fc183bf12a77a8bbf36318db4238a3ded7fa9df1b9a133f1cb79f8fc", size = 4525163, upload-time = "2024-07-01T09:45:35.279Z" }, - { url = "https://files.pythonhosted.org/packages/07/8b/34854bf11a83c248505c8cb0fcf8d3d0b459a2246c8809b967963b6b12ae/pillow-10.4.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:812f7342b0eee081eaec84d91423d1b4650bb9828eb53d8511bcef8ce5aecf1e", size = 4463100, upload-time = "2024-07-01T09:45:37.74Z" }, - { url = "https://files.pythonhosted.org/packages/78/63/0632aee4e82476d9cbe5200c0cdf9ba41ee04ed77887432845264d81116d/pillow-10.4.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:ac1452d2fbe4978c2eec89fb5a23b8387aba707ac72810d9490118817d9c0b46", size = 4592880, upload-time = "2024-07-01T09:45:39.89Z" }, - { url = "https://files.pythonhosted.org/packages/df/56/b8663d7520671b4398b9d97e1ed9f583d4afcbefbda3c6188325e8c297bd/pillow-10.4.0-cp310-cp310-win32.whl", hash = "sha256:bcd5e41a859bf2e84fdc42f4edb7d9aba0a13d29a2abadccafad99de3feff984", size = 2235218, upload-time = "2024-07-01T09:45:42.771Z" }, - { url = "https://files.pythonhosted.org/packages/f4/72/0203e94a91ddb4a9d5238434ae6c1ca10e610e8487036132ea9bf806ca2a/pillow-10.4.0-cp310-cp310-win_amd64.whl", hash = "sha256:ecd85a8d3e79cd7158dec1c9e5808e821feea088e2f69a974db5edf84dc53141", size = 2554487, upload-time = "2024-07-01T09:45:45.176Z" }, - { url = "https://files.pythonhosted.org/packages/bd/52/7e7e93d7a6e4290543f17dc6f7d3af4bd0b3dd9926e2e8a35ac2282bc5f4/pillow-10.4.0-cp310-cp310-win_arm64.whl", hash = "sha256:ff337c552345e95702c5fde3158acb0625111017d0e5f24bf3acdb9cc16b90d1", size = 2243219, upload-time = "2024-07-01T09:45:47.274Z" }, - { url = "https://files.pythonhosted.org/packages/a7/62/c9449f9c3043c37f73e7487ec4ef0c03eb9c9afc91a92b977a67b3c0bbc5/pillow-10.4.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:0a9ec697746f268507404647e531e92889890a087e03681a3606d9b920fbee3c", size = 3509265, upload-time = "2024-07-01T09:45:49.812Z" }, - { url = "https://files.pythonhosted.org/packages/f4/5f/491dafc7bbf5a3cc1845dc0430872e8096eb9e2b6f8161509d124594ec2d/pillow-10.4.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:dfe91cb65544a1321e631e696759491ae04a2ea11d36715eca01ce07284738be", size = 3375655, upload-time = "2024-07-01T09:45:52.462Z" }, - { url = "https://files.pythonhosted.org/packages/73/d5/c4011a76f4207a3c151134cd22a1415741e42fa5ddecec7c0182887deb3d/pillow-10.4.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5dc6761a6efc781e6a1544206f22c80c3af4c8cf461206d46a1e6006e4429ff3", size = 4340304, upload-time = "2024-07-01T09:45:55.006Z" }, - { url = "https://files.pythonhosted.org/packages/ac/10/c67e20445a707f7a610699bba4fe050583b688d8cd2d202572b257f46600/pillow-10.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:5e84b6cc6a4a3d76c153a6b19270b3526a5a8ed6b09501d3af891daa2a9de7d6", size = 4452804, upload-time = "2024-07-01T09:45:58.437Z" }, - { url = "https://files.pythonhosted.org/packages/a9/83/6523837906d1da2b269dee787e31df3b0acb12e3d08f024965a3e7f64665/pillow-10.4.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:bbc527b519bd3aa9d7f429d152fea69f9ad37c95f0b02aebddff592688998abe", size = 4365126, upload-time = "2024-07-01T09:46:00.713Z" }, - { url = "https://files.pythonhosted.org/packages/ba/e5/8c68ff608a4203085158cff5cc2a3c534ec384536d9438c405ed6370d080/pillow-10.4.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:76a911dfe51a36041f2e756b00f96ed84677cdeb75d25c767f296c1c1eda1319", size = 4533541, upload-time = "2024-07-01T09:46:03.235Z" }, - { url = "https://files.pythonhosted.org/packages/f4/7c/01b8dbdca5bc6785573f4cee96e2358b0918b7b2c7b60d8b6f3abf87a070/pillow-10.4.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:59291fb29317122398786c2d44427bbd1a6d7ff54017075b22be9d21aa59bd8d", size = 4471616, upload-time = "2024-07-01T09:46:05.356Z" }, - { url = "https://files.pythonhosted.org/packages/c8/57/2899b82394a35a0fbfd352e290945440e3b3785655a03365c0ca8279f351/pillow-10.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:416d3a5d0e8cfe4f27f574362435bc9bae57f679a7158e0096ad2beb427b8696", size = 4600802, upload-time = "2024-07-01T09:46:08.145Z" }, - { url = "https://files.pythonhosted.org/packages/4d/d7/a44f193d4c26e58ee5d2d9db3d4854b2cfb5b5e08d360a5e03fe987c0086/pillow-10.4.0-cp311-cp311-win32.whl", hash = "sha256:7086cc1d5eebb91ad24ded9f58bec6c688e9f0ed7eb3dbbf1e4800280a896496", size = 2235213, upload-time = "2024-07-01T09:46:10.211Z" }, - { url = "https://files.pythonhosted.org/packages/c1/d0/5866318eec2b801cdb8c82abf190c8343d8a1cd8bf5a0c17444a6f268291/pillow-10.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:cbed61494057c0f83b83eb3a310f0bf774b09513307c434d4366ed64f4128a91", size = 2554498, upload-time = "2024-07-01T09:46:12.685Z" }, - { url = "https://files.pythonhosted.org/packages/d4/c8/310ac16ac2b97e902d9eb438688de0d961660a87703ad1561fd3dfbd2aa0/pillow-10.4.0-cp311-cp311-win_arm64.whl", hash = "sha256:f5f0c3e969c8f12dd2bb7e0b15d5c468b51e5017e01e2e867335c81903046a22", size = 2243219, upload-time = "2024-07-01T09:46:14.83Z" }, - { url = "https://files.pythonhosted.org/packages/05/cb/0353013dc30c02a8be34eb91d25e4e4cf594b59e5a55ea1128fde1e5f8ea/pillow-10.4.0-cp312-cp312-macosx_10_10_x86_64.whl", hash = "sha256:673655af3eadf4df6b5457033f086e90299fdd7a47983a13827acf7459c15d94", size = 3509350, upload-time = "2024-07-01T09:46:17.177Z" }, - { url = "https://files.pythonhosted.org/packages/e7/cf/5c558a0f247e0bf9cec92bff9b46ae6474dd736f6d906315e60e4075f737/pillow-10.4.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:866b6942a92f56300012f5fbac71f2d610312ee65e22f1aa2609e491284e5597", size = 3374980, upload-time = "2024-07-01T09:46:19.169Z" }, - { url = "https://files.pythonhosted.org/packages/84/48/6e394b86369a4eb68b8a1382c78dc092245af517385c086c5094e3b34428/pillow-10.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:29dbdc4207642ea6aad70fbde1a9338753d33fb23ed6956e706936706f52dd80", size = 4343799, upload-time = "2024-07-01T09:46:21.883Z" }, - { url = "https://files.pythonhosted.org/packages/3b/f3/a8c6c11fa84b59b9df0cd5694492da8c039a24cd159f0f6918690105c3be/pillow-10.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:bf2342ac639c4cf38799a44950bbc2dfcb685f052b9e262f446482afaf4bffca", size = 4459973, upload-time = "2024-07-01T09:46:24.321Z" }, - { url = "https://files.pythonhosted.org/packages/7d/1b/c14b4197b80150fb64453585247e6fb2e1d93761fa0fa9cf63b102fde822/pillow-10.4.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:f5b92f4d70791b4a67157321c4e8225d60b119c5cc9aee8ecf153aace4aad4ef", size = 4370054, upload-time = "2024-07-01T09:46:26.825Z" }, - { url = "https://files.pythonhosted.org/packages/55/77/40daddf677897a923d5d33329acd52a2144d54a9644f2a5422c028c6bf2d/pillow-10.4.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:86dcb5a1eb778d8b25659d5e4341269e8590ad6b4e8b44d9f4b07f8d136c414a", size = 4539484, upload-time = "2024-07-01T09:46:29.355Z" }, - { url = "https://files.pythonhosted.org/packages/40/54/90de3e4256b1207300fb2b1d7168dd912a2fb4b2401e439ba23c2b2cabde/pillow-10.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:780c072c2e11c9b2c7ca37f9a2ee8ba66f44367ac3e5c7832afcfe5104fd6d1b", size = 4477375, upload-time = "2024-07-01T09:46:31.756Z" }, - { url = "https://files.pythonhosted.org/packages/13/24/1bfba52f44193860918ff7c93d03d95e3f8748ca1de3ceaf11157a14cf16/pillow-10.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:37fb69d905be665f68f28a8bba3c6d3223c8efe1edf14cc4cfa06c241f8c81d9", size = 4608773, upload-time = "2024-07-01T09:46:33.73Z" }, - { url = "https://files.pythonhosted.org/packages/55/04/5e6de6e6120451ec0c24516c41dbaf80cce1b6451f96561235ef2429da2e/pillow-10.4.0-cp312-cp312-win32.whl", hash = "sha256:7dfecdbad5c301d7b5bde160150b4db4c659cee2b69589705b6f8a0c509d9f42", size = 2235690, upload-time = "2024-07-01T09:46:36.587Z" }, - { url = "https://files.pythonhosted.org/packages/74/0a/d4ce3c44bca8635bd29a2eab5aa181b654a734a29b263ca8efe013beea98/pillow-10.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:1d846aea995ad352d4bdcc847535bd56e0fd88d36829d2c90be880ef1ee4668a", size = 2554951, upload-time = "2024-07-01T09:46:38.777Z" }, - { url = "https://files.pythonhosted.org/packages/b5/ca/184349ee40f2e92439be9b3502ae6cfc43ac4b50bc4fc6b3de7957563894/pillow-10.4.0-cp312-cp312-win_arm64.whl", hash = "sha256:e553cad5179a66ba15bb18b353a19020e73a7921296a7979c4a2b7f6a5cd57f9", size = 2243427, upload-time = "2024-07-01T09:46:43.15Z" }, - { url = "https://files.pythonhosted.org/packages/c3/00/706cebe7c2c12a6318aabe5d354836f54adff7156fd9e1bd6c89f4ba0e98/pillow-10.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:8bc1a764ed8c957a2e9cacf97c8b2b053b70307cf2996aafd70e91a082e70df3", size = 3525685, upload-time = "2024-07-01T09:46:45.194Z" }, - { url = "https://files.pythonhosted.org/packages/cf/76/f658cbfa49405e5ecbfb9ba42d07074ad9792031267e782d409fd8fe7c69/pillow-10.4.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6209bb41dc692ddfee4942517c19ee81b86c864b626dbfca272ec0f7cff5d9fb", size = 3374883, upload-time = "2024-07-01T09:46:47.331Z" }, - { url = "https://files.pythonhosted.org/packages/46/2b/99c28c4379a85e65378211971c0b430d9c7234b1ec4d59b2668f6299e011/pillow-10.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:bee197b30783295d2eb680b311af15a20a8b24024a19c3a26431ff83eb8d1f70", size = 4339837, upload-time = "2024-07-01T09:46:49.647Z" }, - { url = "https://files.pythonhosted.org/packages/f1/74/b1ec314f624c0c43711fdf0d8076f82d9d802afd58f1d62c2a86878e8615/pillow-10.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:1ef61f5dd14c300786318482456481463b9d6b91ebe5ef12f405afbba77ed0be", size = 4455562, upload-time = "2024-07-01T09:46:51.811Z" }, - { url = "https://files.pythonhosted.org/packages/4a/2a/4b04157cb7b9c74372fa867096a1607e6fedad93a44deeff553ccd307868/pillow-10.4.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:297e388da6e248c98bc4a02e018966af0c5f92dfacf5a5ca22fa01cb3179bca0", size = 4366761, upload-time = "2024-07-01T09:46:53.961Z" }, - { url = "https://files.pythonhosted.org/packages/ac/7b/8f1d815c1a6a268fe90481232c98dd0e5fa8c75e341a75f060037bd5ceae/pillow-10.4.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:e4db64794ccdf6cb83a59d73405f63adbe2a1887012e308828596100a0b2f6cc", size = 4536767, upload-time = "2024-07-01T09:46:56.664Z" }, - { url = "https://files.pythonhosted.org/packages/e5/77/05fa64d1f45d12c22c314e7b97398ffb28ef2813a485465017b7978b3ce7/pillow-10.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:bd2880a07482090a3bcb01f4265f1936a903d70bc740bfcb1fd4e8a2ffe5cf5a", size = 4477989, upload-time = "2024-07-01T09:46:58.977Z" }, - { url = "https://files.pythonhosted.org/packages/12/63/b0397cfc2caae05c3fb2f4ed1b4fc4fc878f0243510a7a6034ca59726494/pillow-10.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4b35b21b819ac1dbd1233317adeecd63495f6babf21b7b2512d244ff6c6ce309", size = 4610255, upload-time = "2024-07-01T09:47:01.189Z" }, - { url = "https://files.pythonhosted.org/packages/7b/f9/cfaa5082ca9bc4a6de66ffe1c12c2d90bf09c309a5f52b27759a596900e7/pillow-10.4.0-cp313-cp313-win32.whl", hash = "sha256:551d3fd6e9dc15e4c1eb6fc4ba2b39c0c7933fa113b220057a34f4bb3268a060", size = 2235603, upload-time = "2024-07-01T09:47:03.918Z" }, - { url = "https://files.pythonhosted.org/packages/01/6a/30ff0eef6e0c0e71e55ded56a38d4859bf9d3634a94a88743897b5f96936/pillow-10.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:030abdbe43ee02e0de642aee345efa443740aa4d828bfe8e2eb11922ea6a21ea", size = 2554972, upload-time = "2024-07-01T09:47:06.152Z" }, - { url = "https://files.pythonhosted.org/packages/48/2c/2e0a52890f269435eee38b21c8218e102c621fe8d8df8b9dd06fabf879ba/pillow-10.4.0-cp313-cp313-win_arm64.whl", hash = "sha256:5b001114dd152cfd6b23befeb28d7aee43553e2402c9f159807bf55f33af8a8d", size = 2243375, upload-time = "2024-07-01T09:47:09.065Z" }, - { url = "https://files.pythonhosted.org/packages/38/30/095d4f55f3a053392f75e2eae45eba3228452783bab3d9a920b951ac495c/pillow-10.4.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:5b4815f2e65b30f5fbae9dfffa8636d992d49705723fe86a3661806e069352d4", size = 3493889, upload-time = "2024-07-01T09:48:04.815Z" }, - { url = "https://files.pythonhosted.org/packages/f3/e8/4ff79788803a5fcd5dc35efdc9386af153569853767bff74540725b45863/pillow-10.4.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:8f0aef4ef59694b12cadee839e2ba6afeab89c0f39a3adc02ed51d109117b8da", size = 3346160, upload-time = "2024-07-01T09:48:07.206Z" }, - { url = "https://files.pythonhosted.org/packages/d7/ac/4184edd511b14f760c73f5bb8a5d6fd85c591c8aff7c2229677a355c4179/pillow-10.4.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9f4727572e2918acaa9077c919cbbeb73bd2b3ebcfe033b72f858fc9fbef0026", size = 3435020, upload-time = "2024-07-01T09:48:09.66Z" }, - { url = "https://files.pythonhosted.org/packages/da/21/1749cd09160149c0a246a81d646e05f35041619ce76f6493d6a96e8d1103/pillow-10.4.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:ff25afb18123cea58a591ea0244b92eb1e61a1fd497bf6d6384f09bc3262ec3e", size = 3490539, upload-time = "2024-07-01T09:48:12.529Z" }, - { url = "https://files.pythonhosted.org/packages/b6/f5/f71fe1888b96083b3f6dfa0709101f61fc9e972c0c8d04e9d93ccef2a045/pillow-10.4.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:dc3e2db6ba09ffd7d02ae9141cfa0ae23393ee7687248d46a7507b75d610f4f5", size = 3476125, upload-time = "2024-07-01T09:48:14.891Z" }, - { url = "https://files.pythonhosted.org/packages/96/b9/c0362c54290a31866c3526848583a2f45a535aa9d725fd31e25d318c805f/pillow-10.4.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:02a2be69f9c9b8c1e97cf2713e789d4e398c751ecfd9967c18d0ce304efbf885", size = 3579373, upload-time = "2024-07-01T09:48:17.601Z" }, - { url = "https://files.pythonhosted.org/packages/52/3b/ce7a01026a7cf46e5452afa86f97a5e88ca97f562cafa76570178ab56d8d/pillow-10.4.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:0755ffd4a0c6f267cccbae2e9903d95477ca2f77c4fcf3a3a09570001856c8a5", size = 2554661, upload-time = "2024-07-01T09:48:20.293Z" }, +version = "11.1.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f3/af/c097e544e7bd278333db77933e535098c259609c4eb3b85381109602fb5b/pillow-11.1.0.tar.gz", hash = "sha256:368da70808b36d73b4b390a8ffac11069f8a5c85f29eff1f1b01bcf3ef5b2a20", size = 46742715, upload-time = "2025-01-02T08:13:58.407Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/50/1c/2dcea34ac3d7bc96a1fd1bd0a6e06a57c67167fec2cff8d95d88229a8817/pillow-11.1.0-cp310-cp310-macosx_10_10_x86_64.whl", hash = "sha256:e1abe69aca89514737465752b4bcaf8016de61b3be1397a8fc260ba33321b3a8", size = 3229983, upload-time = "2025-01-02T08:10:16.008Z" }, + { url = "https://files.pythonhosted.org/packages/14/ca/6bec3df25e4c88432681de94a3531cc738bd85dea6c7aa6ab6f81ad8bd11/pillow-11.1.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c640e5a06869c75994624551f45e5506e4256562ead981cce820d5ab39ae2192", size = 3101831, upload-time = "2025-01-02T08:10:18.774Z" }, + { url = "https://files.pythonhosted.org/packages/d4/2c/668e18e5521e46eb9667b09e501d8e07049eb5bfe39d56be0724a43117e6/pillow-11.1.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a07dba04c5e22824816b2615ad7a7484432d7f540e6fa86af60d2de57b0fcee2", size = 4314074, upload-time = "2025-01-02T08:10:21.114Z" }, + { url = "https://files.pythonhosted.org/packages/02/80/79f99b714f0fc25f6a8499ecfd1f810df12aec170ea1e32a4f75746051ce/pillow-11.1.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e267b0ed063341f3e60acd25c05200df4193e15a4a5807075cd71225a2386e26", size = 4394933, upload-time = "2025-01-02T08:10:23.982Z" }, + { url = "https://files.pythonhosted.org/packages/81/aa/8d4ad25dc11fd10a2001d5b8a80fdc0e564ac33b293bdfe04ed387e0fd95/pillow-11.1.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:bd165131fd51697e22421d0e467997ad31621b74bfc0b75956608cb2906dda07", size = 4353349, upload-time = "2025-01-02T08:10:25.887Z" }, + { url = "https://files.pythonhosted.org/packages/84/7a/cd0c3eaf4a28cb2a74bdd19129f7726277a7f30c4f8424cd27a62987d864/pillow-11.1.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:abc56501c3fd148d60659aae0af6ddc149660469082859fa7b066a298bde9482", size = 4476532, upload-time = "2025-01-02T08:10:28.129Z" }, + { url = "https://files.pythonhosted.org/packages/8f/8b/a907fdd3ae8f01c7670dfb1499c53c28e217c338b47a813af8d815e7ce97/pillow-11.1.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:54ce1c9a16a9561b6d6d8cb30089ab1e5eb66918cb47d457bd996ef34182922e", size = 4279789, upload-time = "2025-01-02T08:10:32.976Z" }, + { url = "https://files.pythonhosted.org/packages/6f/9a/9f139d9e8cccd661c3efbf6898967a9a337eb2e9be2b454ba0a09533100d/pillow-11.1.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:73ddde795ee9b06257dac5ad42fcb07f3b9b813f8c1f7f870f402f4dc54b5269", size = 4413131, upload-time = "2025-01-02T08:10:36.912Z" }, + { url = "https://files.pythonhosted.org/packages/a8/68/0d8d461f42a3f37432203c8e6df94da10ac8081b6d35af1c203bf3111088/pillow-11.1.0-cp310-cp310-win32.whl", hash = "sha256:3a5fe20a7b66e8135d7fd617b13272626a28278d0e578c98720d9ba4b2439d49", size = 2291213, upload-time = "2025-01-02T08:10:40.186Z" }, + { url = "https://files.pythonhosted.org/packages/14/81/d0dff759a74ba87715509af9f6cb21fa21d93b02b3316ed43bda83664db9/pillow-11.1.0-cp310-cp310-win_amd64.whl", hash = "sha256:b6123aa4a59d75f06e9dd3dac5bf8bc9aa383121bb3dd9a7a612e05eabc9961a", size = 2625725, upload-time = "2025-01-02T08:10:42.404Z" }, + { url = "https://files.pythonhosted.org/packages/ce/1f/8d50c096a1d58ef0584ddc37e6f602828515219e9d2428e14ce50f5ecad1/pillow-11.1.0-cp310-cp310-win_arm64.whl", hash = "sha256:a76da0a31da6fcae4210aa94fd779c65c75786bc9af06289cd1c184451ef7a65", size = 2375213, upload-time = "2025-01-02T08:10:44.173Z" }, + { url = "https://files.pythonhosted.org/packages/dd/d6/2000bfd8d5414fb70cbbe52c8332f2283ff30ed66a9cde42716c8ecbe22c/pillow-11.1.0-cp311-cp311-macosx_10_10_x86_64.whl", hash = "sha256:e06695e0326d05b06833b40b7ef477e475d0b1ba3a6d27da1bb48c23209bf457", size = 3229968, upload-time = "2025-01-02T08:10:48.172Z" }, + { url = "https://files.pythonhosted.org/packages/d9/45/3fe487010dd9ce0a06adf9b8ff4f273cc0a44536e234b0fad3532a42c15b/pillow-11.1.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:96f82000e12f23e4f29346e42702b6ed9a2f2fea34a740dd5ffffcc8c539eb35", size = 3101806, upload-time = "2025-01-02T08:10:50.981Z" }, + { url = "https://files.pythonhosted.org/packages/e3/72/776b3629c47d9d5f1c160113158a7a7ad177688d3a1159cd3b62ded5a33a/pillow-11.1.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a3cd561ded2cf2bbae44d4605837221b987c216cff94f49dfeed63488bb228d2", size = 4322283, upload-time = "2025-01-02T08:10:54.724Z" }, + { url = "https://files.pythonhosted.org/packages/e4/c2/e25199e7e4e71d64eeb869f5b72c7ddec70e0a87926398785ab944d92375/pillow-11.1.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:f189805c8be5ca5add39e6f899e6ce2ed824e65fb45f3c28cb2841911da19070", size = 4402945, upload-time = "2025-01-02T08:10:57.376Z" }, + { url = "https://files.pythonhosted.org/packages/c1/ed/51d6136c9d5911f78632b1b86c45241c712c5a80ed7fa7f9120a5dff1eba/pillow-11.1.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:dd0052e9db3474df30433f83a71b9b23bd9e4ef1de13d92df21a52c0303b8ab6", size = 4361228, upload-time = "2025-01-02T08:11:02.374Z" }, + { url = "https://files.pythonhosted.org/packages/48/a4/fbfe9d5581d7b111b28f1d8c2762dee92e9821bb209af9fa83c940e507a0/pillow-11.1.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:837060a8599b8f5d402e97197d4924f05a2e0d68756998345c829c33186217b1", size = 4484021, upload-time = "2025-01-02T08:11:04.431Z" }, + { url = "https://files.pythonhosted.org/packages/39/db/0b3c1a5018117f3c1d4df671fb8e47d08937f27519e8614bbe86153b65a5/pillow-11.1.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:aa8dd43daa836b9a8128dbe7d923423e5ad86f50a7a14dc688194b7be5c0dea2", size = 4287449, upload-time = "2025-01-02T08:11:07.412Z" }, + { url = "https://files.pythonhosted.org/packages/d9/58/bc128da7fea8c89fc85e09f773c4901e95b5936000e6f303222490c052f3/pillow-11.1.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:0a2f91f8a8b367e7a57c6e91cd25af510168091fb89ec5146003e424e1558a96", size = 4419972, upload-time = "2025-01-02T08:11:09.508Z" }, + { url = "https://files.pythonhosted.org/packages/5f/bb/58f34379bde9fe197f51841c5bbe8830c28bbb6d3801f16a83b8f2ad37df/pillow-11.1.0-cp311-cp311-win32.whl", hash = "sha256:c12fc111ef090845de2bb15009372175d76ac99969bdf31e2ce9b42e4b8cd88f", size = 2291201, upload-time = "2025-01-02T08:11:13.056Z" }, + { url = "https://files.pythonhosted.org/packages/3a/c6/fce9255272bcf0c39e15abd2f8fd8429a954cf344469eaceb9d0d1366913/pillow-11.1.0-cp311-cp311-win_amd64.whl", hash = "sha256:fbd43429d0d7ed6533b25fc993861b8fd512c42d04514a0dd6337fb3ccf22761", size = 2625686, upload-time = "2025-01-02T08:11:16.547Z" }, + { url = "https://files.pythonhosted.org/packages/c8/52/8ba066d569d932365509054859f74f2a9abee273edcef5cd75e4bc3e831e/pillow-11.1.0-cp311-cp311-win_arm64.whl", hash = "sha256:f7955ecf5609dee9442cbface754f2c6e541d9e6eda87fad7f7a989b0bdb9d71", size = 2375194, upload-time = "2025-01-02T08:11:19.897Z" }, + { url = "https://files.pythonhosted.org/packages/95/20/9ce6ed62c91c073fcaa23d216e68289e19d95fb8188b9fb7a63d36771db8/pillow-11.1.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:2062ffb1d36544d42fcaa277b069c88b01bb7298f4efa06731a7fd6cc290b81a", size = 3226818, upload-time = "2025-01-02T08:11:22.518Z" }, + { url = "https://files.pythonhosted.org/packages/b9/d8/f6004d98579a2596c098d1e30d10b248798cceff82d2b77aa914875bfea1/pillow-11.1.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:a85b653980faad27e88b141348707ceeef8a1186f75ecc600c395dcac19f385b", size = 3101662, upload-time = "2025-01-02T08:11:25.19Z" }, + { url = "https://files.pythonhosted.org/packages/08/d9/892e705f90051c7a2574d9f24579c9e100c828700d78a63239676f960b74/pillow-11.1.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:9409c080586d1f683df3f184f20e36fb647f2e0bc3988094d4fd8c9f4eb1b3b3", size = 4329317, upload-time = "2025-01-02T08:11:30.371Z" }, + { url = "https://files.pythonhosted.org/packages/8c/aa/7f29711f26680eab0bcd3ecdd6d23ed6bce180d82e3f6380fb7ae35fcf3b/pillow-11.1.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:7fdadc077553621911f27ce206ffcbec7d3f8d7b50e0da39f10997e8e2bb7f6a", size = 4412999, upload-time = "2025-01-02T08:11:33.499Z" }, + { url = "https://files.pythonhosted.org/packages/c8/c4/8f0fe3b9e0f7196f6d0bbb151f9fba323d72a41da068610c4c960b16632a/pillow-11.1.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:93a18841d09bcdd774dcdc308e4537e1f867b3dec059c131fde0327899734aa1", size = 4368819, upload-time = "2025-01-02T08:11:37.304Z" }, + { url = "https://files.pythonhosted.org/packages/38/0d/84200ed6a871ce386ddc82904bfadc0c6b28b0c0ec78176871a4679e40b3/pillow-11.1.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:9aa9aeddeed452b2f616ff5507459e7bab436916ccb10961c4a382cd3e03f47f", size = 4496081, upload-time = "2025-01-02T08:11:39.598Z" }, + { url = "https://files.pythonhosted.org/packages/84/9c/9bcd66f714d7e25b64118e3952d52841a4babc6d97b6d28e2261c52045d4/pillow-11.1.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3cdcdb0b896e981678eee140d882b70092dac83ac1cdf6b3a60e2216a73f2b91", size = 4296513, upload-time = "2025-01-02T08:11:43.083Z" }, + { url = "https://files.pythonhosted.org/packages/db/61/ada2a226e22da011b45f7104c95ebda1b63dcbb0c378ad0f7c2a710f8fd2/pillow-11.1.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:36ba10b9cb413e7c7dfa3e189aba252deee0602c86c309799da5a74009ac7a1c", size = 4431298, upload-time = "2025-01-02T08:11:46.626Z" }, + { url = "https://files.pythonhosted.org/packages/e7/c4/fc6e86750523f367923522014b821c11ebc5ad402e659d8c9d09b3c9d70c/pillow-11.1.0-cp312-cp312-win32.whl", hash = "sha256:cfd5cd998c2e36a862d0e27b2df63237e67273f2fc78f47445b14e73a810e7e6", size = 2291630, upload-time = "2025-01-02T08:11:49.401Z" }, + { url = "https://files.pythonhosted.org/packages/08/5c/2104299949b9d504baf3f4d35f73dbd14ef31bbd1ddc2c1b66a5b7dfda44/pillow-11.1.0-cp312-cp312-win_amd64.whl", hash = "sha256:a697cd8ba0383bba3d2d3ada02b34ed268cb548b369943cd349007730c92bddf", size = 2626369, upload-time = "2025-01-02T08:11:52.02Z" }, + { url = "https://files.pythonhosted.org/packages/37/f3/9b18362206b244167c958984b57c7f70a0289bfb59a530dd8af5f699b910/pillow-11.1.0-cp312-cp312-win_arm64.whl", hash = "sha256:4dd43a78897793f60766563969442020e90eb7847463eca901e41ba186a7d4a5", size = 2375240, upload-time = "2025-01-02T08:11:56.193Z" }, + { url = "https://files.pythonhosted.org/packages/b3/31/9ca79cafdce364fd5c980cd3416c20ce1bebd235b470d262f9d24d810184/pillow-11.1.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ae98e14432d458fc3de11a77ccb3ae65ddce70f730e7c76140653048c71bfcbc", size = 3226640, upload-time = "2025-01-02T08:11:58.329Z" }, + { url = "https://files.pythonhosted.org/packages/ac/0f/ff07ad45a1f172a497aa393b13a9d81a32e1477ef0e869d030e3c1532521/pillow-11.1.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:cc1331b6d5a6e144aeb5e626f4375f5b7ae9934ba620c0ac6b3e43d5e683a0f0", size = 3101437, upload-time = "2025-01-02T08:12:01.797Z" }, + { url = "https://files.pythonhosted.org/packages/08/2f/9906fca87a68d29ec4530be1f893149e0cb64a86d1f9f70a7cfcdfe8ae44/pillow-11.1.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:758e9d4ef15d3560214cddbc97b8ef3ef86ce04d62ddac17ad39ba87e89bd3b1", size = 4326605, upload-time = "2025-01-02T08:12:05.224Z" }, + { url = "https://files.pythonhosted.org/packages/b0/0f/f3547ee15b145bc5c8b336401b2d4c9d9da67da9dcb572d7c0d4103d2c69/pillow-11.1.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b523466b1a31d0dcef7c5be1f20b942919b62fd6e9a9be199d035509cbefc0ec", size = 4411173, upload-time = "2025-01-02T08:12:08.281Z" }, + { url = "https://files.pythonhosted.org/packages/b1/df/bf8176aa5db515c5de584c5e00df9bab0713548fd780c82a86cba2c2fedb/pillow-11.1.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:9044b5e4f7083f209c4e35aa5dd54b1dd5b112b108648f5c902ad586d4f945c5", size = 4369145, upload-time = "2025-01-02T08:12:11.411Z" }, + { url = "https://files.pythonhosted.org/packages/de/7c/7433122d1cfadc740f577cb55526fdc39129a648ac65ce64db2eb7209277/pillow-11.1.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:3764d53e09cdedd91bee65c2527815d315c6b90d7b8b79759cc48d7bf5d4f114", size = 4496340, upload-time = "2025-01-02T08:12:15.29Z" }, + { url = "https://files.pythonhosted.org/packages/25/46/dd94b93ca6bd555588835f2504bd90c00d5438fe131cf01cfa0c5131a19d/pillow-11.1.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:31eba6bbdd27dde97b0174ddf0297d7a9c3a507a8a1480e1e60ef914fe23d352", size = 4296906, upload-time = "2025-01-02T08:12:17.485Z" }, + { url = "https://files.pythonhosted.org/packages/a8/28/2f9d32014dfc7753e586db9add35b8a41b7a3b46540e965cb6d6bc607bd2/pillow-11.1.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b5d658fbd9f0d6eea113aea286b21d3cd4d3fd978157cbf2447a6035916506d3", size = 4431759, upload-time = "2025-01-02T08:12:20.382Z" }, + { url = "https://files.pythonhosted.org/packages/33/48/19c2cbe7403870fbe8b7737d19eb013f46299cdfe4501573367f6396c775/pillow-11.1.0-cp313-cp313-win32.whl", hash = "sha256:f86d3a7a9af5d826744fabf4afd15b9dfef44fe69a98541f666f66fbb8d3fef9", size = 2291657, upload-time = "2025-01-02T08:12:23.922Z" }, + { url = "https://files.pythonhosted.org/packages/3b/ad/285c556747d34c399f332ba7c1a595ba245796ef3e22eae190f5364bb62b/pillow-11.1.0-cp313-cp313-win_amd64.whl", hash = "sha256:593c5fd6be85da83656b93ffcccc2312d2d149d251e98588b14fbc288fd8909c", size = 2626304, upload-time = "2025-01-02T08:12:28.069Z" }, + { url = "https://files.pythonhosted.org/packages/e5/7b/ef35a71163bf36db06e9c8729608f78dedf032fc8313d19bd4be5c2588f3/pillow-11.1.0-cp313-cp313-win_arm64.whl", hash = "sha256:11633d58b6ee5733bde153a8dafd25e505ea3d32e261accd388827ee987baf65", size = 2375117, upload-time = "2025-01-02T08:12:30.064Z" }, + { url = "https://files.pythonhosted.org/packages/79/30/77f54228401e84d6791354888549b45824ab0ffde659bafa67956303a09f/pillow-11.1.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:70ca5ef3b3b1c4a0812b5c63c57c23b63e53bc38e758b37a951e5bc466449861", size = 3230060, upload-time = "2025-01-02T08:12:32.362Z" }, + { url = "https://files.pythonhosted.org/packages/ce/b1/56723b74b07dd64c1010fee011951ea9c35a43d8020acd03111f14298225/pillow-11.1.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:8000376f139d4d38d6851eb149b321a52bb8893a88dae8ee7d95840431977081", size = 3106192, upload-time = "2025-01-02T08:12:34.361Z" }, + { url = "https://files.pythonhosted.org/packages/e1/cd/7bf7180e08f80a4dcc6b4c3a0aa9e0b0ae57168562726a05dc8aa8fa66b0/pillow-11.1.0-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9ee85f0696a17dd28fbcfceb59f9510aa71934b483d1f5601d1030c3c8304f3c", size = 4446805, upload-time = "2025-01-02T08:12:36.99Z" }, + { url = "https://files.pythonhosted.org/packages/97/42/87c856ea30c8ed97e8efbe672b58c8304dee0573f8c7cab62ae9e31db6ae/pillow-11.1.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:dd0e081319328928531df7a0e63621caf67652c8464303fd102141b785ef9547", size = 4530623, upload-time = "2025-01-02T08:12:41.912Z" }, + { url = "https://files.pythonhosted.org/packages/ff/41/026879e90c84a88e33fb00cc6bd915ac2743c67e87a18f80270dfe3c2041/pillow-11.1.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:e63e4e5081de46517099dc30abe418122f54531a6ae2ebc8680bcd7096860eab", size = 4465191, upload-time = "2025-01-02T08:12:45.186Z" }, + { url = "https://files.pythonhosted.org/packages/e5/fb/a7960e838bc5df57a2ce23183bfd2290d97c33028b96bde332a9057834d3/pillow-11.1.0-cp313-cp313t-win32.whl", hash = "sha256:dda60aa465b861324e65a78c9f5cf0f4bc713e4309f83bc387be158b077963d9", size = 2295494, upload-time = "2025-01-02T08:12:47.098Z" }, + { url = "https://files.pythonhosted.org/packages/d7/6c/6ec83ee2f6f0fda8d4cf89045c6be4b0373ebfc363ba8538f8c999f63fcd/pillow-11.1.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ad5db5781c774ab9a9b2c4302bbf0c1014960a0a7be63278d13ae6fdf88126fe", size = 2631595, upload-time = "2025-01-02T08:12:50.47Z" }, + { url = "https://files.pythonhosted.org/packages/cf/6c/41c21c6c8af92b9fea313aa47c75de49e2f9a467964ee33eb0135d47eb64/pillow-11.1.0-cp313-cp313t-win_arm64.whl", hash = "sha256:67cd427c68926108778a9005f2a04adbd5e67c442ed21d95389fe1d595458756", size = 2377651, upload-time = "2025-01-02T08:12:53.356Z" }, + { url = "https://files.pythonhosted.org/packages/fa/c5/389961578fb677b8b3244fcd934f720ed25a148b9a5cc81c91bdf59d8588/pillow-11.1.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:8c730dc3a83e5ac137fbc92dfcfe1511ce3b2b5d7578315b63dbbb76f7f51d90", size = 3198345, upload-time = "2025-01-02T08:13:34.091Z" }, + { url = "https://files.pythonhosted.org/packages/c4/fa/803c0e50ffee74d4b965229e816af55276eac1d5806712de86f9371858fd/pillow-11.1.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:7d33d2fae0e8b170b6a6c57400e077412240f6f5bb2a342cf1ee512a787942bb", size = 3072938, upload-time = "2025-01-02T08:13:37.272Z" }, + { url = "https://files.pythonhosted.org/packages/dc/67/2a3a5f8012b5d8c63fe53958ba906c1b1d0482ebed5618057ef4d22f8076/pillow-11.1.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a8d65b38173085f24bc07f8b6c505cbb7418009fa1a1fcb111b1f4961814a442", size = 3400049, upload-time = "2025-01-02T08:13:41.565Z" }, + { url = "https://files.pythonhosted.org/packages/e5/a0/514f0d317446c98c478d1872497eb92e7cde67003fed74f696441e647446/pillow-11.1.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:015c6e863faa4779251436db398ae75051469f7c903b043a48f078e437656f83", size = 3422431, upload-time = "2025-01-02T08:13:43.609Z" }, + { url = "https://files.pythonhosted.org/packages/cd/00/20f40a935514037b7d3f87adfc87d2c538430ea625b63b3af8c3f5578e72/pillow-11.1.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:d44ff19eea13ae4acdaaab0179fa68c0c6f2f45d66a4d8ec1eda7d6cecbcc15f", size = 3446208, upload-time = "2025-01-02T08:13:46.817Z" }, + { url = "https://files.pythonhosted.org/packages/28/3c/7de681727963043e093c72e6c3348411b0185eab3263100d4490234ba2f6/pillow-11.1.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:d3d8da4a631471dfaf94c10c85f5277b1f8e42ac42bade1ac67da4b4a7359b73", size = 3509746, upload-time = "2025-01-02T08:13:50.6Z" }, + { url = "https://files.pythonhosted.org/packages/41/67/936f9814bdd74b2dfd4822f1f7725ab5d8ff4103919a1664eb4874c58b2f/pillow-11.1.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:4637b88343166249fe8aa94e7c4a62a180c4b3898283bb5d3d2fd5fe10d8e4e0", size = 2626353, upload-time = "2025-01-02T08:13:52.725Z" }, ] [[package]] @@ -3737,6 +4378,20 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/54/20/4d324d65cc6d9205fabedc306948156824eb9f0ee1633355a8f7ec5c66bf/pluggy-1.6.0-py3-none-any.whl", hash = "sha256:e920276dd6813095e9377c0bc5566d94c932c33b27a3e3945d8389c374dd4746", size = 20538, upload-time = "2025-05-15T12:30:06.134Z" }, ] +[[package]] +name = "polars-lts-cpu" +version = "1.33.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f4/93/a0c4200a5e0af2eee31ea79330cb1f5f4c58f604cb3de352f654e2010c81/polars_lts_cpu-1.33.1.tar.gz", hash = "sha256:0a5426d95ec9eec937a56d3e7cf7911a4b5486c42f4dbbcc9512aa706039322c", size = 4822741, upload-time = "2025-09-09T08:37:51.491Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/f1/9b/75916636b33724afabe820b0993f60dc243793421d6f680d5fcb531fe170/polars_lts_cpu-1.33.1-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:5db75d1b424bd8aa34c9a670a901592f1931cc94d9fb32bdd428dbaad8c33761", size = 38908638, upload-time = "2025-09-09T08:37:02.258Z" }, + { url = "https://files.pythonhosted.org/packages/81/e2/dc77b81650ba0c631c06f05d8e81faacee87730600fceca372273facf77b/polars_lts_cpu-1.33.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:37cf3a56cf447c69cfb3f9cd0e714d5b0c754705d7b497b9ab86cbf56e36b3e7", size = 35638895, upload-time = "2025-09-09T08:37:07.575Z" }, + { url = "https://files.pythonhosted.org/packages/27/fb/4dcff801d71dfa02ec682d6b32fd0ce5339de48797f663698d5f8348ffe7/polars_lts_cpu-1.33.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:656b530a672fe8fbd4c212b2a8481099e5cef63e84970975619ea7c25faeb833", size = 39585825, upload-time = "2025-09-09T08:37:11.631Z" }, + { url = "https://files.pythonhosted.org/packages/54/31/0474c14dce2c0507bea40069daafb848980ba7c351ad991908e51ac895fb/polars_lts_cpu-1.33.1-cp39-abi3-manylinux_2_24_aarch64.whl", hash = "sha256:64574c784380b37167b3db3a7cfdb9839cd308e89b8818859d2ffb34a9c896b2", size = 36685020, upload-time = "2025-09-09T08:37:15.597Z" }, + { url = "https://files.pythonhosted.org/packages/d4/0a/5ebba9b145388ffbbd09fa84ac3cd7d336b922e34256b1417abf0a1c2fb9/polars_lts_cpu-1.33.1-cp39-abi3-win_amd64.whl", hash = "sha256:6b849e0e1485acb8ac39bf13356d280ea7c924c2b41cd548ea6e4d102d70be77", size = 39191650, upload-time = "2025-09-09T08:37:19.541Z" }, + { url = "https://files.pythonhosted.org/packages/ce/ad/bf3db68d30ac798ca31c80624709a0c03aa890e2e20e5ca987d7e55fcfc2/polars_lts_cpu-1.33.1-cp39-abi3-win_arm64.whl", hash = "sha256:c99ab56b059cee6bcabe9fb89e97f5813be1012a2251bf77f76e15c2d1cba934", size = 35445244, upload-time = "2025-09-09T08:37:22.97Z" }, +] + [[package]] name = "pre-commit" version = "4.5.1" @@ -3917,6 +4572,69 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/05/33/2d74d588408caedd065c2497bdb5ef83ce6082db01289a1e1147f6639802/psutil-5.9.8-cp38-abi3-macosx_11_0_arm64.whl", hash = "sha256:d16bbddf0693323b8c6123dd804100241da461e41d6e332fb0ba6058f630f8c8", size = 249898, upload-time = "2024-01-19T20:47:59.238Z" }, ] +[[package]] +name = "psycopg2-binary" +version = "2.9.11" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/ac/6c/8767aaa597ba424643dc87348c6f1754dd9f48e80fdc1b9f7ca5c3a7c213/psycopg2-binary-2.9.11.tar.gz", hash = "sha256:b6aed9e096bf63f9e75edf2581aa9a7e7186d97ab5c177aa6c87797cd591236c", size = 379620, upload-time = "2025-10-10T11:14:48.041Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/6a/f2/8e377d29c2ecf99f6062d35ea606b036e8800720eccfec5fe3dd672c2b24/psycopg2_binary-2.9.11-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:d6fe6b47d0b42ce1c9f1fa3e35bb365011ca22e39db37074458f27921dca40f2", size = 3756506, upload-time = "2025-10-10T11:10:30.144Z" }, + { url = "https://files.pythonhosted.org/packages/24/cc/dc143ea88e4ec9d386106cac05023b69668bd0be20794c613446eaefafe5/psycopg2_binary-2.9.11-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:a6c0e4262e089516603a09474ee13eabf09cb65c332277e39af68f6233911087", size = 3863943, upload-time = "2025-10-10T11:10:34.586Z" }, + { url = "https://files.pythonhosted.org/packages/8c/df/16848771155e7c419c60afeb24950b8aaa3ab09c0a091ec3ccca26a574d0/psycopg2_binary-2.9.11-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c47676e5b485393f069b4d7a811267d3168ce46f988fa602658b8bb901e9e64d", size = 4410873, upload-time = "2025-10-10T11:10:38.951Z" }, + { url = "https://files.pythonhosted.org/packages/43/79/5ef5f32621abd5a541b89b04231fe959a9b327c874a1d41156041c75494b/psycopg2_binary-2.9.11-cp310-cp310-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:a28d8c01a7b27a1e3265b11250ba7557e5f72b5ee9e5f3a2fa8d2949c29bf5d2", size = 4468016, upload-time = "2025-10-10T11:10:43.319Z" }, + { url = "https://files.pythonhosted.org/packages/f0/9b/d7542d0f7ad78f57385971f426704776d7b310f5219ed58da5d605b1892e/psycopg2_binary-2.9.11-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5f3f2732cf504a1aa9e9609d02f79bea1067d99edf844ab92c247bbca143303b", size = 4164996, upload-time = "2025-10-10T11:10:46.705Z" }, + { url = "https://files.pythonhosted.org/packages/14/ed/e409388b537fa7414330687936917c522f6a77a13474e4238219fcfd9a84/psycopg2_binary-2.9.11-cp310-cp310-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:865f9945ed1b3950d968ec4690ce68c55019d79e4497366d36e090327ce7db14", size = 3981881, upload-time = "2025-10-30T02:54:57.182Z" }, + { url = "https://files.pythonhosted.org/packages/bf/30/50e330e63bb05efc6fa7c1447df3e08954894025ca3dcb396ecc6739bc26/psycopg2_binary-2.9.11-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:91537a8df2bde69b1c1db01d6d944c831ca793952e4f57892600e96cee95f2cd", size = 3650857, upload-time = "2025-10-10T11:10:50.112Z" }, + { url = "https://files.pythonhosted.org/packages/f0/e0/4026e4c12bb49dd028756c5b0bc4c572319f2d8f1c9008e0dad8cc9addd7/psycopg2_binary-2.9.11-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:4dca1f356a67ecb68c81a7bc7809f1569ad9e152ce7fd02c2f2036862ca9f66b", size = 3296063, upload-time = "2025-10-10T11:10:54.089Z" }, + { url = "https://files.pythonhosted.org/packages/2c/34/eb172be293c886fef5299fe5c3fcf180a05478be89856067881007934a7c/psycopg2_binary-2.9.11-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:0da4de5c1ac69d94ed4364b6cbe7190c1a70d325f112ba783d83f8440285f152", size = 3043464, upload-time = "2025-10-30T02:55:02.483Z" }, + { url = "https://files.pythonhosted.org/packages/18/1c/532c5d2cb11986372f14b798a95f2eaafe5779334f6a80589a68b5fcf769/psycopg2_binary-2.9.11-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:37d8412565a7267f7d79e29ab66876e55cb5e8e7b3bbf94f8206f6795f8f7e7e", size = 3345378, upload-time = "2025-10-10T11:11:01.039Z" }, + { url = "https://files.pythonhosted.org/packages/70/e7/de420e1cf16f838e1fa17b1120e83afff374c7c0130d088dba6286fcf8ea/psycopg2_binary-2.9.11-cp310-cp310-win_amd64.whl", hash = "sha256:c665f01ec8ab273a61c62beeb8cce3014c214429ced8a308ca1fc410ecac3a39", size = 2713904, upload-time = "2025-10-10T11:11:04.81Z" }, + { url = "https://files.pythonhosted.org/packages/c7/ae/8d8266f6dd183ab4d48b95b9674034e1b482a3f8619b33a0d86438694577/psycopg2_binary-2.9.11-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0e8480afd62362d0a6a27dd09e4ca2def6fa50ed3a4e7c09165266106b2ffa10", size = 3756452, upload-time = "2025-10-10T11:11:11.583Z" }, + { url = "https://files.pythonhosted.org/packages/4b/34/aa03d327739c1be70e09d01182619aca8ebab5970cd0cfa50dd8b9cec2ac/psycopg2_binary-2.9.11-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:763c93ef1df3da6d1a90f86ea7f3f806dc06b21c198fa87c3c25504abec9404a", size = 3863957, upload-time = "2025-10-10T11:11:16.932Z" }, + { url = "https://files.pythonhosted.org/packages/48/89/3fdb5902bdab8868bbedc1c6e6023a4e08112ceac5db97fc2012060e0c9a/psycopg2_binary-2.9.11-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:2e164359396576a3cc701ba8af4751ae68a07235d7a380c631184a611220d9a4", size = 4410955, upload-time = "2025-10-10T11:11:21.21Z" }, + { url = "https://files.pythonhosted.org/packages/ce/24/e18339c407a13c72b336e0d9013fbbbde77b6fd13e853979019a1269519c/psycopg2_binary-2.9.11-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:d57c9c387660b8893093459738b6abddbb30a7eab058b77b0d0d1c7d521ddfd7", size = 4468007, upload-time = "2025-10-10T11:11:24.831Z" }, + { url = "https://files.pythonhosted.org/packages/91/7e/b8441e831a0f16c159b5381698f9f7f7ed54b77d57bc9c5f99144cc78232/psycopg2_binary-2.9.11-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:2c226ef95eb2250974bf6fa7a842082b31f68385c4f3268370e3f3870e7859ee", size = 4165012, upload-time = "2025-10-10T11:11:29.51Z" }, + { url = "https://files.pythonhosted.org/packages/0d/61/4aa89eeb6d751f05178a13da95516c036e27468c5d4d2509bb1e15341c81/psycopg2_binary-2.9.11-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a311f1edc9967723d3511ea7d2708e2c3592e3405677bf53d5c7246753591fbb", size = 3981881, upload-time = "2025-10-30T02:55:07.332Z" }, + { url = "https://files.pythonhosted.org/packages/76/a1/2f5841cae4c635a9459fe7aca8ed771336e9383b6429e05c01267b0774cf/psycopg2_binary-2.9.11-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ebb415404821b6d1c47353ebe9c8645967a5235e6d88f914147e7fd411419e6f", size = 3650985, upload-time = "2025-10-10T11:11:34.975Z" }, + { url = "https://files.pythonhosted.org/packages/84/74/4defcac9d002bca5709951b975173c8c2fa968e1a95dc713f61b3a8d3b6a/psycopg2_binary-2.9.11-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f07c9c4a5093258a03b28fab9b4f151aa376989e7f35f855088234e656ee6a94", size = 3296039, upload-time = "2025-10-10T11:11:40.432Z" }, + { url = "https://files.pythonhosted.org/packages/6d/c2/782a3c64403d8ce35b5c50e1b684412cf94f171dc18111be8c976abd2de1/psycopg2_binary-2.9.11-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:00ce1830d971f43b667abe4a56e42c1e2d594b32da4802e44a73bacacb25535f", size = 3043477, upload-time = "2025-10-30T02:55:11.182Z" }, + { url = "https://files.pythonhosted.org/packages/c8/31/36a1d8e702aa35c38fc117c2b8be3f182613faa25d794b8aeaab948d4c03/psycopg2_binary-2.9.11-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:cffe9d7697ae7456649617e8bb8d7a45afb71cd13f7ab22af3e5c61f04840908", size = 3345842, upload-time = "2025-10-10T11:11:45.366Z" }, + { url = "https://files.pythonhosted.org/packages/6e/b4/a5375cda5b54cb95ee9b836930fea30ae5a8f14aa97da7821722323d979b/psycopg2_binary-2.9.11-cp311-cp311-win_amd64.whl", hash = "sha256:304fd7b7f97eef30e91b8f7e720b3db75fee010b520e434ea35ed1ff22501d03", size = 2713894, upload-time = "2025-10-10T11:11:48.775Z" }, + { url = "https://files.pythonhosted.org/packages/d8/91/f870a02f51be4a65987b45a7de4c2e1897dd0d01051e2b559a38fa634e3e/psycopg2_binary-2.9.11-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:be9b840ac0525a283a96b556616f5b4820e0526addb8dcf6525a0fa162730be4", size = 3756603, upload-time = "2025-10-10T11:11:52.213Z" }, + { url = "https://files.pythonhosted.org/packages/27/fa/cae40e06849b6c9a95eb5c04d419942f00d9eaac8d81626107461e268821/psycopg2_binary-2.9.11-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:f090b7ddd13ca842ebfe301cd587a76a4cf0913b1e429eb92c1be5dbeb1a19bc", size = 3864509, upload-time = "2025-10-10T11:11:56.452Z" }, + { url = "https://files.pythonhosted.org/packages/2d/75/364847b879eb630b3ac8293798e380e441a957c53657995053c5ec39a316/psycopg2_binary-2.9.11-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ab8905b5dcb05bf3fb22e0cf90e10f469563486ffb6a96569e51f897c750a76a", size = 4411159, upload-time = "2025-10-10T11:12:00.49Z" }, + { url = "https://files.pythonhosted.org/packages/6f/a0/567f7ea38b6e1c62aafd58375665a547c00c608a471620c0edc364733e13/psycopg2_binary-2.9.11-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:bf940cd7e7fec19181fdbc29d76911741153d51cab52e5c21165f3262125685e", size = 4468234, upload-time = "2025-10-10T11:12:04.892Z" }, + { url = "https://files.pythonhosted.org/packages/30/da/4e42788fb811bbbfd7b7f045570c062f49e350e1d1f3df056c3fb5763353/psycopg2_binary-2.9.11-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fa0f693d3c68ae925966f0b14b8edda71696608039f4ed61b1fe9ffa468d16db", size = 4166236, upload-time = "2025-10-10T11:12:11.674Z" }, + { url = "https://files.pythonhosted.org/packages/3c/94/c1777c355bc560992af848d98216148be5f1be001af06e06fc49cbded578/psycopg2_binary-2.9.11-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:a1cf393f1cdaf6a9b57c0a719a1068ba1069f022a59b8b1fe44b006745b59757", size = 3983083, upload-time = "2025-10-30T02:55:15.73Z" }, + { url = "https://files.pythonhosted.org/packages/bd/42/c9a21edf0e3daa7825ed04a4a8588686c6c14904344344a039556d78aa58/psycopg2_binary-2.9.11-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:ef7a6beb4beaa62f88592ccc65df20328029d721db309cb3250b0aae0fa146c3", size = 3652281, upload-time = "2025-10-10T11:12:17.713Z" }, + { url = "https://files.pythonhosted.org/packages/12/22/dedfbcfa97917982301496b6b5e5e6c5531d1f35dd2b488b08d1ebc52482/psycopg2_binary-2.9.11-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:31b32c457a6025e74d233957cc9736742ac5a6cb196c6b68499f6bb51390bd6a", size = 3298010, upload-time = "2025-10-10T11:12:22.671Z" }, + { url = "https://files.pythonhosted.org/packages/66/ea/d3390e6696276078bd01b2ece417deac954dfdd552d2edc3d03204416c0c/psycopg2_binary-2.9.11-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:edcb3aeb11cb4bf13a2af3c53a15b3d612edeb6409047ea0b5d6a21a9d744b34", size = 3044641, upload-time = "2025-10-30T02:55:19.929Z" }, + { url = "https://files.pythonhosted.org/packages/12/9a/0402ded6cbd321da0c0ba7d34dc12b29b14f5764c2fc10750daa38e825fc/psycopg2_binary-2.9.11-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:62b6d93d7c0b61a1dd6197d208ab613eb7dcfdcca0a49c42ceb082257991de9d", size = 3347940, upload-time = "2025-10-10T11:12:26.529Z" }, + { url = "https://files.pythonhosted.org/packages/b1/d2/99b55e85832ccde77b211738ff3925a5d73ad183c0b37bcbbe5a8ff04978/psycopg2_binary-2.9.11-cp312-cp312-win_amd64.whl", hash = "sha256:b33fabeb1fde21180479b2d4667e994de7bbf0eec22832ba5d9b5e4cf65b6c6d", size = 2714147, upload-time = "2025-10-10T11:12:29.535Z" }, + { url = "https://files.pythonhosted.org/packages/ff/a8/a2709681b3ac11b0b1786def10006b8995125ba268c9a54bea6f5ae8bd3e/psycopg2_binary-2.9.11-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:b8fb3db325435d34235b044b199e56cdf9ff41223a4b9752e8576465170bb38c", size = 3756572, upload-time = "2025-10-10T11:12:32.873Z" }, + { url = "https://files.pythonhosted.org/packages/62/e1/c2b38d256d0dafd32713e9f31982a5b028f4a3651f446be70785f484f472/psycopg2_binary-2.9.11-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:366df99e710a2acd90efed3764bb1e28df6c675d33a7fb40df9b7281694432ee", size = 3864529, upload-time = "2025-10-10T11:12:36.791Z" }, + { url = "https://files.pythonhosted.org/packages/11/32/b2ffe8f3853c181e88f0a157c5fb4e383102238d73c52ac6d93a5c8bffe6/psycopg2_binary-2.9.11-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8c55b385daa2f92cb64b12ec4536c66954ac53654c7f15a203578da4e78105c0", size = 4411242, upload-time = "2025-10-10T11:12:42.388Z" }, + { url = "https://files.pythonhosted.org/packages/10/04/6ca7477e6160ae258dc96f67c371157776564679aefd247b66f4661501a2/psycopg2_binary-2.9.11-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:c0377174bf1dd416993d16edc15357f6eb17ac998244cca19bc67cdc0e2e5766", size = 4468258, upload-time = "2025-10-10T11:12:48.654Z" }, + { url = "https://files.pythonhosted.org/packages/3c/7e/6a1a38f86412df101435809f225d57c1a021307dd0689f7a5e7fe83588b1/psycopg2_binary-2.9.11-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5c6ff3335ce08c75afaed19e08699e8aacf95d4a260b495a4a8545244fe2ceb3", size = 4166295, upload-time = "2025-10-10T11:12:52.525Z" }, + { url = "https://files.pythonhosted.org/packages/f2/7d/c07374c501b45f3579a9eb761cbf2604ddef3d96ad48679112c2c5aa9c25/psycopg2_binary-2.9.11-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:84011ba3109e06ac412f95399b704d3d6950e386b7994475b231cf61eec2fc1f", size = 3983133, upload-time = "2025-10-30T02:55:24.329Z" }, + { url = "https://files.pythonhosted.org/packages/82/56/993b7104cb8345ad7d4516538ccf8f0d0ac640b1ebd8c754a7b024e76878/psycopg2_binary-2.9.11-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ba34475ceb08cccbdd98f6b46916917ae6eeb92b5ae111df10b544c3a4621dc4", size = 3652383, upload-time = "2025-10-10T11:12:56.387Z" }, + { url = "https://files.pythonhosted.org/packages/2d/ac/eaeb6029362fd8d454a27374d84c6866c82c33bfc24587b4face5a8e43ef/psycopg2_binary-2.9.11-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:b31e90fdd0f968c2de3b26ab014314fe814225b6c324f770952f7d38abf17e3c", size = 3298168, upload-time = "2025-10-10T11:13:00.403Z" }, + { url = "https://files.pythonhosted.org/packages/2b/39/50c3facc66bded9ada5cbc0de867499a703dc6bca6be03070b4e3b65da6c/psycopg2_binary-2.9.11-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d526864e0f67f74937a8fce859bd56c979f5e2ec57ca7c627f5f1071ef7fee60", size = 3044712, upload-time = "2025-10-30T02:55:27.975Z" }, + { url = "https://files.pythonhosted.org/packages/9c/8e/b7de019a1f562f72ada81081a12823d3c1590bedc48d7d2559410a2763fe/psycopg2_binary-2.9.11-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:04195548662fa544626c8ea0f06561eb6203f1984ba5b4562764fbeb4c3d14b1", size = 3347549, upload-time = "2025-10-10T11:13:03.971Z" }, + { url = "https://files.pythonhosted.org/packages/80/2d/1bb683f64737bbb1f86c82b7359db1eb2be4e2c0c13b947f80efefa7d3e5/psycopg2_binary-2.9.11-cp313-cp313-win_amd64.whl", hash = "sha256:efff12b432179443f54e230fdf60de1f6cc726b6c832db8701227d089310e8aa", size = 2714215, upload-time = "2025-10-10T11:13:07.14Z" }, + { url = "https://files.pythonhosted.org/packages/64/12/93ef0098590cf51d9732b4f139533732565704f45bdc1ffa741b7c95fb54/psycopg2_binary-2.9.11-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:92e3b669236327083a2e33ccfa0d320dd01b9803b3e14dd986a4fc54aa00f4e1", size = 3756567, upload-time = "2025-10-10T11:13:11.885Z" }, + { url = "https://files.pythonhosted.org/packages/7c/a9/9d55c614a891288f15ca4b5209b09f0f01e3124056924e17b81b9fa054cc/psycopg2_binary-2.9.11-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:e0deeb03da539fa3577fcb0b3f2554a97f7e5477c246098dbb18091a4a01c16f", size = 3864755, upload-time = "2025-10-10T11:13:17.727Z" }, + { url = "https://files.pythonhosted.org/packages/13/1e/98874ce72fd29cbde93209977b196a2edae03f8490d1bd8158e7f1daf3a0/psycopg2_binary-2.9.11-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9b52a3f9bb540a3e4ec0f6ba6d31339727b2950c9772850d6545b7eae0b9d7c5", size = 4411646, upload-time = "2025-10-10T11:13:24.432Z" }, + { url = "https://files.pythonhosted.org/packages/5a/bd/a335ce6645334fb8d758cc358810defca14a1d19ffbc8a10bd38a2328565/psycopg2_binary-2.9.11-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.whl", hash = "sha256:db4fd476874ccfdbb630a54426964959e58da4c61c9feba73e6094d51303d7d8", size = 4468701, upload-time = "2025-10-10T11:13:29.266Z" }, + { url = "https://files.pythonhosted.org/packages/44/d6/c8b4f53f34e295e45709b7568bf9b9407a612ea30387d35eb9fa84f269b4/psycopg2_binary-2.9.11-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:47f212c1d3be608a12937cc131bd85502954398aaa1320cb4c14421a0ffccf4c", size = 4166293, upload-time = "2025-10-10T11:13:33.336Z" }, + { url = "https://files.pythonhosted.org/packages/4b/e0/f8cc36eadd1b716ab36bb290618a3292e009867e5c97ce4aba908cb99644/psycopg2_binary-2.9.11-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e35b7abae2b0adab776add56111df1735ccc71406e56203515e228a8dc07089f", size = 3983184, upload-time = "2025-10-30T02:55:32.483Z" }, + { url = "https://files.pythonhosted.org/packages/53/3e/2a8fe18a4e61cfb3417da67b6318e12691772c0696d79434184a511906dc/psycopg2_binary-2.9.11-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fcf21be3ce5f5659daefd2b3b3b6e4727b028221ddc94e6c1523425579664747", size = 3652650, upload-time = "2025-10-10T11:13:38.181Z" }, + { url = "https://files.pythonhosted.org/packages/76/36/03801461b31b29fe58d228c24388f999fe814dfc302856e0d17f97d7c54d/psycopg2_binary-2.9.11-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:9bd81e64e8de111237737b29d68039b9c813bdf520156af36d26819c9a979e5f", size = 3298663, upload-time = "2025-10-10T11:13:44.878Z" }, + { url = "https://files.pythonhosted.org/packages/97/77/21b0ea2e1a73aa5fa9222b2a6b8ba325c43c3a8d54272839c991f2345656/psycopg2_binary-2.9.11-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:32770a4d666fbdafab017086655bcddab791d7cb260a16679cc5a7338b64343b", size = 3044737, upload-time = "2025-10-30T02:55:35.69Z" }, + { url = "https://files.pythonhosted.org/packages/67/69/f36abe5f118c1dca6d3726ceae164b9356985805480731ac6712a63f24f0/psycopg2_binary-2.9.11-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c3cb3a676873d7506825221045bd70e0427c905b9c8ee8d6acd70cfcbd6e576d", size = 3347643, upload-time = "2025-10-10T11:13:53.499Z" }, + { url = "https://files.pythonhosted.org/packages/e1/36/9c0c326fe3a4227953dfb29f5d0c8ae3b8eb8c1cd2967aa569f50cb3c61f/psycopg2_binary-2.9.11-cp314-cp314-win_amd64.whl", hash = "sha256:4012c9c954dfaccd28f94e84ab9f94e12df76b4afb22331b1f0d3154893a6316", size = 2803913, upload-time = "2025-10-10T11:13:57.058Z" }, +] + [[package]] name = "ptyprocess" version = "0.7.0" @@ -3935,6 +4653,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl", hash = "sha256:1db8e35b67b3d218d818ae653e27f06c3aa420901fa7b081ca98cbedc874e0d0", size = 11842, upload-time = "2024-07-21T12:58:20.04Z" }, ] +[[package]] +name = "puremagic" +version = "1.27" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/d5/ce/dc3a664654f1abed89d4e8a95ac3af02a2a0449c776ccea5ef9f48bde267/puremagic-1.27.tar.gz", hash = "sha256:7cb316f40912f56f34149f8ebdd77a91d099212d2ed936feb2feacfc7cbce2c1", size = 312737, upload-time = "2024-08-08T19:08:37.771Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b1/5c/c277e7638815795a8fd6487e70eeeb30698e5033f4d562619e1571c660d2/puremagic-1.27-py3-none-any.whl", hash = "sha256:b5519ad89e9b7c96a5fd9947d9a907e44f97cc30eae6dcf746d90a58e3681936", size = 40728, upload-time = "2024-08-08T19:08:36.134Z" }, +] + [[package]] name = "pyarrow" version = "23.0.0" @@ -4205,6 +4932,24 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fb/35/b763e8fbcd51968329b9adc52d188fc97859f85f2ee15fe9f379987d99c5/pymdown_extensions-10.19.1-py3-none-any.whl", hash = "sha256:e8698a66055b1dc0dca2a7f2c9d0ea6f5faa7834a9c432e3535ab96c0c4e509b", size = 266693, upload-time = "2025-12-14T17:25:22.999Z" }, ] +[[package]] +name = "pymysql" +version = "1.1.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f5/ae/1fe3fcd9f959efa0ebe200b8de88b5a5ce3e767e38c7ac32fb179f16a388/pymysql-1.1.2.tar.gz", hash = "sha256:4961d3e165614ae65014e361811a724e2044ad3ea3739de9903ae7c21f539f03", size = 48258, upload-time = "2025-08-24T12:55:55.146Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7c/4c/ad33b92b9864cbde84f259d5df035a6447f91891f5be77788e2a3892bce3/pymysql-1.1.2-py3-none-any.whl", hash = "sha256:e6b1d89711dd51f8f74b1631fe08f039e7d76cf67a42a323d3178f0f25762ed9", size = 45300, upload-time = "2025-08-24T12:55:53.394Z" }, +] + +[[package]] +name = "pypdf2" +version = "3.0.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9f/bb/18dc3062d37db6c491392007dfd1a7f524bb95886eb956569ac38a23a784/PyPDF2-3.0.1.tar.gz", hash = "sha256:a74408f69ba6271f71b9352ef4ed03dc53a31aa404d29b5d31f53bfecfee1440", size = 227419, upload-time = "2022-12-31T10:36:13.13Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8e/5e/c86a5643653825d3c913719e788e41386bee415c2b87b4f955432f2de6b2/pypdf2-3.0.1-py3-none-any.whl", hash = "sha256:d16e4205cfee272fbdc0568b68d82be796540b1537508cef59388f839c191928", size = 232572, upload-time = "2022-12-31T10:36:10.327Z" }, +] + [[package]] name = "pytest" version = "9.0.2" @@ -4276,6 +5021,21 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/aa/76/03af049af4dcee5d27442f71b6924f01f3efb5d2bd34f23fcd563f2cc5f5/python_multipart-0.0.21-py3-none-any.whl", hash = "sha256:cf7a6713e01c87aa35387f4774e812c4361150938d20d232800f75ffcf266090", size = 24541, upload-time = "2025-12-17T09:24:21.153Z" }, ] +[[package]] +name = "python-pptx" +version = "1.0.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "lxml" }, + { name = "pillow" }, + { name = "typing-extensions" }, + { name = "xlsxwriter" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/52/a9/0c0db8d37b2b8a645666f7fd8accea4c6224e013c42b1d5c17c93590cd06/python_pptx-1.0.2.tar.gz", hash = "sha256:479a8af0eaf0f0d76b6f00b0887732874ad2e3188230315290cd1f9dd9cc7095", size = 10109297, upload-time = "2024-08-07T17:33:37.772Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/d9/4f/00be2196329ebbff56ce564aa94efb0fbc828d00de250b1980de1a34ab49/python_pptx-1.0.2-py3-none-any.whl", hash = "sha256:160838e0b8565a8b1f67947675886e9fea18aa5e795db7ae531606d68e785cba", size = 472788, upload-time = "2024-08-07T17:33:28.192Z" }, +] + [[package]] name = "pytz" version = "2025.2" @@ -4471,6 +5231,80 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/01/1b/5dbe84eefc86f48473947e2f41711aded97eecef1231f4558f1f02713c12/pyzmq-27.1.0-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:c9f7f6e13dff2e44a6afeaf2cf54cee5929ad64afaf4d40b50f93c58fc687355", size = 544862, upload-time = "2025-09-08T23:09:56.509Z" }, ] +[[package]] +name = "rapidfuzz" +version = "3.12.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c9/df/c300ead8c2962f54ad87872e6372a6836f0181a7f20b433c987bd106bfce/rapidfuzz-3.12.1.tar.gz", hash = "sha256:6a98bbca18b4a37adddf2d8201856441c26e9c981d8895491b5bc857b5f780eb", size = 57907552, upload-time = "2025-01-30T17:10:28.279Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/29/e6/a56a87edff979559ce1e5486bf148c5f8905c9159ebdb14f217b3a3eeb2b/rapidfuzz-3.12.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:dbb7ea2fd786e6d66f225ef6eef1728832314f47e82fee877cb2a793ebda9579", size = 1959669, upload-time = "2025-01-30T17:06:50.82Z" }, + { url = "https://files.pythonhosted.org/packages/2e/6d/010a33d3425494f9967025897ad5283a159cf72e4552cc443d5f646cd040/rapidfuzz-3.12.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:1ae41361de05762c1eaa3955e5355de7c4c6f30d1ef1ea23d29bf738a35809ab", size = 1433648, upload-time = "2025-01-30T17:06:52.758Z" }, + { url = "https://files.pythonhosted.org/packages/43/a8/2964c7dac65f147098145598e265a434a55a6a6be13ce1bca4c8b822e77f/rapidfuzz-3.12.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:dc3c39e0317e7f68ba01bac056e210dd13c7a0abf823e7b6a5fe7e451ddfc496", size = 1423317, upload-time = "2025-01-30T17:06:55.165Z" }, + { url = "https://files.pythonhosted.org/packages/d6/9c/a8d376fcad2f4b48483b5a54a45bd71d75d9401fd12227dae7cfe565f2db/rapidfuzz-3.12.1-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:69f2520296f1ae1165b724a3aad28c56fd0ac7dd2e4cff101a5d986e840f02d4", size = 5641782, upload-time = "2025-01-30T17:06:57.56Z" }, + { url = "https://files.pythonhosted.org/packages/98/69/26b21a1c3ccd4960a82493396e90db5e81a73d5fbbad98fc9b913b96e557/rapidfuzz-3.12.1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:34dcbf5a7daecebc242f72e2500665f0bde9dd11b779246c6d64d106a7d57c99", size = 1683506, upload-time = "2025-01-30T17:06:59.626Z" }, + { url = "https://files.pythonhosted.org/packages/6a/a0/87323883234508bd0ebc599004aab25319c1e296644e73f94c8fbee7c57d/rapidfuzz-3.12.1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:773ab37fccf6e0513891f8eb4393961ddd1053c6eb7e62eaa876e94668fc6d31", size = 1685813, upload-time = "2025-01-30T17:07:02.124Z" }, + { url = "https://files.pythonhosted.org/packages/91/ea/e99bea5218805d28a5df7b39a35239e3209e8dce25d0b5a3e1146a9b9d40/rapidfuzz-3.12.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:3ecf0e6de84c0bc2c0f48bc03ba23cef2c5f1245db7b26bc860c11c6fd7a097c", size = 3142162, upload-time = "2025-01-30T17:07:04.582Z" }, + { url = "https://files.pythonhosted.org/packages/da/cd/89751db1dd8b020ccce6d83e59fcf7f4f4090d093900b52552c5561a438c/rapidfuzz-3.12.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:4dc2ebad4adb29d84a661f6a42494df48ad2b72993ff43fad2b9794804f91e45", size = 2339376, upload-time = "2025-01-30T17:07:06.419Z" }, + { url = "https://files.pythonhosted.org/packages/eb/85/b6e46c3d686cc3f53457468d46499e88492980a447e34f12ce1f81fc246d/rapidfuzz-3.12.1-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:8389d98b9f54cb4f8a95f1fa34bf0ceee639e919807bb931ca479c7a5f2930bf", size = 6941790, upload-time = "2025-01-30T17:07:08.32Z" }, + { url = "https://files.pythonhosted.org/packages/54/20/9309eb912ffd701e6a1d1961475b9607f8cd0a793d6011c44a1f0e306f45/rapidfuzz-3.12.1-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:165bcdecbfed9978962da1d3ec9c191b2ff9f1ccc2668fbaf0613a975b9aa326", size = 2719567, upload-time = "2025-01-30T17:07:10.425Z" }, + { url = "https://files.pythonhosted.org/packages/43/74/449c1680b30f640ed380bef6cdd8837b69b0325e4e9e7a8bc3dd106bd8cb/rapidfuzz-3.12.1-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:129d536740ab0048c1a06ccff73c683f282a2347c68069affae8dbc423a37c50", size = 3268295, upload-time = "2025-01-30T17:07:12.938Z" }, + { url = "https://files.pythonhosted.org/packages/47/71/949eacc7b5a69b5d0aeca27eab295b2a3481116dc26959aa9a063e3876d0/rapidfuzz-3.12.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:1b67e390261ffe98ec86c771b89425a78b60ccb610c3b5874660216fcdbded4b", size = 4172971, upload-time = "2025-01-30T17:07:15.687Z" }, + { url = "https://files.pythonhosted.org/packages/bf/38/a023f9f11e59a2124581814bb22693e0cbd919dd63273c2736526512ee34/rapidfuzz-3.12.1-cp310-cp310-win32.whl", hash = "sha256:a66520180d3426b9dc2f8d312f38e19bc1fc5601f374bae5c916f53fa3534a7d", size = 1851232, upload-time = "2025-01-30T17:07:17.942Z" }, + { url = "https://files.pythonhosted.org/packages/c6/b5/afa8c28c9a0f9ad15c2af8bb7c66a5b9b832ff2ebd00f380bda1bb3287d7/rapidfuzz-3.12.1-cp310-cp310-win_amd64.whl", hash = "sha256:82260b20bc7a76556cecb0c063c87dad19246a570425d38f8107b8404ca3ac97", size = 1620845, upload-time = "2025-01-30T17:07:20.164Z" }, + { url = "https://files.pythonhosted.org/packages/59/42/d7b9a120051dc9dbde1ee2db558e0fbe9a9074c1e27f00d89a67835bc0eb/rapidfuzz-3.12.1-cp310-cp310-win_arm64.whl", hash = "sha256:3a860d103bbb25c69c2e995fdf4fac8cb9f77fb69ec0a00469d7fd87ff148f46", size = 869032, upload-time = "2025-01-30T17:07:22.759Z" }, + { url = "https://files.pythonhosted.org/packages/a3/f2/9146cee62060dfe1de4beebe349fe4c007f5de4611cf3fbfb61e4b61b500/rapidfuzz-3.12.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:6d9afad7b16d01c9e8929b6a205a18163c7e61b6cd9bcf9c81be77d5afc1067a", size = 1960497, upload-time = "2025-01-30T17:07:24.427Z" }, + { url = "https://files.pythonhosted.org/packages/3e/54/7fee154f9a00c97b4eb12b223c184ca9be1ec0725b9f9e5e913dc6266c69/rapidfuzz-3.12.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:bb424ae7240f2d2f7d8dda66a61ebf603f74d92f109452c63b0dbf400204a437", size = 1434283, upload-time = "2025-01-30T17:07:26.296Z" }, + { url = "https://files.pythonhosted.org/packages/ef/c5/8138e48c1ee31b5bd38facbb78c859e4e58aa306f5f753ffee82166390b7/rapidfuzz-3.12.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:42149e6d13bd6d06437d2a954dae2184dadbbdec0fdb82dafe92860d99f80519", size = 1417803, upload-time = "2025-01-30T17:07:28.802Z" }, + { url = "https://files.pythonhosted.org/packages/03/0a/be43022744d79f1f0725cb21fe2a9656fb8a509547dbef120b4b335ca9bd/rapidfuzz-3.12.1-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:760ac95d788f2964b73da01e0bdffbe1bf2ad8273d0437565ce9092ae6ad1fbc", size = 5620489, upload-time = "2025-01-30T17:07:30.749Z" }, + { url = "https://files.pythonhosted.org/packages/21/d8/fa4b5ce056c4c2e2506706058cb14c44b77de897e70396643ea3bfa75ed0/rapidfuzz-3.12.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2cf27e8e4bf7bf9d92ef04f3d2b769e91c3f30ba99208c29f5b41e77271a2614", size = 1671236, upload-time = "2025-01-30T17:07:33.372Z" }, + { url = "https://files.pythonhosted.org/packages/db/21/5b171401ac92189328ba680a1f68c54c89b18a410d8c865794c433839ea1/rapidfuzz-3.12.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:00ceb8ff3c44ab0d6014106c71709c85dee9feedd6890eff77c814aa3798952b", size = 1683376, upload-time = "2025-01-30T17:07:35.866Z" }, + { url = "https://files.pythonhosted.org/packages/1d/ce/f209f437c6df46ba523a6898ebd854b30196650f77dcddf203191f09bf9b/rapidfuzz-3.12.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b8b61c558574fbc093d85940c3264c08c2b857b8916f8e8f222e7b86b0bb7d12", size = 3139202, upload-time = "2025-01-30T17:07:37.583Z" }, + { url = "https://files.pythonhosted.org/packages/41/3a/6821bddb2af8412b340a7258c89a7519e7ebece58c6b3027859138bb3142/rapidfuzz-3.12.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:346a2d8f17224e99f9ef988606c83d809d5917d17ad00207237e0965e54f9730", size = 2346575, upload-time = "2025-01-30T17:07:39.519Z" }, + { url = "https://files.pythonhosted.org/packages/44/db/f76a211e050024f11d0d2b0dfca6378e949d6d81f9bdaac15c7c30280942/rapidfuzz-3.12.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:d60d1db1b7e470e71ae096b6456e20ec56b52bde6198e2dbbc5e6769fa6797dc", size = 6944232, upload-time = "2025-01-30T17:07:41.617Z" }, + { url = "https://files.pythonhosted.org/packages/16/a5/670287316f7f3591141c9ab3752f295705547f8075bf1616b76ad8f64069/rapidfuzz-3.12.1-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:2477da227e266f9c712f11393182c69a99d3c8007ea27f68c5afc3faf401cc43", size = 2722753, upload-time = "2025-01-30T17:07:43.672Z" }, + { url = "https://files.pythonhosted.org/packages/ba/68/5be0dfd2b3fc0dfac7f4b251b18121b2809f244f16b2c44a54b0ffa733a6/rapidfuzz-3.12.1-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:8499c7d963ddea8adb6cffac2861ee39a1053e22ca8a5ee9de1197f8dc0275a5", size = 3262227, upload-time = "2025-01-30T17:07:47.926Z" }, + { url = "https://files.pythonhosted.org/packages/02/c6/a747b4103d3a96b4e5d022326b764d2493190dd5240e4aeb1a791c5a26f9/rapidfuzz-3.12.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:12802e5c4d8ae104fb6efeeb436098325ce0dca33b461c46e8df015c84fbef26", size = 4175381, upload-time = "2025-01-30T17:07:50.31Z" }, + { url = "https://files.pythonhosted.org/packages/77/72/d5c9d5fe02a0f2b66a0669aafdc8875a4d09e3a77a50d1fc9e524ec098ca/rapidfuzz-3.12.1-cp311-cp311-win32.whl", hash = "sha256:e1061311d07e7cdcffa92c9b50c2ab4192907e70ca01b2e8e1c0b6b4495faa37", size = 1851445, upload-time = "2025-01-30T17:07:52.246Z" }, + { url = "https://files.pythonhosted.org/packages/12/24/f7bd6618e4f2463f1f3574476a06b8d9041f9c69e431df1ab9c924da5cc3/rapidfuzz-3.12.1-cp311-cp311-win_amd64.whl", hash = "sha256:c6e4ed63e204daa863a802eec09feea5448617981ba5d150f843ad8e3ae071a4", size = 1626995, upload-time = "2025-01-30T17:07:54.541Z" }, + { url = "https://files.pythonhosted.org/packages/9c/ec/fb8244f3ce12caf3caea54c4f79ab9fac9855beec12beacd7edca7b017a6/rapidfuzz-3.12.1-cp311-cp311-win_arm64.whl", hash = "sha256:920733a28c3af47870835d59ca9879579f66238f10de91d2b4b3f809d1ebfc5b", size = 870216, upload-time = "2025-01-30T17:07:57.003Z" }, + { url = "https://files.pythonhosted.org/packages/1a/20/6049061411df87f2814a2677db0f15e673bb9795bfeff57dc9708121374d/rapidfuzz-3.12.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:f6235b57ae3faa3f85cb3f90c9fee49b21bd671b76e90fc99e8ca2bdf0b5e4a3", size = 1944328, upload-time = "2025-01-30T17:07:59.244Z" }, + { url = "https://files.pythonhosted.org/packages/25/73/199383c4c21ae3b4b6ea6951c6896ab38e9dc96942462fa01f9d3fb047da/rapidfuzz-3.12.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:af4585e5812632c357fee5ab781c29f00cd06bea58f8882ff244cc4906ba6c9e", size = 1430203, upload-time = "2025-01-30T17:08:00.917Z" }, + { url = "https://files.pythonhosted.org/packages/7b/51/77ebaeec5413c53c3e6d8b800f2b979551adbed7b5efa094d1fad5c5b751/rapidfuzz-3.12.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:5942dc4460e5030c5f9e1d4c9383de2f3564a2503fe25e13e89021bcbfea2f44", size = 1403662, upload-time = "2025-01-30T17:08:02.881Z" }, + { url = "https://files.pythonhosted.org/packages/54/06/1fadd2704db0a7eecf78de812e2f4fab37c4ae105a5ce4578c9fc66bb0c5/rapidfuzz-3.12.1-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:0b31ab59e1a0df5afc21f3109b6cfd77b34040dbf54f1bad3989f885cfae1e60", size = 5555849, upload-time = "2025-01-30T17:08:05.746Z" }, + { url = "https://files.pythonhosted.org/packages/19/45/da128c3952bd09cef2935df58db5273fc4eb67f04a69dcbf9e25af9e4432/rapidfuzz-3.12.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:97c885a7a480b21164f57a706418c9bbc9a496ec6da087e554424358cadde445", size = 1655273, upload-time = "2025-01-30T17:08:07.913Z" }, + { url = "https://files.pythonhosted.org/packages/03/ee/bf2b2a95b5af4e6d36105dd9284dc5335fdcc7f0326186d4ab0b5aa4721e/rapidfuzz-3.12.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2d844c0587d969ce36fbf4b7cbf0860380ffeafc9ac5e17a7cbe8abf528d07bb", size = 1678041, upload-time = "2025-01-30T17:08:10.043Z" }, + { url = "https://files.pythonhosted.org/packages/7f/4f/36ea4d7f306a23e30ea1a6cabf545d2a794e8ca9603d2ee48384314cde3a/rapidfuzz-3.12.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a93c95dce8917bf428064c64024de43ffd34ec5949dd4425780c72bd41f9d969", size = 3137099, upload-time = "2025-01-30T17:08:12.943Z" }, + { url = "https://files.pythonhosted.org/packages/70/ef/48195d94b018e7340a60c9a642ab0081bf9dc64fb0bd01dfafd93757d2a2/rapidfuzz-3.12.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:834f6113d538af358f39296604a1953e55f8eeffc20cb4caf82250edbb8bf679", size = 2307388, upload-time = "2025-01-30T17:08:15.029Z" }, + { url = "https://files.pythonhosted.org/packages/e5/cd/53d5dbc4791df3e1a8640fc4ad5e328ebb040cc01c10c66f891aa6b83ed5/rapidfuzz-3.12.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:a940aa71a7f37d7f0daac186066bf6668d4d3b7e7ef464cb50bc7ba89eae1f51", size = 6906504, upload-time = "2025-01-30T17:08:17.704Z" }, + { url = "https://files.pythonhosted.org/packages/1b/99/c27e7db1d49cfd77780cb73978f81092682c2bdbc6de75363df6aaa086d6/rapidfuzz-3.12.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:ec9eaf73501c9a7de2c6938cb3050392e2ee0c5ca3921482acf01476b85a7226", size = 2684757, upload-time = "2025-01-30T17:08:19.821Z" }, + { url = "https://files.pythonhosted.org/packages/02/8c/2474d6282fdd4aae386a6b16272e544a3f9ea2dcdcf2f3b0b286549bc3d5/rapidfuzz-3.12.1-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:3c5ec360694ac14bfaeb6aea95737cf1a6cf805b5fe8ea7fd28814706c7fa838", size = 3229940, upload-time = "2025-01-30T17:08:21.639Z" }, + { url = "https://files.pythonhosted.org/packages/ac/27/95d5a8ebe5fcc5462dd0fd265553c8a2ec4a770e079afabcff978442bcb3/rapidfuzz-3.12.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:6b5e176524653ac46f1802bdd273a4b44a5f8d0054ed5013a8e8a4b72f254599", size = 4148489, upload-time = "2025-01-30T17:08:24.032Z" }, + { url = "https://files.pythonhosted.org/packages/8d/2c/e509bc24b6514de4d6f2c5480201568e1d9a3c7e4692cc969ef899227ba5/rapidfuzz-3.12.1-cp312-cp312-win32.whl", hash = "sha256:6f463c6f1c42ec90e45d12a6379e18eddd5cdf74138804d8215619b6f4d31cea", size = 1834110, upload-time = "2025-01-30T17:08:26.75Z" }, + { url = "https://files.pythonhosted.org/packages/cc/ab/900b8d57090b30269258e3ae31752ec9c31042cd58660fcc96d50728487d/rapidfuzz-3.12.1-cp312-cp312-win_amd64.whl", hash = "sha256:b894fa2b30cd6498a29e5c470cb01c6ea898540b7e048a0342775a5000531334", size = 1612461, upload-time = "2025-01-30T17:08:28.862Z" }, + { url = "https://files.pythonhosted.org/packages/a0/df/3f51a0a277185b3f28b2941e071aff62908a6b81527efc67a643bcb59fb8/rapidfuzz-3.12.1-cp312-cp312-win_arm64.whl", hash = "sha256:43bb17056c5d1332f517b888c4e57846c4b5f936ed304917eeb5c9ac85d940d4", size = 864251, upload-time = "2025-01-30T17:08:31.364Z" }, + { url = "https://files.pythonhosted.org/packages/62/d2/ceebc2446d1f3d3f2cae2597116982e50c2eed9ff2f5a322a51736981405/rapidfuzz-3.12.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:97f824c15bc6933a31d6e3cbfa90188ba0e5043cf2b6dd342c2b90ee8b3fd47c", size = 1936794, upload-time = "2025-01-30T17:08:33.83Z" }, + { url = "https://files.pythonhosted.org/packages/88/38/37f7ea800aa959a4f7a63477fc9ad7f3cd024e46bfadce5d23420af6c7e5/rapidfuzz-3.12.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a973b3f5cabf931029a3ae4a0f72e3222e53d412ea85fc37ddc49e1774f00fbf", size = 1424155, upload-time = "2025-01-30T17:08:36.774Z" }, + { url = "https://files.pythonhosted.org/packages/3f/14/409d0aa84430451488177fcc5cba8babcdf5a45cee772a2a265b9b5f4c7e/rapidfuzz-3.12.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:df7880e012228722dec1be02b9ef3898ed023388b8a24d6fa8213d7581932510", size = 1398013, upload-time = "2025-01-30T17:08:39.607Z" }, + { url = "https://files.pythonhosted.org/packages/4b/2c/601e3ad0bbe61e65f99e72c8cefed9713606cf4b297cc4c3876051db7722/rapidfuzz-3.12.1-cp313-cp313-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9c78582f50e75e6c2bc38c791ed291cb89cf26a3148c47860c1a04d6e5379c8e", size = 5526157, upload-time = "2025-01-30T17:08:42.749Z" }, + { url = "https://files.pythonhosted.org/packages/97/ce/deb7b00ce6e06713fc4df81336402b7fa062f2393c8a47401c228ee906c3/rapidfuzz-3.12.1-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:2d7d9e6a04d8344b0198c96394c28874086888d0a2b2f605f30d1b27b9377b7d", size = 1648446, upload-time = "2025-01-30T17:08:44.927Z" }, + { url = "https://files.pythonhosted.org/packages/ec/6f/2b8eae1748a022290815999594b438dbc1e072c38c76178ea996920a6253/rapidfuzz-3.12.1-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:5620001fd4d6644a2f56880388179cc8f3767670f0670160fcb97c3b46c828af", size = 1676038, upload-time = "2025-01-30T17:08:47.4Z" }, + { url = "https://files.pythonhosted.org/packages/b9/6c/5c831197aca7148ed85c86bbe940e66073fea0fa97f30307bb5850ed8858/rapidfuzz-3.12.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0666ab4c52e500af7ba5cc17389f5d15c0cdad06412c80312088519fdc25686d", size = 3114137, upload-time = "2025-01-30T17:08:50.234Z" }, + { url = "https://files.pythonhosted.org/packages/fc/f2/d66ac185eeb0ee3fc0fe208dab1e72feece2c883bc0ab2097570a8159a7b/rapidfuzz-3.12.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:27b4d440fa50b50c515a91a01ee17e8ede719dca06eef4c0cccf1a111a4cfad3", size = 2305754, upload-time = "2025-01-30T17:08:52.618Z" }, + { url = "https://files.pythonhosted.org/packages/6c/61/9bf74d7ea9bebc7a1bed707591617bba7901fce414d346a7c5532ef02dbd/rapidfuzz-3.12.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:83dccfd5a754f2a0e8555b23dde31f0f7920601bfa807aa76829391ea81e7c67", size = 6901746, upload-time = "2025-01-30T17:08:54.747Z" }, + { url = "https://files.pythonhosted.org/packages/81/73/d8dddf73e168f723ef21272e8abb7d34d9244da395eb90ed5a617f870678/rapidfuzz-3.12.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:b572b634740e047c53743ed27a1bb3b4f93cf4abbac258cd7af377b2c4a9ba5b", size = 2673947, upload-time = "2025-01-30T17:08:57.525Z" }, + { url = "https://files.pythonhosted.org/packages/2e/31/3c473cea7d76af162819a5b84f5e7bdcf53b9e19568fc37cfbdab4f4512a/rapidfuzz-3.12.1-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:7fa7b81fb52902d5f78dac42b3d6c835a6633b01ddf9b202a3ca8443be4b2d6a", size = 3233070, upload-time = "2025-01-30T17:09:00.624Z" }, + { url = "https://files.pythonhosted.org/packages/c0/b7/73227dcbf8586f0ca4a77be2720311367288e2db142ae00a1404f42e712d/rapidfuzz-3.12.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b1d4fbff980cb6baef4ee675963c081f7b5d6580a105d6a4962b20f1f880e1fb", size = 4146828, upload-time = "2025-01-30T17:09:03.787Z" }, + { url = "https://files.pythonhosted.org/packages/3a/c8/fea749c662e268d348a77501995b51ac95cdc3624f3f95ba261f30b000ff/rapidfuzz-3.12.1-cp313-cp313-win32.whl", hash = "sha256:3fe8da12ea77271097b303fa7624cfaf5afd90261002314e3b0047d36f4afd8d", size = 1831797, upload-time = "2025-01-30T17:09:05.803Z" }, + { url = "https://files.pythonhosted.org/packages/66/18/11052be5984d9972eb04a52e2931e19e95b2e87731d179f60b79707b7efd/rapidfuzz-3.12.1-cp313-cp313-win_amd64.whl", hash = "sha256:6f7e92fc7d2a7f02e1e01fe4f539324dfab80f27cb70a30dd63a95445566946b", size = 1610169, upload-time = "2025-01-30T17:09:07.819Z" }, + { url = "https://files.pythonhosted.org/packages/db/c1/66427c618f000298edbd24e46dd3dd2d3fa441a602701ba6a260d41dd62b/rapidfuzz-3.12.1-cp313-cp313-win_arm64.whl", hash = "sha256:e31be53d7f4905a6a038296d8b773a79da9ee9f0cd19af9490c5c5a22e37d2e5", size = 863036, upload-time = "2025-01-30T17:09:09.715Z" }, + { url = "https://files.pythonhosted.org/packages/0b/5f/82352d6e68ddd45973cbc9f4c89a2a6b6b93907b0f775b8095f34bef654e/rapidfuzz-3.12.1-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:b7cba636c32a6fc3a402d1cb2c70c6c9f8e6319380aaf15559db09d868a23e56", size = 1858389, upload-time = "2025-01-30T17:09:51.512Z" }, + { url = "https://files.pythonhosted.org/packages/05/17/76bab0b29b78171cde746d180258b93aa66a80503291c813b7d8b2a2b927/rapidfuzz-3.12.1-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:b79286738a43e8df8420c4b30a92712dec6247430b130f8e015c3a78b6d61ac2", size = 1368428, upload-time = "2025-01-30T17:09:53.654Z" }, + { url = "https://files.pythonhosted.org/packages/71/77/0ad39429d25b52e21fa2ecbc1f577e62d77c76c8db562bb93c56fe19ccd3/rapidfuzz-3.12.1-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:8dc1937198e7ff67e217e60bfa339f05da268d91bb15fec710452d11fe2fdf60", size = 1364376, upload-time = "2025-01-30T17:09:56.616Z" }, + { url = "https://files.pythonhosted.org/packages/c0/1d/724670d13222f9959634d3dfa832e7cec889e62fca5f9f4acf65f83fa1d5/rapidfuzz-3.12.1-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b85817a57cf8db32dd5d2d66ccfba656d299b09eaf86234295f89f91be1a0db2", size = 5486472, upload-time = "2025-01-30T17:09:59.734Z" }, + { url = "https://files.pythonhosted.org/packages/2c/69/e5cb280ce99dea2de60fa1c80ffab2ebc6e38694a98d7c2b25d2337f87eb/rapidfuzz-3.12.1-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:04283c6f3e79f13a784f844cd5b1df4f518ad0f70c789aea733d106c26e1b4fb", size = 3064862, upload-time = "2025-01-30T17:10:02.323Z" }, + { url = "https://files.pythonhosted.org/packages/eb/56/22227bc9da19687d052fc43d5045f90526a2cb41c6b8e23c860acf1674b5/rapidfuzz-3.12.1-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:a718f740553aad5f4daef790191511da9c6eae893ee1fc2677627e4b624ae2db", size = 1549445, upload-time = "2025-01-30T17:10:04.568Z" }, +] + [[package]] name = "referencing" version = "0.37.0" @@ -4592,15 +5426,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/20/31/32c0c4610cbc070362bf1d2e4ea86d1ea29014d400a6d6c2486fcfd57766/regex-2025.11.3-cp314-cp314t-win_arm64.whl", hash = "sha256:c54f768482cef41e219720013cd05933b6f971d9562544d691c68699bf2b6801", size = 274741, upload-time = "2025-11-03T21:33:45.557Z" }, ] -[[package]] -name = "reiter" -version = "0.8.0" -source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/79/2b/6aa02fc71217f5119f095cfb68acf379c3378a54a6cc52860a5a289e85e2/reiter-0.8.0.tar.gz", hash = "sha256:ee08ae52515c165f8389054ec47df26c341877e86ce89c2722dc232896901472", size = 8966, upload-time = "2023-05-27T03:52:08.237Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/8e/0a/83ab615858ef88d47690818cb05418daffeefdb31bd7e8609c9bb45c5d1d/reiter-0.8.0-py3-none-any.whl", hash = "sha256:03d99dbdb6a3b99144cf3d2704a885b27d465d441b99e3fe57ad87253d35f0dd", size = 7539, upload-time = "2023-05-27T03:52:06.49Z" }, -] - [[package]] name = "requests" version = "2.32.5" @@ -4820,6 +5645,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/64/8d/0133e4eb4beed9e425d9a98ed6e081a55d195481b7632472be1af08d2f6b/rsa-4.9.1-py3-none-any.whl", hash = "sha256:68635866661c6836b8d39430f97a996acbd61bfa49406748ea243539fe239762", size = 34696, upload-time = "2025-04-16T09:51:17.142Z" }, ] +[[package]] +name = "ruamel-yaml" +version = "0.19.1" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/c7/3b/ebda527b56beb90cb7652cb1c7e4f91f48649fbcd8d2eb2fb6e77cd3329b/ruamel_yaml-0.19.1.tar.gz", hash = "sha256:53eb66cd27849eff968ebf8f0bf61f46cdac2da1d1f3576dd4ccee9b25c31993", size = 142709, upload-time = "2026-01-02T16:50:31.84Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/b8/0c/51f6841f1d84f404f92463fc2b1ba0da357ca1e3db6b7fbda26956c3b82a/ruamel_yaml-0.19.1-py3-none-any.whl", hash = "sha256:27592957fedf6e0b62f281e96effd28043345e0e66001f97683aa9a40c667c93", size = 118102, upload-time = "2026-01-02T16:50:29.201Z" }, +] + [[package]] name = "ruff" version = "0.14.10" @@ -4872,6 +5706,265 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/58/5b/632a58724221ef03d78ab65062e82a1010e1bef8e8e0b9d7c6d7b8044841/safetensors-0.7.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:473b32699f4200e69801bf5abf93f1a4ecd432a70984df164fc22ccf39c4a6f3", size = 531885, upload-time = "2025-11-19T15:18:27.146Z" }, ] +[[package]] +name = "scikit-learn" +version = "1.7.2" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.11' and sys_platform == 'linux'", + "python_full_version < '3.11' and sys_platform != 'linux'", +] +dependencies = [ + { name = "joblib", marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "threadpoolctl", marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/98/c2/a7855e41c9d285dfe86dc50b250978105dce513d6e459ea66a6aeb0e1e0c/scikit_learn-1.7.2.tar.gz", hash = "sha256:20e9e49ecd130598f1ca38a1d85090e1a600147b9c02fa6f15d69cb53d968fda", size = 7193136, upload-time = "2025-09-09T08:21:29.075Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ba/3e/daed796fd69cce768b8788401cc464ea90b306fb196ae1ffed0b98182859/scikit_learn-1.7.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6b33579c10a3081d076ab403df4a4190da4f4432d443521674637677dc91e61f", size = 9336221, upload-time = "2025-09-09T08:20:19.328Z" }, + { url = "https://files.pythonhosted.org/packages/1c/ce/af9d99533b24c55ff4e18d9b7b4d9919bbc6cd8f22fe7a7be01519a347d5/scikit_learn-1.7.2-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:36749fb62b3d961b1ce4fedf08fa57a1986cd409eff2d783bca5d4b9b5fce51c", size = 8653834, upload-time = "2025-09-09T08:20:22.073Z" }, + { url = "https://files.pythonhosted.org/packages/58/0e/8c2a03d518fb6bd0b6b0d4b114c63d5f1db01ff0f9925d8eb10960d01c01/scikit_learn-1.7.2-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7a58814265dfc52b3295b1900cfb5701589d30a8bb026c7540f1e9d3499d5ec8", size = 9660938, upload-time = "2025-09-09T08:20:24.327Z" }, + { url = "https://files.pythonhosted.org/packages/2b/75/4311605069b5d220e7cf5adabb38535bd96f0079313cdbb04b291479b22a/scikit_learn-1.7.2-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4a847fea807e278f821a0406ca01e387f97653e284ecbd9750e3ee7c90347f18", size = 9477818, upload-time = "2025-09-09T08:20:26.845Z" }, + { url = "https://files.pythonhosted.org/packages/7f/9b/87961813c34adbca21a6b3f6b2bea344c43b30217a6d24cc437c6147f3e8/scikit_learn-1.7.2-cp310-cp310-win_amd64.whl", hash = "sha256:ca250e6836d10e6f402436d6463d6c0e4d8e0234cfb6a9a47835bd392b852ce5", size = 8886969, upload-time = "2025-09-09T08:20:29.329Z" }, + { url = "https://files.pythonhosted.org/packages/43/83/564e141eef908a5863a54da8ca342a137f45a0bfb71d1d79704c9894c9d1/scikit_learn-1.7.2-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:c7509693451651cd7361d30ce4e86a1347493554f172b1c72a39300fa2aea79e", size = 9331967, upload-time = "2025-09-09T08:20:32.421Z" }, + { url = "https://files.pythonhosted.org/packages/18/d6/ba863a4171ac9d7314c4d3fc251f015704a2caeee41ced89f321c049ed83/scikit_learn-1.7.2-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:0486c8f827c2e7b64837c731c8feff72c0bd2b998067a8a9cbc10643c31f0fe1", size = 8648645, upload-time = "2025-09-09T08:20:34.436Z" }, + { url = "https://files.pythonhosted.org/packages/ef/0e/97dbca66347b8cf0ea8b529e6bb9367e337ba2e8be0ef5c1a545232abfde/scikit_learn-1.7.2-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:89877e19a80c7b11a2891a27c21c4894fb18e2c2e077815bcade10d34287b20d", size = 9715424, upload-time = "2025-09-09T08:20:36.776Z" }, + { url = "https://files.pythonhosted.org/packages/f7/32/1f3b22e3207e1d2c883a7e09abb956362e7d1bd2f14458c7de258a26ac15/scikit_learn-1.7.2-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8da8bf89d4d79aaec192d2bda62f9b56ae4e5b4ef93b6a56b5de4977e375c1f1", size = 9509234, upload-time = "2025-09-09T08:20:38.957Z" }, + { url = "https://files.pythonhosted.org/packages/9f/71/34ddbd21f1da67c7a768146968b4d0220ee6831e4bcbad3e03dd3eae88b6/scikit_learn-1.7.2-cp311-cp311-win_amd64.whl", hash = "sha256:9b7ed8d58725030568523e937c43e56bc01cadb478fc43c042a9aca1dacb3ba1", size = 8894244, upload-time = "2025-09-09T08:20:41.166Z" }, + { url = "https://files.pythonhosted.org/packages/a7/aa/3996e2196075689afb9fce0410ebdb4a09099d7964d061d7213700204409/scikit_learn-1.7.2-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:8d91a97fa2b706943822398ab943cde71858a50245e31bc71dba62aab1d60a96", size = 9259818, upload-time = "2025-09-09T08:20:43.19Z" }, + { url = "https://files.pythonhosted.org/packages/43/5d/779320063e88af9c4a7c2cf463ff11c21ac9c8bd730c4a294b0000b666c9/scikit_learn-1.7.2-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:acbc0f5fd2edd3432a22c69bed78e837c70cf896cd7993d71d51ba6708507476", size = 8636997, upload-time = "2025-09-09T08:20:45.468Z" }, + { url = "https://files.pythonhosted.org/packages/5c/d0/0c577d9325b05594fdd33aa970bf53fb673f051a45496842caee13cfd7fe/scikit_learn-1.7.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e5bf3d930aee75a65478df91ac1225ff89cd28e9ac7bd1196853a9229b6adb0b", size = 9478381, upload-time = "2025-09-09T08:20:47.982Z" }, + { url = "https://files.pythonhosted.org/packages/82/70/8bf44b933837ba8494ca0fc9a9ab60f1c13b062ad0197f60a56e2fc4c43e/scikit_learn-1.7.2-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b4d6e9deed1a47aca9fe2f267ab8e8fe82ee20b4526b2c0cd9e135cea10feb44", size = 9300296, upload-time = "2025-09-09T08:20:50.366Z" }, + { url = "https://files.pythonhosted.org/packages/c6/99/ed35197a158f1fdc2fe7c3680e9c70d0128f662e1fee4ed495f4b5e13db0/scikit_learn-1.7.2-cp312-cp312-win_amd64.whl", hash = "sha256:6088aa475f0785e01bcf8529f55280a3d7d298679f50c0bb70a2364a82d0b290", size = 8731256, upload-time = "2025-09-09T08:20:52.627Z" }, + { url = "https://files.pythonhosted.org/packages/ae/93/a3038cb0293037fd335f77f31fe053b89c72f17b1c8908c576c29d953e84/scikit_learn-1.7.2-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0b7dacaa05e5d76759fb071558a8b5130f4845166d88654a0f9bdf3eb57851b7", size = 9212382, upload-time = "2025-09-09T08:20:54.731Z" }, + { url = "https://files.pythonhosted.org/packages/40/dd/9a88879b0c1104259136146e4742026b52df8540c39fec21a6383f8292c7/scikit_learn-1.7.2-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:abebbd61ad9e1deed54cca45caea8ad5f79e1b93173dece40bb8e0c658dbe6fe", size = 8592042, upload-time = "2025-09-09T08:20:57.313Z" }, + { url = "https://files.pythonhosted.org/packages/46/af/c5e286471b7d10871b811b72ae794ac5fe2989c0a2df07f0ec723030f5f5/scikit_learn-1.7.2-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:502c18e39849c0ea1a5d681af1dbcf15f6cce601aebb657aabbfe84133c1907f", size = 9434180, upload-time = "2025-09-09T08:20:59.671Z" }, + { url = "https://files.pythonhosted.org/packages/f1/fd/df59faa53312d585023b2da27e866524ffb8faf87a68516c23896c718320/scikit_learn-1.7.2-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7a4c328a71785382fe3fe676a9ecf2c86189249beff90bf85e22bdb7efaf9ae0", size = 9283660, upload-time = "2025-09-09T08:21:01.71Z" }, + { url = "https://files.pythonhosted.org/packages/a7/c7/03000262759d7b6f38c836ff9d512f438a70d8a8ddae68ee80de72dcfb63/scikit_learn-1.7.2-cp313-cp313-win_amd64.whl", hash = "sha256:63a9afd6f7b229aad94618c01c252ce9e6fa97918c5ca19c9a17a087d819440c", size = 8702057, upload-time = "2025-09-09T08:21:04.234Z" }, + { url = "https://files.pythonhosted.org/packages/55/87/ef5eb1f267084532c8e4aef98a28b6ffe7425acbfd64b5e2f2e066bc29b3/scikit_learn-1.7.2-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:9acb6c5e867447b4e1390930e3944a005e2cb115922e693c08a323421a6966e8", size = 9558731, upload-time = "2025-09-09T08:21:06.381Z" }, + { url = "https://files.pythonhosted.org/packages/93/f8/6c1e3fc14b10118068d7938878a9f3f4e6d7b74a8ddb1e5bed65159ccda8/scikit_learn-1.7.2-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:2a41e2a0ef45063e654152ec9d8bcfc39f7afce35b08902bfe290c2498a67a6a", size = 9038852, upload-time = "2025-09-09T08:21:08.628Z" }, + { url = "https://files.pythonhosted.org/packages/83/87/066cafc896ee540c34becf95d30375fe5cbe93c3b75a0ee9aa852cd60021/scikit_learn-1.7.2-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:98335fb98509b73385b3ab2bd0639b1f610541d3988ee675c670371d6a87aa7c", size = 9527094, upload-time = "2025-09-09T08:21:11.486Z" }, + { url = "https://files.pythonhosted.org/packages/9c/2b/4903e1ccafa1f6453b1ab78413938c8800633988c838aa0be386cbb33072/scikit_learn-1.7.2-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:191e5550980d45449126e23ed1d5e9e24b2c68329ee1f691a3987476e115e09c", size = 9367436, upload-time = "2025-09-09T08:21:13.602Z" }, + { url = "https://files.pythonhosted.org/packages/b5/aa/8444be3cfb10451617ff9d177b3c190288f4563e6c50ff02728be67ad094/scikit_learn-1.7.2-cp313-cp313t-win_amd64.whl", hash = "sha256:57dc4deb1d3762c75d685507fbd0bc17160144b2f2ba4ccea5dc285ab0d0e973", size = 9275749, upload-time = "2025-09-09T08:21:15.96Z" }, + { url = "https://files.pythonhosted.org/packages/d9/82/dee5acf66837852e8e68df6d8d3a6cb22d3df997b733b032f513d95205b7/scikit_learn-1.7.2-cp314-cp314-macosx_10_13_x86_64.whl", hash = "sha256:fa8f63940e29c82d1e67a45d5297bdebbcb585f5a5a50c4914cc2e852ab77f33", size = 9208906, upload-time = "2025-09-09T08:21:18.557Z" }, + { url = "https://files.pythonhosted.org/packages/3c/30/9029e54e17b87cb7d50d51a5926429c683d5b4c1732f0507a6c3bed9bf65/scikit_learn-1.7.2-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:f95dc55b7902b91331fa4e5845dd5bde0580c9cd9612b1b2791b7e80c3d32615", size = 8627836, upload-time = "2025-09-09T08:21:20.695Z" }, + { url = "https://files.pythonhosted.org/packages/60/18/4a52c635c71b536879f4b971c2cedf32c35ee78f48367885ed8025d1f7ee/scikit_learn-1.7.2-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:9656e4a53e54578ad10a434dc1f993330568cfee176dff07112b8785fb413106", size = 9426236, upload-time = "2025-09-09T08:21:22.645Z" }, + { url = "https://files.pythonhosted.org/packages/99/7e/290362f6ab582128c53445458a5befd471ed1ea37953d5bcf80604619250/scikit_learn-1.7.2-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96dc05a854add0e50d3f47a1ef21a10a595016da5b007c7d9cd9d0bffd1fcc61", size = 9312593, upload-time = "2025-09-09T08:21:24.65Z" }, + { url = "https://files.pythonhosted.org/packages/8e/87/24f541b6d62b1794939ae6422f8023703bbf6900378b2b34e0b4384dfefd/scikit_learn-1.7.2-cp314-cp314-win_amd64.whl", hash = "sha256:bb24510ed3f9f61476181e4db51ce801e2ba37541def12dc9333b946fc7a9cf8", size = 8820007, upload-time = "2025-09-09T08:21:26.713Z" }, +] + +[[package]] +name = "scikit-learn" +version = "1.8.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'linux'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'linux'", + "python_full_version == '3.11.*' and sys_platform == 'linux'", + "python_full_version >= '3.14' and sys_platform != 'linux'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'linux'", + "python_full_version == '3.11.*' and sys_platform != 'linux'", +] +dependencies = [ + { name = "joblib", marker = "python_full_version >= '3.11'" }, + { name = "numpy", version = "2.4.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "scipy", version = "1.17.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "threadpoolctl", marker = "python_full_version >= '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0e/d4/40988bf3b8e34feec1d0e6a051446b1f66225f8529b9309becaeef62b6c4/scikit_learn-1.8.0.tar.gz", hash = "sha256:9bccbb3b40e3de10351f8f5068e105d0f4083b1a65fa07b6634fbc401a6287fd", size = 7335585, upload-time = "2025-12-10T07:08:53.618Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/c9/92/53ea2181da8ac6bf27170191028aee7251f8f841f8d3edbfdcaf2008fde9/scikit_learn-1.8.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:146b4d36f800c013d267b29168813f7a03a43ecd2895d04861f1240b564421da", size = 8595835, upload-time = "2025-12-10T07:07:39.385Z" }, + { url = "https://files.pythonhosted.org/packages/01/18/d154dc1638803adf987910cdd07097d9c526663a55666a97c124d09fb96a/scikit_learn-1.8.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:f984ca4b14914e6b4094c5d52a32ea16b49832c03bd17a110f004db3c223e8e1", size = 8080381, upload-time = "2025-12-10T07:07:41.93Z" }, + { url = "https://files.pythonhosted.org/packages/8a/44/226142fcb7b7101e64fdee5f49dbe6288d4c7af8abf593237b70fca080a4/scikit_learn-1.8.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5e30adb87f0cc81c7690a84f7932dd66be5bac57cfe16b91cb9151683a4a2d3b", size = 8799632, upload-time = "2025-12-10T07:07:43.899Z" }, + { url = "https://files.pythonhosted.org/packages/36/4d/4a67f30778a45d542bbea5db2dbfa1e9e100bf9ba64aefe34215ba9f11f6/scikit_learn-1.8.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ada8121bcb4dac28d930febc791a69f7cb1673c8495e5eee274190b73a4559c1", size = 9103788, upload-time = "2025-12-10T07:07:45.982Z" }, + { url = "https://files.pythonhosted.org/packages/89/3c/45c352094cfa60050bcbb967b1faf246b22e93cb459f2f907b600f2ceda5/scikit_learn-1.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:c57b1b610bd1f40ba43970e11ce62821c2e6569e4d74023db19c6b26f246cb3b", size = 8081706, upload-time = "2025-12-10T07:07:48.111Z" }, + { url = "https://files.pythonhosted.org/packages/3d/46/5416595bb395757f754feb20c3d776553a386b661658fb21b7c814e89efe/scikit_learn-1.8.0-cp311-cp311-win_arm64.whl", hash = "sha256:2838551e011a64e3053ad7618dda9310175f7515f1742fa2d756f7c874c05961", size = 7688451, upload-time = "2025-12-10T07:07:49.873Z" }, + { url = "https://files.pythonhosted.org/packages/90/74/e6a7cc4b820e95cc38cf36cd74d5aa2b42e8ffc2d21fe5a9a9c45c1c7630/scikit_learn-1.8.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:5fb63362b5a7ddab88e52b6dbb47dac3fd7dafeee740dc6c8d8a446ddedade8e", size = 8548242, upload-time = "2025-12-10T07:07:51.568Z" }, + { url = "https://files.pythonhosted.org/packages/49/d8/9be608c6024d021041c7f0b3928d4749a706f4e2c3832bbede4fb4f58c95/scikit_learn-1.8.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:5025ce924beccb28298246e589c691fe1b8c1c96507e6d27d12c5fadd85bfd76", size = 8079075, upload-time = "2025-12-10T07:07:53.697Z" }, + { url = "https://files.pythonhosted.org/packages/dd/47/f187b4636ff80cc63f21cd40b7b2d177134acaa10f6bb73746130ee8c2e5/scikit_learn-1.8.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4496bb2cf7a43ce1a2d7524a79e40bc5da45cf598dbf9545b7e8316ccba47bb4", size = 8660492, upload-time = "2025-12-10T07:07:55.574Z" }, + { url = "https://files.pythonhosted.org/packages/97/74/b7a304feb2b49df9fafa9382d4d09061a96ee9a9449a7cbea7988dda0828/scikit_learn-1.8.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a0bcfe4d0d14aec44921545fd2af2338c7471de9cb701f1da4c9d85906ab847a", size = 8931904, upload-time = "2025-12-10T07:07:57.666Z" }, + { url = "https://files.pythonhosted.org/packages/9f/c4/0ab22726a04ede56f689476b760f98f8f46607caecff993017ac1b64aa5d/scikit_learn-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:35c007dedb2ffe38fe3ee7d201ebac4a2deccd2408e8621d53067733e3c74809", size = 8019359, upload-time = "2025-12-10T07:07:59.838Z" }, + { url = "https://files.pythonhosted.org/packages/24/90/344a67811cfd561d7335c1b96ca21455e7e472d281c3c279c4d3f2300236/scikit_learn-1.8.0-cp312-cp312-win_arm64.whl", hash = "sha256:8c497fff237d7b4e07e9ef1a640887fa4fb765647f86fbe00f969ff6280ce2bb", size = 7641898, upload-time = "2025-12-10T07:08:01.36Z" }, + { url = "https://files.pythonhosted.org/packages/03/aa/e22e0768512ce9255eba34775be2e85c2048da73da1193e841707f8f039c/scikit_learn-1.8.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0d6ae97234d5d7079dc0040990a6f7aeb97cb7fa7e8945f1999a429b23569e0a", size = 8513770, upload-time = "2025-12-10T07:08:03.251Z" }, + { url = "https://files.pythonhosted.org/packages/58/37/31b83b2594105f61a381fc74ca19e8780ee923be2d496fcd8d2e1147bd99/scikit_learn-1.8.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:edec98c5e7c128328124a029bceb09eda2d526997780fef8d65e9a69eead963e", size = 8044458, upload-time = "2025-12-10T07:08:05.336Z" }, + { url = "https://files.pythonhosted.org/packages/2d/5a/3f1caed8765f33eabb723596666da4ebbf43d11e96550fb18bdec42b467b/scikit_learn-1.8.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:74b66d8689d52ed04c271e1329f0c61635bcaf5b926db9b12d58914cdc01fe57", size = 8610341, upload-time = "2025-12-10T07:08:07.732Z" }, + { url = "https://files.pythonhosted.org/packages/38/cf/06896db3f71c75902a8e9943b444a56e727418f6b4b4a90c98c934f51ed4/scikit_learn-1.8.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8fdf95767f989b0cfedb85f7ed8ca215d4be728031f56ff5a519ee1e3276dc2e", size = 8900022, upload-time = "2025-12-10T07:08:09.862Z" }, + { url = "https://files.pythonhosted.org/packages/1c/f9/9b7563caf3ec8873e17a31401858efab6b39a882daf6c1bfa88879c0aa11/scikit_learn-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:2de443b9373b3b615aec1bb57f9baa6bb3a9bd093f1269ba95c17d870422b271", size = 7989409, upload-time = "2025-12-10T07:08:12.028Z" }, + { url = "https://files.pythonhosted.org/packages/49/bd/1f4001503650e72c4f6009ac0c4413cb17d2d601cef6f71c0453da2732fc/scikit_learn-1.8.0-cp313-cp313-win_arm64.whl", hash = "sha256:eddde82a035681427cbedded4e6eff5e57fa59216c2e3e90b10b19ab1d0a65c3", size = 7619760, upload-time = "2025-12-10T07:08:13.688Z" }, + { url = "https://files.pythonhosted.org/packages/d2/7d/a630359fc9dcc95496588c8d8e3245cc8fd81980251079bc09c70d41d951/scikit_learn-1.8.0-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:7cc267b6108f0a1499a734167282c00c4ebf61328566b55ef262d48e9849c735", size = 8826045, upload-time = "2025-12-10T07:08:15.215Z" }, + { url = "https://files.pythonhosted.org/packages/cc/56/a0c86f6930cfcd1c7054a2bc417e26960bb88d32444fe7f71d5c2cfae891/scikit_learn-1.8.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:fe1c011a640a9f0791146011dfd3c7d9669785f9fed2b2a5f9e207536cf5c2fd", size = 8420324, upload-time = "2025-12-10T07:08:17.561Z" }, + { url = "https://files.pythonhosted.org/packages/46/1e/05962ea1cebc1cf3876667ecb14c283ef755bf409993c5946ade3b77e303/scikit_learn-1.8.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:72358cce49465d140cc4e7792015bb1f0296a9742d5622c67e31399b75468b9e", size = 8680651, upload-time = "2025-12-10T07:08:19.952Z" }, + { url = "https://files.pythonhosted.org/packages/fe/56/a85473cd75f200c9759e3a5f0bcab2d116c92a8a02ee08ccd73b870f8bb4/scikit_learn-1.8.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:80832434a6cc114f5219211eec13dcbc16c2bac0e31ef64c6d346cde3cf054cb", size = 8925045, upload-time = "2025-12-10T07:08:22.11Z" }, + { url = "https://files.pythonhosted.org/packages/cc/b7/64d8cfa896c64435ae57f4917a548d7ac7a44762ff9802f75a79b77cb633/scikit_learn-1.8.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ee787491dbfe082d9c3013f01f5991658b0f38aa8177e4cd4bf434c58f551702", size = 8507994, upload-time = "2025-12-10T07:08:23.943Z" }, + { url = "https://files.pythonhosted.org/packages/5e/37/e192ea709551799379958b4c4771ec507347027bb7c942662c7fbeba31cb/scikit_learn-1.8.0-cp313-cp313t-win_arm64.whl", hash = "sha256:bf97c10a3f5a7543f9b88cbf488d33d175e9146115a451ae34568597ba33dcde", size = 7869518, upload-time = "2025-12-10T07:08:25.71Z" }, + { url = "https://files.pythonhosted.org/packages/24/05/1af2c186174cc92dcab2233f327336058c077d38f6fe2aceb08e6ab4d509/scikit_learn-1.8.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c22a2da7a198c28dd1a6e1136f19c830beab7fdca5b3e5c8bba8394f8a5c45b3", size = 8528667, upload-time = "2025-12-10T07:08:27.541Z" }, + { url = "https://files.pythonhosted.org/packages/a8/25/01c0af38fe969473fb292bba9dc2b8f9b451f3112ff242c647fee3d0dfe7/scikit_learn-1.8.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:6b595b07a03069a2b1740dc08c2299993850ea81cce4fe19b2421e0c970de6b7", size = 8066524, upload-time = "2025-12-10T07:08:29.822Z" }, + { url = "https://files.pythonhosted.org/packages/be/ce/a0623350aa0b68647333940ee46fe45086c6060ec604874e38e9ab7d8e6c/scikit_learn-1.8.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:29ffc74089f3d5e87dfca4c2c8450f88bdc61b0fc6ed5d267f3988f19a1309f6", size = 8657133, upload-time = "2025-12-10T07:08:31.865Z" }, + { url = "https://files.pythonhosted.org/packages/b8/cb/861b41341d6f1245e6ca80b1c1a8c4dfce43255b03df034429089ca2a2c5/scikit_learn-1.8.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:fb65db5d7531bccf3a4f6bec3462223bea71384e2cda41da0f10b7c292b9e7c4", size = 8923223, upload-time = "2025-12-10T07:08:34.166Z" }, + { url = "https://files.pythonhosted.org/packages/76/18/a8def8f91b18cd1ba6e05dbe02540168cb24d47e8dcf69e8d00b7da42a08/scikit_learn-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:56079a99c20d230e873ea40753102102734c5953366972a71d5cb39a32bc40c6", size = 8096518, upload-time = "2025-12-10T07:08:36.339Z" }, + { url = "https://files.pythonhosted.org/packages/d1/77/482076a678458307f0deb44e29891d6022617b2a64c840c725495bee343f/scikit_learn-1.8.0-cp314-cp314-win_arm64.whl", hash = "sha256:3bad7565bc9cf37ce19a7c0d107742b320c1285df7aab1a6e2d28780df167242", size = 7754546, upload-time = "2025-12-10T07:08:38.128Z" }, + { url = "https://files.pythonhosted.org/packages/2d/d1/ef294ca754826daa043b2a104e59960abfab4cf653891037d19dd5b6f3cf/scikit_learn-1.8.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:4511be56637e46c25721e83d1a9cea9614e7badc7040c4d573d75fbe257d6fd7", size = 8848305, upload-time = "2025-12-10T07:08:41.013Z" }, + { url = "https://files.pythonhosted.org/packages/5b/e2/b1f8b05138ee813b8e1a4149f2f0d289547e60851fd1bb268886915adbda/scikit_learn-1.8.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:a69525355a641bf8ef136a7fa447672fb54fe8d60cab5538d9eb7c6438543fb9", size = 8432257, upload-time = "2025-12-10T07:08:42.873Z" }, + { url = "https://files.pythonhosted.org/packages/26/11/c32b2138a85dcb0c99f6afd13a70a951bfdff8a6ab42d8160522542fb647/scikit_learn-1.8.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c2656924ec73e5939c76ac4c8b026fc203b83d8900362eb2599d8aee80e4880f", size = 8678673, upload-time = "2025-12-10T07:08:45.362Z" }, + { url = "https://files.pythonhosted.org/packages/c7/57/51f2384575bdec454f4fe4e7a919d696c9ebce914590abf3e52d47607ab8/scikit_learn-1.8.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:15fc3b5d19cc2be65404786857f2e13c70c83dd4782676dd6814e3b89dc8f5b9", size = 8922467, upload-time = "2025-12-10T07:08:47.408Z" }, + { url = "https://files.pythonhosted.org/packages/35/4d/748c9e2872637a57981a04adc038dacaa16ba8ca887b23e34953f0b3f742/scikit_learn-1.8.0-cp314-cp314t-win_amd64.whl", hash = "sha256:00d6f1d66fbcf4eba6e356e1420d33cc06c70a45bb1363cd6f6a8e4ebbbdece2", size = 8774395, upload-time = "2025-12-10T07:08:49.337Z" }, + { url = "https://files.pythonhosted.org/packages/60/22/d7b2ebe4704a5e50790ba089d5c2ae308ab6bb852719e6c3bd4f04c3a363/scikit_learn-1.8.0-cp314-cp314t-win_arm64.whl", hash = "sha256:f28dd15c6bb0b66ba09728cf09fd8736c304be29409bd8445a080c1280619e8c", size = 8002647, upload-time = "2025-12-10T07:08:51.601Z" }, +] + +[[package]] +name = "scipy" +version = "1.15.3" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version < '3.11' and sys_platform == 'linux'", + "python_full_version < '3.11' and sys_platform != 'linux'", +] +dependencies = [ + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/0f/37/6964b830433e654ec7485e45a00fc9a27cf868d622838f6b6d9c5ec0d532/scipy-1.15.3.tar.gz", hash = "sha256:eae3cf522bc7df64b42cad3925c876e1b0b6c35c1337c93e12c0f366f55b0eaf", size = 59419214, upload-time = "2025-05-08T16:13:05.955Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/78/2f/4966032c5f8cc7e6a60f1b2e0ad686293b9474b65246b0c642e3ef3badd0/scipy-1.15.3-cp310-cp310-macosx_10_13_x86_64.whl", hash = "sha256:a345928c86d535060c9c2b25e71e87c39ab2f22fc96e9636bd74d1dbf9de448c", size = 38702770, upload-time = "2025-05-08T16:04:20.849Z" }, + { url = "https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:ad3432cb0f9ed87477a8d97f03b763fd1d57709f1bbde3c9369b1dff5503b253", size = 30094511, upload-time = "2025-05-08T16:04:27.103Z" }, + { url = "https://files.pythonhosted.org/packages/ea/b1/4deb37252311c1acff7f101f6453f0440794f51b6eacb1aad4459a134081/scipy-1.15.3-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:aef683a9ae6eb00728a542b796f52a5477b78252edede72b8327a886ab63293f", size = 22368151, upload-time = "2025-05-08T16:04:31.731Z" }, + { url = "https://files.pythonhosted.org/packages/38/7d/f457626e3cd3c29b3a49ca115a304cebb8cc6f31b04678f03b216899d3c6/scipy-1.15.3-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:1c832e1bd78dea67d5c16f786681b28dd695a8cb1fb90af2e27580d3d0967e92", size = 25121732, upload-time = "2025-05-08T16:04:36.596Z" }, + { url = "https://files.pythonhosted.org/packages/db/0a/92b1de4a7adc7a15dcf5bddc6e191f6f29ee663b30511ce20467ef9b82e4/scipy-1.15.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:263961f658ce2165bbd7b99fa5135195c3a12d9bef045345016b8b50c315cb82", size = 35547617, upload-time = "2025-05-08T16:04:43.546Z" }, + { url = "https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9e2abc762b0811e09a0d3258abee2d98e0c703eee49464ce0069590846f31d40", size = 37662964, upload-time = "2025-05-08T16:04:49.431Z" }, + { url = "https://files.pythonhosted.org/packages/25/e1/3df8f83cb15f3500478c889be8fb18700813b95e9e087328230b98d547ff/scipy-1.15.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:ed7284b21a7a0c8f1b6e5977ac05396c0d008b89e05498c8b7e8f4a1423bba0e", size = 37238749, upload-time = "2025-05-08T16:04:55.215Z" }, + { url = "https://files.pythonhosted.org/packages/93/3e/b3257cf446f2a3533ed7809757039016b74cd6f38271de91682aa844cfc5/scipy-1.15.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:5380741e53df2c566f4d234b100a484b420af85deb39ea35a1cc1be84ff53a5c", size = 40022383, upload-time = "2025-05-08T16:05:01.914Z" }, + { url = "https://files.pythonhosted.org/packages/d1/84/55bc4881973d3f79b479a5a2e2df61c8c9a04fcb986a213ac9c02cfb659b/scipy-1.15.3-cp310-cp310-win_amd64.whl", hash = "sha256:9d61e97b186a57350f6d6fd72640f9e99d5a4a2b8fbf4b9ee9a841eab327dc13", size = 41259201, upload-time = "2025-05-08T16:05:08.166Z" }, + { url = "https://files.pythonhosted.org/packages/96/ab/5cc9f80f28f6a7dff646c5756e559823614a42b1939d86dd0ed550470210/scipy-1.15.3-cp311-cp311-macosx_10_13_x86_64.whl", hash = "sha256:993439ce220d25e3696d1b23b233dd010169b62f6456488567e830654ee37a6b", size = 38714255, upload-time = "2025-05-08T16:05:14.596Z" }, + { url = "https://files.pythonhosted.org/packages/4a/4a/66ba30abe5ad1a3ad15bfb0b59d22174012e8056ff448cb1644deccbfed2/scipy-1.15.3-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:34716e281f181a02341ddeaad584205bd2fd3c242063bd3423d61ac259ca7eba", size = 30111035, upload-time = "2025-05-08T16:05:20.152Z" }, + { url = "https://files.pythonhosted.org/packages/4b/fa/a7e5b95afd80d24313307f03624acc65801846fa75599034f8ceb9e2cbf6/scipy-1.15.3-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:3b0334816afb8b91dab859281b1b9786934392aa3d527cd847e41bb6f45bee65", size = 22384499, upload-time = "2025-05-08T16:05:24.494Z" }, + { url = "https://files.pythonhosted.org/packages/17/99/f3aaddccf3588bb4aea70ba35328c204cadd89517a1612ecfda5b2dd9d7a/scipy-1.15.3-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:6db907c7368e3092e24919b5e31c76998b0ce1684d51a90943cb0ed1b4ffd6c1", size = 25152602, upload-time = "2025-05-08T16:05:29.313Z" }, + { url = "https://files.pythonhosted.org/packages/56/c5/1032cdb565f146109212153339f9cb8b993701e9fe56b1c97699eee12586/scipy-1.15.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:721d6b4ef5dc82ca8968c25b111e307083d7ca9091bc38163fb89243e85e3889", size = 35503415, upload-time = "2025-05-08T16:05:34.699Z" }, + { url = "https://files.pythonhosted.org/packages/bd/37/89f19c8c05505d0601ed5650156e50eb881ae3918786c8fd7262b4ee66d3/scipy-1.15.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:39cb9c62e471b1bb3750066ecc3a3f3052b37751c7c3dfd0fd7e48900ed52982", size = 37652622, upload-time = "2025-05-08T16:05:40.762Z" }, + { url = "https://files.pythonhosted.org/packages/7e/31/be59513aa9695519b18e1851bb9e487de66f2d31f835201f1b42f5d4d475/scipy-1.15.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:795c46999bae845966368a3c013e0e00947932d68e235702b5c3f6ea799aa8c9", size = 37244796, upload-time = "2025-05-08T16:05:48.119Z" }, + { url = "https://files.pythonhosted.org/packages/10/c0/4f5f3eeccc235632aab79b27a74a9130c6c35df358129f7ac8b29f562ac7/scipy-1.15.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:18aaacb735ab38b38db42cb01f6b92a2d0d4b6aabefeb07f02849e47f8fb3594", size = 40047684, upload-time = "2025-05-08T16:05:54.22Z" }, + { url = "https://files.pythonhosted.org/packages/ab/a7/0ddaf514ce8a8714f6ed243a2b391b41dbb65251affe21ee3077ec45ea9a/scipy-1.15.3-cp311-cp311-win_amd64.whl", hash = "sha256:ae48a786a28412d744c62fd7816a4118ef97e5be0bee968ce8f0a2fba7acf3bb", size = 41246504, upload-time = "2025-05-08T16:06:00.437Z" }, + { url = "https://files.pythonhosted.org/packages/37/4b/683aa044c4162e10ed7a7ea30527f2cbd92e6999c10a8ed8edb253836e9c/scipy-1.15.3-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:6ac6310fdbfb7aa6612408bd2f07295bcbd3fda00d2d702178434751fe48e019", size = 38766735, upload-time = "2025-05-08T16:06:06.471Z" }, + { url = "https://files.pythonhosted.org/packages/7b/7e/f30be3d03de07f25dc0ec926d1681fed5c732d759ac8f51079708c79e680/scipy-1.15.3-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:185cd3d6d05ca4b44a8f1595af87f9c372bb6acf9c808e99aa3e9aa03bd98cf6", size = 30173284, upload-time = "2025-05-08T16:06:11.686Z" }, + { url = "https://files.pythonhosted.org/packages/07/9c/0ddb0d0abdabe0d181c1793db51f02cd59e4901da6f9f7848e1f96759f0d/scipy-1.15.3-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:05dc6abcd105e1a29f95eada46d4a3f251743cfd7d3ae8ddb4088047f24ea477", size = 22446958, upload-time = "2025-05-08T16:06:15.97Z" }, + { url = "https://files.pythonhosted.org/packages/af/43/0bce905a965f36c58ff80d8bea33f1f9351b05fad4beaad4eae34699b7a1/scipy-1.15.3-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:06efcba926324df1696931a57a176c80848ccd67ce6ad020c810736bfd58eb1c", size = 25242454, upload-time = "2025-05-08T16:06:20.394Z" }, + { url = "https://files.pythonhosted.org/packages/56/30/a6f08f84ee5b7b28b4c597aca4cbe545535c39fe911845a96414700b64ba/scipy-1.15.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:c05045d8b9bfd807ee1b9f38761993297b10b245f012b11b13b91ba8945f7e45", size = 35210199, upload-time = "2025-05-08T16:06:26.159Z" }, + { url = "https://files.pythonhosted.org/packages/0b/1f/03f52c282437a168ee2c7c14a1a0d0781a9a4a8962d84ac05c06b4c5b555/scipy-1.15.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:271e3713e645149ea5ea3e97b57fdab61ce61333f97cfae392c28ba786f9bb49", size = 37309455, upload-time = "2025-05-08T16:06:32.778Z" }, + { url = "https://files.pythonhosted.org/packages/89/b1/fbb53137f42c4bf630b1ffdfc2151a62d1d1b903b249f030d2b1c0280af8/scipy-1.15.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:6cfd56fc1a8e53f6e89ba3a7a7251f7396412d655bca2aa5611c8ec9a6784a1e", size = 36885140, upload-time = "2025-05-08T16:06:39.249Z" }, + { url = "https://files.pythonhosted.org/packages/2e/2e/025e39e339f5090df1ff266d021892694dbb7e63568edcfe43f892fa381d/scipy-1.15.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0ff17c0bb1cb32952c09217d8d1eed9b53d1463e5f1dd6052c7857f83127d539", size = 39710549, upload-time = "2025-05-08T16:06:45.729Z" }, + { url = "https://files.pythonhosted.org/packages/e6/eb/3bf6ea8ab7f1503dca3a10df2e4b9c3f6b3316df07f6c0ded94b281c7101/scipy-1.15.3-cp312-cp312-win_amd64.whl", hash = "sha256:52092bc0472cfd17df49ff17e70624345efece4e1a12b23783a1ac59a1b728ed", size = 40966184, upload-time = "2025-05-08T16:06:52.623Z" }, + { url = "https://files.pythonhosted.org/packages/73/18/ec27848c9baae6e0d6573eda6e01a602e5649ee72c27c3a8aad673ebecfd/scipy-1.15.3-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:2c620736bcc334782e24d173c0fdbb7590a0a436d2fdf39310a8902505008759", size = 38728256, upload-time = "2025-05-08T16:06:58.696Z" }, + { url = "https://files.pythonhosted.org/packages/74/cd/1aef2184948728b4b6e21267d53b3339762c285a46a274ebb7863c9e4742/scipy-1.15.3-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:7e11270a000969409d37ed399585ee530b9ef6aa99d50c019de4cb01e8e54e62", size = 30109540, upload-time = "2025-05-08T16:07:04.209Z" }, + { url = "https://files.pythonhosted.org/packages/5b/d8/59e452c0a255ec352bd0a833537a3bc1bfb679944c4938ab375b0a6b3a3e/scipy-1.15.3-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:8c9ed3ba2c8a2ce098163a9bdb26f891746d02136995df25227a20e71c396ebb", size = 22383115, upload-time = "2025-05-08T16:07:08.998Z" }, + { url = "https://files.pythonhosted.org/packages/08/f5/456f56bbbfccf696263b47095291040655e3cbaf05d063bdc7c7517f32ac/scipy-1.15.3-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:0bdd905264c0c9cfa74a4772cdb2070171790381a5c4d312c973382fc6eaf730", size = 25163884, upload-time = "2025-05-08T16:07:14.091Z" }, + { url = "https://files.pythonhosted.org/packages/a2/66/a9618b6a435a0f0c0b8a6d0a2efb32d4ec5a85f023c2b79d39512040355b/scipy-1.15.3-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:79167bba085c31f38603e11a267d862957cbb3ce018d8b38f79ac043bc92d825", size = 35174018, upload-time = "2025-05-08T16:07:19.427Z" }, + { url = "https://files.pythonhosted.org/packages/b5/09/c5b6734a50ad4882432b6bb7c02baf757f5b2f256041da5df242e2d7e6b6/scipy-1.15.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:c9deabd6d547aee2c9a81dee6cc96c6d7e9a9b1953f74850c179f91fdc729cb7", size = 37269716, upload-time = "2025-05-08T16:07:25.712Z" }, + { url = "https://files.pythonhosted.org/packages/77/0a/eac00ff741f23bcabd352731ed9b8995a0a60ef57f5fd788d611d43d69a1/scipy-1.15.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:dde4fc32993071ac0c7dd2d82569e544f0bdaff66269cb475e0f369adad13f11", size = 36872342, upload-time = "2025-05-08T16:07:31.468Z" }, + { url = "https://files.pythonhosted.org/packages/fe/54/4379be86dd74b6ad81551689107360d9a3e18f24d20767a2d5b9253a3f0a/scipy-1.15.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f77f853d584e72e874d87357ad70f44b437331507d1c311457bed8ed2b956126", size = 39670869, upload-time = "2025-05-08T16:07:38.002Z" }, + { url = "https://files.pythonhosted.org/packages/87/2e/892ad2862ba54f084ffe8cc4a22667eaf9c2bcec6d2bff1d15713c6c0703/scipy-1.15.3-cp313-cp313-win_amd64.whl", hash = "sha256:b90ab29d0c37ec9bf55424c064312930ca5f4bde15ee8619ee44e69319aab163", size = 40988851, upload-time = "2025-05-08T16:08:33.671Z" }, + { url = "https://files.pythonhosted.org/packages/1b/e9/7a879c137f7e55b30d75d90ce3eb468197646bc7b443ac036ae3fe109055/scipy-1.15.3-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:3ac07623267feb3ae308487c260ac684b32ea35fd81e12845039952f558047b8", size = 38863011, upload-time = "2025-05-08T16:07:44.039Z" }, + { url = "https://files.pythonhosted.org/packages/51/d1/226a806bbd69f62ce5ef5f3ffadc35286e9fbc802f606a07eb83bf2359de/scipy-1.15.3-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:6487aa99c2a3d509a5227d9a5e889ff05830a06b2ce08ec30df6d79db5fcd5c5", size = 30266407, upload-time = "2025-05-08T16:07:49.891Z" }, + { url = "https://files.pythonhosted.org/packages/e5/9b/f32d1d6093ab9eeabbd839b0f7619c62e46cc4b7b6dbf05b6e615bbd4400/scipy-1.15.3-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:50f9e62461c95d933d5c5ef4a1f2ebf9a2b4e83b0db374cb3f1de104d935922e", size = 22540030, upload-time = "2025-05-08T16:07:54.121Z" }, + { url = "https://files.pythonhosted.org/packages/e7/29/c278f699b095c1a884f29fda126340fcc201461ee8bfea5c8bdb1c7c958b/scipy-1.15.3-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:14ed70039d182f411ffc74789a16df3835e05dc469b898233a245cdfd7f162cb", size = 25218709, upload-time = "2025-05-08T16:07:58.506Z" }, + { url = "https://files.pythonhosted.org/packages/24/18/9e5374b617aba742a990581373cd6b68a2945d65cc588482749ef2e64467/scipy-1.15.3-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0a769105537aa07a69468a0eefcd121be52006db61cdd8cac8a0e68980bbb723", size = 34809045, upload-time = "2025-05-08T16:08:03.929Z" }, + { url = "https://files.pythonhosted.org/packages/e1/fe/9c4361e7ba2927074360856db6135ef4904d505e9b3afbbcb073c4008328/scipy-1.15.3-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9db984639887e3dffb3928d118145ffe40eff2fa40cb241a306ec57c219ebbbb", size = 36703062, upload-time = "2025-05-08T16:08:09.558Z" }, + { url = "https://files.pythonhosted.org/packages/b7/8e/038ccfe29d272b30086b25a4960f757f97122cb2ec42e62b460d02fe98e9/scipy-1.15.3-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:40e54d5c7e7ebf1aa596c374c49fa3135f04648a0caabcb66c52884b943f02b4", size = 36393132, upload-time = "2025-05-08T16:08:15.34Z" }, + { url = "https://files.pythonhosted.org/packages/10/7e/5c12285452970be5bdbe8352c619250b97ebf7917d7a9a9e96b8a8140f17/scipy-1.15.3-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:5e721fed53187e71d0ccf382b6bf977644c533e506c4d33c3fb24de89f5c3ed5", size = 38979503, upload-time = "2025-05-08T16:08:21.513Z" }, + { url = "https://files.pythonhosted.org/packages/81/06/0a5e5349474e1cbc5757975b21bd4fad0e72ebf138c5592f191646154e06/scipy-1.15.3-cp313-cp313t-win_amd64.whl", hash = "sha256:76ad1fb5f8752eabf0fa02e4cc0336b4e8f021e2d5f061ed37d6d264db35e3ca", size = 40308097, upload-time = "2025-05-08T16:08:27.627Z" }, +] + +[[package]] +name = "scipy" +version = "1.17.0" +source = { registry = "https://pypi.org/simple" } +resolution-markers = [ + "python_full_version >= '3.14' and sys_platform == 'linux'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform == 'linux'", + "python_full_version == '3.11.*' and sys_platform == 'linux'", + "python_full_version >= '3.14' and sys_platform != 'linux'", + "python_full_version >= '3.12' and python_full_version < '3.14' and sys_platform != 'linux'", + "python_full_version == '3.11.*' and sys_platform != 'linux'", +] +dependencies = [ + { name = "numpy", version = "2.4.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/56/3e/9cca699f3486ce6bc12ff46dc2031f1ec8eb9ccc9a320fdaf925f1417426/scipy-1.17.0.tar.gz", hash = "sha256:2591060c8e648d8b96439e111ac41fd8342fdeff1876be2e19dea3fe8930454e", size = 30396830, upload-time = "2026-01-10T21:34:23.009Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/1e/4b/c89c131aa87cad2b77a54eb0fb94d633a842420fa7e919dc2f922037c3d8/scipy-1.17.0-cp311-cp311-macosx_10_14_x86_64.whl", hash = "sha256:2abd71643797bd8a106dff97894ff7869eeeb0af0f7a5ce02e4227c6a2e9d6fd", size = 31381316, upload-time = "2026-01-10T21:24:33.42Z" }, + { url = "https://files.pythonhosted.org/packages/5e/5f/a6b38f79a07d74989224d5f11b55267714707582908a5f1ae854cf9a9b84/scipy-1.17.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:ef28d815f4d2686503e5f4f00edc387ae58dfd7a2f42e348bb53359538f01558", size = 27966760, upload-time = "2026-01-10T21:24:38.911Z" }, + { url = "https://files.pythonhosted.org/packages/c1/20/095ad24e031ee8ed3c5975954d816b8e7e2abd731e04f8be573de8740885/scipy-1.17.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:272a9f16d6bb4667e8b50d25d71eddcc2158a214df1b566319298de0939d2ab7", size = 20138701, upload-time = "2026-01-10T21:24:43.249Z" }, + { url = "https://files.pythonhosted.org/packages/89/11/4aad2b3858d0337756f3323f8960755704e530b27eb2a94386c970c32cbe/scipy-1.17.0-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:7204fddcbec2fe6598f1c5fdf027e9f259106d05202a959a9f1aecf036adc9f6", size = 22480574, upload-time = "2026-01-10T21:24:47.266Z" }, + { url = "https://files.pythonhosted.org/packages/85/bd/f5af70c28c6da2227e510875cadf64879855193a687fb19951f0f44cfd6b/scipy-1.17.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fc02c37a5639ee67d8fb646ffded6d793c06c5622d36b35cfa8fe5ececb8f042", size = 32862414, upload-time = "2026-01-10T21:24:52.566Z" }, + { url = "https://files.pythonhosted.org/packages/ef/df/df1457c4df3826e908879fe3d76bc5b6e60aae45f4ee42539512438cfd5d/scipy-1.17.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dac97a27520d66c12a34fd90a4fe65f43766c18c0d6e1c0a80f114d2260080e4", size = 35112380, upload-time = "2026-01-10T21:24:58.433Z" }, + { url = "https://files.pythonhosted.org/packages/5f/bb/88e2c16bd1dd4de19d80d7c5e238387182993c2fb13b4b8111e3927ad422/scipy-1.17.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:ebb7446a39b3ae0fe8f416a9a3fdc6fba3f11c634f680f16a239c5187bc487c0", size = 34922676, upload-time = "2026-01-10T21:25:04.287Z" }, + { url = "https://files.pythonhosted.org/packages/02/ba/5120242cc735f71fc002cff0303d536af4405eb265f7c60742851e7ccfe9/scipy-1.17.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:474da16199f6af66601a01546144922ce402cb17362e07d82f5a6cf8f963e449", size = 37507599, upload-time = "2026-01-10T21:25:09.851Z" }, + { url = "https://files.pythonhosted.org/packages/52/c8/08629657ac6c0da198487ce8cd3de78e02cfde42b7f34117d56a3fe249dc/scipy-1.17.0-cp311-cp311-win_amd64.whl", hash = "sha256:255c0da161bd7b32a6c898e7891509e8a9289f0b1c6c7d96142ee0d2b114c2ea", size = 36380284, upload-time = "2026-01-10T21:25:15.632Z" }, + { url = "https://files.pythonhosted.org/packages/6c/4a/465f96d42c6f33ad324a40049dfd63269891db9324aa66c4a1c108c6f994/scipy-1.17.0-cp311-cp311-win_arm64.whl", hash = "sha256:85b0ac3ad17fa3be50abd7e69d583d98792d7edc08367e01445a1e2076005379", size = 24370427, upload-time = "2026-01-10T21:25:20.514Z" }, + { url = "https://files.pythonhosted.org/packages/0b/11/7241a63e73ba5a516f1930ac8d5b44cbbfabd35ac73a2d08ca206df007c4/scipy-1.17.0-cp312-cp312-macosx_10_14_x86_64.whl", hash = "sha256:0d5018a57c24cb1dd828bcf51d7b10e65986d549f52ef5adb6b4d1ded3e32a57", size = 31364580, upload-time = "2026-01-10T21:25:25.717Z" }, + { url = "https://files.pythonhosted.org/packages/ed/1d/5057f812d4f6adc91a20a2d6f2ebcdb517fdbc87ae3acc5633c9b97c8ba5/scipy-1.17.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:88c22af9e5d5a4f9e027e26772cc7b5922fab8bcc839edb3ae33de404feebd9e", size = 27969012, upload-time = "2026-01-10T21:25:30.921Z" }, + { url = "https://files.pythonhosted.org/packages/e3/21/f6ec556c1e3b6ec4e088da667d9987bb77cc3ab3026511f427dc8451187d/scipy-1.17.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:f3cd947f20fe17013d401b64e857c6b2da83cae567adbb75b9dcba865abc66d8", size = 20140691, upload-time = "2026-01-10T21:25:34.802Z" }, + { url = "https://files.pythonhosted.org/packages/7a/fe/5e5ad04784964ba964a96f16c8d4676aa1b51357199014dce58ab7ec5670/scipy-1.17.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:e8c0b331c2c1f531eb51f1b4fc9ba709521a712cce58f1aa627bc007421a5306", size = 22463015, upload-time = "2026-01-10T21:25:39.277Z" }, + { url = "https://files.pythonhosted.org/packages/4a/69/7c347e857224fcaf32a34a05183b9d8a7aca25f8f2d10b8a698b8388561a/scipy-1.17.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5194c445d0a1c7a6c1a4a4681b6b7c71baad98ff66d96b949097e7513c9d6742", size = 32724197, upload-time = "2026-01-10T21:25:44.084Z" }, + { url = "https://files.pythonhosted.org/packages/d1/fe/66d73b76d378ba8cc2fe605920c0c75092e3a65ae746e1e767d9d020a75a/scipy-1.17.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9eeb9b5f5997f75507814ed9d298ab23f62cf79f5a3ef90031b1ee2506abdb5b", size = 35009148, upload-time = "2026-01-10T21:25:50.591Z" }, + { url = "https://files.pythonhosted.org/packages/af/07/07dec27d9dc41c18d8c43c69e9e413431d20c53a0339c388bcf72f353c4b/scipy-1.17.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:40052543f7bbe921df4408f46003d6f01c6af109b9e2c8a66dd1cf6cf57f7d5d", size = 34798766, upload-time = "2026-01-10T21:25:59.41Z" }, + { url = "https://files.pythonhosted.org/packages/81/61/0470810c8a093cdacd4ba7504b8a218fd49ca070d79eca23a615f5d9a0b0/scipy-1.17.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:0cf46c8013fec9d3694dc572f0b54100c28405d55d3e2cb15e2895b25057996e", size = 37405953, upload-time = "2026-01-10T21:26:07.75Z" }, + { url = "https://files.pythonhosted.org/packages/92/ce/672ed546f96d5d41ae78c4b9b02006cedd0b3d6f2bf5bb76ea455c320c28/scipy-1.17.0-cp312-cp312-win_amd64.whl", hash = "sha256:0937a0b0d8d593a198cededd4c439a0ea216a3f36653901ea1f3e4be949056f8", size = 36328121, upload-time = "2026-01-10T21:26:16.509Z" }, + { url = "https://files.pythonhosted.org/packages/9d/21/38165845392cae67b61843a52c6455d47d0cc2a40dd495c89f4362944654/scipy-1.17.0-cp312-cp312-win_arm64.whl", hash = "sha256:f603d8a5518c7426414d1d8f82e253e454471de682ce5e39c29adb0df1efb86b", size = 24314368, upload-time = "2026-01-10T21:26:23.087Z" }, + { url = "https://files.pythonhosted.org/packages/0c/51/3468fdfd49387ddefee1636f5cf6d03ce603b75205bf439bbf0e62069bfd/scipy-1.17.0-cp313-cp313-macosx_10_14_x86_64.whl", hash = "sha256:65ec32f3d32dfc48c72df4291345dae4f048749bc8d5203ee0a3f347f96c5ce6", size = 31344101, upload-time = "2026-01-10T21:26:30.25Z" }, + { url = "https://files.pythonhosted.org/packages/b2/9a/9406aec58268d437636069419e6977af953d1e246df941d42d3720b7277b/scipy-1.17.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:1f9586a58039d7229ce77b52f8472c972448cded5736eaf102d5658bbac4c269", size = 27950385, upload-time = "2026-01-10T21:26:36.801Z" }, + { url = "https://files.pythonhosted.org/packages/4f/98/e7342709e17afdfd1b26b56ae499ef4939b45a23a00e471dfb5375eea205/scipy-1.17.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:9fad7d3578c877d606b1150135c2639e9de9cecd3705caa37b66862977cc3e72", size = 20122115, upload-time = "2026-01-10T21:26:42.107Z" }, + { url = "https://files.pythonhosted.org/packages/fd/0e/9eeeb5357a64fd157cbe0302c213517c541cc16b8486d82de251f3c68ede/scipy-1.17.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:423ca1f6584fc03936972b5f7c06961670dbba9f234e71676a7c7ccf938a0d61", size = 22442402, upload-time = "2026-01-10T21:26:48.029Z" }, + { url = "https://files.pythonhosted.org/packages/c9/10/be13397a0e434f98e0c79552b2b584ae5bb1c8b2be95db421533bbca5369/scipy-1.17.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fe508b5690e9eaaa9467fc047f833af58f1152ae51a0d0aed67aa5801f4dd7d6", size = 32696338, upload-time = "2026-01-10T21:26:55.521Z" }, + { url = "https://files.pythonhosted.org/packages/63/1e/12fbf2a3bb240161651c94bb5cdd0eae5d4e8cc6eaeceb74ab07b12a753d/scipy-1.17.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6680f2dfd4f6182e7d6db161344537da644d1cf85cf293f015c60a17ecf08752", size = 34977201, upload-time = "2026-01-10T21:27:03.501Z" }, + { url = "https://files.pythonhosted.org/packages/19/5b/1a63923e23ccd20bd32156d7dd708af5bbde410daa993aa2500c847ab2d2/scipy-1.17.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:eec3842ec9ac9de5917899b277428886042a93db0b227ebbe3a333b64ec7643d", size = 34777384, upload-time = "2026-01-10T21:27:11.423Z" }, + { url = "https://files.pythonhosted.org/packages/39/22/b5da95d74edcf81e540e467202a988c50fef41bd2011f46e05f72ba07df6/scipy-1.17.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:d7425fcafbc09a03731e1bc05581f5fad988e48c6a861f441b7ab729a49a55ea", size = 37379586, upload-time = "2026-01-10T21:27:20.171Z" }, + { url = "https://files.pythonhosted.org/packages/b9/b6/8ac583d6da79e7b9e520579f03007cb006f063642afd6b2eeb16b890bf93/scipy-1.17.0-cp313-cp313-win_amd64.whl", hash = "sha256:87b411e42b425b84777718cc41516b8a7e0795abfa8e8e1d573bf0ef014f0812", size = 36287211, upload-time = "2026-01-10T21:28:43.122Z" }, + { url = "https://files.pythonhosted.org/packages/55/fb/7db19e0b3e52f882b420417644ec81dd57eeef1bd1705b6f689d8ff93541/scipy-1.17.0-cp313-cp313-win_arm64.whl", hash = "sha256:357ca001c6e37601066092e7c89cca2f1ce74e2a520ca78d063a6d2201101df2", size = 24312646, upload-time = "2026-01-10T21:28:49.893Z" }, + { url = "https://files.pythonhosted.org/packages/20/b6/7feaa252c21cc7aff335c6c55e1b90ab3e3306da3f048109b8b639b94648/scipy-1.17.0-cp313-cp313t-macosx_10_14_x86_64.whl", hash = "sha256:ec0827aa4d36cb79ff1b81de898e948a51ac0b9b1c43e4a372c0508c38c0f9a3", size = 31693194, upload-time = "2026-01-10T21:27:27.454Z" }, + { url = "https://files.pythonhosted.org/packages/76/bb/bbb392005abce039fb7e672cb78ac7d158700e826b0515cab6b5b60c26fb/scipy-1.17.0-cp313-cp313t-macosx_12_0_arm64.whl", hash = "sha256:819fc26862b4b3c73a60d486dbb919202f3d6d98c87cf20c223511429f2d1a97", size = 28365415, upload-time = "2026-01-10T21:27:34.26Z" }, + { url = "https://files.pythonhosted.org/packages/37/da/9d33196ecc99fba16a409c691ed464a3a283ac454a34a13a3a57c0d66f3a/scipy-1.17.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:363ad4ae2853d88ebcde3ae6ec46ccca903ea9835ee8ba543f12f575e7b07e4e", size = 20537232, upload-time = "2026-01-10T21:27:40.306Z" }, + { url = "https://files.pythonhosted.org/packages/56/9d/f4b184f6ddb28e9a5caea36a6f98e8ecd2a524f9127354087ce780885d83/scipy-1.17.0-cp313-cp313t-macosx_14_0_x86_64.whl", hash = "sha256:979c3a0ff8e5ba254d45d59ebd38cde48fce4f10b5125c680c7a4bfe177aab07", size = 22791051, upload-time = "2026-01-10T21:27:46.539Z" }, + { url = "https://files.pythonhosted.org/packages/9b/9d/025cccdd738a72140efc582b1641d0dd4caf2e86c3fb127568dc80444e6e/scipy-1.17.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:130d12926ae34399d157de777472bf82e9061c60cc081372b3118edacafe1d00", size = 32815098, upload-time = "2026-01-10T21:27:54.389Z" }, + { url = "https://files.pythonhosted.org/packages/48/5f/09b879619f8bca15ce392bfc1894bd9c54377e01d1b3f2f3b595a1b4d945/scipy-1.17.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6e886000eb4919eae3a44f035e63f0fd8b651234117e8f6f29bad1cd26e7bc45", size = 35031342, upload-time = "2026-01-10T21:28:03.012Z" }, + { url = "https://files.pythonhosted.org/packages/f2/9a/f0f0a9f0aa079d2f106555b984ff0fbb11a837df280f04f71f056ea9c6e4/scipy-1.17.0-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:13c4096ac6bc31d706018f06a49abe0485f96499deb82066b94d19b02f664209", size = 34893199, upload-time = "2026-01-10T21:28:10.832Z" }, + { url = "https://files.pythonhosted.org/packages/90/b8/4f0f5cf0c5ea4d7548424e6533e6b17d164f34a6e2fb2e43ffebb6697b06/scipy-1.17.0-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:cacbaddd91fcffde703934897c5cd2c7cb0371fac195d383f4e1f1c5d3f3bd04", size = 37438061, upload-time = "2026-01-10T21:28:19.684Z" }, + { url = "https://files.pythonhosted.org/packages/f9/cc/2bd59140ed3b2fa2882fb15da0a9cb1b5a6443d67cfd0d98d4cec83a57ec/scipy-1.17.0-cp313-cp313t-win_amd64.whl", hash = "sha256:edce1a1cf66298cccdc48a1bdf8fb10a3bf58e8b58d6c3883dd1530e103f87c0", size = 36328593, upload-time = "2026-01-10T21:28:28.007Z" }, + { url = "https://files.pythonhosted.org/packages/13/1b/c87cc44a0d2c7aaf0f003aef2904c3d097b422a96c7e7c07f5efd9073c1b/scipy-1.17.0-cp313-cp313t-win_arm64.whl", hash = "sha256:30509da9dbec1c2ed8f168b8d8aa853bc6723fede1dbc23c7d43a56f5ab72a67", size = 24625083, upload-time = "2026-01-10T21:28:35.188Z" }, + { url = "https://files.pythonhosted.org/packages/1a/2d/51006cd369b8e7879e1c630999a19d1fbf6f8b5ed3e33374f29dc87e53b3/scipy-1.17.0-cp314-cp314-macosx_10_14_x86_64.whl", hash = "sha256:c17514d11b78be8f7e6331b983a65a7f5ca1fd037b95e27b280921fe5606286a", size = 31346803, upload-time = "2026-01-10T21:28:57.24Z" }, + { url = "https://files.pythonhosted.org/packages/d6/2e/2349458c3ce445f53a6c93d4386b1c4c5c0c540917304c01222ff95ff317/scipy-1.17.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:4e00562e519c09da34c31685f6acc3aa384d4d50604db0f245c14e1b4488bfa2", size = 27967182, upload-time = "2026-01-10T21:29:04.107Z" }, + { url = "https://files.pythonhosted.org/packages/5e/7c/df525fbfa77b878d1cfe625249529514dc02f4fd5f45f0f6295676a76528/scipy-1.17.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:f7df7941d71314e60a481e02d5ebcb3f0185b8d799c70d03d8258f6c80f3d467", size = 20139125, upload-time = "2026-01-10T21:29:10.179Z" }, + { url = "https://files.pythonhosted.org/packages/33/11/fcf9d43a7ed1234d31765ec643b0515a85a30b58eddccc5d5a4d12b5f194/scipy-1.17.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:aabf057c632798832f071a8dde013c2e26284043934f53b00489f1773b33527e", size = 22443554, upload-time = "2026-01-10T21:29:15.888Z" }, + { url = "https://files.pythonhosted.org/packages/80/5c/ea5d239cda2dd3d31399424967a24d556cf409fbea7b5b21412b0fd0a44f/scipy-1.17.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a38c3337e00be6fd8a95b4ed66b5d988bac4ec888fd922c2ea9fe5fb1603dd67", size = 32757834, upload-time = "2026-01-10T21:29:23.406Z" }, + { url = "https://files.pythonhosted.org/packages/b8/7e/8c917cc573310e5dc91cbeead76f1b600d3fb17cf0969db02c9cf92e3cfa/scipy-1.17.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:00fb5f8ec8398ad90215008d8b6009c9db9fa924fd4c7d6be307c6f945f9cd73", size = 34995775, upload-time = "2026-01-10T21:29:31.915Z" }, + { url = "https://files.pythonhosted.org/packages/c5/43/176c0c3c07b3f7df324e7cdd933d3e2c4898ca202b090bd5ba122f9fe270/scipy-1.17.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f2a4942b0f5f7c23c7cd641a0ca1955e2ae83dedcff537e3a0259096635e186b", size = 34841240, upload-time = "2026-01-10T21:29:39.995Z" }, + { url = "https://files.pythonhosted.org/packages/44/8c/d1f5f4b491160592e7f084d997de53a8e896a3ac01cd07e59f43ca222744/scipy-1.17.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:dbf133ced83889583156566d2bdf7a07ff89228fe0c0cb727f777de92092ec6b", size = 37394463, upload-time = "2026-01-10T21:29:48.723Z" }, + { url = "https://files.pythonhosted.org/packages/9f/ec/42a6657f8d2d087e750e9a5dde0b481fd135657f09eaf1cf5688bb23c338/scipy-1.17.0-cp314-cp314-win_amd64.whl", hash = "sha256:3625c631a7acd7cfd929e4e31d2582cf00f42fcf06011f59281271746d77e061", size = 37053015, upload-time = "2026-01-10T21:30:51.418Z" }, + { url = "https://files.pythonhosted.org/packages/27/58/6b89a6afd132787d89a362d443a7bddd511b8f41336a1ae47f9e4f000dc4/scipy-1.17.0-cp314-cp314-win_arm64.whl", hash = "sha256:9244608d27eafe02b20558523ba57f15c689357c85bdcfe920b1828750aa26eb", size = 24951312, upload-time = "2026-01-10T21:30:56.771Z" }, + { url = "https://files.pythonhosted.org/packages/e9/01/f58916b9d9ae0112b86d7c3b10b9e685625ce6e8248df139d0fcb17f7397/scipy-1.17.0-cp314-cp314t-macosx_10_14_x86_64.whl", hash = "sha256:2b531f57e09c946f56ad0b4a3b2abee778789097871fc541e267d2eca081cff1", size = 31706502, upload-time = "2026-01-10T21:29:56.326Z" }, + { url = "https://files.pythonhosted.org/packages/59/8e/2912a87f94a7d1f8b38aabc0faf74b82d3b6c9e22be991c49979f0eceed8/scipy-1.17.0-cp314-cp314t-macosx_12_0_arm64.whl", hash = "sha256:13e861634a2c480bd237deb69333ac79ea1941b94568d4b0efa5db5e263d4fd1", size = 28380854, upload-time = "2026-01-10T21:30:01.554Z" }, + { url = "https://files.pythonhosted.org/packages/bd/1c/874137a52dddab7d5d595c1887089a2125d27d0601fce8c0026a24a92a0b/scipy-1.17.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:eb2651271135154aa24f6481cbae5cc8af1f0dd46e6533fb7b56aa9727b6a232", size = 20552752, upload-time = "2026-01-10T21:30:05.93Z" }, + { url = "https://files.pythonhosted.org/packages/3f/f0/7518d171cb735f6400f4576cf70f756d5b419a07fe1867da34e2c2c9c11b/scipy-1.17.0-cp314-cp314t-macosx_14_0_x86_64.whl", hash = "sha256:c5e8647f60679790c2f5c76be17e2e9247dc6b98ad0d3b065861e082c56e078d", size = 22803972, upload-time = "2026-01-10T21:30:10.651Z" }, + { url = "https://files.pythonhosted.org/packages/7c/74/3498563a2c619e8a3ebb4d75457486c249b19b5b04a30600dfd9af06bea5/scipy-1.17.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5fb10d17e649e1446410895639f3385fd2bf4c3c7dfc9bea937bddcbc3d7b9ba", size = 32829770, upload-time = "2026-01-10T21:30:16.359Z" }, + { url = "https://files.pythonhosted.org/packages/48/d1/7b50cedd8c6c9d6f706b4b36fa8544d829c712a75e370f763b318e9638c1/scipy-1.17.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8547e7c57f932e7354a2319fab613981cde910631979f74c9b542bb167a8b9db", size = 35051093, upload-time = "2026-01-10T21:30:22.987Z" }, + { url = "https://files.pythonhosted.org/packages/e2/82/a2d684dfddb87ba1b3ea325df7c3293496ee9accb3a19abe9429bce94755/scipy-1.17.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:33af70d040e8af9d5e7a38b5ed3b772adddd281e3062ff23fec49e49681c38cf", size = 34909905, upload-time = "2026-01-10T21:30:28.704Z" }, + { url = "https://files.pythonhosted.org/packages/ef/5e/e565bd73991d42023eb82bb99e51c5b3d9e2c588ca9d4b3e2cc1d3ca62a6/scipy-1.17.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:f9eb55bb97d00f8b7ab95cb64f873eb0bf54d9446264d9f3609130381233483f", size = 37457743, upload-time = "2026-01-10T21:30:34.819Z" }, + { url = "https://files.pythonhosted.org/packages/58/a8/a66a75c3d8f1fb2b83f66007d6455a06a6f6cf5618c3dc35bc9b69dd096e/scipy-1.17.0-cp314-cp314t-win_amd64.whl", hash = "sha256:1ff269abf702f6c7e67a4b7aad981d42871a11b9dd83c58d2d2ea624efbd1088", size = 37098574, upload-time = "2026-01-10T21:30:40.782Z" }, + { url = "https://files.pythonhosted.org/packages/56/a5/df8f46ef7da168f1bc52cd86e09a9de5c6f19cc1da04454d51b7d4f43408/scipy-1.17.0-cp314-cp314t-win_arm64.whl", hash = "sha256:031121914e295d9791319a1875444d55079885bbae5bdc9c5e0f2ee5f09d34ff", size = 25246266, upload-time = "2026-01-10T21:30:45.923Z" }, +] + +[[package]] +name = "semanticscholar" +version = "0.11.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "httpx" }, + { name = "nest-asyncio" }, + { name = "tenacity" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/8a/5c/18302aad8f094b3e4b8bb79ee82dd0fafceee7811f9844c41f393f8a0e5f/semanticscholar-0.11.0.tar.gz", hash = "sha256:91e1233aedbf8e290daf8556fc7a3a6a6db78fe833adee88fd99ddb0a07a0c17", size = 28283, upload-time = "2025-09-14T01:14:51.832Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7e/af/0627cf6bb64054d03a1fe8e9b0e659b496794000c30ba4fb921ca8aef20a/semanticscholar-0.11.0-py3-none-any.whl", hash = "sha256:824b7c3d11237ec829a211480ed1ed05f4ee9dfdf03e226b04c3d2051ea19b6e", size = 26048, upload-time = "2025-09-14T01:14:50.575Z" }, +] + [[package]] name = "send2trash" version = "1.8.3" @@ -4881,6 +5974,28 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/40/b0/4562db6223154aa4e22f939003cb92514c79f3d4dccca3444253fd17f902/Send2Trash-1.8.3-py3-none-any.whl", hash = "sha256:0c31227e0bd08961c7665474a3d1ef7193929fedda4233843689baa056be46c9", size = 18072, upload-time = "2024-04-07T00:01:07.438Z" }, ] +[[package]] +name = "sentence-transformers" +version = "5.2.2" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "huggingface-hub" }, + { name = "numpy", version = "2.2.6", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "numpy", version = "2.4.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "scikit-learn", version = "1.7.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scikit-learn", version = "1.8.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "scipy", version = "1.15.3", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "scipy", version = "1.17.0", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "torch" }, + { name = "tqdm" }, + { name = "transformers" }, + { name = "typing-extensions" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/a6/bc/0bc9c0ec1cf83ab2ec6e6f38667d167349b950fff6dd2086b79bd360eeca/sentence_transformers-5.2.2.tar.gz", hash = "sha256:7033ee0a24bc04c664fd490abf2ef194d387b3a58a97adcc528783ff505159fa", size = 381607, upload-time = "2026-01-27T11:11:02.658Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/cc/21/7e925890636791386e81b52878134f114d63072e79fffe14cdcc5e7a5e6a/sentence_transformers-5.2.2-py3-none-any.whl", hash = "sha256:280ac54bffb84c110726b4d8848ba7b7c60813b9034547f8aea6e9a345cd1c23", size = 494106, upload-time = "2026-01-27T11:11:00.983Z" }, +] + [[package]] name = "sentry-sdk" version = "2.48.0" @@ -4903,6 +6018,12 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/a3/dc/17031897dae0efacfea57dfd3a82fdd2a2aeb58e0ff71b77b87e44edc772/setuptools-80.9.0-py3-none-any.whl", hash = "sha256:062d34222ad13e0cc312a4c02d73f059e86a4acbfbdea8f8f76b28c99f306922", size = 1201486, upload-time = "2025-05-27T00:56:49.664Z" }, ] +[[package]] +name = "sgmllib3k" +version = "1.0.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/9e/bd/3704a8c3e0942d711c1299ebf7b9091930adae6675d7c8f476a7ce48653c/sgmllib3k-1.0.0.tar.gz", hash = "sha256:7868fb1c8bfa764c1ac563d3cf369c381d1325d36124933a726f29fcdaa812e9", size = 5750, upload-time = "2010-08-24T14:33:52.445Z" } + [[package]] name = "six" version = "1.17.0" @@ -5050,6 +6171,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/d9/52/1064f510b141bd54025f9b55105e26d1fa970b9be67ad766380a3c9b74b0/starlette-0.50.0-py3-none-any.whl", hash = "sha256:9e5391843ec9b6e472eed1365a78c8098cfceb7a74bfd4d6b1c0c0095efb3bca", size = 74033, upload-time = "2025-11-01T15:25:25.461Z" }, ] +[[package]] +name = "sympy" +version = "1.14.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "mpmath" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517", size = 7793921, upload-time = "2025-04-27T18:05:01.611Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5", size = 6299353, upload-time = "2025-04-27T18:04:59.103Z" }, +] + [[package]] name = "tenacity" version = "9.1.2" @@ -5059,6 +6192,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/e5/30/643397144bfbfec6f6ef821f36f33e57d35946c44a2352d3c9f0ae847619/tenacity-9.1.2-py3-none-any.whl", hash = "sha256:f77bf36710d8b73a50b2dd155c97b870017ad21afe6ab300326b0371b3b05138", size = 28248, upload-time = "2025-04-02T08:25:07.678Z" }, ] +[[package]] +name = "termcolor" +version = "2.5.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/37/72/88311445fd44c455c7d553e61f95412cf89054308a1aa2434ab835075fc5/termcolor-2.5.0.tar.gz", hash = "sha256:998d8d27da6d48442e8e1f016119076b690d962507531df4890fcd2db2ef8a6f", size = 13057, upload-time = "2024-10-06T19:50:04.115Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7f/be/df630c387a0a054815d60be6a97eb4e8f17385d5d6fe660e1c02750062b4/termcolor-2.5.0-py3-none-any.whl", hash = "sha256:37b17b5fc1e604945c2642c872a3764b5d547a48009871aea3edd3afa180afb8", size = 7755, upload-time = "2024-10-06T19:50:02.097Z" }, +] + [[package]] name = "terminado" version = "0.18.1" @@ -5073,6 +6215,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6a/9e/2064975477fdc887e47ad42157e214526dcad8f317a948dee17e1659a62f/terminado-0.18.1-py3-none-any.whl", hash = "sha256:a4468e1b37bb318f8a86514f65814e1afc977cf29b3992a4500d9dd305dcceb0", size = 14154, upload-time = "2024-03-12T14:34:36.569Z" }, ] +[[package]] +name = "threadpoolctl" +version = "3.6.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/b7/4d/08c89e34946fce2aec4fbb45c9016efd5f4d7f24af8e5d93296e935631d8/threadpoolctl-3.6.0.tar.gz", hash = "sha256:8ab8b4aa3491d812b623328249fab5302a68d2d71745c8a4c719a2fcaba9f44e", size = 21274, upload-time = "2025-03-13T13:49:23.031Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/32/d5/f9a850d79b0851d1d4ef6456097579a9005b31fea68726a4ae5f2d82ddd9/threadpoolctl-3.6.0-py3-none-any.whl", hash = "sha256:43a0b8fd5a2928500110039e43a5eed8480b918967083ea48dc3ab9f13c4a7fb", size = 18638, upload-time = "2025-03-13T13:49:21.846Z" }, +] + [[package]] name = "tiktoken" version = "0.7.0" @@ -5120,27 +6271,27 @@ wheels = [ [[package]] name = "tokenizers" -version = "0.22.1" +version = "0.21.4" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "huggingface-hub" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/1c/46/fb6854cec3278fbfa4a75b50232c77622bc517ac886156e6afbfa4d8fc6e/tokenizers-0.22.1.tar.gz", hash = "sha256:61de6522785310a309b3407bac22d99c4db5dba349935e99e4d15ea2226af2d9", size = 363123, upload-time = "2025-09-19T09:49:23.424Z" } +sdist = { url = "https://files.pythonhosted.org/packages/c2/2f/402986d0823f8d7ca139d969af2917fefaa9b947d1fb32f6168c509f2492/tokenizers-0.21.4.tar.gz", hash = "sha256:fa23f85fbc9a02ec5c6978da172cdcbac23498c3ca9f3645c5c68740ac007880", size = 351253, upload-time = "2025-07-28T15:48:54.325Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/bf/33/f4b2d94ada7ab297328fc671fed209368ddb82f965ec2224eb1892674c3a/tokenizers-0.22.1-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:59fdb013df17455e5f950b4b834a7b3ee2e0271e6378ccb33aa74d178b513c73", size = 3069318, upload-time = "2025-09-19T09:49:11.848Z" }, - { url = "https://files.pythonhosted.org/packages/1c/58/2aa8c874d02b974990e89ff95826a4852a8b2a273c7d1b4411cdd45a4565/tokenizers-0.22.1-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:8d4e484f7b0827021ac5f9f71d4794aaef62b979ab7608593da22b1d2e3c4edc", size = 2926478, upload-time = "2025-09-19T09:49:09.759Z" }, - { url = "https://files.pythonhosted.org/packages/1e/3b/55e64befa1e7bfea963cf4b787b2cea1011362c4193f5477047532ce127e/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:19d2962dd28bc67c1f205ab180578a78eef89ac60ca7ef7cbe9635a46a56422a", size = 3256994, upload-time = "2025-09-19T09:48:56.701Z" }, - { url = "https://files.pythonhosted.org/packages/71/0b/fbfecf42f67d9b7b80fde4aabb2b3110a97fac6585c9470b5bff103a80cb/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:38201f15cdb1f8a6843e6563e6e79f4abd053394992b9bbdf5213ea3469b4ae7", size = 3153141, upload-time = "2025-09-19T09:48:59.749Z" }, - { url = "https://files.pythonhosted.org/packages/17/a9/b38f4e74e0817af8f8ef925507c63c6ae8171e3c4cb2d5d4624bf58fca69/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:d1cbe5454c9a15df1b3443c726063d930c16f047a3cc724b9e6e1a91140e5a21", size = 3508049, upload-time = "2025-09-19T09:49:05.868Z" }, - { url = "https://files.pythonhosted.org/packages/d2/48/dd2b3dac46bb9134a88e35d72e1aa4869579eacc1a27238f1577270773ff/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e7d094ae6312d69cc2a872b54b91b309f4f6fbce871ef28eb27b52a98e4d0214", size = 3710730, upload-time = "2025-09-19T09:49:01.832Z" }, - { url = "https://files.pythonhosted.org/packages/93/0e/ccabc8d16ae4ba84a55d41345207c1e2ea88784651a5a487547d80851398/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:afd7594a56656ace95cdd6df4cca2e4059d294c5cfb1679c57824b605556cb2f", size = 3412560, upload-time = "2025-09-19T09:49:03.867Z" }, - { url = "https://files.pythonhosted.org/packages/d0/c6/dc3a0db5a6766416c32c034286d7c2d406da1f498e4de04ab1b8959edd00/tokenizers-0.22.1-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:e2ef6063d7a84994129732b47e7915e8710f27f99f3a3260b8a38fc7ccd083f4", size = 3250221, upload-time = "2025-09-19T09:49:07.664Z" }, - { url = "https://files.pythonhosted.org/packages/d7/a6/2c8486eef79671601ff57b093889a345dd3d576713ef047776015dc66de7/tokenizers-0.22.1-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:ba0a64f450b9ef412c98f6bcd2a50c6df6e2443b560024a09fa6a03189726879", size = 9345569, upload-time = "2025-09-19T09:49:14.214Z" }, - { url = "https://files.pythonhosted.org/packages/6b/16/32ce667f14c35537f5f605fe9bea3e415ea1b0a646389d2295ec348d5657/tokenizers-0.22.1-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:331d6d149fa9c7d632cde4490fb8bbb12337fa3a0232e77892be656464f4b446", size = 9271599, upload-time = "2025-09-19T09:49:16.639Z" }, - { url = "https://files.pythonhosted.org/packages/51/7c/a5f7898a3f6baa3fc2685c705e04c98c1094c523051c805cdd9306b8f87e/tokenizers-0.22.1-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:607989f2ea68a46cb1dfbaf3e3aabdf3f21d8748312dbeb6263d1b3b66c5010a", size = 9533862, upload-time = "2025-09-19T09:49:19.146Z" }, - { url = "https://files.pythonhosted.org/packages/36/65/7e75caea90bc73c1dd8d40438adf1a7bc26af3b8d0a6705ea190462506e1/tokenizers-0.22.1-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:a0f307d490295717726598ef6fa4f24af9d484809223bbc253b201c740a06390", size = 9681250, upload-time = "2025-09-19T09:49:21.501Z" }, - { url = "https://files.pythonhosted.org/packages/30/2c/959dddef581b46e6209da82df3b78471e96260e2bc463f89d23b1bf0e52a/tokenizers-0.22.1-cp39-abi3-win32.whl", hash = "sha256:b5120eed1442765cd90b903bb6cfef781fd8fe64e34ccaecbae4c619b7b12a82", size = 2472003, upload-time = "2025-09-19T09:49:27.089Z" }, - { url = "https://files.pythonhosted.org/packages/b3/46/e33a8c93907b631a99377ef4c5f817ab453d0b34f93529421f42ff559671/tokenizers-0.22.1-cp39-abi3-win_amd64.whl", hash = "sha256:65fd6e3fb11ca1e78a6a93602490f134d1fdeb13bcef99389d5102ea318ed138", size = 2674684, upload-time = "2025-09-19T09:49:24.953Z" }, + { url = "https://files.pythonhosted.org/packages/98/c6/fdb6f72bf6454f52eb4a2510be7fb0f614e541a2554d6210e370d85efff4/tokenizers-0.21.4-cp39-abi3-macosx_10_12_x86_64.whl", hash = "sha256:2ccc10a7c3bcefe0f242867dc914fc1226ee44321eb618cfe3019b5df3400133", size = 2863987, upload-time = "2025-07-28T15:48:44.877Z" }, + { url = "https://files.pythonhosted.org/packages/8d/a6/28975479e35ddc751dc1ddc97b9b69bf7fcf074db31548aab37f8116674c/tokenizers-0.21.4-cp39-abi3-macosx_11_0_arm64.whl", hash = "sha256:5e2f601a8e0cd5be5cc7506b20a79112370b9b3e9cb5f13f68ab11acd6ca7d60", size = 2732457, upload-time = "2025-07-28T15:48:43.265Z" }, + { url = "https://files.pythonhosted.org/packages/aa/8f/24f39d7b5c726b7b0be95dca04f344df278a3fe3a4deb15a975d194cbb32/tokenizers-0.21.4-cp39-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:39b376f5a1aee67b4d29032ee85511bbd1b99007ec735f7f35c8a2eb104eade5", size = 3012624, upload-time = "2025-07-28T13:22:43.895Z" }, + { url = "https://files.pythonhosted.org/packages/58/47/26358925717687a58cb74d7a508de96649544fad5778f0cd9827398dc499/tokenizers-0.21.4-cp39-abi3-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:2107ad649e2cda4488d41dfd031469e9da3fcbfd6183e74e4958fa729ffbf9c6", size = 2939681, upload-time = "2025-07-28T13:22:47.499Z" }, + { url = "https://files.pythonhosted.org/packages/99/6f/cc300fea5db2ab5ddc2c8aea5757a27b89c84469899710c3aeddc1d39801/tokenizers-0.21.4-cp39-abi3-manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:3c73012da95afafdf235ba80047699df4384fdc481527448a078ffd00e45a7d9", size = 3247445, upload-time = "2025-07-28T15:48:39.711Z" }, + { url = "https://files.pythonhosted.org/packages/be/bf/98cb4b9c3c4afd8be89cfa6423704337dc20b73eb4180397a6e0d456c334/tokenizers-0.21.4-cp39-abi3-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:f23186c40395fc390d27f519679a58023f368a0aad234af145e0f39ad1212732", size = 3428014, upload-time = "2025-07-28T13:22:49.569Z" }, + { url = "https://files.pythonhosted.org/packages/75/c7/96c1cc780e6ca7f01a57c13235dd05b7bc1c0f3588512ebe9d1331b5f5ae/tokenizers-0.21.4-cp39-abi3-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:cc88bb34e23a54cc42713d6d98af5f1bf79c07653d24fe984d2d695ba2c922a2", size = 3193197, upload-time = "2025-07-28T13:22:51.471Z" }, + { url = "https://files.pythonhosted.org/packages/f2/90/273b6c7ec78af547694eddeea9e05de771278bd20476525ab930cecaf7d8/tokenizers-0.21.4-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:51b7eabb104f46c1c50b486520555715457ae833d5aee9ff6ae853d1130506ff", size = 3115426, upload-time = "2025-07-28T15:48:41.439Z" }, + { url = "https://files.pythonhosted.org/packages/91/43/c640d5a07e95f1cf9d2c92501f20a25f179ac53a4f71e1489a3dcfcc67ee/tokenizers-0.21.4-cp39-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:714b05b2e1af1288bd1bc56ce496c4cebb64a20d158ee802887757791191e6e2", size = 9089127, upload-time = "2025-07-28T15:48:46.472Z" }, + { url = "https://files.pythonhosted.org/packages/44/a1/dd23edd6271d4dca788e5200a807b49ec3e6987815cd9d0a07ad9c96c7c2/tokenizers-0.21.4-cp39-abi3-musllinux_1_2_armv7l.whl", hash = "sha256:1340ff877ceedfa937544b7d79f5b7becf33a4cfb58f89b3b49927004ef66f78", size = 9055243, upload-time = "2025-07-28T15:48:48.539Z" }, + { url = "https://files.pythonhosted.org/packages/21/2b/b410d6e9021c4b7ddb57248304dc817c4d4970b73b6ee343674914701197/tokenizers-0.21.4-cp39-abi3-musllinux_1_2_i686.whl", hash = "sha256:3c1f4317576e465ac9ef0d165b247825a2a4078bcd01cba6b54b867bdf9fdd8b", size = 9298237, upload-time = "2025-07-28T15:48:50.443Z" }, + { url = "https://files.pythonhosted.org/packages/b7/0a/42348c995c67e2e6e5c89ffb9cfd68507cbaeb84ff39c49ee6e0a6dd0fd2/tokenizers-0.21.4-cp39-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:c212aa4e45ec0bb5274b16b6f31dd3f1c41944025c2358faaa5782c754e84c24", size = 9461980, upload-time = "2025-07-28T15:48:52.325Z" }, + { url = "https://files.pythonhosted.org/packages/3d/d3/dacccd834404cd71b5c334882f3ba40331ad2120e69ded32cf5fda9a7436/tokenizers-0.21.4-cp39-abi3-win32.whl", hash = "sha256:6c42a930bc5f4c47f4ea775c91de47d27910881902b0f20e4990ebe045a415d0", size = 2329871, upload-time = "2025-07-28T15:48:56.841Z" }, + { url = "https://files.pythonhosted.org/packages/41/f2/fd673d979185f5dcbac4be7d09461cbb99751554ffb6718d0013af8604cb/tokenizers-0.21.4-cp39-abi3-win_amd64.whl", hash = "sha256:475d807a5c3eb72c59ad9b5fcdb254f6e17f53dfcbb9903233b0dfa9c943b597", size = 2507568, upload-time = "2025-07-28T15:48:55.456Z" }, ] [[package]] @@ -5192,6 +6343,72 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/77/b8/0135fadc89e73be292b473cb820b4f5a08197779206b33191e801feeae40/tomli-2.3.0-py3-none-any.whl", hash = "sha256:e95b1af3c5b07d9e643909b5abbec77cd9f1217e6d0bca72b0234736b9fb1f1b", size = 14408, upload-time = "2025-10-08T22:01:46.04Z" }, ] +[[package]] +name = "torch" +version = "2.10.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "cuda-bindings", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "filelock" }, + { name = "fsspec" }, + { name = "jinja2" }, + { name = "networkx", version = "3.4.2", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.11'" }, + { name = "networkx", version = "3.6.1", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.11'" }, + { name = "nvidia-cublas-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cuda-cupti-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cuda-nvrtc-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cuda-runtime-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cudnn-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cufft-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cufile-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-curand-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cusolver-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cusparse-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-cusparselt-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nccl-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nvjitlink-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nvshmem-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "nvidia-nvtx-cu12", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "setuptools", marker = "python_full_version >= '3.12'" }, + { name = "sympy" }, + { name = "triton", marker = "platform_machine == 'x86_64' and sys_platform == 'linux'" }, + { name = "typing-extensions" }, +] +wheels = [ + { url = "https://files.pythonhosted.org/packages/5b/30/bfebdd8ec77db9a79775121789992d6b3b75ee5494971294d7b4b7c999bc/torch-2.10.0-2-cp310-none-macosx_11_0_arm64.whl", hash = "sha256:2b980edd8d7c0a68c4e951ee1856334a43193f98730d97408fbd148c1a933313", size = 79411457, upload-time = "2026-02-10T21:44:59.189Z" }, + { url = "https://files.pythonhosted.org/packages/0f/8b/4b61d6e13f7108f36910df9ab4b58fd389cc2520d54d81b88660804aad99/torch-2.10.0-2-cp311-none-macosx_11_0_arm64.whl", hash = "sha256:418997cb02d0a0f1497cf6a09f63166f9f5df9f3e16c8a716ab76a72127c714f", size = 79423467, upload-time = "2026-02-10T21:44:48.711Z" }, + { url = "https://files.pythonhosted.org/packages/d3/54/a2ba279afcca44bbd320d4e73675b282fcee3d81400ea1b53934efca6462/torch-2.10.0-2-cp312-none-macosx_11_0_arm64.whl", hash = "sha256:13ec4add8c3faaed8d13e0574f5cd4a323c11655546f91fbe6afa77b57423574", size = 79498202, upload-time = "2026-02-10T21:44:52.603Z" }, + { url = "https://files.pythonhosted.org/packages/ec/23/2c9fe0c9c27f7f6cb865abcea8a4568f29f00acaeadfc6a37f6801f84cb4/torch-2.10.0-2-cp313-none-macosx_11_0_arm64.whl", hash = "sha256:e521c9f030a3774ed770a9c011751fb47c4d12029a3d6522116e48431f2ff89e", size = 79498254, upload-time = "2026-02-10T21:44:44.095Z" }, + { url = "https://files.pythonhosted.org/packages/0c/1a/c61f36cfd446170ec27b3a4984f072fd06dab6b5d7ce27e11adb35d6c838/torch-2.10.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:5276fa790a666ee8becaffff8acb711922252521b28fbce5db7db5cf9cb2026d", size = 145992962, upload-time = "2026-01-21T16:24:14.04Z" }, + { url = "https://files.pythonhosted.org/packages/b5/60/6662535354191e2d1555296045b63e4279e5a9dbad49acf55a5d38655a39/torch-2.10.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:aaf663927bcd490ae971469a624c322202a2a1e68936eb952535ca4cd3b90444", size = 915599237, upload-time = "2026-01-21T16:23:25.497Z" }, + { url = "https://files.pythonhosted.org/packages/40/b8/66bbe96f0d79be2b5c697b2e0b187ed792a15c6c4b8904613454651db848/torch-2.10.0-cp310-cp310-win_amd64.whl", hash = "sha256:a4be6a2a190b32ff5c8002a0977a25ea60e64f7ba46b1be37093c141d9c49aeb", size = 113720931, upload-time = "2026-01-21T16:24:23.743Z" }, + { url = "https://files.pythonhosted.org/packages/76/bb/d820f90e69cda6c8169b32a0c6a3ab7b17bf7990b8f2c680077c24a3c14c/torch-2.10.0-cp310-none-macosx_11_0_arm64.whl", hash = "sha256:35e407430795c8d3edb07a1d711c41cc1f9eaddc8b2f1cc0a165a6767a8fb73d", size = 79411450, upload-time = "2026-01-21T16:25:30.692Z" }, + { url = "https://files.pythonhosted.org/packages/78/89/f5554b13ebd71e05c0b002f95148033e730d3f7067f67423026cc9c69410/torch-2.10.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:3282d9febd1e4e476630a099692b44fdc214ee9bf8ee5377732d9d9dfe5712e4", size = 145992610, upload-time = "2026-01-21T16:25:26.327Z" }, + { url = "https://files.pythonhosted.org/packages/ae/30/a3a2120621bf9c17779b169fc17e3dc29b230c29d0f8222f499f5e159aa8/torch-2.10.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:a2f9edd8dbc99f62bc4dfb78af7bf89499bca3d753423ac1b4e06592e467b763", size = 915607863, upload-time = "2026-01-21T16:25:06.696Z" }, + { url = "https://files.pythonhosted.org/packages/6f/3d/c87b33c5f260a2a8ad68da7147e105f05868c281c63d65ed85aa4da98c66/torch-2.10.0-cp311-cp311-win_amd64.whl", hash = "sha256:29b7009dba4b7a1c960260fc8ac85022c784250af43af9fb0ebafc9883782ebd", size = 113723116, upload-time = "2026-01-21T16:25:21.916Z" }, + { url = "https://files.pythonhosted.org/packages/61/d8/15b9d9d3a6b0c01b883787bd056acbe5cc321090d4b216d3ea89a8fcfdf3/torch-2.10.0-cp311-none-macosx_11_0_arm64.whl", hash = "sha256:b7bd80f3477b830dd166c707c5b0b82a898e7b16f59a7d9d42778dd058272e8b", size = 79423461, upload-time = "2026-01-21T16:24:50.266Z" }, + { url = "https://files.pythonhosted.org/packages/cc/af/758e242e9102e9988969b5e621d41f36b8f258bb4a099109b7a4b4b50ea4/torch-2.10.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:5fd4117d89ffd47e3dcc71e71a22efac24828ad781c7e46aaaf56bf7f2796acf", size = 145996088, upload-time = "2026-01-21T16:24:44.171Z" }, + { url = "https://files.pythonhosted.org/packages/23/8e/3c74db5e53bff7ed9e34c8123e6a8bfef718b2450c35eefab85bb4a7e270/torch-2.10.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:787124e7db3b379d4f1ed54dd12ae7c741c16a4d29b49c0226a89bea50923ffb", size = 915711952, upload-time = "2026-01-21T16:23:53.503Z" }, + { url = "https://files.pythonhosted.org/packages/6e/01/624c4324ca01f66ae4c7cd1b74eb16fb52596dce66dbe51eff95ef9e7a4c/torch-2.10.0-cp312-cp312-win_amd64.whl", hash = "sha256:2c66c61f44c5f903046cc696d088e21062644cbe541c7f1c4eaae88b2ad23547", size = 113757972, upload-time = "2026-01-21T16:24:39.516Z" }, + { url = "https://files.pythonhosted.org/packages/c9/5c/dee910b87c4d5c0fcb41b50839ae04df87c1cfc663cf1b5fca7ea565eeaa/torch-2.10.0-cp312-none-macosx_11_0_arm64.whl", hash = "sha256:6d3707a61863d1c4d6ebba7be4ca320f42b869ee657e9b2c21c736bf17000294", size = 79498198, upload-time = "2026-01-21T16:24:34.704Z" }, + { url = "https://files.pythonhosted.org/packages/c9/6f/f2e91e34e3fcba2e3fc8d8f74e7d6c22e74e480bbd1db7bc8900fdf3e95c/torch-2.10.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:5c4d217b14741e40776dd7074d9006fd28b8a97ef5654db959d8635b2fe5f29b", size = 146004247, upload-time = "2026-01-21T16:24:29.335Z" }, + { url = "https://files.pythonhosted.org/packages/98/fb/5160261aeb5e1ee12ee95fe599d0541f7c976c3701d607d8fc29e623229f/torch-2.10.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:6b71486353fce0f9714ca0c9ef1c850a2ae766b409808acd58e9678a3edb7738", size = 915716445, upload-time = "2026-01-21T16:22:45.353Z" }, + { url = "https://files.pythonhosted.org/packages/6a/16/502fb1b41e6d868e8deb5b0e3ae926bbb36dab8ceb0d1b769b266ad7b0c3/torch-2.10.0-cp313-cp313-win_amd64.whl", hash = "sha256:c2ee399c644dc92ef7bc0d4f7e74b5360c37cdbe7c5ba11318dda49ffac2bc57", size = 113757050, upload-time = "2026-01-21T16:24:19.204Z" }, + { url = "https://files.pythonhosted.org/packages/1a/0b/39929b148f4824bc3ad6f9f72a29d4ad865bcf7ebfc2fa67584773e083d2/torch-2.10.0-cp313-cp313t-macosx_14_0_arm64.whl", hash = "sha256:3202429f58309b9fa96a614885eace4b7995729f44beb54d3e4a47773649d382", size = 79851305, upload-time = "2026-01-21T16:24:09.209Z" }, + { url = "https://files.pythonhosted.org/packages/d8/14/21fbce63bc452381ba5f74a2c0a959fdf5ad5803ccc0c654e752e0dbe91a/torch-2.10.0-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:aae1b29cd68e50a9397f5ee897b9c24742e9e306f88a807a27d617f07adb3bd8", size = 146005472, upload-time = "2026-01-21T16:22:29.022Z" }, + { url = "https://files.pythonhosted.org/packages/54/fd/b207d1c525cb570ef47f3e9f836b154685011fce11a2f444ba8a4084d042/torch-2.10.0-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:6021db85958db2f07ec94e1bc77212721ba4920c12a18dc552d2ae36a3eb163f", size = 915612644, upload-time = "2026-01-21T16:21:47.019Z" }, + { url = "https://files.pythonhosted.org/packages/36/53/0197f868c75f1050b199fe58f9bf3bf3aecac9b4e85cc9c964383d745403/torch-2.10.0-cp313-cp313t-win_amd64.whl", hash = "sha256:ff43db38af76fda183156153983c9a096fc4c78d0cd1e07b14a2314c7f01c2c8", size = 113997015, upload-time = "2026-01-21T16:23:00.767Z" }, + { url = "https://files.pythonhosted.org/packages/0e/13/e76b4d9c160e89fff48bf16b449ea324bda84745d2ab30294c37c2434c0d/torch-2.10.0-cp313-none-macosx_11_0_arm64.whl", hash = "sha256:cdf2a523d699b70d613243211ecaac14fe9c5df8a0b0a9c02add60fb2a413e0f", size = 79498248, upload-time = "2026-01-21T16:23:09.315Z" }, + { url = "https://files.pythonhosted.org/packages/4f/93/716b5ac0155f1be70ed81bacc21269c3ece8dba0c249b9994094110bfc51/torch-2.10.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:bf0d9ff448b0218e0433aeb198805192346c4fd659c852370d5cc245f602a06a", size = 79464992, upload-time = "2026-01-21T16:23:05.162Z" }, + { url = "https://files.pythonhosted.org/packages/69/2b/51e663ff190c9d16d4a8271203b71bc73a16aa7619b9f271a69b9d4a936b/torch-2.10.0-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:233aed0659a2503b831d8a67e9da66a62c996204c0bba4f4c442ccc0c68a3f60", size = 146018567, upload-time = "2026-01-21T16:22:23.393Z" }, + { url = "https://files.pythonhosted.org/packages/5e/cd/4b95ef7f293b927c283db0b136c42be91c8ec6845c44de0238c8c23bdc80/torch-2.10.0-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:682497e16bdfa6efeec8cde66531bc8d1fbbbb4d8788ec6173c089ed3cc2bfe5", size = 915721646, upload-time = "2026-01-21T16:21:16.983Z" }, + { url = "https://files.pythonhosted.org/packages/56/97/078a007208f8056d88ae43198833469e61a0a355abc0b070edd2c085eb9a/torch-2.10.0-cp314-cp314-win_amd64.whl", hash = "sha256:6528f13d2a8593a1a412ea07a99812495bec07e9224c28b2a25c0a30c7da025c", size = 113752373, upload-time = "2026-01-21T16:22:13.471Z" }, + { url = "https://files.pythonhosted.org/packages/d8/94/71994e7d0d5238393df9732fdab607e37e2b56d26a746cb59fdb415f8966/torch-2.10.0-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:f5ab4ba32383061be0fb74bda772d470140a12c1c3b58a0cfbf3dae94d164c28", size = 79850324, upload-time = "2026-01-21T16:22:09.494Z" }, + { url = "https://files.pythonhosted.org/packages/e2/65/1a05346b418ea8ccd10360eef4b3e0ce688fba544e76edec26913a8d0ee0/torch-2.10.0-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:716b01a176c2a5659c98f6b01bf868244abdd896526f1c692712ab36dbaf9b63", size = 146006482, upload-time = "2026-01-21T16:22:18.42Z" }, + { url = "https://files.pythonhosted.org/packages/1d/b9/5f6f9d9e859fc3235f60578fa64f52c9c6e9b4327f0fe0defb6de5c0de31/torch-2.10.0-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:d8f5912ba938233f86361e891789595ff35ca4b4e2ac8fe3670895e5976731d6", size = 915613050, upload-time = "2026-01-21T16:20:49.035Z" }, + { url = "https://files.pythonhosted.org/packages/66/4d/35352043ee0eaffdeff154fad67cd4a31dbed7ff8e3be1cc4549717d6d51/torch-2.10.0-cp314-cp314t-win_amd64.whl", hash = "sha256:71283a373f0ee2c89e0f0d5f446039bdabe8dbc3c9ccf35f0f784908b0acd185", size = 113995816, upload-time = "2026-01-21T16:22:05.312Z" }, +] + [[package]] name = "tornado" version = "6.5.4" @@ -5234,7 +6451,7 @@ wheels = [ [[package]] name = "transformers" -version = "4.57.3" +version = "4.53.3" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "filelock" }, @@ -5249,9 +6466,23 @@ dependencies = [ { name = "tokenizers" }, { name = "tqdm" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/dd/70/d42a739e8dfde3d92bb2fff5819cbf331fe9657323221e79415cd5eb65ee/transformers-4.57.3.tar.gz", hash = "sha256:df4945029aaddd7c09eec5cad851f30662f8bd1746721b34cc031d70c65afebc", size = 10139680, upload-time = "2025-11-25T15:51:30.139Z" } +sdist = { url = "https://files.pythonhosted.org/packages/f1/5c/49182918b58eaa0b4c954fd0e37c79fc299e5643e69d70089d0b0eb0cd9b/transformers-4.53.3.tar.gz", hash = "sha256:b2eda1a261de79b78b97f7888fe2005fc0c3fabf5dad33d52cc02983f9f675d8", size = 9197478, upload-time = "2025-07-22T07:30:51.51Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/41/b1/d7520cc5cb69c825599042eb3a7c986fa9baa8a8d2dea9acd78e152c81e2/transformers-4.53.3-py3-none-any.whl", hash = "sha256:5aba81c92095806b6baf12df35d756cf23b66c356975fb2a7fa9e536138d7c75", size = 10826382, upload-time = "2025-07-22T07:30:48.458Z" }, +] + +[[package]] +name = "triton" +version = "3.6.0" +source = { registry = "https://pypi.org/simple" } wheels = [ - { url = "https://files.pythonhosted.org/packages/6a/6b/2f416568b3c4c91c96e5a365d164f8a4a4a88030aa8ab4644181fdadce97/transformers-4.57.3-py3-none-any.whl", hash = "sha256:c77d353a4851b1880191603d36acb313411d3577f6e2897814f333841f7003f4", size = 11993463, upload-time = "2025-11-25T15:51:26.493Z" }, + { url = "https://files.pythonhosted.org/packages/8c/f7/f1c9d3424ab199ac53c2da567b859bcddbb9c9e7154805119f8bd95ec36f/triton-3.6.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a6550fae429e0667e397e5de64b332d1e5695b73650ee75a6146e2e902770bea", size = 188105201, upload-time = "2026-01-20T16:00:29.272Z" }, + { url = "https://files.pythonhosted.org/packages/e0/12/b05ba554d2c623bffa59922b94b0775673de251f468a9609bc9e45de95e9/triton-3.6.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e8e323d608e3a9bfcc2d9efcc90ceefb764a82b99dea12a86d643c72539ad5d3", size = 188214640, upload-time = "2026-01-20T16:00:35.869Z" }, + { url = "https://files.pythonhosted.org/packages/ab/a8/cdf8b3e4c98132f965f88c2313a4b493266832ad47fb52f23d14d4f86bb5/triton-3.6.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74caf5e34b66d9f3a429af689c1c7128daba1d8208df60e81106b115c00d6fca", size = 188266850, upload-time = "2026-01-20T16:00:43.041Z" }, + { url = "https://files.pythonhosted.org/packages/f9/0b/37d991d8c130ce81a8728ae3c25b6e60935838e9be1b58791f5997b24a54/triton-3.6.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10c7f76c6e72d2ef08df639e3d0d30729112f47a56b0c81672edc05ee5116ac9", size = 188289450, upload-time = "2026-01-20T16:00:49.136Z" }, + { url = "https://files.pythonhosted.org/packages/35/f8/9c66bfc55361ec6d0e4040a0337fb5924ceb23de4648b8a81ae9d33b2b38/triton-3.6.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d002e07d7180fd65e622134fbd980c9a3d4211fb85224b56a0a0efbd422ab72f", size = 188400296, upload-time = "2026-01-20T16:00:56.042Z" }, + { url = "https://files.pythonhosted.org/packages/df/3d/9e7eee57b37c80cec63322c0231bb6da3cfe535a91d7a4d64896fcb89357/triton-3.6.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a17a5d5985f0ac494ed8a8e54568f092f7057ef60e1b0fa09d3fd1512064e803", size = 188273063, upload-time = "2026-01-20T16:01:07.278Z" }, + { url = "https://files.pythonhosted.org/packages/f6/56/6113c23ff46c00aae423333eb58b3e60bdfe9179d542781955a5e1514cb3/triton-3.6.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:46bd1c1af4b6704e554cad2eeb3b0a6513a980d470ccfa63189737340c7746a7", size = 188397994, upload-time = "2026-01-20T16:01:14.236Z" }, ] [[package]] @@ -5398,6 +6629,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/6a/2a/dc2228b2888f51192c7dc766106cd475f1b768c10caaf9727659726f7391/virtualenv-20.36.1-py3-none-any.whl", hash = "sha256:575a8d6b124ef88f6f51d56d656132389f961062a9177016a50e4f507bbcc19f", size = 6008258, upload-time = "2026-01-09T18:20:59.425Z" }, ] +[[package]] +name = "waitress" +version = "3.0.2" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/bf/cb/04ddb054f45faa306a230769e868c28b8065ea196891f09004ebace5b184/waitress-3.0.2.tar.gz", hash = "sha256:682aaaf2af0c44ada4abfb70ded36393f0e307f4ab9456a215ce0020baefc31f", size = 179901, upload-time = "2024-11-16T20:02:35.195Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/8d/57/a27182528c90ef38d82b636a11f606b0cbb0e17588ed205435f8affe3368/waitress-3.0.2-py3-none-any.whl", hash = "sha256:c56d67fd6e87c2ee598b76abdd4e96cfad1f24cacdea5078d382b1f9d7b5ed2e", size = 56232, upload-time = "2024-11-16T20:02:33.858Z" }, +] + [[package]] name = "wandb" version = "0.23.1" @@ -5554,6 +6794,18 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/fa/a8/5b41e0da817d64113292ab1f8247140aac61cbf6cfd085d6a0fa77f4984f/websockets-15.0.1-py3-none-any.whl", hash = "sha256:f7a866fbc1e97b5c617ee4116daaa09b722101d4a3c170c787450ba409f9736f", size = 169743, upload-time = "2025-03-05T20:03:39.41Z" }, ] +[[package]] +name = "werkzeug" +version = "3.1.5" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "markupsafe" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/5a/70/1469ef1d3542ae7c2c7b72bd5e3a4e6ee69d7978fa8a3af05a38eca5becf/werkzeug-3.1.5.tar.gz", hash = "sha256:6a548b0e88955dd07ccb25539d7d0cc97417ee9e179677d22c7041c8f078ce67", size = 864754, upload-time = "2026-01-08T17:49:23.247Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/ad/e4/8d97cca767bcc1be76d16fb76951608305561c6e056811587f36cb1316a8/werkzeug-3.1.5-py3-none-any.whl", hash = "sha256:5111e36e91086ece91f93268bb39b4a35c1e6f1feac762c9c822ded0a4e322dc", size = 225025, upload-time = "2026-01-08T17:49:21.859Z" }, +] + [[package]] name = "widgetsnbextension" version = "4.0.15" @@ -5632,6 +6884,15 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/1f/f6/a933bd70f98e9cf3e08167fc5cd7aaaca49147e48411c0bd5ae701bb2194/wrapt-1.17.3-py3-none-any.whl", hash = "sha256:7171ae35d2c33d326ac19dd8facb1e82e5fd04ef8c6c0e394d7af55a55051c22", size = 23591, upload-time = "2025-08-12T05:53:20.674Z" }, ] +[[package]] +name = "xlsxwriter" +version = "3.2.9" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/46/2c/c06ef49dc36e7954e55b802a8b231770d286a9758b3d936bd1e04ce5ba88/xlsxwriter-3.2.9.tar.gz", hash = "sha256:254b1c37a368c444eac6e2f867405cc9e461b0ed97a3233b2ac1e574efb4140c", size = 215940, upload-time = "2025-09-16T00:16:21.63Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/3a/0c/3662f4a66880196a590b202f0db82d919dd2f89e99a27fadef91c4a33d41/xlsxwriter-3.2.9-py3-none-any.whl", hash = "sha256:9a5db42bc5dff014806c58a20b9eae7322a134abb6fce3c92c181bfb275ec5b3", size = 175315, upload-time = "2025-09-16T00:16:20.108Z" }, +] + [[package]] name = "xxhash" version = "3.6.0"