From bd48f05b309702698d40c806662620d8dcdf3bfd Mon Sep 17 00:00:00 2001 From: "J.Jason" <130959319+JJasonSun@users.noreply.github.com> Date: Mon, 24 Aug 2026 00:07:53 +0800 Subject: [PATCH 1/2] Add budgeted live validation evidence and harden ECNU skill --- .github/workflows/validate.yml | 11 + .gitignore | 1 + AGENTS.md | 101 +- README.md | 54 +- SKILL.md | 78 +- references/api_reference.md | 68 +- references/examples.md | 223 ++- references/known_deviations.md | 284 ++- references/models.md | 22 +- references/workflows.md | 131 +- scripts/smoke_test.py | 2882 +++++++++++++++++++++++++--- scripts/validate_skill.py | 292 ++- tests/test_repository_contracts.py | 114 ++ tests/test_smoke_test.py | 1009 +++++++++- 14 files changed, 4656 insertions(+), 614 deletions(-) create mode 100644 tests/test_repository_contracts.py diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 0139ca3..c0cf0ff 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -13,6 +13,17 @@ jobs: steps: - name: Check out repository uses: actions/checkout@v4 + - name: Set up Python + uses: actions/setup-python@v5 + with: + python-version: "3.11" + - name: Install compatibility-test dependencies + run: >- + python -m pip install + anthropic==0.125.0 + httpx==0.28.1 + langchain-openai==0.3.35 + openai==2.48.0 - name: Run repository validation run: python scripts/validate_skill.py - name: Run unit tests diff --git a/.gitignore b/.gitignore index 0bced4e..9389eef 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,4 @@ __pycache__/ *.py[cod] smoke-results*.json smoke-results/ +.live-artifacts/ diff --git a/AGENTS.md b/AGENTS.md index 4c1d42d..183ac2c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,85 +1,50 @@ # ecnu-api repository guide +## Purpose + This repository is an Agent Skills package for the ECNU LLM Open Platform API. The repository content is the skill; there is no production application. -## Scope - -- `SKILL.md` — concise activation and execution instructions -- `references/api_reference.md` — documented endpoint contracts -- `references/models.md` — models, aliases, credits, quotas, and deployment -- `references/examples.md` — minimal, safe examples -- `references/workflows.md` — implementation, debugging, retry, and test flows -- `references/known_deviations.md` — dated live observations only -- `scripts/smoke_test.py` — opt-in live structural checks -- `scripts/validate_skill.py` — deterministic repository validation -- `tests/` — offline tests for helper behavior +## File routing -Do not write outside the repository unless the user explicitly asks to install -or synchronize the skill into a client-specific directory. +- `SKILL.md` is the concise task entry point. +- `references/api_reference.md` contains documented endpoint contracts. +- `references/models.md` contains models, aliases, credits, and quotas. +- `references/examples.md` contains minimal safe examples. +- `references/workflows.md` contains executable integration and test flows. +- `references/known_deviations.md` contains dated live observations only. +- `scripts/smoke_test.py` performs opt-in live structural checks. +- `scripts/validate_skill.py` and `tests/` provide offline validation. -## Source precedence +## Editing rules -When ECNU documentation pages disagree: +- Treat current official ECNU documentation as the documented contract. +- Keep documented facts, live observations, application policy, and unverified + claims distinct. +- Do not invent undocumented fields, limits, model capabilities, or prices. +- Put point-in-time behavior only in `known_deviations.md` with dated evidence. +- Keep examples sequential, timeout-bounded, and environment-key based. +- Do not modify files outside this repository unless the user explicitly asks. -1. Use the current model page for model identity, context figures, aliases, and - capability labels. -2. Use the endpoint page for JSON fields, types, and endpoint-specific limits. -3. Use the quota page for current prices and quota periods. -4. Use release notes to establish when a change occurred. -5. Use `GET /models` for runtime visibility, not as the sole source of - capability truth. -6. Keep live probes in `known_deviations.md`; never let a single observation - silently override a documented contract. +## Validation -## Change workflow - -1. Work on a branch. -2. Identify the exact official pages affected by the change. -3. Update only the relevant focused reference. -4. If live testing is needed, use `ECNU_API_KEY` from the environment. -5. Never paste or persist a real key in a file, command example, report, issue, - commit, or pull request. -6. Run: +Run before committing: ```bash -python scripts/validate_skill.py -python -m unittest discover -s tests -v +python3 scripts/validate_skill.py +python3 -m unittest discover -s tests -v +python3 -m compileall scripts tests uvx --from skills-ref agentskills validate . ``` -7. Review the diff for secrets, machine-specific paths, duplicated guidance, - undocumented request fields, and accidental billable calls. -8. Summarize whether each changed claim is documented, observed, or unverified. - -## Live verification rules - -- The default smoke test performs model-list checks only. -- Chat, embedding, and Anthropic probes require `--low-cost` or `--anthropic`. -- Do not add image generation to an automatic or CI smoke test. -- Do not blindly retry image, TTS, or any other billable request after an - ambiguous network failure. -- Record SDK or Python version, account type, date, endpoint, status, content - type, and structural result. -- Sanitize reports before sharing or committing them. -- Update the date in `known_deviations.md` only when the behavior was actually - reproduced. - -## Content conventions - -- Write documentation in English; retain official Chinese UI labels where - needed. -- Prefer imperative, stepwise instructions over broad prose. -- Keep `SKILL.md` below the Agent Skills recommended size and route detail to - focused references. -- Never describe `skills-ref` validation as an API correctness test. -- Never claim undocumented limits. -- Never describe an output dimension as a request parameter unless ECNU - documents it. -- Use environment variables in every credential example. +Review `git diff --check` and scan tracked content for secrets and personal paths. -## Current state +## Safety -The reference content is aligned with the ECNU documentation and repository -observations available on 2026-08-22. The live deviations remain dated -2026-08-21 until a new authenticated run reproduces or supersedes them. +- Read live credentials only from `ECNU_API_KEY`; never accept a CLI key. +- Never commit keys, Authorization values, private inputs, raw responses, + generated media, one-time URLs, or full reasoning content. +- Keep API calls serial and enforce the declared credit ceiling. +- Do not automatically retry POST requests after ambiguous transport failures. +- Record only sanitized response structure and allowlisted diagnostic headers. +- Do not update an observation date unless the behavior was reproduced. diff --git a/README.md b/README.md index a35867e..8ff4f63 100644 --- a/README.md +++ b/README.md @@ -46,6 +46,7 @@ ecnu-api/ │ ├── smoke_test.py │ └── validate_skill.py ├── tests/ +│ ├── test_repository_contracts.py │ └── test_smoke_test.py └── .github/workflows/validate.yml ``` @@ -74,41 +75,64 @@ export ECNU_API_KEY="your-api-key" A key pasted into a chat or public location should be revoked or rotated after testing. -## Reproducible smoke tests +## Reproducible live validation -The default profile performs model-list checks and does not send chat, -embedding, Anthropic, image, or TTS POST requests: +The runner reads only `ECNU_API_KEY`, sends requests serially, and does not +retry POST requests. Select the smallest profile that answers the question: + +| Profile | Scope | +|---|---| +| `auth` | Service status plus valid, invalid, and missing-token model discovery; no billable POST requests. This is the default. | +| `core` | Low-cost Chat Completions, Responses, embeddings, rerank, vision, structured output, error-shape, OpenAI SDK, and LangChain probes. | +| `compatibility` | Responses vision and Anthropic-compatible models, aliases, effort controls, long-context suffix behavior, vision, and SDK probes. | +| `billable` | Fixed-price TTS and one documented image-generation probe, subject to the credit ceiling. | +| `all` | The union of all four profiles; later billable cases are skipped when the ceiling is reached. | + +Examples: ```bash -python scripts/smoke_test.py +python3 scripts/smoke_test.py --profile auth --max-credits 0 --output .live-artifacts/auth.json +python3 scripts/smoke_test.py --profile core --max-credits 50 --output .live-artifacts/core.json +python3 scripts/smoke_test.py --profile compatibility --max-credits 50 --output .live-artifacts/compatibility.json +python3 scripts/smoke_test.py --profile billable --max-credits 50 --output .live-artifacts/billable.json +python3 scripts/smoke_test.py --profile all --max-credits 50 --output .live-artifacts/all.json ``` -Low-cost POST probes are explicit: +`--max-credits` is a conservative planned-cost gate, defaulting to 50. The +runner reserves each case's estimate before sending it and skips a case that +would exceed the ceiling. The estimate is not proof of the service's actual +debit. Recheck the official quota and pricing page before a live run. + +Use `--case` to rerun only named cases within the selected profile; repeat the +flag to select more than one: ```bash -python scripts/smoke_test.py --low-cost --anthropic \ - --account-type personal-token \ - --output smoke-results.json +python3 scripts/smoke_test.py --profile core --case openai_sdk_chat \ + --max-credits 1 --output .live-artifacts/openai-sdk-chat.json ``` -The report contains statuses and structural summaries. It does not print the -API key or successful model content. Image generation is intentionally absent -from the automated smoke test because it is comparatively expensive and a -retry after an ambiguous failure could duplicate charges. +Keep reports under `.live-artifacts/`, which is Git-ignored. Reports contain +statuses and structural summaries, not the API key, generated content, +reasoning text, media, or one-time URLs. + +Selected SDK probes on 2026-08-23 passed with OpenAI Python SDK 2.48.0, +Anthropic Python SDK 0.125.0, `langchain-openai` 0.3.35, and `httpx` 0.28.1. +This is dated, point-in-time evidence, not a blanket compatibility guarantee; +see `references/known_deviations.md` for the observed scope and divergences. ## Validate the skill Run deterministic repository checks and unit tests: ```bash -python scripts/validate_skill.py -python -m unittest discover -s tests -v +python3 scripts/validate_skill.py +python3 -m unittest discover -s tests -v ``` Run the Agent Skills reference validator separately: ```bash -uvx --from skills-ref agentskills validate . +uvx --from skills-ref agentskills validate "$PWD" ``` The reference validator checks format and naming conventions; it does not diff --git a/SKILL.md b/SKILL.md index 48dfd06..634bb52 100644 --- a/SKILL.md +++ b/SKILL.md @@ -15,7 +15,7 @@ description: > Use this skill to turn ECNU API documentation into a safe, verifiable integration. The current official ECNU developer documentation is the authority for documented contracts. Keep documented facts, live observations, -and application policy separate. +application policy, and unverified claims separate. ## Core rules @@ -131,55 +131,89 @@ names as compatibility aliases. Before a real request: -- use an environment variable such as `ECNU_API_KEY`; +- recheck the current official quota and pricing page, then calculate a + conservative planned cost from those documented prices; +- use 50 credits as the default ceiling and do not run a larger plan without + separate user authorization; +- use only `ECNU_API_KEY` from the environment; never accept a key through a + command-line argument; - remove secrets and unnecessary personal or confidential data; - confirm the user intended to send the supplied content to ECNU; -- state when image generation, TTS, or other calls may consume credits; -- never blindly retry a billable request after an ambiguous timeout. +- execute requests serially; and +- never retry a POST after an ambiguous timeout or connection failure. + +If the full plan exceeds 50 credits, preserve the core dialog, embedding, +rerank, compatibility, and error checks; prefer one TTS PCM check; run at most +one documented image-generation case; and skip expanded voices and +undocumented model probes. If a key has already been pasted into a chat or public location, recommend revoking or rotating it after testing. ### 7. Execute and verify -For reproducible checks, run: - -```bash -python scripts/smoke_test.py -``` - -This default profile performs model-list checks only. Low-cost POST probes are -opt-in: +Use the smallest profile that answers the question. Sanitized reports belong +under the ignored `.live-artifacts/` directory: ```bash -python scripts/smoke_test.py --low-cost --anthropic +python3 scripts/smoke_test.py --profile auth --max-credits 0 --output .live-artifacts/auth.json +python3 scripts/smoke_test.py --profile core --max-credits 50 --output .live-artifacts/core.json +python3 scripts/smoke_test.py --profile compatibility --max-credits 50 --output .live-artifacts/compatibility.json +python3 scripts/smoke_test.py --profile billable --max-credits 50 --output .live-artifacts/billable.json +python3 scripts/smoke_test.py --profile all --max-credits 50 --output .live-artifacts/all.json ``` -The script reads `ECNU_API_KEY`, redacts key-shaped strings, and emits a -structural JSON report rather than model output. +The default profile is `auth`. The runner reads `ECNU_API_KEY`, executes +serially with POST retries disabled, reserves estimated credits before each +request, skips cases that would exceed the ceiling, and emits response +structure rather than generated content. A credit estimate is not proof of the +service's actual debit. ### 8. Report provenance Label important conclusions as one of: -- **Documented** — supported by the current official ECNU documentation. -- **Observed** — reproduced against the live service at a stated date. -- **Unverified** — inferred, historical, or not reproducible in the current +- **`documented`** — supported by the current official ECNU documentation. +- **`observed`** — reproduced against the live service at a stated date. +- **`application-policy`** — a local safety, cost, or reliability constraint; + not an ECNU platform guarantee. +- **`unverified`** — inferred, historical, or not reproducible in the current environment. Do not silently promote an observed deviation into a documented guarantee. +### 9. Finish repository work + +When repository files changed, finish with offline, format, and secret checks: + +```bash +python3 scripts/validate_skill.py +python3 -m unittest discover -s tests -v +python3 -m compileall scripts tests +uvx --from skills-ref agentskills validate "$PWD" +git grep -nE 'sk-[A-Za-z0-9_-]{16,}' +git grep -nE 'Authorization:[[:space:]]*Bearer[[:space:]]+[^<"$]' +git diff --check +``` + +Review every secret-scan match; no tracked literal credential may remain. +Variable-based test fixtures may match the coarse Bearer expression. If `uvx` +is not available, report that validator as not run rather than treating it as +live API evidence. + ## High-value gotchas - `GET /models` is runtime discovery, not a reliable authentication test. - A model appearing in `/models` does not prove that a capability is usable. - Use `ecnu-max[1m]` only when an Anthropic tool requires the suffix to - advertise long context. Fall back to plain `ecnu-max` if the suffix returns - an authentication or metadata error. + advertise long context. Consider plain `ecnu-max` only when the same + credential already succeeds with that model, the suffixed request returns + the observed suffix-specific `401` metadata error, and the caller accepts + the shorter advertised context. - TTS errors may not match the documented JSON shape; preserve the HTTP status, content type, and a bounded redacted body sample. -- Do not depend on optional PCM metadata headers without checking them at - runtime. +- Do not assume the documented PCM metadata headers are present; check them at + runtime and configure the format explicitly when they are absent. - `422` means request validation failed; inspect `detail`. - `429` may represent quota exhaustion, rate control, or short-term service protection. Stop parallel retries and inspect credits first. diff --git a/references/api_reference.md b/references/api_reference.md index ed38d7e..daf6a3f 100644 --- a/references/api_reference.md +++ b/references/api_reference.md @@ -76,6 +76,10 @@ Documented request fields include: | `response_format` | object | Structured output | | `max_tokens` | integer | Use enough room for complete output | +A live-verified tool-result continuation preserves the assistant tool call, +then adds a message with `role: "tool"`, the matching `tool_call_id`, and the +tool result in `content`. + Pass ECNU-specific fields through `extra_body` when using the OpenAI Python SDK. @@ -91,6 +95,10 @@ A non-streaming response follows the OpenAI completion-list shape with `choices[].message`, `finish_reason`, and `usage`. For streaming, parse SSE `data:` lines and stop at `[DONE]`. +Do not require a response `model` value to equal the requested model name. The +official examples either omit that field or show a backend label different from +the requested name. Treat it as response metadata, not a stable alias echo. + Native `search_mode` web search was removed. Implement search through tool calling or an external search service. @@ -216,10 +224,37 @@ POST https://chat.ecnu.edu.cn/open/api/v1/audio/speech | `response_format` | string | No | `mp3`, `opus`, `aac`, `flac`, `wav`, `pcm` | | `speed` | number | No | 0.25 through 4.0 | -The success body is binary audio with a format-specific content type. Do not -parse it as JSON. The official docs and model page may not list voices in the -same place or at the same update time; consult the current TTS page before -validating a voice ID. +The 16 documented voice IDs are: + +| Category | Voice IDs | +|---|---| +| Campus | `xiayu`, `liwa` | +| Male | `male_warm`, `male_steady`, `male_news`, `male_philosophy`, `yunze` | +| Female | `female_sweet`, `female_literary`, `female_news` | +| Dialect | `sichuan`, `tianjin`, `shaanxi` | +| Multilingual and roles | `japanese`, `lindaiyu`, `labixiaoxin` | + +The success body is binary audio with a format-specific `Content-Type` and a +`Content-Disposition` header containing a suggested filename. Do not parse it +as JSON. For `pcm`, the documented response also includes `Content-Rate` +(sampling rate), `Content-Channels` (fixed at 1), and `Content-Bits` (fixed at +16). + +Invalid parameters are documented to return `400` JSON with this shape: + +```json +{ + "error": "", + "request_id": "", + "details": { + "available_voices": ["xiayu", "liwa"] + } +} +``` + +The documented `details` object supplies applicable supplemental information; +the invalid-voice example uses `available_voices`. Other documented messages +cover missing input, out-of-range speed, and unsupported response formats. "Batch TTS" examples are sequential client loops, not one batch request. @@ -229,10 +264,16 @@ validating a voice ID. GET https://chat.ecnu.edu.cn/open/api/v1/models ``` -The documented response is an OpenAI-style list with `data[].id`, `object`, -`created`, and `owned_by`. Use it for runtime visibility, then consult the model -documentation for capabilities, aliases, and prices. Do not treat visibility -alone as a capability guarantee or an authentication check. +The official request example uses bearer authentication and has no request +parameters. The documented response is an OpenAI-style list with a top-level +`object: "list"` and model entries in `data`; each entry has `id`, `object` +(fixed to `model`), `created`, and `owned_by`. + +Use this endpoint for runtime visibility, then consult the model documentation +for capabilities, aliases, and prices. Do not treat visibility alone as a +capability guarantee. Dated runtime differences, including authentication +behavior, belong in [known_deviations.md](known_deviations.md) and do not change +the documented contract here. ## Structured output @@ -266,9 +307,13 @@ Set: ```text ANTHROPIC_BASE_URL=https://chat.ecnu.edu.cn/open/api/anthropic -ANTHROPIC_AUTH_TOKEN= +ECNU_API_KEY= ``` +Pass `ECNU_API_KEY` explicitly to the Anthropic SDK as its `api_key`. Only if a +generic Anthropic client cannot accept that variable name, map the same runtime +value to the client-specific token variable without logging or persisting it. + Documented mappings: | Requested name | Effective model | @@ -283,6 +328,10 @@ The documentation describes `ecnu-max[1m]` for Anthropic tools that inspect the model name to advertise a 1M-character context. Treat the suffix as compatibility metadata, not a model name for OpenAI-compatible endpoints. +These mappings describe internal compatibility routing. The Anthropic page does +not document whether a response `model` value echoes the requested alias or +names the effective ECNU model, so clients must not depend on either behavior. + `output_config.effort` controls thinking intensity for `ecnu-max`; `none` disables thinking. `ecnu-plus` ignores this field. @@ -300,6 +349,7 @@ or persist them. Tickets are one-time use and expire. | Status | Typical meaning | |---|---| +| `400` | Invalid TTS parameters; `/audio/speech` documents an endpoint-specific JSON error | | `401` | Missing or invalid credentials | | `403` | Application or client IP is not authorized | | `422` | Request body validation failed | diff --git a/references/examples.md b/references/examples.md index 39651d4..4359619 100644 --- a/references/examples.md +++ b/references/examples.md @@ -3,10 +3,16 @@ These examples use environment variables, explicit request shapes, and sequential calls. They avoid undocumented parameters. +The OpenAI SDK chat, Responses, and embedding paths, the Anthropic SDK +`ecnu-plus` path, and the LangChain embedding path below were live-verified on +**2026-08-23** with Python 3.9.6, OpenAI 2.48.0, Anthropic 0.125.0, and +langchain-openai 0.3.35. This is dated compatibility evidence, not a guarantee +for other versions. + ## Setup ```bash -pip install openai requests +python3 -m pip install openai requests export ECNU_API_KEY="your-api-key" ``` @@ -19,9 +25,14 @@ api_key = os.environ["ECNU_API_KEY"] client = OpenAI( api_key=api_key, base_url="https://chat.ecnu.edu.cn/open/api/v1", + timeout=60.0, + max_retries=0, ) ``` +Disabling SDK retries prevents an ambiguous POST failure from being submitted +again. Treat such a transport failure as inconclusive. + ## Chat Completions ```python @@ -33,7 +44,10 @@ completion = client.chat.completions.create( ], ) -print(completion.choices[0].message.content) +if not completion.choices: + raise RuntimeError("chat response contained no choices") +content = completion.choices[0].message.content or "" +print(content) ``` ## Responses API @@ -44,7 +58,10 @@ response = client.responses.create( input="用一句话介绍华东师范大学。", ) -print(response.output_text) +text = getattr(response, "output_text", None) or "" +if not text: + raise RuntimeError("response contained no text output") +print(text) ``` Start with text input. Verify advanced Responses tools and streaming events @@ -64,13 +81,16 @@ completion = client.chat.completions.create( }, ) +if not completion.choices: + raise RuntimeError("thinking response contained no choices") message = completion.choices[0].message -answer = message.content +answer = message.content or "" print(answer) ``` -Do not print hidden reasoning in user-facing applications. Preserve -`reasoning_content` only when required for a subsequent tool-using turn. +Do not print or persist hidden reasoning. Keep `reasoning_content` only in +memory when it is required for the immediately following tool-using turn, then +discard it. ## Streaming @@ -82,9 +102,12 @@ stream = client.chat.completions.create( ) for chunk in stream: - delta = chunk.choices[0].delta.content - if delta: - print(delta, end="", flush=True) + choices = getattr(chunk, "choices", None) or [] + if not choices: # Usage-only and keepalive chunks may have no choices. + continue + text = getattr(choices[0].delta, "content", None) + if text: + print(text, end="", flush=True) ``` ## Tool calling @@ -113,7 +136,9 @@ completion = client.chat.completions.create( tools=tools, ) -for call in completion.choices[0].message.tool_calls or []: +if not completion.choices: + raise RuntimeError("tool response contained no choices") +for call in getattr(completion.choices[0].message, "tool_calls", None) or []: print(call.id, call.function.name, call.function.arguments) ``` @@ -139,7 +164,9 @@ completion = client.chat.completions.create( ], ) -print(completion.choices[0].message.content) +if not completion.choices: + raise RuntimeError("vision response contained no choices") +print(completion.choices[0].message.content or "") ``` For a local image, base64-encode it into a data URL. Do not assume undocumented @@ -155,6 +182,8 @@ response = client.embeddings.create( input="华东师范大学", ) +if not response.data: + raise RuntimeError("embedding response contained no vectors") vector = response.data[0].embedding assert len(vector) == 1024 ``` @@ -177,6 +206,8 @@ response = client.embeddings.create( ordered = sorted(response.data, key=lambda item: item.index) vectors = [item.embedding for item in ordered] +if len(vectors) != len(texts): + raise RuntimeError("embedding response count did not match input count") assert all(len(vector) == 1024 for vector in vectors) ``` @@ -185,7 +216,7 @@ Do not send integer token IDs. ### LangChain embeddings ```bash -pip install langchain-openai +python3 -m pip install langchain-openai ``` ```python @@ -196,6 +227,8 @@ embeddings = OpenAIEmbeddings( api_key=api_key, model="ecnu-embedding-small", check_embedding_ctx_length=False, + timeout=60.0, + max_retries=0, ) vector = embeddings.embed_query("Hello world") @@ -269,9 +302,18 @@ The local `top_n` check is application policy, not a published ECNU maximum. ## Image generation -This call consumes credits. Do not run it merely to validate code. +This call consumes credits. Do not run it merely to validate code. Use a +one-time URL only in memory and transfer the response to controlled storage +before its 24-hour expiry. ```python +import os +from pathlib import Path +from tempfile import NamedTemporaryFile +from urllib.parse import urlsplit + +import requests + response = client.images.generate( model="ecnu-image", prompt="水墨风，竹林，渔船，湖泊，带斗笠的老翁", @@ -279,11 +321,59 @@ response = client.images.generate( response_format="url", ) -print(response.data[0].url) +item = response.data[0] if response.data else None +image_url = getattr(item, "url", None) +if not image_url: + raise RuntimeError("image response contained no URL") + +approved_hosts = { + host.strip().lower() + for host in os.environ["APPROVED_IMAGE_HOSTS"].split(",") + if host.strip() +} +parsed_url = urlsplit(str(image_url)) +hostname = parsed_url.hostname +if parsed_url.scheme.lower() != "https" or not hostname or hostname.lower() not in approved_hosts: + raise RuntimeError("image URL host is not approved") + +target = Path(".live-artifacts") / "generated-image.bin" +target.parent.mkdir(parents=True, exist_ok=True) +temporary = None +max_image_bytes = 20 * 1024 * 1024 # Application policy, not an ECNU limit. +try: + try: + with requests.get( + str(image_url), stream=True, timeout=(5, 60), allow_redirects=False + ) as download: + if download.is_redirect: + raise RuntimeError("redirect target requires separate validation") + download.raise_for_status() + media_type = download.headers.get("Content-Type", "").split(";", 1)[0] + if not media_type.startswith("image/"): + raise RuntimeError("download did not return an image") + with NamedTemporaryFile("wb", dir=target.parent, delete=False) as output: + temporary = Path(output.name) + total = 0 + for block in download.iter_content(64 * 1024): + if not block: + continue + total += len(block) + if total > max_image_bytes: + raise RuntimeError("download exceeded the application limit") + output.write(block) + if total == 0: + raise RuntimeError("download returned an empty image") + except requests.RequestException: + raise RuntimeError("image download failed") from None + temporary.replace(target) +finally: + if temporary is not None and temporary.exists(): + temporary.unlink() ``` -Transfer URL results before their 24-hour expiry. Do not blindly retry after an -ambiguous timeout. +Do not print or log `image_url`. Configure `APPROVED_IMAGE_HOSTS` from an +application-owned egress policy and validate every redirect separately. Do not +retry generation after an ambiguous timeout. ## Text-to-speech @@ -298,7 +388,7 @@ response = client.audio.speech.create( speed=1.0, ) -response.stream_to_file("output.mp3") +response.stream_to_file(".live-artifacts/output.mp3") ``` Multiple texts require separate sequential calls. They are not one batch API @@ -341,6 +431,8 @@ completion = client.chat.completions.create( max_tokens=512, ) +if not completion.choices or not completion.choices[0].message.content: + raise RuntimeError("structured response contained no content") result = json.loads(completion.choices[0].message.content) print(result) ``` @@ -348,15 +440,20 @@ print(result) ## Anthropic compatibility ```bash -pip install anthropic -export ANTHROPIC_BASE_URL="https://chat.ecnu.edu.cn/open/api/anthropic" -export ANTHROPIC_AUTH_TOKEN="$ECNU_API_KEY" +python3 -m pip install anthropic ``` ```python +import os + import anthropic -anthropic_client = anthropic.Anthropic() +anthropic_client = anthropic.Anthropic( + api_key=os.environ["ECNU_API_KEY"], + base_url="https://chat.ecnu.edu.cn/open/api/anthropic", + timeout=60.0, + max_retries=0, +) message = anthropic_client.messages.create( model="ecnu-plus", @@ -369,21 +466,30 @@ message = anthropic_client.messages.create( ], ) -print(message.content) +text = "".join( + block.text + for block in message.content + if getattr(block, "type", None) == "text" +) +print(text) ``` Use plain `ecnu-max` by default. Only try `ecnu-max[1m]` when an Anthropic tool must recognize the long-context suffix: ```python -def create_long_context_message(client, messages): +def create_long_context_message( + client, messages, *, plain_max_previously_verified=False +): try: return client.messages.create( model="ecnu-max[1m]", max_tokens=1000, messages=messages, ) - except anthropic.AuthenticationError: + except anthropic.AuthenticationError as exc: + if exc.status_code != 401 or not plain_max_previously_verified: + raise return client.messages.create( model="ecnu-max", max_tokens=1000, @@ -391,30 +497,73 @@ def create_long_context_message(client, messages): ) ``` -Log the fallback without logging prompts or credentials. The fallback preserves -model access but may not advertise the same context capability to the client. +Use this one-shot fallback only after plain `ecnu-max` has already succeeded +with the same credential and the caller accepts losing the `[1m]` capability +signal. Log only that the fallback occurred, never the prompt, credential, or +error body. ## Error handling -Preserve string and array forms of `detail`, and tolerate non-JSON errors: +Preserve JSON error structure, tolerate non-JSON errors, and retain only a +bounded, redacted sample plus an allowlisted request ID: ```python -def raise_ecnu_error(response): - if response.ok: - return +import json +import re + +SENSITIVE_KEYS = { + "authorization", "x_api_key", "api_key", "auth_token", "token", + "access_token", "client_secret", "ticket", "reasoning_content", + "messages", "input", "prompt", "content", "url", "download_url", + "image_url", "b64_json", "base64", "audio", "data", +} +SECRET_TEXT = re.compile( + r"(?i)(bearer\s+)[^\s\"']+|\bsk-[A-Za-z0-9_-]{8,}\b|" + r"https?://[^\s\"']+|data:[^,\s]+;base64,[A-Za-z0-9+/=_-]+" +) + + +def sanitize_error(value): + if isinstance(value, dict): + return { + str(key): ( + "[REDACTED]" + if str(key).lower().replace("-", "_") in SENSITIVE_KEYS + else sanitize_error(item) + ) + for key, item in value.items() + } + if isinstance(value, list): + return [sanitize_error(item) for item in value] + if isinstance(value, str): + return SECRET_TEXT.sub("[REDACTED]", value) + return value + +def bounded_error_sample(response, limit=1000): try: - payload = response.json() + value = response.json() + sample = json.dumps(sanitize_error(value), ensure_ascii=False) except ValueError: - payload = None + sample = SECRET_TEXT.sub("[REDACTED]", response.text) + return sample if len(sample) <= limit else sample[: limit - 14] + "...[truncated]" - if isinstance(payload, dict) and "detail" in payload: - detail = payload["detail"] - else: - detail = response.text[:1000] +def raise_ecnu_error(response): + if response.ok: + return + + request_id = next( + ( + response.headers.get(name) + for name in ("X-Request-ID", "Request-ID", "X-Trace-ID") + if response.headers.get(name) + ), + "unavailable", + ) raise RuntimeError( - f"ECNU API HTTP {response.status_code}: {detail}" + f"ECNU API HTTP {response.status_code} " + f"request_id={str(request_id)[:200]}: {bounded_error_sample(response)}" ) ``` diff --git a/references/known_deviations.md b/references/known_deviations.md index 16339c3..123262d 100644 --- a/references/known_deviations.md +++ b/references/known_deviations.md @@ -1,52 +1,236 @@ # Known Service Deviations -These are point-in-time observations from a personal-token test on -**2026-08-21**. They are not official contracts. Re-run a controlled smoke test -before relying on them, and update this file only when a result is actually -reproduced. - -## Observation matrix - -| Area | Documented expectation | Observed behavior | Application fallback | -|---|---|---|---| -| `/models`, invalid token | authentication error | `200` with an empty model list | never use an empty list as an authentication check | -| `/models`, missing auth | authentication error | `500` with an HTML error embedded in the response | preserve content type/body; report auth as inconclusive | -| runtime model list | documented models | included `ecnu-image-pro`, absent from model page | do not select undocumented IDs without an endpoint probe | -| `ecnu-image-pro` generation | not documented | probe returned plain-text `500` | use documented `ecnu-image` | -| TTS invalid voice | documented `400` JSON details | plain-text `500` | preserve status/content type; do not assume JSON | -| TTS PCM metadata | documented format metadata headers | `Content-Rate`, `Content-Channels`, and `Content-Bits` absent | inspect headers and require caller-side audio configuration | -| Anthropic `ecnu-max[1m]` | suffix stripped and long context advertised | `401` third-party metadata failure; plain `ecnu-max` worked | fall back to plain `ecnu-max` and report context-advertising difference | - -## Verified in the same 2026-08-21 run - -The repository maintainer recorded successful checks for: - -- Chat Completions; -- thinking with `reasoning_effort`; -- non-tool multi-turn handling; -- tool calling; -- vision content parts; -- structured output; -- Responses API and `reasoning.effort`; -- scalar and array embeddings with 1024-value outputs; -- rerank; -- Anthropic model mapping and `output_config.effort`; -- documented image generation; -- default and additional TTS voices; -- `422` validation response shape. - -These statements remain dated observations. They are not substitutes for -current production monitoring. - -## How to update this file - -1. Run a controlled authenticated test. -2. Record UTC date, account type, Python/SDK version, endpoint, status, content - type, and structural result. -3. Remove keys, prompts, generated content, one-time URLs, and personal data. -4. Reproduce an anomaly before replacing an existing observation. -5. Keep the documented expectation in `api_reference.md`; keep only the - deviation here. -6. Add or update an application fallback. -7. Do not write "fixed" merely because one subsequent request succeeded; note - the evidence and date. +These entries are point-in-time observations, not official contracts. Preserve +the documented expectation in `api_reference.md`. Do not change a test date or +status unless the behavior was actually exercised again. + +Test environment `live-2026-08-23-a` was a personal token on macOS arm64 +(Darwin 25.6.0), Python 3.9.6, OpenAI 2.48.0, Anthropic 0.125.0, +langchain-openai 0.3.35, httpx 0.28.1, direct HTTP where noted, and UTC report +timestamps. No private input or generated content was retained. + +## Invalid bearer on model discovery + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `GET /models` +- **Documented expectation:** Missing or invalid authentication returns `401`. +- **Observed behavior:** An obviously invalid bearer returned `200` JSON with an empty model list. +- **Reproduction conditions:** Send `GET /models` with a non-secret invalid bearer value. +- **Impact:** An empty list can be mistaken for authenticated discovery. +- **Recommended fallback:** Require a non-empty valid-token list; do not use an empty list as an auth check. +- **Status:** active + +## Missing authorization on model discovery + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `GET /models` +- **Documented expectation:** Missing authentication returns `401`. +- **Observed behavior:** The response was `500 application/json` with an HTML 500 page stored in the JSON `error` string. +- **Reproduction conditions:** Send `GET /models` without an Authorization header. +- **Impact:** Status-only or HTML-only handling can misclassify the failure. +- **Recommended fallback:** Preserve status, content type, request ID, and a bounded redacted body sample. +- **Status:** active + +## Undocumented model visible at runtime + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `GET /models` +- **Documented expectation:** Runtime discovery is interpreted with the current model page. +- **Observed behavior:** The seven-model list included `ecnu-image-pro`, which was absent from the model page. +- **Reproduction conditions:** Compare a valid-token model list with the current official model page. +- **Impact:** Visibility alone may lead callers to select an undocumented ID. +- **Recommended fallback:** Classify it as visible-but-undocumented and do not select it without an authorized endpoint probe. +- **Status:** active + +## Undocumented image model generation + +- **Tested at:** 2026-08-21 +- **Environment:** Personal token; exact OS, Python, SDK, and transport versions were not recorded. +- **Protocol and endpoint:** OpenAI-compatible `POST /images/generations` +- **Documented expectation:** `ecnu-image-pro` has no documented generation contract. +- **Observed behavior:** One probe returned plain-text `500 Internal Server Error`. +- **Reproduction conditions:** Submit one minimal request using the runtime-only model ID. +- **Impact:** A visible ID was not verifiably usable and the POST may have cost implications. +- **Recommended fallback:** Use documented `ecnu-image`; do not retry or routinely probe undocumented image models. +- **Status:** not-retested + +## Invalid TTS voice error shape + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /audio/speech` +- **Documented expectation:** Invalid voice input returns `400` JSON with `error`, `request_id`, and `details`. +- **Observed behavior:** An invalid voice returned `500 text/plain` with a bounded internal-error string. +- **Reproduction conditions:** Submit the short text `你好。` with an obviously invalid voice ID. +- **Impact:** Clients cannot rely on the documented JSON voice list. +- **Recommended fallback:** Tolerate non-JSON errors and retain only a bounded redacted sample. +- **Status:** active + +## Successful TTS response headers + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /audio/speech` +- **Documented expectation:** Successful audio includes `Content-Disposition`; PCM also includes rate, channel, and bit-depth headers. +- **Observed behavior:** Short `xiayu` PCM and MP3 returned non-empty audio-typed bodies but no `Content-Disposition`; PCM also omitted all three format headers. +- **Reproduction conditions:** Request `你好。` with `xiayu` as PCM and MP3 in separate sequential calls. +- **Impact:** Callers cannot derive filenames or raw PCM configuration from the response. +- **Recommended fallback:** Validate MIME and bytes, choose a local filename, and configure PCM explicitly when headers are absent. +- **Status:** active + +## Anthropic long-context suffix metadata + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP with Bearer auth +- **Protocol and endpoint:** Anthropic-compatible `POST /v1/messages` +- **Documented expectation:** The proxy strips `[1m]`, routes to `ecnu-max`, and advertises 1M-character context. +- **Observed behavior:** `ecnu-max[1m]` returned metadata-related `401`; the immediate plain `ecnu-max` control returned `200`. +- **Reproduction conditions:** Compare otherwise identical short requests for suffixed and plain model names. +- **Impact:** Model access can work while the long-context compatibility signal fails. +- **Recommended fallback:** Only after the same credential has passed a plain `ecnu-max` control, fall back once when the suffix returns this exact metadata-related `401`; disclose the loss of the 1M signal and require the caller to accept it. +- **Status:** active + +## Anthropic effort `none` + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP with Bearer auth +- **Protocol and endpoint:** Anthropic-compatible `POST /v1/messages` +- **Documented expectation:** `output_config.effort: none` disables thinking. +- **Observed behavior:** The request returned structured `422`; the accepted-value message listed `low`, `medium`, `high`, `xhigh`, and `max` only. +- **Reproduction conditions:** Send a short `ecnu-max` request with effort `none`. +- **Impact:** Clients following the documented disable value fail validation. +- **Recommended fallback:** Omit `output_config` when thinking is not required; do not translate `none` into a supported tier silently. +- **Status:** active + +## Empty embedding array error + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /embeddings` +- **Documented expectation:** `input` is a string or string array; invalid request shapes use structured validation errors. +- **Observed behavior:** An empty string array returned plain-text `500 Internal Server Error`. +- **Reproduction conditions:** Send `input: []` with `ecnu-embedding-small`. +- **Impact:** A deterministic client error appears as a server failure. +- **Recommended fallback:** Reject empty arrays locally and do not retry the unchanged POST. +- **Status:** active + +## Embedding 8192-character boundary + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /embeddings` +- **Documented expectation:** Input is limited to 8192 characters. +- **Observed behavior:** Both 8192 and 8193 ASCII characters returned `200` with one 1024-value vector. +- **Reproduction conditions:** Send separate scalar inputs of exactly 8192 and 8193 characters. +- **Impact:** The documented boundary was not enforced by this deployment. +- **Recommended fallback:** Continue enforcing 8192 characters in clients; do not promote one accepted over-limit request into a new limit. +- **Status:** active + +## Rerank document 8192-character boundary + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** Cohere-compatible `POST /rerank` +- **Documented expectation:** Each document is limited to 8192 characters. +- **Observed behavior:** A single 8193-character document returned `200` with a scored result. +- **Reproduction conditions:** Rerank one 8193-character document against a short query. +- **Impact:** The documented boundary was not enforced by this deployment. +- **Recommended fallback:** Keep the documented 8192-character client guard. +- **Status:** active + +## Invalid Chat reasoning effort error + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** Direct `reasoning_effort` accepts `low`, `high`, or `max`; invalid fields normally produce structured validation errors. +- **Observed behavior:** An invalid value returned plain-text `500 Internal Server Error`. +- **Reproduction conditions:** Enable thinking for `ecnu-max` and send an obviously invalid effort value. +- **Impact:** Callers cannot inspect a field path or accepted values. +- **Recommended fallback:** Validate the three documented values locally and never retry the same invalid request. +- **Status:** active + +## Out-of-range Chat temperature + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** `temperature` is between 0 and 1. +- **Observed behavior:** A short `ecnu-plus` request with `temperature: 2` returned `200` and a normal completion shape. +- **Reproduction conditions:** Send the otherwise minimal deterministic Chat request with temperature 2. +- **Impact:** Current server acceptance can hide invalid application configuration. +- **Recommended fallback:** Enforce the documented range client-side. +- **Status:** active + +## Unsupported Chat model error + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** `401` indicates credential failure; unsupported request values should not be confused with invalid bearer auth. +- **Observed behavior:** An obviously unsupported model returned `401` JSON indicating third-party metadata retrieval failure while the same bearer passed `/models` and primary-model calls. +- **Reproduction conditions:** Use a valid bearer with a synthetic unsupported model name. +- **Impact:** A model-resolution failure can trigger incorrect credential rotation or a global auth stop. +- **Recommended fallback:** Verify auth with a documented model; treat this 401 as case-specific after a valid control succeeds. +- **Status:** active + +## Direct `ecnu-max` image input + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** `ecnu-max` does not support vision; use `ecnu-plus`. +- **Observed behavior:** A tiny PNG data URL sent directly to `ecnu-max` returned plain-text `500`; the `ecnu-plus` control returned `200` and recognized the image. +- **Reproduction conditions:** Send the same synthetic red-square image to each primary model. +- **Impact:** Direct unsupported vision fails without a structured validation body. +- **Recommended fallback:** Route all image understanding to `ecnu-plus` before sending. +- **Status:** active + +## Thinking-tool continuation without reasoning content + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP +- **Protocol and endpoint:** OpenAI-compatible `POST /chat/completions` +- **Documented expectation:** A thinking-mode assistant tool call must retain `reasoning_content` in subsequent turns. +- **Observed behavior:** The correct preserved-reasoning flow returned `200`, and a single negative continuation with the field omitted also returned `200`. +- **Reproduction conditions:** Reuse one deterministic `echo` tool call, then submit preserved and omitted variants once each. +- **Impact:** Current permissive behavior can hide a client bug that may fail on another model or deployment. +- **Recommended fallback:** Continue preserving the field ephemerally; never rely on the observed permissiveness. +- **Status:** active + +## Generated image URL verification + +- **Tested at:** 2026-08-23 +- **Environment:** live-2026-08-23-a, direct HTTP plus bounded safe downloader +- **Protocol and endpoint:** OpenAI-compatible `POST /images/generations` +- **Documented expectation:** A documented `ecnu-image` URL result is available for transfer for 24 hours. +- **Observed behavior:** The single generation returned `200` with a URL, but its host did not resolve exclusively to public IP addresses, so the low-risk downloader refused access and no image bytes were retained. +- **Reproduction conditions:** Generate one 512x512 URL result and apply public-HTTPS DNS and redirect checks before download. +- **Impact:** Content type, byte hash, and pixel dimensions could not be verified without relaxing the safety policy or paying for another generation. +- **Recommended fallback:** Treat the generation as inconclusive; transfer through an approved network path and never blindly retry the POST. +- **Status:** inconclusive + +## Verified coverage on 2026-08-23 + +Across the 2026-08-23 runs and targeted reruns, the probes successfully checked +valid model discovery; direct and OpenAI-SDK Chat; bounded SSE with empty chunks +and `[DONE]`; `ecnu-max` thinking low; +deterministic tools and the correct two-turn thinking-tool splice; Responses +with effort `none` and `low`; scalar, array, SDK, and LangChain embeddings; +integer-token rejection; standard rerank behavior; `ecnu-plus` vision; JSON +Schema structure; Anthropic direct names, alias requests, valid effort, plain-max +fallback, and SDK use; and image stripping in the Responses and Anthropic +compatibility layers. Successful Chat responses may expose backend model labels, +while Anthropic alias responses may echo the requested alias; neither label alone +proves or disproves internal routing. + +## Update rules + +1. Record the date, non-secret environment, protocol, endpoint, expectation, + observed structure, reproduction conditions, impact, and fallback. +2. Use only `active`, `resolved`, `inconclusive`, or `not-retested` as status. +3. Remove keys, prompts, generated content, one-time URLs, and reasoning text. +4. Do not mark an item resolved from an unrelated success or an unexecuted case. diff --git a/references/models.md b/references/models.md index ee86b70..d7d3c8e 100644 --- a/references/models.md +++ b/references/models.md @@ -45,14 +45,27 @@ Prefer the primary names for new integrations. Anthropic mappings are broader: `opus` maps to `ecnu-max`; `sonnet` and `haiku` map to `ecnu-plus`; other unrecognized names map to `ecnu-plus`. +These are internal compatibility-routing rules. The official Anthropic page +does not document whether the response `model` field echoes the requested alias +or identifies the effective ECNU model. + +## Model identifiers in responses + +A request model name selects a documented primary model or compatibility route; +it is not guaranteed to be echoed as the response label. Official Chat +Completions examples omit `model` in some responses and use a backend label in +others. Treat a returned `model` value as response metadata and do not require +equality with the requested name. The Anthropic response-label behavior is not +documented. + ## Specialized models | Model | Underlying model | Contract | |---|---|---| -| `ecnu-embedding-small` | bge-m3 | Raw string or string array; 1024-float output | +| `ecnu-embedding-small` | bge-m3 | Raw string or string array; documented input limit 8192 characters; 1024-float output | | `ecnu-rerank` | bge-reranker-v2-m3 | String documents; 8192 characters per document | | `ecnu-image` | Z-Image-Turbo | Prompt at most 1024 characters | -| `ecnu-tts` | Fun-CosyVoice3-0.5B | Input at most 4096 characters | +| `ecnu-tts` | Fun-CosyVoice3-0.5B | Input at most 4096 characters; 16 documented voices | The model page labels embedding and rerank context as `8K`, while endpoint pages express request limits in characters. Use the endpoint wording when validating @@ -94,8 +107,7 @@ hidden reasoning to end users. ## Shared credits quotas -The repository documentation available on 2026-08-22 records these defaults -for personal tokens: +The current official quota page documents these defaults for personal tokens: | Period | Default quota | |---|---| @@ -121,7 +133,7 @@ it is not a guarantee for an application. ## Fixed-cost capabilities -The repository documentation available on 2026-08-22 records: +The current official quota page documents: | Capability | Model | Cost | |---|---|---| diff --git a/references/workflows.md b/references/workflows.md index 2817555..f263de6 100644 --- a/references/workflows.md +++ b/references/workflows.md @@ -31,6 +31,7 @@ Check: - no undocumented embedding dimension request; - `ecnu-plus` for image understanding; - explicit timeout; +- SDK retries disabled with `max_retries=0` for live probes; - bounded error-body capture; - no automatic parallel batch; - no blind retry for billable POST requests; @@ -66,14 +67,16 @@ Do not collect the API key or full sensitive prompt. | `5xx` JSON | proxy or backend error details | | `5xx` plain text/HTML | preserve content type and bounded body | | `200` with empty data | do not assume success; validate semantics | -| timeout | determine whether the server may still have accepted the request | +| timeout or connection drop | mark inconclusive; the server may still have accepted the request | ### 3. Retry safely -Safe read-style requests may use limited exponential backoff with jitter. +Safe GET requests may use limited exponential backoff with jitter. The live +validator never retries POST requests. -For chat, embedding, and rerank, retry only when the application can tolerate -duplicate work and the error is clearly transient. +Treat a POST timeout, connection drop, or truncated response as inconclusive. +Stop the flow and do not resubmit automatically; a later rerun is a new, +explicitly authorized request because the first request may have succeeded. For image generation, TTS, or any billed operation, do not automatically resubmit after a timeout or connection drop unless the service provides an @@ -99,8 +102,9 @@ Do not use a single `/models` result as proof of endpoint support. ## Live-verification workflow -Use `scripts/smoke_test.py`. Its default profile limits itself to model-list -requests. +Use `scripts/smoke_test.py`. Its default `auth` profile performs only service +status and model-list GET requests. The script uses fixed ECNU hosts, serial +requests, explicit timeouts, no POST retry, and a credit ceiling. Before opt-in POST probes: @@ -112,24 +116,58 @@ Before opt-in POST probes: 6. avoid parallel execution; 7. write only a sanitized structural report. -Example: +Create reports only under the ignored artifact directory. Model discovery and +the documented `401` expectations are non-billable: ```bash +mkdir -p .live-artifacts export ECNU_API_KEY="your-api-key" -python scripts/smoke_test.py --low-cost --anthropic \ +python3 scripts/smoke_test.py --profile auth --max-credits 0 --timeout 30 \ --account-type personal-token \ - --output smoke-results.json + --output .live-artifacts/auth.json ``` -A valid report should record: +Exercise a valid-token gate, one invalid-token POST, and two request-shape +checks with a conservative allowance. These are real POST requests and require +account-owner authorization: + +```bash +python3 scripts/smoke_test.py --profile core --max-credits 0.06 --timeout 60 \ + --case models_valid \ + --case error_invalid_token_post \ + --case error_missing_model \ + --case error_wrong_messages_type \ + --account-type personal-token \ + --output .live-artifacts/auth-and-422.json +``` + +The report stores the actual status and JSON shape even when the observed +service behavior differs from the documented `401` or `422` expectation. Do +not include `--strict` when the purpose is to collect deviation evidence. + +A valid report records: - test date and Python version; - enabled profiles; - status and content type per request; - model IDs for `/models`; - vector count and output length for embeddings; -- response structure, not successful model text; -- transport errors without credentials. +- response structure, not successful model text or reasoning; +- bounded, redacted errors and allowlisted request IDs; +- transport errors as `inconclusive`, without credentials. + +Interpret evidence per case: + +| `result` | Meaning | +|---|---| +| `pass` | structural expectation passed in this dated run | +| `mismatch` | observed evidence differs from the documented expectation | +| `inconclusive` | no supportable endpoint conclusion, including transport failure | +| `skipped` | the case did not run and provides no live evidence | + +`classification: observed` is point-in-time evidence, not a platform +guarantee. Reproduce a mismatch before changing `known_deviations.md`; never +promote an inconclusive or skipped case into a claim. If the environment cannot reach the ECNU host, label the behavior unverified. Do not update the known-deviation date. @@ -145,6 +183,29 @@ Do not update the known-deviation date. across every endpoint. 6. Prefer documented primary names even when aliases are visible. +The `auth` command above is the executable discovery flow. A `200` response +with an empty `data` array does not prove authentication. + +## Thinking-and-tool workflow + +Thinking tool calls are a two-turn protocol. The second request must splice the +returned assistant message, including its `reasoning_content`, immediately +before the tool result. Keep that assistant message in process memory only; +never print it or write it to the report, and clear it after the second turn. + +```bash +python3 scripts/smoke_test.py --profile core --max-credits 1.2 --timeout 60 \ + --case models_valid \ + --case chat_thinking_tool_first \ + --case chat_thinking_tool_continue \ + --account-type personal-token \ + --output .live-artifacts/thinking-tool.json +``` + +The validator retains the first assistant message only in ephemeral run state, +submits one tool result, then clears that state before writing the sanitized +structural report. + ## Embedding workflow 1. Validate `input` as `str` or non-empty `list[str]`. @@ -159,15 +220,53 @@ Do not update the known-deviation date. ## Anthropic workflow -1. Set `ANTHROPIC_BASE_URL` to the Anthropic root, not the OpenAI root. -2. Use `ANTHROPIC_AUTH_TOKEN` from the environment. +1. Use the Anthropic root, not the OpenAI root. +2. Read `ECNU_API_KEY` from the environment and pass it to the SDK; set an + explicit timeout and `max_retries=0`. 3. Prefer `ecnu-plus` or plain `ecnu-max`. 4. Use `ecnu-max[1m]` only when a tool requires the suffix to advertise long context. -5. On suffix-specific metadata or authentication failure, fall back to plain - `ecnu-max` and report the capability difference. +5. After plain `ecnu-max` was verified with the same credential, fall back once + on the known suffix-specific `401` and report the capability difference. 6. Do not generalize the suffix to OpenAI-compatible APIs. +Probe the suffix and its plain-model control narrowly: + +```bash +python3 scripts/smoke_test.py --profile compatibility --max-credits 0.16 \ + --timeout 60 \ + --case models_valid \ + --case anthropic_max_1m \ + --case anthropic_max_1m_fallback_plain_max \ + --account-type personal-token \ + --output .live-artifacts/anthropic-1m.json +``` + +In application code, fall back once only when the `[1m]` request itself returns +the known suffix-specific `401`, plain `ecnu-max` was already verified with the +same credential, and the caller accepts losing the long-context capability +signal. Otherwise surface the error. + +## Budgeted billable workflow + +Select exact billable cases and set a ceiling equal to their conservative +planned cost. The script reserves budget before each request and stops before +the next case would exceed it. For example, after explicit authorization for a +5-credit TTS probe and a 30-credit image probe: + +```bash +python3 scripts/smoke_test.py --profile billable --max-credits 35 --timeout 60 \ + --case models_valid \ + --case tts_xiayu_pcm \ + --case image_generation_documented \ + --account-type personal-token \ + --output .live-artifacts/billable.json +``` + +The image case attempts a bounded, public-network-safe validation without +persisting its one-time URL or generated bytes. A rejected download target or +transport failure is `inconclusive`; the billable POST is not retried. + ## Security and privacy workflow - Do not request a key in chat when an environment variable or secret manager diff --git a/scripts/smoke_test.py b/scripts/smoke_test.py index e3e99c2..ee4f8a7 100755 --- a/scripts/smoke_test.py +++ b/scripts/smoke_test.py @@ -1,30 +1,147 @@ #!/usr/bin/env python3 -"""Sanitized structural smoke tests for the ECNU API. +"""Serial, sanitized, budgeted live checks for the ECNU API. -The default profile performs model-list checks only. POST probes that may -consume credits are opt-in. The script never prints successful model content -or the API key. +The default ``auth`` profile does not make billable POST requests. API hosts +are fixed, credentials come only from ``ECNU_API_KEY``, and POST requests are +never retried. Reports contain response structure, never generated content. """ from __future__ import annotations import argparse +import base64 +import copy +import hashlib +import http.client +import importlib.metadata +import ipaddress import json +import math import os import platform import re +import socket +import ssl +import struct import sys -from dataclasses import dataclass +import tempfile +import time +import zlib +from contextlib import contextmanager +from dataclasses import dataclass, field from datetime import datetime, timezone from pathlib import Path -from typing import Any, Iterable, Mapping +from typing import Any, Callable, Iterable, Mapping, Optional, Sequence, Union from urllib.error import HTTPError, URLError -from urllib.request import Request, urlopen +from urllib.parse import urljoin, urlsplit +from urllib.request import HTTPRedirectHandler, Request, build_opener -DEFAULT_OPENAI_BASE = "https://chat.ecnu.edu.cn/open/api/v1" -DEFAULT_ANTHROPIC_BASE = "https://chat.ecnu.edu.cn/open/api/anthropic" -KEY_PATTERN = re.compile(r"\bsk-[A-Za-z0-9_-]{8,}\b") +OPENAI_BASE = "https://chat.ecnu.edu.cn/open/api/v1" +ANTHROPIC_BASE = "https://chat.ecnu.edu.cn/open/api/anthropic" +ANTHROPIC_MESSAGES_URL = ANTHROPIC_BASE + "/v1/messages" +STATUS_URL = "https://chat.ecnu.edu.cn/status" +DEFAULT_MAX_CREDITS = 50.0 MAX_ERROR_TEXT = 1000 +MAX_SANITIZED_ITEMS = 30 +MAX_RESPONSE_BYTES = 8 * 1024 * 1024 +MAX_STREAM_BYTES = 1024 * 1024 +MAX_STREAM_EVENTS = 1000 +MAX_STREAM_SECONDS = 120.0 +MAX_IMAGE_REDIRECTS = 3 +PROFILES = ("auth", "core", "compatibility", "billable", "all") +CASE_LOCAL_401 = frozenset({"anthropic_max_1m", "error_unsupported_model"}) + +KEY_PATTERN = re.compile(r"\bsk-[A-Za-z0-9_-]{8,}\b") +BEARER_PATTERN = re.compile( + r"(?i)(authorization\s*[:=]\s*bearer\s+)[^\s,;\"']+" +) +DATA_URL_PATTERN = re.compile(r"data:[^\s,;]+;base64,[A-Za-z0-9+/=_-]+", re.I) +BASE64_PATTERN = re.compile(r"(?]+", re.I) + +CASE_FIELDS = ( + "case_id", + "tested_at", + "protocol", + "endpoint", + "model", + "request_shape", + "documented_expectation", + "actual_http_status", + "actual_content_type", + "actual_response_shape", + "important_headers", + "sdk_or_transport", + "result", + "classification", + "notes", +) + +IMPORTANT_RESPONSE_HEADERS = frozenset( + { + "content-type", + "content-length", + "content-rate", + "content-channels", + "content-bits", + "request-id", + "x-request-id", + "x-trace-id", + "trace-id", + "x-ratelimit-limit", + "x-ratelimit-remaining", + "x-ratelimit-reset", + "retry-after", + } +) + +DOCUMENTED_MODELS = frozenset( + { + "ecnu-plus", + "ecnu-max", + "ecnu-embedding-small", + "ecnu-rerank", + "ecnu-image", + "ecnu-tts", + } +) +ALIASES = frozenset( + { + "ecnu-reasoner", + "ecnu-reasoner-lite", + "ecnu-turbo", + "ecnu-vl", + "InnoSpark", + "educhat-r1", + "educhat-general", + "educhat-psychology", + "ChatECNU", + "gpt-4", + } +) + +ECHO_TOOL = { + "type": "function", + "function": { + "name": "echo", + "description": "Return the supplied value.", + "parameters": { + "type": "object", + "properties": {"value": {"type": "string"}}, + "required": ["value"], + }, + }, +} + +STRUCTURED_SCHEMA = { + "type": "object", + "properties": { + "name": {"type": "string"}, + "department": {"type": "string"}, + }, + "required": ["name", "department"], + "additionalProperties": False, +} @dataclass(frozen=True) @@ -35,38 +152,260 @@ class HttpResult: transport_error: str | None = None +@dataclass(frozen=True) +class Execution: + response: HttpResult + transport: str + attempts: int = 1 + shape_updates: Mapping[str, Any] = field(default_factory=dict) + + +@dataclass(frozen=True) +class SkipExecution: + reason: str + classification: str = "unverified" + + +class CaseUnavailable(RuntimeError): + pass + + +PayloadFactory = Callable[["RunContext"], Optional[Mapping[str, Any]]] +CustomExecutor = Callable[["RunContext", "CaseSpec"], Union[Execution, SkipExecution]] + + +@dataclass(frozen=True) +class CaseSpec: + case_id: str + profiles: frozenset[str] + protocol: str + endpoint: str + model: str | None + method: str + request_shape: Mapping[str, Any] + documented_expectation: str + response_kind: str + expected_statuses: tuple[int, ...] + estimated_credits: float = 0.0 + headers_kind: str = "valid" + payload_factory: PayloadFactory | None = None + custom_executor: CustomExecutor | None = None + requires_valid_auth: bool = True + required: bool = True + capture_assistant_as: str | None = None + evidence_classification: str = "observed" + + +@dataclass +class CreditBudget: + limit: float + planned: float = 0.0 + reserved: float = 0.0 + exhausted: bool = False + + def __post_init__(self) -> None: + if not math.isfinite(self.limit) or self.limit < 0: + raise ValueError("credit limit must be finite and non-negative") + + def reserve(self, credits: float) -> bool: + """Reserve before sending; an ambiguous request can still consume credits.""" + if credits <= 0: + return True + if self.exhausted or self.reserved + credits > self.limit + 1e-9: + self.exhausted = True + return False + self.reserved += credits + return True + + def release_unattempted(self, credits: float) -> None: + """Release a reservation only when execution proves no wire attempt occurred.""" + if credits > 0: + self.reserved = max(0.0, self.reserved - credits) + + +@dataclass +class RunContext: + api_key: str + timeout: float + artifact_dir: Path + budget: CreditBudget + state: dict[str, Any] = field(default_factory=dict) + auth_gate_reason: str | None = None + stop_reason: str | None = None + estimated_consumed_credits: float = 0.0 + + def image_data_url(self) -> str: + path = self.artifact_dir / "vision-test.png" + if not path.exists(): + path.write_bytes(make_test_png()) + encoded = base64.b64encode(path.read_bytes()).decode("ascii") + return "data:image/png;base64," + encoded + + +def utc_now() -> str: + return datetime.now(timezone.utc).isoformat() + + +def package_version(name: str) -> str | None: + try: + return importlib.metadata.version(name) + except importlib.metadata.PackageNotFoundError: + return None + + def redact_text(text: str, secrets: Iterable[str] = ()) -> str: - """Redact exact secrets and key-shaped strings.""" + """Redact credentials, encoded media, and potentially one-time URLs.""" redacted = text for secret in secrets: if secret: redacted = redacted.replace(secret, "[REDACTED_API_KEY]") - return KEY_PATTERN.sub("[REDACTED_API_KEY]", redacted) + redacted = BEARER_PATTERN.sub(r"\1[REDACTED_API_KEY]", redacted) + redacted = KEY_PATTERN.sub("[REDACTED_API_KEY]", redacted) + redacted = DATA_URL_PATTERN.sub("[REDACTED_BASE64_DATA]", redacted) + redacted = BASE64_PATTERN.sub("[REDACTED_BASE64_DATA]", redacted) + return URL_PATTERN.sub("[REDACTED_URL]", redacted) + + +def _content_length(value: Any) -> int: + if value is None: + return 0 + if isinstance(value, str): + return len(value) + try: + return len(json.dumps(value, ensure_ascii=False)) + except (TypeError, ValueError): + return len(str(value)) def sanitize_value(value: Any, secrets: Iterable[str] = ()) -> Any: - """Bound and redact a JSON-compatible value for a report.""" + """Return a bounded JSON value without auth, reasoning, URLs, or media.""" if isinstance(value, str): return redact_text(value, secrets)[:MAX_ERROR_TEXT] + if isinstance(value, bytes): + return {"byte_count": len(value), "sha256": hashlib.sha256(value).hexdigest()} if isinstance(value, list): - return [sanitize_value(item, secrets) for item in value[:20]] + return [sanitize_value(item, secrets) for item in value[:MAX_SANITIZED_ITEMS]] if isinstance(value, dict): result: dict[str, Any] = {} for index, (key, item) in enumerate(value.items()): - if index >= 30: + if index >= MAX_SANITIZED_ITEMS: result["..."] = "truncated" break - lowered = str(key).lower() - if lowered in {"authorization", "x-api-key", "api_key", "token"}: - result[str(key)] = "[REDACTED]" + text_key = str(key) + lowered = text_key.lower().replace("-", "_") + if lowered in { + "authorization", + "x_api_key", + "api_key", + "token", + "access_token", + "client_secret", + "ticket", + }: + result[text_key] = "[REDACTED]" + elif lowered == "reasoning_content": + result["reasoning_content_present"] = item is not None + result["reasoning_content_length"] = _content_length(item) + elif lowered in { + "messages", + "input", + "inputs", + "prompt", + "prompts", + "content", + "contents", + "output", + "outputs", + "completion", + "completions", + "query", + "queries", + "document", + "documents", + "image_url", + "text", + "data", + "request", + "body", + "payload", + "argument", + "arguments", + "parameters", + "source", + }: + result[text_key + "_present"] = item is not None + result[text_key + "_length"] = _content_length(item) + elif lowered in {"b64_json", "base64", "audio", "image_data"} and isinstance(item, str) and len(item) > 100: + result[text_key + "_present"] = bool(item) + result[text_key + "_length"] = len(item) + elif lowered in {"url", "download_url", "one_time_url"}: + result[text_key + "_present"] = bool(item) else: - result[str(key)] = sanitize_value(item, secrets) + result[text_key] = sanitize_value(item, secrets) return result - return value + if value is None or isinstance(value, (bool, int, float)): + return value + return redact_text(str(value), secrets)[:MAX_ERROR_TEXT] + + +def parse_json(body: bytes) -> Any | None: + try: + return json.loads(body.decode("utf-8")) + except (UnicodeDecodeError, json.JSONDecodeError): + return None + + +def bounded_error_sample(body: bytes, secrets: Iterable[str] = ()) -> str: + parsed = parse_json(body) + if parsed is None: + sample = redact_text(body.decode("utf-8", errors="replace"), secrets) + else: + sample = json.dumps( + sanitize_value(parsed, secrets), ensure_ascii=False, separators=(",", ":") + ) + if len(sample) > MAX_ERROR_TEXT: + return sample[: MAX_ERROR_TEXT - 14] + "...[truncated]" + return sample + + +def extract_important_headers(headers: Mapping[str, str]) -> dict[str, str]: + lowered = {str(key).lower(): str(value) for key, value in headers.items()} + return { + key: redact_text(lowered[key])[:MAX_ERROR_TEXT] + for key in sorted(IMPORTANT_RESPONSE_HEADERS) + if key in lowered + } + + +def json_shape(value: Any) -> dict[str, Any]: + """Describe JSON types and lengths without retaining scalar content.""" + if isinstance(value, dict): + return { + "type": "object", + "fields": {str(key): json_shape(item) for key, item in value.items()}, + } + if isinstance(value, list): + unique: list[dict[str, Any]] = [] + for item in value[:3]: + shape = json_shape(item) + if shape not in unique: + unique.append(shape) + return {"type": "array", "length": len(value), "items": unique} + if isinstance(value, str): + return {"type": "string", "length": len(value)} + if value is None: + return {"type": "null"} + if isinstance(value, bool): + return {"type": "boolean"} + if isinstance(value, int): + return {"type": "integer"} + if isinstance(value, float): + return {"type": "number"} + return {"type": type(value).__name__} def build_embedding_payload(input_value: str | list[str]) -> dict[str, Any]: - """Build only the documented ECNU embedding request fields.""" + """Build only the two documented fields and reject token-ID input.""" if isinstance(input_value, str): if not input_value: raise ValueError("embedding input must not be empty") @@ -77,11 +416,33 @@ def build_embedding_payload(input_value: str | list[str]) -> dict[str, Any]: raise TypeError("embedding input must be a non-empty string array") else: raise TypeError("embedding input must be a string or string array") + return {"model": "ecnu-embedding-small", "input": input_value} - return { - "model": "ecnu-embedding-small", - "input": input_value, - } + +def _read_bounded(stream: Any, limit: int) -> tuple[bytes, bool]: + data = stream.read(limit + 1) + return data[:limit], len(data) > limit + + +class _NoRedirect(HTTPRedirectHandler): + def redirect_request( + self, + req: Any, + fp: Any, + code: int, + msg: str, + headers: Any, + newurl: str, + ) -> None: + return None + + +_API_OPENER = build_opener(_NoRedirect()) + + +def _open_api_request(req: Request, timeout: float) -> Any: + """Open one request without following or forwarding auth across redirects.""" + return _API_OPENER.open(req, timeout=timeout) def request( @@ -92,335 +453,2262 @@ def request( payload: Mapping[str, Any] | None = None, timeout: float = 30.0, ) -> HttpResult: + """Make exactly one request; there is deliberately no retry loop.""" data = None request_headers = dict(headers or {}) if payload is not None: data = json.dumps(payload, ensure_ascii=False).encode("utf-8") request_headers.setdefault("Content-Type", "application/json") + req = Request(url=url, data=data, headers=request_headers, method=method) + try: + with _open_api_request(req, timeout) as response: + body, exceeded = _read_bounded(response, MAX_RESPONSE_BYTES) + return HttpResult( + response.status, + {key.lower(): value for key, value in response.headers.items()}, + body, + "response byte limit exceeded" if exceeded else None, + ) + except HTTPError as exc: + body, exceeded = _read_bounded(exc, MAX_RESPONSE_BYTES) + return HttpResult( + exc.code, + {key.lower(): value for key, value in (exc.headers or {}).items()}, + body, + "error response byte limit exceeded" if exceeded else None, + ) + except (URLError, TimeoutError, OSError) as exc: + return HttpResult(None, {}, b"", type(exc).__name__) + +def stream_request( + url: str, + *, + headers: Mapping[str, str], + payload: Mapping[str, Any], + timeout: float, +) -> HttpResult: + """Read SSE incrementally and stop immediately after the [DONE] event.""" + request_headers = dict(headers) + request_headers.setdefault("Content-Type", "application/json") req = Request( url=url, - data=data, + data=json.dumps(payload, ensure_ascii=False).encode("utf-8"), headers=request_headers, - method=method, + method="POST", ) - + deadline = time.monotonic() + min(timeout, MAX_STREAM_SECONDS) try: - with urlopen(req, timeout=timeout) as response: + with _open_api_request(req, min(timeout, MAX_STREAM_SECONDS)) as response: + chunks: list[bytes] = [] + byte_count = 0 + event_count = 0 + while True: + if time.monotonic() > deadline: + return HttpResult( + response.status, + {key.lower(): value for key, value in response.headers.items()}, + b"".join(chunks), + "stream wall-time limit exceeded", + ) + line = response.readline(MAX_STREAM_BYTES + 1) + if not line: + break + byte_count += len(line) + if byte_count > MAX_STREAM_BYTES: + return HttpResult( + response.status, + {key.lower(): value for key, value in response.headers.items()}, + b"".join(chunks), + "stream byte limit exceeded", + ) + chunks.append(line) + if line.startswith(b"data:"): + event_count += 1 + if event_count > MAX_STREAM_EVENTS: + return HttpResult( + response.status, + {key.lower(): value for key, value in response.headers.items()}, + b"".join(chunks), + "stream event limit exceeded", + ) + if line[5:].strip() == b"[DONE]": + break return HttpResult( - status=response.status, - headers={key.lower(): value for key, value in response.headers.items()}, - body=response.read(), + response.status, + {key.lower(): value for key, value in response.headers.items()}, + b"".join(chunks), ) except HTTPError as exc: + body, exceeded = _read_bounded(exc, MAX_ERROR_TEXT) return HttpResult( - status=exc.code, - headers={key.lower(): value for key, value in exc.headers.items()}, - body=exc.read(), + exc.code, + {key.lower(): value for key, value in (exc.headers or {}).items()}, + body, + "stream error response byte limit exceeded" if exceeded else None, ) except (URLError, TimeoutError, OSError) as exc: - return HttpResult( - status=None, - headers={}, - body=b"", - transport_error=f"{type(exc).__name__}: {exc}", + return HttpResult(None, {}, b"", type(exc).__name__) + + +def _png_chunk(kind: bytes, data: bytes) -> bytes: + return ( + struct.pack(">I", len(data)) + + kind + + data + + struct.pack(">I", zlib.crc32(kind + data) & 0xFFFFFFFF) + ) + + +def make_test_png(width: int = 16, height: int = 16) -> bytes: + """Return a tiny white PNG containing a centered red square.""" + rows = [] + for y in range(height): + row = bytearray([0]) + for x in range(width): + red = width // 4 <= x < 3 * width // 4 and height // 4 <= y < 3 * height // 4 + row.extend((255, 0, 0, 255) if red else (255, 255, 255, 255)) + rows.append(bytes(row)) + header = struct.pack(">IIBBBBB", width, height, 8, 6, 0, 0, 0) + return ( + b"\x89PNG\r\n\x1a\n" + + _png_chunk(b"IHDR", header) + + _png_chunk(b"IDAT", zlib.compress(b"".join(rows))) + + _png_chunk(b"IEND", b"") + ) + + +@contextmanager +def temporary_artifacts() -> Iterable[Path]: + with tempfile.TemporaryDirectory(prefix="ecnu-api-smoke-") as directory: + yield Path(directory) + + +def _image_dimensions(data: bytes) -> list[int] | None: + if data.startswith(b"\x89PNG\r\n\x1a\n"): + offset = 8 + dimensions: list[int] | None = None + pixel_layout: tuple[int, int] | None = None + compressed = bytearray() + while offset + 12 <= len(data): + length = struct.unpack(">I", data[offset : offset + 4])[0] + end = offset + 12 + length + if length > MAX_RESPONSE_BYTES or end > len(data): + return None + kind = data[offset + 4 : offset + 8] + chunk = data[offset + 8 : offset + 8 + length] + expected_crc = struct.unpack(">I", data[offset + 8 + length : end])[0] + if zlib.crc32(kind + chunk) & 0xFFFFFFFF != expected_crc: + return None + if kind == b"IHDR": + if dimensions is not None or offset != 8 or length != 13: + return None + width, height, bit_depth, color_type, compression, filtering, interlace = ( + struct.unpack(">IIBBBBB", chunk) + ) + valid_depths = { + 0: {1, 2, 4, 8, 16}, + 2: {8, 16}, + 4: {8, 16}, + 6: {8, 16}, + } + if ( + width <= 0 + or height <= 0 + or compression != 0 + or filtering != 0 + or interlace != 0 + or bit_depth not in valid_depths.get(color_type, set()) + ): + return None + dimensions = [width, height] + channels = {0: 1, 2: 3, 4: 2, 6: 4}[color_type] + pixel_layout = (bit_depth, channels) + elif kind == b"IDAT": + if dimensions is None: + return None + compressed.extend(chunk) + elif kind == b"IEND": + if ( + length != 0 + or dimensions is None + or pixel_layout is None + or not compressed + or end != len(data) + ): + return None + width, height = dimensions + bit_depth, channels = pixel_layout + row_bytes = (width * bit_depth * channels + 7) // 8 + expected_size = height * (row_bytes + 1) + if expected_size <= 0 or expected_size > MAX_RESPONSE_BYTES: + return None + try: + decompressor = zlib.decompressobj() + decoded = decompressor.decompress( + bytes(compressed), expected_size + 1 + ) + except zlib.error: + return None + return ( + dimensions + if len(decoded) == expected_size + and decompressor.eof + and not decompressor.unused_data + and not decompressor.unconsumed_tail + and all( + decoded[row * (row_bytes + 1)] <= 4 + for row in range(height) + ) + else None + ) + offset = end + return None + return None + + +def _image_mime(data: bytes) -> str | None: + dimensions = _image_dimensions(data) + if dimensions and data.startswith(b"\x89PNG\r\n\x1a\n"): + return "image/png" + return None + + +def _public_ip(address: str) -> bool: + try: + parsed = ipaddress.ip_address(address) + except ValueError: + return False + if ( + not parsed.is_global + or parsed.is_private + or parsed.is_loopback + or parsed.is_link_local + or parsed.is_multicast + or parsed.is_reserved + or parsed.is_unspecified + ): + return False + if isinstance(parsed, ipaddress.IPv6Address): + embedded: list[ipaddress.IPv4Address] = [] + if parsed.ipv4_mapped: + embedded.append(parsed.ipv4_mapped) + if parsed.sixtofour: + embedded.append(parsed.sixtofour) + for prefix in ( + ipaddress.IPv6Network("64:ff9b::/96"), + ipaddress.IPv6Network("64:ff9b:1::/48"), + ): + if parsed in prefix: + embedded.append(ipaddress.IPv4Address(parsed.packed[-4:])) + if any(not _public_ip(str(candidate)) for candidate in embedded): + return False + return True + + +def _resolve_public_https(url: str) -> tuple[Any, str] | None: + if any(character in url for character in "\r\n\x00"): + return None + parsed = urlsplit(url) + if parsed.scheme.lower() != "https" or not parsed.hostname: + return None + try: + port = parsed.port + except ValueError: + return None + if parsed.username or parsed.password or port not in {None, 443}: + return None + if parsed.hostname.lower() == "localhost": + return None + try: + addresses = socket.getaddrinfo( + parsed.hostname, port or 443, type=socket.SOCK_STREAM ) + except socket.gaierror: + return None + resolved = {item[4][0] for item in addresses} + if not resolved or not all(_public_ip(address) for address in resolved): + return None + address = sorted(resolved, key=lambda item: (ipaddress.ip_address(item).version, item))[0] + return parsed, address -def parse_json(body: bytes) -> Any | None: +def _pinned_https_get(parsed: Any, address: str, timeout: float) -> HttpResult: + hostname = parsed.hostname + if not hostname: + return HttpResult(None, {}, b"", "image URL host validation failed") + target = parsed.path or "/" + if parsed.query: + target += "?" + parsed.query try: - return json.loads(body.decode("utf-8")) - except (UnicodeDecodeError, json.JSONDecodeError): + target_bytes = target.encode("ascii") + host_bytes = hostname.encode("idna") + except UnicodeError: + return HttpResult(None, {}, b"", "image URL encoding validation failed") + try: + with socket.create_connection((address, 443), timeout=timeout) as raw_socket: + context = ssl.create_default_context() + with context.wrap_socket(raw_socket, server_hostname=hostname) as tls_socket: + tls_socket.settimeout(timeout) + tls_socket.sendall( + b"GET " + + target_bytes + + b" HTTP/1.1\r\nHost: " + + host_bytes + + b"\r\nAccept: image/*\r\nConnection: close\r\n\r\n" + ) + response = http.client.HTTPResponse(tls_socket, method="GET") + response.begin() + body, exceeded = _read_bounded(response, MAX_RESPONSE_BYTES) + return HttpResult( + response.status, + {key.lower(): value for key, value in response.headers.items()}, + body, + "image download byte limit exceeded" if exceeded else None, + ) + except (OSError, ssl.SSLError, http.client.HTTPException) as exc: + return HttpResult(None, {}, b"", type(exc).__name__) + + +def fetch_public_image(url: str, timeout: float) -> HttpResult: + """Fetch one bounded image with each DNS result pinned to its TLS socket.""" + current = url + for redirect_count in range(MAX_IMAGE_REDIRECTS + 1): + resolved = _resolve_public_https(current) + if resolved is None: + return HttpResult( + None, + {}, + b"", + "image URL did not resolve exclusively to public HTTPS addresses", + ) + parsed, address = resolved + result = _pinned_https_get(parsed, address, timeout) + if result.status is not None and 300 <= result.status < 400: + location = result.headers.get("location") + if not location or redirect_count >= MAX_IMAGE_REDIRECTS: + return HttpResult(None, {}, b"", "image redirect limit or location failure") + current = urljoin(current, location) + continue + return result + return HttpResult(None, {}, b"", "image redirect limit exceeded") + + +def classify_models(model_ids: Sequence[str]) -> dict[str, list[str]]: + visible = set(model_ids) + return { + "documented-and-visible": sorted(visible & DOCUMENTED_MODELS), + "documented-but-not-visible": sorted(DOCUMENTED_MODELS - visible), + "visible-but-undocumented": sorted(visible - DOCUMENTED_MODELS - ALIASES), + "alias": sorted(visible & ALIASES), + "unknown": [], + } + + +def _usage_keys(payload: Mapping[str, Any]) -> list[str]: + usage = payload.get("usage") + return sorted(str(key) for key in usage) if isinstance(usage, dict) else [] + + +def _safe_model_label(value: Any) -> str | None: + if not isinstance(value, str): return None + if value.startswith(("/", "\\")) or re.match(r"^[A-Za-z]:[\\/]", value): + return "[REDACTED_BACKEND_PATH]" + return redact_text(value)[:200] -def summarize_response( - name: str, - result: HttpResult, - *, - secrets: Iterable[str] = (), -) -> dict[str, Any]: - summary: dict[str, Any] = { - "status": result.status, - "content_type": result.headers.get("content-type", ""), - "body_bytes": len(result.body), +def _numeric_counters(value: Any) -> Any: + if isinstance(value, bool): + return None + if isinstance(value, (int, float)) and math.isfinite(float(value)) and value >= 0: + return value + if isinstance(value, dict): + result = { + str(key): counter + for key, item in list(value.items())[:MAX_SANITIZED_ITEMS] + if (counter := _numeric_counters(item)) is not None + } + return result or None + return None + + +def _usage_counters(payload: Mapping[str, Any]) -> dict[str, Any]: + usage = _numeric_counters(payload.get("usage")) + return usage if isinstance(usage, dict) else {} + + +def _chat_message(payload: Any) -> Mapping[str, Any] | None: + if not isinstance(payload, dict): + return None + choices = payload.get("choices") + if not isinstance(choices, list) or not choices or not isinstance(choices[0], dict): + return None + message = choices[0].get("message") + return message if isinstance(message, dict) else None + + +def _text_length(value: Any) -> int: + return len(value) if isinstance(value, str) else 0 + + +def _summarize_chat(payload: Mapping[str, Any]) -> dict[str, Any]: + choices = payload.get("choices") + message = _chat_message(payload) or {} + content = message.get("content") + reasoning = message.get("reasoning_content") + calls = message.get("tool_calls") + calls = calls if isinstance(calls, list) else [] + names: list[str | None] = [] + valid_arguments: list[bool] = [] + ping_arguments: list[bool] = [] + for call in calls: + function = call.get("function") if isinstance(call, dict) else None + function = function if isinstance(function, dict) else {} + names.append(function.get("name") if isinstance(function.get("name"), str) else None) + raw = function.get("arguments") + try: + arguments = json.loads(raw) if isinstance(raw, str) else None + except json.JSONDecodeError: + arguments = None + valid_arguments.append(isinstance(arguments, dict)) + ping_arguments.append(isinstance(arguments, dict) and arguments.get("value") == "ping") + return { + "valid_json": True, + "top_level_keys": sorted(payload.keys()), + "choice_count": len(choices) if isinstance(choices, list) else 0, + "model": _safe_model_label(payload.get("model")), + "content_present": isinstance(content, str) and bool(content), + "content_length": _text_length(content), + "reasoning_content_present": isinstance(reasoning, str) and bool(reasoning), + "reasoning_content_length": _text_length(reasoning), + "tool_call_count": len(calls), + "tool_names": names, + "tool_arguments_json_valid": valid_arguments, + "tool_ping_argument": ping_arguments, + "usage_keys": _usage_keys(payload), + "usage_counters": _usage_counters(payload), } - if result.transport_error: - summary["transport_error"] = redact_text( - result.transport_error, - secrets, - )[:MAX_ERROR_TEXT] - return summary - payload = parse_json(result.body) +def _summarize_stream(body: bytes) -> dict[str, Any]: + data_count = json_count = empty_count = text_count = reasoning_count = 0 + reasoning_length = 0 + usage_counters: dict[str, Any] = {} + done = False + for line in body.decode("utf-8", errors="replace").splitlines(): + if not line.startswith("data:"): + continue + data_count += 1 + data = line[5:].strip() + if data == "[DONE]": + done = True + continue + if not data: + empty_count += 1 + continue + try: + event = json.loads(data) + except json.JSONDecodeError: + continue + json_count += 1 + if isinstance(event, dict) and isinstance(event.get("usage"), dict): + usage_counters = _usage_counters(event) + choices = event.get("choices") if isinstance(event, dict) else None + delta = choices[0].get("delta") if isinstance(choices, list) and choices and isinstance(choices[0], dict) else None + if not isinstance(delta, dict): + empty_count += 1 + continue + content = delta.get("content") + reasoning = delta.get("reasoning_content") + if isinstance(content, str) and content: + text_count += 1 + if isinstance(reasoning, str) and reasoning: + reasoning_count += 1 + reasoning_length += len(reasoning) + if not content and not reasoning: + empty_count += 1 + return { + "sse_data_event_count": data_count, + "json_event_count": json_count, + "text_delta_event_count": text_count, + "empty_chunk_count": empty_count, + "done_event_present": done, + "reasoning_content_present": reasoning_count > 0, + "reasoning_content_length": reasoning_length, + "usage_keys": sorted(usage_counters), + "usage_counters": usage_counters, + } + + +def _summarize_responses(payload: Mapping[str, Any]) -> dict[str, Any]: + output = payload.get("output") + items = output if isinstance(output, list) else [] + text_length = 0 + reasoning_length = 0 + for item in items: + if not isinstance(item, dict): + continue + content = item.get("content") + for part in content if isinstance(content, list) else []: + if isinstance(part, dict): + text_length += _text_length(part.get("text")) + if item.get("type") == "reasoning": + reasoning_length += _content_length(item) + return { + "valid_json": True, + "top_level_keys": sorted(payload.keys()), + "output_count": len(items), + "output_types": [item.get("type") for item in items if isinstance(item, dict)], + "output_text_present": text_length > 0, + "output_text_length": text_length, + "reasoning_content_present": reasoning_length > 0, + "reasoning_content_length": reasoning_length, + "model": _safe_model_label(payload.get("model")), + "usage_keys": _usage_keys(payload), + "usage_counters": _usage_counters(payload), + } - if name.startswith("models_") and isinstance(payload, dict): + +def _responses_text(payload: Mapping[str, Any]) -> str: + parts: list[str] = [] + output = payload.get("output") + for item in output if isinstance(output, list) else []: + if not isinstance(item, dict): + continue + content = item.get("content") + for part in content if isinstance(content, list) else []: + if isinstance(part, dict) and isinstance(part.get("text"), str): + parts.append(part["text"]) + return "".join(parts) + + +def _visual_behavior(text: str, compatibility: bool) -> str: + lowered = text.lower() + if ("red" in lowered or "红" in text) and ("square" in lowered or "方" in text): + return "accept" + return "strip-image" if compatibility and text else "other" + + +def _summarize_embedding(payload: Mapping[str, Any]) -> dict[str, Any]: + data = payload.get("data") + items = data if isinstance(data, list) else [] + return { + "valid_json": True, + "top_level_keys": sorted(payload.keys()), + "count": len(items), + "indexes": [item.get("index") for item in items if isinstance(item, dict)], + "vector_lengths": [len(item.get("embedding", [])) for item in items if isinstance(item, dict) and isinstance(item.get("embedding"), list)], + "model": _safe_model_label(payload.get("model")), + "usage_keys": _usage_keys(payload), + "usage_counters": _usage_counters(payload), + } + + +def _summarize_rerank(payload: Mapping[str, Any]) -> dict[str, Any]: + results = payload.get("results") + items = results if isinstance(results, list) else [] + return { + "valid_json": True, + "top_level_keys": sorted(payload.keys()), + "result_count": len(items), + "indexes": [item.get("index") for item in items if isinstance(item, dict)], + "score_types": [type(item.get("relevance_score")).__name__ for item in items if isinstance(item, dict)], + "documents_present": ["document" in item for item in items if isinstance(item, dict)], + } + + +def _summarize_anthropic(payload: Mapping[str, Any]) -> dict[str, Any]: + content = payload.get("content") + items = content if isinstance(content, list) else [] + text_length = sum(_text_length(item.get("text")) for item in items if isinstance(item, dict)) + reasoning_length = sum(_content_length(item) for item in items if isinstance(item, dict) and item.get("type") == "thinking") + return { + "valid_json": True, + "top_level_keys": sorted(payload.keys()), + "type": payload.get("type"), + "model": _safe_model_label(payload.get("model")), + "content_count": len(items), + "content_types": [item.get("type") for item in items if isinstance(item, dict)], + "text_present": text_length > 0, + "text_length": text_length, + "reasoning_content_present": reasoning_length > 0, + "reasoning_content_length": reasoning_length, + "usage_keys": _usage_keys(payload), + "usage_counters": _usage_counters(payload), + } + + +def _summarize_structured(payload: Mapping[str, Any]) -> dict[str, Any]: + result = _summarize_chat(payload) + content = (_chat_message(payload) or {}).get("content") + parsed = None + if isinstance(content, str): + try: + parsed = json.loads(content) + except json.JSONDecodeError: + pass + required = isinstance(parsed, dict) and all(isinstance(parsed.get(key), str) for key in ("name", "department")) + extras = set(parsed) - {"name", "department"} if isinstance(parsed, dict) else set() + result.update( + { + "structured_json_valid": isinstance(parsed, dict), + "required_fields_valid": required, + "additional_property_count": len(extras), + "schema_valid": required and not extras, + "semantic_match": isinstance(parsed, dict) and parsed.get("name") == "张三" and parsed.get("department") == "数据科学部", + } + ) + return result + + +def _vision_behavior(payload: Mapping[str, Any], compatibility: bool) -> str: + content = (_chat_message(payload) or {}).get("content") + if not isinstance(content, str): + return "other" + behavior = _visual_behavior(content, compatibility) + return "ignore-image" if behavior == "other" and content else behavior + + +def summarize_response(kind: str, response: HttpResult, *, secrets: Iterable[str] = ()) -> dict[str, Any]: + """Summarize successful structure or one bounded sanitized error sample.""" + if response.transport_error: + return {"transport_error": redact_text(response.transport_error, secrets)[:MAX_ERROR_TEXT]} + if response.status is None: + return {} + if kind == "tts" and response.status < 400: + pcm_headers = { + key: response.headers.get(key) + for key in ("content-rate", "content-channels", "content-bits") + } + return { + "byte_count": len(response.body), + "sha256": hashlib.sha256(response.body).hexdigest(), + "content_type": response.headers.get("content-type", "").split(";", 1)[0].lower(), + "content_disposition_present": bool(response.headers.get("content-disposition")), + "pcm_headers": pcm_headers, + "pcm_headers_complete": all(pcm_headers.values()), + } + if kind == "stream" and response.status < 400: + return _summarize_stream(response.body) + payload = parse_json(response.body) + if response.status >= 400: + result = { + "valid_json": payload is not None, + "error_body_sample": bounded_error_sample(response.body, secrets), + "detail_type": type(payload.get("detail")).__name__ if isinstance(payload, dict) and "detail" in payload else None, + "error_top_level_keys": sorted(payload.keys()) if isinstance(payload, dict) else [], + "error_value_type": type(payload.get("error")).__name__ if isinstance(payload, dict) and "error" in payload else None, + } + if kind in {"vision", "vision_compat", "responses_vision_compat", "anthropic_vision_compat"}: + result["vision_behavior"] = "reject" + return result + if not isinstance(payload, dict): + return {"valid_json": False, "body_bytes": len(response.body)} + if kind == "models": data = payload.get("data") - summary["object"] = payload.get("object") - summary["model_ids"] = [ - item.get("id") - for item in data or [] - if isinstance(item, dict) and isinstance(item.get("id"), str) - ][:100] - - elif name.startswith("chat_") and isinstance(payload, dict): - summary["has_choices"] = bool(payload.get("choices")) - summary["model"] = payload.get("model") - usage = payload.get("usage") - summary["usage_keys"] = sorted(usage.keys()) if isinstance(usage, dict) else [] - - elif name.startswith("embedding_") and isinstance(payload, dict): + items = data if isinstance(data, list) else [] + ids = [item.get("id") for item in items if isinstance(item, dict) and isinstance(item.get("id"), str)][:100] + return {"valid_json": True, "top_level_keys": sorted(payload.keys()), "object": payload.get("object"), "model_count": len(ids), "model_ids": ids, "model_classification": classify_models(ids)} + if kind in {"chat", "thinking", "tool", "thinking_tool"}: + return _summarize_chat(payload) + if kind in {"vision", "vision_compat"}: + result = _summarize_chat(payload) + result["vision_behavior"] = _vision_behavior(payload, kind == "vision_compat") + return result + if kind in {"responses", "responses_vision_compat"}: + result = _summarize_responses(payload) + if kind == "responses_vision_compat": + result["vision_behavior"] = _visual_behavior( + _responses_text(payload), True + ) + return result + if kind == "embedding": + return _summarize_embedding(payload) + if kind == "rerank": + return _summarize_rerank(payload) + if kind == "structured": + return _summarize_structured(payload) + if kind in {"anthropic", "anthropic_vision_compat"}: + result = _summarize_anthropic(payload) + if kind == "anthropic_vision_compat": + content = payload.get("content") + items = content if isinstance(content, list) else [] + text = "".join( + item.get("text", "") + for item in items if isinstance(item, dict) + ) + result["vision_behavior"] = _visual_behavior(text, True) + return result + if kind == "image": data = payload.get("data") items = data if isinstance(data, list) else [] - summary["count"] = len(items) - summary["dimensions"] = [ - len(item.get("embedding", [])) - for item in items - if isinstance(item, dict) and isinstance(item.get("embedding"), list) - ] + first = items[0] if items and isinstance(items[0], dict) else {} + return {"valid_json": True, "top_level_keys": sorted(payload.keys()), "data_count": len(items), "url_present": isinstance(first.get("url"), str), "b64_json_present": isinstance(first.get("b64_json"), str)} + if kind in {"capture", "sdk"}: + return sanitize_value(payload, secrets) + return {"valid_json": True, "top_level_keys": sorted(payload.keys())} - elif name.startswith("anthropic_") and isinstance(payload, dict): - content = payload.get("content") - summary["type"] = payload.get("type") - summary["model"] = payload.get("model") - summary["content_types"] = [ - item.get("type") - for item in content or [] - if isinstance(item, dict) - ] - if result.status is None or result.status >= 400: - if payload is not None: - summary["error"] = sanitize_value(payload, secrets) - else: - text = result.body.decode("utf-8", errors="replace") - summary["error_text"] = redact_text(text, secrets)[:MAX_ERROR_TEXT] +def response_matches(kind: str, status: int | None, shape: Mapping[str, Any]) -> bool: + if status is None: + return False + if status >= 400: + return True + if kind == "models": + return bool(shape.get("valid_json")) and bool(shape.get("model_ids")) + if kind == "chat": + return ( + bool(shape.get("choice_count")) + and bool(shape.get("content_present")) + and bool(shape.get("usage_keys")) + ) + if kind == "thinking": + return ( + bool(shape.get("choice_count")) + and bool(shape.get("content_present")) + and bool(shape.get("usage_keys")) + ) + if kind in {"vision", "vision_compat"}: + return bool(shape.get("choice_count")) and bool(shape.get("content_present")) + if kind == "structured": + return bool(shape.get("choice_count")) and bool(shape.get("schema_valid")) + if kind in {"tool", "thinking_tool"}: + matches = ( + bool(shape.get("tool_call_count")) + and all(name == "echo" for name in shape.get("tool_names", [])) + and all(shape.get("tool_arguments_json_valid", [])) + and all(shape.get("tool_ping_argument", [])) + ) + return matches and ( + kind != "thinking_tool" or bool(shape.get("reasoning_content_present")) + ) + if kind == "stream": + return bool(shape.get("done_event_present")) and bool(shape.get("text_delta_event_count")) + if kind in {"responses", "responses_vision_compat"}: + return ( + bool(shape.get("output_count")) + and bool(shape.get("output_text_present")) + and bool(shape.get("usage_keys")) + ) + if kind == "embedding": + lengths = shape.get("vector_lengths") + return ( + bool(shape.get("count")) + and isinstance(lengths, list) + and bool(lengths) + and all(length == 1024 for length in lengths) + ) + if kind == "rerank": + count = shape.get("result_count") + indexes = shape.get("indexes") + score_types = shape.get("score_types") + return ( + isinstance(count, int) + and not isinstance(count, bool) + and count > 0 + and isinstance(indexes, list) + and len(indexes) == count + and all(isinstance(index, int) and not isinstance(index, bool) and index >= 0 for index in indexes) + and len(set(indexes)) == count + and isinstance(score_types, list) + and len(score_types) == count + and all(score_type in {"float", "int"} for score_type in score_types) + ) + if kind in {"anthropic", "anthropic_vision_compat"}: + return bool(shape.get("content_count")) and bool(shape.get("text_present")) + if kind == "tts": + return ( + bool(shape.get("byte_count")) + and str(shape.get("content_type", "")).startswith("audio/") + and bool(shape.get("content_disposition_present")) + ) + if kind == "image": + return ( + bool(shape.get("data_count")) + and bool(shape.get("media_verified")) + and str(shape.get("media_content_type", "")).startswith("image/") + and shape.get("pixel_dimensions") == [512, 512] + and len(str(shape.get("sha256", ""))) == 64 + ) + return True - return summary +def _official_api_endpoint(endpoint: str) -> bool: + return endpoint.startswith(OPENAI_BASE) or endpoint.startswith(ANTHROPIC_BASE) -def run_case( - name: str, - method: str, - url: str, + +def _counter(counters: Mapping[str, Any], *path: str) -> float: + value: Any = counters + for key in path: + if not isinstance(value, dict): + return 0.0 + value = value.get(key) + if isinstance(value, bool) or not isinstance(value, (int, float)): + return 0.0 + number = float(value) + return number if math.isfinite(number) and number >= 0 else 0.0 + + +def _effective_dialog_model(model: str | None) -> str: + lowered = (model or "").lower().split("[", 1)[0] + if lowered in {"ecnu-max", "ecnu-reasoner"} or re.search( + r"(?:^|[-_/])opus(?:$|[-_/0-9])", lowered + ): + return "ecnu-max" + return "ecnu-plus" + + +def _minimum_output_credit(model: str | None, payload: Mapping[str, Any] | None) -> float: + if payload is None: + return 0.0 + raw_limit = payload.get("max_output_tokens", payload.get("max_tokens")) + if isinstance(raw_limit, bool) or not isinstance(raw_limit, (int, float)): + return 0.0 + limit = max(float(raw_limit), 0.0) + output_rate = 1200.0 if _effective_dialog_model(model) == "ecnu-max" else 400.0 + return limit * output_rate / 1_000_000 + + +def estimate_consumed_credits( + spec: CaseSpec, status: int | None, shape: Mapping[str, Any] +) -> tuple[float, str]: + """Estimate credits conservatively from fixed prices or numeric usage.""" + if spec.method != "POST" or "MockTransport" in spec.protocol: + return 0.0, "no billable ECNU POST" + if spec.model == "ecnu-embedding-small" or spec.endpoint.endswith("/embeddings"): + return 0.05, "official fixed embedding price per attempted call" + if spec.endpoint.endswith("/rerank"): + return 0.1, "official fixed rerank price per attempted call" + if spec.endpoint.endswith("/audio/speech"): + return 5.0, "official fixed TTS price per attempted call" + if spec.endpoint.endswith("/images/generations"): + if status is None or status == 200: + return 30.0, "official image price; ambiguous attempts counted conservatively" + return 0.0, "definite failed image response is not counted as a successful generation" + + counters = shape.get("usage_counters") + if not isinstance(counters, dict) or not counters: + return spec.estimated_credits, "planned conservative allowance; usage unavailable" + input_tokens = _counter(counters, "prompt_tokens") or _counter(counters, "input_tokens") + output_tokens = _counter(counters, "completion_tokens") or _counter(counters, "output_tokens") + cached_subset = _counter(counters, "prompt_tokens_details", "cached_tokens") or _counter(counters, "input_tokens_details", "cached_tokens") + separate_cache = _counter(counters, "cache_read_input_tokens") + uncached = max(input_tokens - cached_subset, 0.0) + model = _effective_dialog_model(spec.model) + miss_rate, hit_rate, output_rate = ( + (300.0, 60.0, 1200.0) + if model == "ecnu-max" + else (100.0, 20.0, 400.0) + ) + credits = ( + uncached * miss_rate + + (cached_subset + separate_cache) * hit_rate + + output_tokens * output_rate + ) / 1_000_000 + return credits, f"official {model} miss/hit/output token formula" + + +def case_response_matches( + spec: CaseSpec, status: int | None, shape: Mapping[str, Any] +) -> bool: + """Apply the generic structure check plus small case-specific invariants.""" + if ( + spec.case_id == "embedding_empty_array" + and status == 200 + and shape.get("count") == 0 + ): + return True + if status is not None and status >= 400 and _official_api_endpoint(spec.endpoint): + if not shape.get("valid_json"): + return False + if spec.case_id == "tts_invalid_voice": + keys = set(shape.get("error_top_level_keys", [])) + return {"error", "request_id", "details"} <= keys + if not response_matches(spec.response_kind, status, shape): + return False + if status != 200: + return True + expected_embedding_counts = { + "embedding_scalar": 1, + "embedding_array": 2, + "embedding_8192_chars": 1, + "embedding_two_long_strings": 2, + "openai_sdk_embedding": 1, + } + if spec.case_id in expected_embedding_counts: + expected = expected_embedding_counts[spec.case_id] + return shape.get("count") == expected and shape.get("indexes") == list( + range(expected) + ) + if spec.case_id == "langchain_embedding_wire_capture": + return ( + shape.get("input_is_string_array") is True + and shape.get("dimensions_present") is False + and shape.get("vector_count") == 2 + and shape.get("vector_lengths") == [1024, 1024] + ) + if spec.case_id == "langchain_embedding_live": + return ( + shape.get("count") == 1 + and shape.get("dimensions_present") is False + and shape.get("vector_lengths") == [1024] + ) + if spec.case_id == "rerank_top_two": + return shape.get("result_count") == 2 + if spec.case_id == "rerank_default": + return shape.get("result_count") == 3 + if spec.case_id == "rerank_without_documents": + return bool(shape.get("result_count")) and not any( + shape.get("documents_present", []) + ) + if spec.case_id == "rerank_with_documents": + return bool(shape.get("result_count")) and all( + shape.get("documents_present", []) + ) + if spec.case_id == "tts_xiayu_pcm": + return bool(shape.get("pcm_headers_complete")) + if spec.case_id == "vision_direct_ecnu_plus": + return shape.get("vision_behavior") == "accept" + if spec.case_id in { + "responses_max_vision_compatibility", + "anthropic_max_vision_compatibility", + }: + return shape.get("vision_behavior") in {"accept", "strip-image", "other"} + if spec.case_id in {"responses_max_effort_none", "anthropic_effort_none"}: + return not bool(shape.get("reasoning_content_present")) + if spec.case_id in {"responses_max_effort_low", "anthropic_effort_low"}: + return bool(shape.get("reasoning_content_present")) + expected_anthropic_models = { + "anthropic_plus": "ecnu-plus", + "anthropic_max": "ecnu-max", + "anthropic_max_1m": "ecnu-max", + "anthropic_max_1m_fallback_plain_max": "ecnu-max", + } + if spec.case_id in expected_anthropic_models: + return shape.get("model") == expected_anthropic_models[spec.case_id] + expected_anthropic_aliases = { + "anthropic_sonnet_mapping": { + "claude-sonnet-4-20250514", + "ecnu-plus", + }, + "anthropic_opus_mapping": { + "claude-opus-4-1-20250805", + "ecnu-max", + }, + } + if spec.case_id in expected_anthropic_aliases: + return shape.get("model") in expected_anthropic_aliases[spec.case_id] + return True + + +def make_case_record( + spec: CaseSpec, *, - headers: Mapping[str, str] | None, - payload: Mapping[str, Any] | None, - timeout: float, - secret: str, + tested_at: str, + status: int | None, + content_type: str, + response_shape: Mapping[str, Any], + headers: Mapping[str, str], + transport: str, + result: str, + classification: str, + notes: Sequence[str], ) -> dict[str, Any]: - result = request( - method, - url, - headers=headers, - payload=payload, - timeout=timeout, + record = { + "case_id": spec.case_id, + "tested_at": tested_at, + "protocol": spec.protocol, + "endpoint": spec.endpoint, + "model": spec.model, + "request_shape": spec.request_shape, + "documented_expectation": spec.documented_expectation, + "actual_http_status": status, + "actual_content_type": content_type, + "actual_response_shape": dict(response_shape), + "important_headers": dict(headers), + "sdk_or_transport": transport, + "result": result, + "classification": classification, + "notes": list(notes), + } + if tuple(record.keys()) != CASE_FIELDS: + raise AssertionError("case evidence schema drifted") + return record + + +def skipped_record(spec: CaseSpec, reason: str, classification: str = "application-policy") -> dict[str, Any]: + return make_case_record( + spec, + tested_at=utc_now(), + status=None, + content_type="", + response_shape={}, + headers={}, + transport="not executed", + result="skipped", + classification=classification, + notes=[reason], ) - return summarize_response(name, result, secrets=(secret,)) -def build_parser() -> argparse.ArgumentParser: - parser = argparse.ArgumentParser( - description="Run sanitized structural ECNU API smoke tests." +def _headers(kind: str, api_key: str) -> dict[str, str] | None: + if kind in {"public", "missing"}: + return None + if kind == "invalid": + return {"Authorization": "Bearer invalid-smoke-test-token"} + if kind == "anthropic": + return {"Authorization": f"Bearer {api_key}", "anthropic-version": "2023-06-01", "Content-Type": "application/json"} + return {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"} + + +def raw_executor(context: RunContext, spec: CaseSpec) -> Execution | SkipExecution: + try: + payload = spec.payload_factory(context) if spec.payload_factory else None + except CaseUnavailable as exc: + return SkipExecution(str(exc)) + headers = _headers(spec.headers_kind, context.api_key) or {} + if spec.response_kind == "stream" and payload is not None: + response = stream_request( + spec.endpoint, + headers=headers, + payload=payload, + timeout=context.timeout, + ) + else: + response = request( + spec.method, + spec.endpoint, + headers=headers, + payload=payload, + timeout=context.timeout, + ) + return Execution(response, "urllib.request (single attempt)") + + +def _bounded_sdk_content(response: Any, transport: Any) -> tuple[bytes, bool]: + try: + content = response.content + except Exception: + try: + content = response.read() + except Exception: + content = b"" + if isinstance(content, str): + content = content.encode("utf-8", errors="replace") + body = bytes(content) + exceeded = bool(getattr(transport, "response_limit_exceeded", False)) or len( + body + ) > MAX_RESPONSE_BYTES + return body[:MAX_RESPONSE_BYTES], exceeded + + +def _sdk_error(exc: Exception, transport: Any = None) -> HttpResult: + response = getattr(exc, "response", None) + if response is not None: + content, exceeded = _bounded_sdk_content(response, transport) + return HttpResult( + getattr(response, "status_code", None), + { + str(key).lower(): str(value) + for key, value in getattr(response, "headers", {}).items() + }, + content, + "SDK response byte limit exceeded" if exceeded else None, + ) + return HttpResult(None, {}, b"", type(exc).__name__) + + +class RecordingHttpxTransport: + """Sync wrapper used to prove SDK POST attempt counts.""" + + def __init__(self, inner: Any): + self.inner = inner + self.attempt_count = 0 + self.json_bodies: list[Any] = [] + self.response_limit_exceeded = False + + def handle_request(self, request_obj: Any) -> Any: + self.attempt_count += 1 + content = request_obj.read() + try: + self.json_bodies.append(json.loads(content)) + except (TypeError, UnicodeDecodeError, json.JSONDecodeError): + self.json_bodies.append(None) + response = self.inner.handle_request(request_obj) + import httpx + + owner = self + inner_stream = response.stream + + class BoundedStream(httpx.SyncByteStream): + def __iter__(self) -> Iterable[bytes]: + total = 0 + for chunk in inner_stream: + remaining = MAX_RESPONSE_BYTES - total + if remaining <= 0: + owner.response_limit_exceeded = True + break + if len(chunk) > remaining: + owner.response_limit_exceeded = True + yield chunk[:remaining] + break + yield chunk + total += len(chunk) + + def close(self) -> None: + inner_stream.close() + + response.stream = BoundedStream() + return response + + def close(self) -> None: + close = getattr(self.inner, "close", None) + if close: + close() + + +def _raw_sdk_response(raw: Any, transport: Any) -> HttpResult: + content, exceeded = _bounded_sdk_content(raw, transport) + return HttpResult( + raw.status_code, + {str(key).lower(): str(value) for key, value in raw.headers.items()}, + content, + "SDK response byte limit exceeded" if exceeded else None, ) - parser.add_argument( - "--low-cost", - action="store_true", - help="Add small Chat Completions and embedding POST probes.", + + +def openai_sdk_executor(resource: str, payload: Mapping[str, Any]) -> CustomExecutor: + def execute(context: RunContext, spec: CaseSpec) -> Execution | SkipExecution: + try: + import httpx + from openai import OpenAI + except ImportError: + return SkipExecution("optional openai/httpx dependency is not installed") + transport = RecordingHttpxTransport(httpx.HTTPTransport()) + client_http = httpx.Client(transport=transport, timeout=context.timeout) + try: + client = OpenAI(api_key=context.api_key, base_url=OPENAI_BASE, timeout=context.timeout, max_retries=0, http_client=client_http) + target: Any = client + for component in resource.split("."): + target = getattr(target, component) + response = _raw_sdk_response( + target.with_raw_response.create(**dict(payload)), transport + ) + except Exception as exc: + response = _sdk_error(exc, transport) + finally: + client_http.close() + return Execution(response, f"openai {package_version('openai')} (max_retries=0)", transport.attempt_count, {"request_attempt_count": transport.attempt_count}) + return execute + + +def anthropic_sdk_executor(payload: Mapping[str, Any]) -> CustomExecutor: + def execute(context: RunContext, spec: CaseSpec) -> Execution | SkipExecution: + try: + import httpx + from anthropic import Anthropic + except ImportError: + return SkipExecution("optional anthropic/httpx dependency is not installed") + transport = RecordingHttpxTransport(httpx.HTTPTransport()) + client_http = httpx.Client(transport=transport, timeout=context.timeout) + try: + client = Anthropic(api_key=context.api_key, base_url=ANTHROPIC_BASE, timeout=context.timeout, max_retries=0, http_client=client_http) + response = _raw_sdk_response( + client.messages.with_raw_response.create(**dict(payload)), transport + ) + except Exception as exc: + response = _sdk_error(exc, transport) + finally: + client_http.close() + return Execution(response, f"anthropic {package_version('anthropic')} (max_retries=0)", transport.attempt_count, {"request_attempt_count": transport.attempt_count}) + return execute + + +def run_langchain_mock_capture() -> tuple[dict[str, Any], int]: + """Use local MockTransport to observe the exact LangChain request body.""" + import httpx + from langchain_openai import OpenAIEmbeddings + + def handler(request_obj: Any) -> Any: + body = json.loads(request_obj.content) + input_value = body.get("input") + count = len(input_value) if isinstance(input_value, list) else 1 + return httpx.Response(200, json={"object": "list", "data": [{"object": "embedding", "index": index, "embedding": [0.0] * 1024} for index in range(count)], "model": "ecnu-embedding-small", "usage": {"prompt_tokens": 1, "total_tokens": 1}}) + + transport = RecordingHttpxTransport(httpx.MockTransport(handler)) + client_http = httpx.Client(transport=transport) + try: + embeddings = OpenAIEmbeddings(api_key="offline-test-key", base_url=OPENAI_BASE, model="ecnu-embedding-small", check_embedding_ctx_length=False, max_retries=0, http_client=client_http) + vectors = embeddings.embed_documents(["one", "two"]) + finally: + client_http.close() + body = transport.json_bodies[-1] + return { + "request_keys": sorted(body.keys()) if isinstance(body, dict) else [], + "input_is_string_array": isinstance(body, dict) and isinstance(body.get("input"), list) and all(isinstance(item, str) for item in body["input"]), + "dimensions_present": isinstance(body, dict) and "dimensions" in body, + "vector_count": len(vectors), + "vector_lengths": [len(vector) for vector in vectors], + }, transport.attempt_count + + +def langchain_capture_executor(context: RunContext, spec: CaseSpec) -> Execution | SkipExecution: + try: + shape, attempts = run_langchain_mock_capture() + except ImportError: + return SkipExecution("optional langchain-openai/httpx dependency is not installed") + except Exception as exc: + return Execution(HttpResult(None, {}, b"", type(exc).__name__), "langchain-openai local mock", 0) + response = HttpResult(200, {"content-type": "application/json"}, json.dumps(shape).encode()) + return Execution(response, f"langchain-openai {package_version('langchain-openai')} local MockTransport", attempts) + + +def langchain_live_executor(context: RunContext, spec: CaseSpec) -> Execution | SkipExecution: + try: + import httpx + from langchain_openai import OpenAIEmbeddings + except ImportError: + return SkipExecution("optional langchain-openai/httpx dependency is not installed") + transport = RecordingHttpxTransport(httpx.HTTPTransport()) + client_http = httpx.Client(transport=transport, timeout=context.timeout) + try: + embeddings = OpenAIEmbeddings(api_key=context.api_key, base_url=OPENAI_BASE, model="ecnu-embedding-small", check_embedding_ctx_length=False, max_retries=0, timeout=context.timeout, http_client=client_http) + vector = embeddings.embed_query("smoke test") + last = transport.json_bodies[-1] if transport.json_bodies else None + body = json.dumps({"count": 1, "vector_lengths": [len(vector)], "dimensions_present": isinstance(last, dict) and "dimensions" in last}).encode() + response = HttpResult(200, {"content-type": "application/json"}, body) + except Exception as exc: + response = _sdk_error(exc, transport) + finally: + client_http.close() + return Execution(response, f"langchain-openai {package_version('langchain-openai')} (max_retries=0)", transport.attempt_count, {"request_attempt_count": transport.attempt_count}) + + +def image_executor(context: RunContext, spec: CaseSpec) -> Execution | SkipExecution: + executed = raw_executor(context, spec) + if isinstance(executed, SkipExecution): + return executed + response = executed.response + updates: dict[str, Any] = {} + if response.status == 200: + payload = parse_json(response.body) + data = payload.get("data") if isinstance(payload, dict) else None + first = data[0] if isinstance(data, list) and data and isinstance(data[0], dict) else {} + media = None + declared_mime = None + b64_value = first.get("b64_json") + url_value = first.get("url") + if isinstance(b64_value, str): + try: + media = base64.b64decode(b64_value, validate=True) + updates["media_decode_valid"] = True + updates["media_source"] = "b64_json" + except (ValueError, base64.binascii.Error): + updates["media_decode_valid"] = False + elif isinstance(url_value, str): + updates["download_url_present"] = True + updates["media_source"] = "url" + downloaded = fetch_public_image(url_value, context.timeout) + updates["download_http_status"] = downloaded.status + declared_mime = downloaded.headers.get("content-type", "").split(";", 1)[0].lower() + updates["download_content_type"] = declared_mime + if downloaded.transport_error: + updates["media_verification_inconclusive"] = True + updates["download_error"] = downloaded.transport_error + elif downloaded.status == 200: + media = downloaded.body + if media is not None: + inferred_mime = _image_mime(media) + dimensions = _image_dimensions(media) + verified = bool( + media + and inferred_mime + and dimensions + and (declared_mime is None or declared_mime.startswith("image/")) + ) + updates.update( + { + "byte_count": len(media), + "sha256": hashlib.sha256(media).hexdigest(), + "media_content_type": inferred_mime, + "pixel_dimensions": dimensions, + "media_verified": verified, + } + ) + if not verified and not media.startswith(b"\x89PNG\r\n\x1a\n"): + updates["media_verification_inconclusive"] = True + updates["media_validation_scope"] = "non-interlaced PNG only" + return Execution(response, executed.transport, executed.attempts, updates) + + +def _payload(value: Mapping[str, Any]) -> PayloadFactory: + return lambda context: copy.deepcopy(value) + + +def _chat_payload(model: str, max_tokens: int = 16) -> dict[str, Any]: + return { + "model": model, + "messages": [ + {"role": "user", "content": "只回复字符串 ECNU_OK，不要添加其他文字。"} + ], + "max_tokens": max_tokens, + } + + +def _vision_payload(context: RunContext, model: str) -> dict[str, Any]: + return { + "model": model, + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "说出图中的颜色和形状。"}, + { + "type": "image_url", + "image_url": {"url": context.image_data_url()}, + }, + ], + } + ], + "max_tokens": 32, + } + + +def _responses_vision_payload(context: RunContext) -> dict[str, Any]: + return { + "model": "ecnu-max", + "input": [ + { + "role": "user", + "content": [ + {"type": "input_text", "text": "说出图中的颜色和形状。"}, + {"type": "input_image", "image_url": context.image_data_url()}, + ], + } + ], + "max_output_tokens": 32, + } + + +def _anthropic_vision_payload(context: RunContext) -> Mapping[str, Any]: + encoded = context.image_data_url().split(",", 1)[1] + return { + "model": "ecnu-max", + "max_tokens": 32, + "messages": [ + { + "role": "user", + "content": [ + { + "type": "image", + "source": { + "type": "base64", + "media_type": "image/png", + "data": encoded, + }, + }, + {"type": "text", "text": "说出图中的颜色和形状。"}, + ], + } + ], + } + + +def _thinking_tool_followup( + context: RunContext, *, include_reasoning: bool +) -> Mapping[str, Any]: + assistant = context.state.get("thinking_tool_assistant") + if not isinstance(assistant, dict): + raise CaseUnavailable("thinking-tool first turn did not yield an assistant message") + calls = assistant.get("tool_calls") + if not isinstance(calls, list) or not calls or not isinstance(calls[0], dict): + raise CaseUnavailable("thinking-tool first turn did not yield a tool call") + call_id = calls[0].get("id") + if not isinstance(call_id, str): + raise CaseUnavailable("thinking-tool call did not include an id") + reasoning = assistant.get("reasoning_content") + if include_reasoning and not (isinstance(reasoning, str) and reasoning): + raise CaseUnavailable("thinking-tool assistant message did not include reasoning_content") + assistant_copy = copy.deepcopy(assistant) + if not include_reasoning: + assistant_copy.pop("reasoning_content", None) + return { + "model": "ecnu-max", + "thinking": {"type": "enabled"}, + "reasoning_effort": "low", + "messages": [ + { + "role": "user", + "content": "请调用 echo 工具，value 必须是 ping，不要直接回答。", + }, + assistant_copy, + {"role": "tool", "tool_call_id": call_id, "content": "ping"}, + ], + "tools": [ECHO_TOOL], + "max_tokens": 256, + } + + +def _case( + case_id: str, + profiles: Sequence[str], + protocol: str, + endpoint: str, + model: str | None, + payload: Mapping[str, Any] | None, + expectation: str, + kind: str, + statuses: Sequence[int], + *, + method: str = "POST", + cost: float = 0.0, + headers_kind: str = "valid", + payload_factory: PayloadFactory | None = None, + custom_executor: CustomExecutor | None = None, + requires_valid_auth: bool = True, + required: bool = True, + capture: str | None = None, + classification: str = "observed", +) -> CaseSpec: + output_floor = _minimum_output_credit(model, payload) + if method == "POST" and cost + 1e-9 < output_floor: + raise ValueError( + f"{case_id} reserves {cost:g} credits below its {output_floor:g} output-only floor" + ) + return CaseSpec( + case_id=case_id, + profiles=frozenset(profiles), + protocol=protocol, + endpoint=endpoint, + model=model, + method=method, + request_shape=json_shape(payload) if payload is not None else {"type": "none"}, + documented_expectation=expectation, + response_kind=kind, + expected_statuses=tuple(statuses), + estimated_credits=cost, + headers_kind=headers_kind, + payload_factory=payload_factory or (_payload(payload) if payload is not None else None), + custom_executor=custom_executor, + requires_valid_auth=requires_valid_auth, + required=required, + capture_assistant_as=capture, + evidence_classification=classification, ) - parser.add_argument( - "--anthropic", - action="store_true", - help=( - "Add small Anthropic probes for ecnu-max and ecnu-max[1m]; " - "these requests may consume credits." + + +def _auth_cases() -> list[CaseSpec]: + all_profiles = ("auth", "core", "compatibility", "billable") + return [ + _case( + "service_status", + all_profiles, + "HTTPS", + STATUS_URL, + None, + None, + "The public service-status page should be reachable.", + "generic", + (200,), + method="GET", + headers_kind="public", + requires_valid_auth=False, + ), + _case( + "models_valid", + all_profiles, + "OpenAI-compatible", + OPENAI_BASE + "/models", + None, + None, + "A valid token should return an OpenAI-style model list.", + "models", + (200,), + method="GET", + ), + _case( + "models_invalid_token", + ("auth",), + "OpenAI-compatible", + OPENAI_BASE + "/models", + None, + None, + "Invalid authentication is documented as 401.", + "models", + (401,), + method="GET", + headers_kind="invalid", + requires_valid_auth=False, ), + _case( + "models_missing_auth", + ("auth",), + "OpenAI-compatible", + OPENAI_BASE + "/models", + None, + None, + "Missing authentication is documented as 401.", + "models", + (401,), + method="GET", + headers_kind="missing", + requires_valid_auth=False, + ), + ] + + +def _chat_response_cases() -> list[CaseSpec]: + endpoint = OPENAI_BASE + "/chat/completions" + cases: list[CaseSpec] = [] + for model, cost in (("ecnu-plus", 0.03), ("ecnu-max", 0.06)): + cases.append( + _case( + "chat_basic_" + model.replace("-", "_"), + ("core",), + "OpenAI-compatible", + endpoint, + model, + _chat_payload(model), + "Chat Completions returns choices[].message.content and usage.", + "chat", + (200,), + cost=cost, + ) + ) + stream = _chat_payload("ecnu-plus", 24) + stream["stream"] = True + cases.append( + _case( + "chat_stream_ecnu_plus", + ("core",), + "OpenAI-compatible SSE", + endpoint, + "ecnu-plus", + stream, + "Streaming emits data events, text deltas, and [DONE].", + "stream", + (200,), + cost=0.04, + ) ) - parser.add_argument( - "--timeout", - type=float, - default=30.0, - help="Per-request timeout in seconds (default: 30).", + thinking = _chat_payload("ecnu-max", 64) + thinking.update({"thinking": {"type": "enabled"}, "reasoning_effort": "low"}) + cases.append( + _case( + "chat_thinking_low", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-max", + thinking, + "ecnu-max accepts thinking enabled with reasoning_effort low.", + "thinking", + (200,), + cost=0.15, + ) ) - parser.add_argument( - "--openai-base", - default=DEFAULT_OPENAI_BASE, - help="OpenAI-compatible base URL.", + invalid_effort = copy.deepcopy(thinking) + invalid_effort["reasoning_effort"] = "invalid" + cases.append( + _case( + "chat_invalid_reasoning_effort", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-max", + invalid_effort, + "reasoning_effort accepts only low, high, or max.", + "generic", + (400, 422), + cost=0.15, + ) ) - parser.add_argument( - "--anthropic-base", - default=DEFAULT_ANTHROPIC_BASE, - help="Anthropic-compatible base URL.", + tool = { + "model": "ecnu-plus", + "messages": [{"role": "user", "content": "请调用 echo 工具，value 必须是 ping，不要直接回答。"}], + "tools": [ECHO_TOOL], + "max_tokens": 64, + } + cases.append( + _case( + "chat_tool_echo", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-plus", + tool, + "Tool calling returns a JSON echo(value=ping) tool call.", + "tool", + (200,), + cost=0.08, + ) ) - parser.add_argument( - "--account-type", - default="unspecified", - help="Non-secret account label for the report, such as personal-token.", + thinking_tool = copy.deepcopy(tool) + thinking_tool.update({"model": "ecnu-max", "thinking": {"type": "enabled"}, "reasoning_effort": "low"}) + thinking_tool["max_tokens"] = 256 + cases.append( + _case( + "chat_thinking_tool_first", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-max", + thinking_tool, + "Thinking plus tools returns an assistant tool call.", + "thinking_tool", + (200,), + cost=0.55, + capture="thinking_tool_assistant", + ) ) - parser.add_argument( - "--output", - type=Path, - help="Optional JSON report path; stdout is always printed.", + followup_shape = { + "model": "ecnu-max", + "thinking": {"type": "enabled"}, + "reasoning_effort": "low", + "messages": [ + {"role": "user", "content": "synthetic"}, + {"role": "assistant", "reasoning_content": "not-persisted", "tool_calls": [ECHO_TOOL]}, + {"role": "tool", "tool_call_id": "synthetic", "content": "ping"}, + ], + "tools": [ECHO_TOOL], + "max_tokens": 256, + } + cases.append( + _case( + "chat_thinking_tool_continue", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-max", + followup_shape, + "Splice the returned assistant message, including reasoning_content, before the tool result.", + "chat", + (200,), + cost=0.65, + payload_factory=lambda context: _thinking_tool_followup(context, include_reasoning=True), + ) ) - parser.add_argument( - "--strict", - action="store_true", - help="Return non-zero when required enabled checks fail.", + cases.append( + _case( + "chat_thinking_tool_omit_reasoning", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-max", + followup_shape, + "Omitting reasoning_content after a thinking tool call may be rejected.", + "generic", + (200, 400, 422), + cost=0.65, + payload_factory=lambda context: _thinking_tool_followup(context, include_reasoning=False), + required=False, + ) ) - return parser + responses_endpoint = OPENAI_BASE + "/responses" + response_rows = [ + ("responses_plus_basic", "ecnu-plus", {"model": "ecnu-plus", "input": "只回复 ECNU_OK。", "max_output_tokens": 16}, "Basic Responses input produces output and usage.", 0.03), + ("responses_max_effort_none", "ecnu-max", {"model": "ecnu-max", "input": "只回复 ECNU_OK。", "reasoning": {"effort": "none"}, "max_output_tokens": 16}, "reasoning.effort none disables thinking.", 0.07), + ("responses_max_effort_low", "ecnu-max", {"model": "ecnu-max", "input": "只回复 ECNU_OK。", "reasoning": {"effort": "low"}, "max_output_tokens": 64}, "A valid non-none reasoning effort is accepted.", 0.13), + ] + for case_id, model, payload, expectation, cost in response_rows: + cases.append(_case(case_id, ("core",), "OpenAI Responses-compatible", responses_endpoint, model, payload, expectation, "responses", (200,), cost=cost)) + return cases -def required_check_failed( - name: str, - summary: Mapping[str, Any], - *, - enabled_low_cost: bool, - enabled_anthropic: bool, -) -> bool: - status = summary.get("status") - if name == "models_valid": - return status != 200 or not summary.get("model_ids") - if enabled_low_cost and name == "chat_plus": - return status != 200 or not summary.get("has_choices") - if enabled_low_cost and name.startswith("embedding_"): - return status != 200 or not summary.get("dimensions") - if enabled_anthropic and name == "anthropic_ecnu_max": - return status != 200 - # Invalid/missing-auth behavior and the [1m] suffix are diagnostic probes. - return False +def _embedding_rerank_cases() -> list[CaseSpec]: + cases: list[CaseSpec] = [] + endpoint = OPENAI_BASE + "/embeddings" + rows: list[tuple[str, Any, str, tuple[int, ...]]] = [ + ("embedding_scalar", "one text", "A scalar string is accepted.", (200,)), + ("embedding_array", ["first text", "second text"], "A string array returns one ordered vector per input.", (200,)), + ("embedding_empty_array", [], "Empty-array behavior is not documented; observe validation.", (200, 400, 422)), + ("embedding_token_ids", [123, 456], "OpenAI integer token-ID arrays are unsupported.", (400, 422)), + ("embedding_8192_chars", "a" * 8192, "The published input limit is 8192 characters.", (200,)), + ("embedding_8193_chars", "a" * 8193, "Input over the 8192-character limit is rejected.", (400, 422)), + ("embedding_two_long_strings", ["a" * 5000, "b" * 5000], "Array limit scope is undocumented; observe per-item versus total behavior.", (200, 400, 422)), + ] + for case_id, input_value, expectation, statuses in rows: + payload = {"model": "ecnu-embedding-small", "input": input_value} + cases.append(_case(case_id, ("core",), "OpenAI-compatible", endpoint, "ecnu-embedding-small", payload, expectation, "embedding", statuses, cost=0.05)) + documents = ["华东师范大学位于上海。", "量子计算使用量子比特。", "校园图书馆提供学习空间。"] + rerank_rows = [ + ("rerank_default", {"model": "ecnu-rerank", "query": "大学在哪里", "documents": documents}, (200,), "With three documents and no top_n, all three return; this does not independently prove the documented default of 5."), + ("rerank_top_two", {"model": "ecnu-rerank", "query": "大学在哪里", "documents": documents, "top_n": 2}, (200,), "Explicit top_n=2 returns at most two results."), + ("rerank_without_documents", {"model": "ecnu-rerank", "query": "大学在哪里", "documents": documents, "return_documents": False}, (200,), "return_documents=false omits document text."), + ("rerank_with_documents", {"model": "ecnu-rerank", "query": "大学在哪里", "documents": documents, "return_documents": True}, (200,), "return_documents=true includes document text."), + ("rerank_document_8193", {"model": "ecnu-rerank", "query": "a", "documents": ["a" * 8193]}, (400, 422), "Each document is limited to 8192 characters."), + ("rerank_top_n_over_count", {"model": "ecnu-rerank", "query": "大学在哪里", "documents": documents, "top_n": 5}, (200, 400, 422), "No maximum top_n is documented; observe values above document count."), + ] + endpoint = OPENAI_BASE + "/rerank" + for case_id, payload, statuses, expectation in rerank_rows: + cases.append(_case(case_id, ("core",), "Cohere-compatible rerank", endpoint, "ecnu-rerank", payload, expectation, "rerank", statuses, cost=0.1)) + return cases -def main(argv: list[str] | None = None) -> int: - args = build_parser().parse_args(argv) - api_key = os.environ.get("ECNU_API_KEY") - if not api_key: - print( - "ECNU_API_KEY is required. Store it in the environment; " - "do not pass it as a command-line argument.", - file=sys.stderr, - ) - return 2 - openai_base = args.openai_base.rstrip("/") - anthropic_messages = args.anthropic_base.rstrip("/") + "/v1/messages" - auth_headers = { - "Authorization": f"Bearer {api_key}", - "Content-Type": "application/json", +def _vision_structured_error_cases() -> list[CaseSpec]: + cases: list[CaseSpec] = [] + endpoint = OPENAI_BASE + "/chat/completions" + representative_vision = { + "model": "ecnu-plus", + "messages": [ + { + "role": "user", + "content": [ + {"type": "text", "text": "synthetic"}, + {"type": "image_url", "image_url": {"url": "data-url-omitted"}}, + ], + } + ], + "max_tokens": 32, } - - report: dict[str, Any] = { - "meta": { - "tested_at_utc": datetime.now(timezone.utc).isoformat(), - "python": platform.python_version(), - "account_type": args.account_type, - "openai_base": openai_base, - "anthropic_base": args.anthropic_base.rstrip("/"), - "profiles": { - "models": True, - "low_cost": args.low_cost, - "anthropic": args.anthropic, - }, - "notes": [ - "Successful model output is intentionally omitted.", - "Image generation and TTS are intentionally not automated.", - "Statuses are observations, not permanent API contracts.", - ], + for model, statuses in (("ecnu-plus", (200,)), ("ecnu-max", (400, 422))): + cases.append( + _case( + "vision_direct_" + model.replace("-", "_"), + ("core",), + "OpenAI-compatible", + endpoint, + model, + {**representative_vision, "model": model}, + ( + "ecnu-plus accepts structured text and image_url data parts." + if model == "ecnu-plus" + else "ecnu-max does not support direct Chat Completions vision." + ), + "vision", + statuses, + cost=0.1 if model == "ecnu-plus" else 0.25, + payload_factory=lambda context, selected=model: _vision_payload(context, selected), + ) + ) + structured = { + "model": "ecnu-plus", + "messages": [{"role": "user", "content": "姓名张三，部门数据科学部。仅按 schema 输出。"}], + "response_format": { + "type": "json_schema", + "json_schema": {"name": "person", "schema": STRUCTURED_SCHEMA}, }, - "tests": {}, + "max_tokens": 128, } - tests: dict[str, Any] = report["tests"] - - tests["models_valid"] = run_case( - "models_valid", - "GET", - openai_base + "/models", - headers={"Authorization": f"Bearer {api_key}"}, - payload=None, - timeout=args.timeout, - secret=api_key, - ) - tests["models_invalid"] = run_case( - "models_invalid", - "GET", - openai_base + "/models", - headers={"Authorization": "Bearer invalid-smoke-test-token"}, - payload=None, - timeout=args.timeout, - secret=api_key, + cases.append( + _case( + "structured_output_ecnu_plus", + ("core",), + "OpenAI-compatible", + endpoint, + "ecnu-plus", + structured, + "json_schema constrains structure, not semantic correctness.", + "structured", + (200,), + cost=0.15, + ) ) - tests["models_missing"] = run_case( - "models_missing", - "GET", - openai_base + "/models", - headers=None, - payload=None, - timeout=args.timeout, - secret=api_key, + error_rows = [ + ("error_missing_model", {"messages": [{"role": "user", "content": "test"}]}, (400, 422), "model is required.", "valid"), + ("error_wrong_messages_type", {"model": "ecnu-plus", "messages": "wrong-type"}, (400, 422), "messages must be an array.", "valid"), + ("error_invalid_token_post", _chat_payload("ecnu-plus"), (401,), "Invalid POST authentication is documented as 401.", "invalid"), + ("error_unsupported_model", _chat_payload("definitely-not-an-ecnu-model"), (400, 404, 422), "Unsupported models return a bounded error.", "valid"), + ("error_unsupported_parameter_value", {**_chat_payload("ecnu-plus"), "temperature": 2}, (400, 422), "temperature outside 0 through 1 is unsupported.", "valid"), + ] + for case_id, payload, statuses, expectation, headers_kind in error_rows: + cases.append( + _case( + case_id, + ("core",), + "OpenAI-compatible", + endpoint, + payload.get("model"), + payload, + expectation, + "generic", + statuses, + cost=0.02, + headers_kind=headers_kind, + requires_valid_auth=headers_kind == "valid", + ) + ) + return cases + + +def _optional_sdk_cases() -> list[CaseSpec]: + cases: list[CaseSpec] = [] + sdk_rows = [ + ("openai_sdk_chat", "chat.completions", _chat_payload("ecnu-plus"), "chat", 0.03), + ("openai_sdk_responses", "responses", {"model": "ecnu-plus", "input": "只回复 ECNU_OK。", "max_output_tokens": 16}, "responses", 0.03), + ("openai_sdk_embedding", "embeddings", build_embedding_payload("smoke test"), "embedding", 0.05), + ] + for case_id, resource, payload, kind, cost in sdk_rows: + cases.append( + _case( + case_id, + ("core",), + "OpenAI Python SDK", + OPENAI_BASE, + payload.get("model"), + payload, + "The optional SDK preserves the wire contract with retries disabled.", + kind, + (200,), + cost=cost, + custom_executor=openai_sdk_executor(resource, payload), + ) + ) + cases.extend( + [ + _case( + "langchain_embedding_wire_capture", + ("core",), + "LangChain local MockTransport", + OPENAI_BASE + "/embeddings", + "ecnu-embedding-small", + build_embedding_payload(["one", "two"]), + "check_embedding_ctx_length=false sends strings and no dimensions field.", + "capture", + (200,), + custom_executor=langchain_capture_executor, + requires_valid_auth=False, + classification="application-policy", + ), + _case( + "langchain_embedding_live", + ("core",), + "LangChain OpenAIEmbeddings", + OPENAI_BASE + "/embeddings", + "ecnu-embedding-small", + build_embedding_payload("smoke test"), + "The live vector has length 1024 without sending dimensions.", + "capture", + (200,), + cost=0.05, + custom_executor=langchain_live_executor, + ), + ] ) + return cases + - if args.low_cost: - tests["chat_plus"] = run_case( - "chat_plus", - "POST", - openai_base + "/chat/completions", - headers=auth_headers, - payload={ - "model": "ecnu-plus", - "messages": [ - {"role": "user", "content": "Reply with exactly: ok"} +def _compatibility_cases() -> list[CaseSpec]: + cases: list[CaseSpec] = [] + representative = { + "model": "ecnu-max", + "input": [ + { + "role": "user", + "content": [ + {"type": "input_text", "text": "synthetic"}, + {"type": "input_image", "image_url": "data-url-omitted"}, ], - "max_tokens": 8, - }, - timeout=args.timeout, - secret=api_key, - ) - tests["embedding_scalar"] = run_case( - "embedding_scalar", - "POST", - openai_base + "/embeddings", - headers=auth_headers, - payload=build_embedding_payload("smoke test"), - timeout=args.timeout, - secret=api_key, - ) - tests["embedding_array"] = run_case( - "embedding_array", - "POST", - openai_base + "/embeddings", - headers=auth_headers, - payload=build_embedding_payload(["one", "two"]), - timeout=args.timeout, - secret=api_key, - ) - - if args.anthropic: - anthropic_headers = { - "Authorization": f"Bearer {api_key}", - "x-api-key": api_key, - "anthropic-version": "2023-06-01", - "Content-Type": "application/json", + } + ], + } + cases.append( + _case( + "responses_max_vision_compatibility", + ("compatibility",), + "OpenAI Responses-compatible", + OPENAI_BASE + "/responses", + "ecnu-max", + representative, + "Not documented as a stable contract; observe current ecnu-max image handling.", + "responses_vision_compat", + (200,), + cost=0.25, + payload_factory=_responses_vision_payload, + ) + ) + rows = [ + ("anthropic_plus", "ecnu-plus", {}, (200,), "ecnu-plus is accepted directly."), + ("anthropic_max", "ecnu-max", {}, (200,), "ecnu-max is accepted directly."), + ("anthropic_sonnet_mapping", "claude-sonnet-4-20250514", {}, (200,), "The sonnet alias is accepted; its response label does not prove effective ecnu-plus routing."), + ("anthropic_opus_mapping", "claude-opus-4-1-20250805", {}, (200,), "The opus alias is accepted; its response label does not prove effective ecnu-max routing."), + ("anthropic_max_1m_fallback_plain_max", "ecnu-max", {}, (200,), "Plain ecnu-max is the explicit control and fallback for suffix-specific failures."), + ("anthropic_max_1m", "ecnu-max[1m]", {}, (200,), "The [1m] suffix is documented compatibility metadata."), + ("anthropic_effort_none", "ecnu-max", {"output_config": {"effort": "none"}}, (200,), "output_config.effort none disables thinking."), + ("anthropic_effort_low", "ecnu-max", {"output_config": {"effort": "low"}}, (200,), "A valid non-none effort is accepted."), + ("anthropic_invalid_effort", "ecnu-max", {"output_config": {"effort": "invalid"}}, (400, 422), "Unsupported effort values are rejected."), + ("anthropic_missing_model", None, {}, (400, 422), "model is required."), + ] + for case_id, model, extra, statuses, expectation in rows: + payload: dict[str, Any] = { + "max_tokens": 32, + "messages": [{"role": "user", "content": "只回复 ECNU_OK。"}], } - for model, name in ( - ("ecnu-max", "anthropic_ecnu_max"), - ("ecnu-max[1m]", "anthropic_ecnu_max_1m"), - ): - tests[name] = run_case( - name, - "POST", - anthropic_messages, - headers=anthropic_headers, - payload={ - "model": model, - "max_tokens": 8, - "messages": [ - {"role": "user", "content": "Reply with exactly: ok"} - ], - }, - timeout=args.timeout, - secret=api_key, + if model is not None: + payload["model"] = model + payload.update(extra) + cases.append( + _case( + case_id, + ("compatibility",), + "Anthropic-compatible", + ANTHROPIC_MESSAGES_URL, + model, + payload, + expectation, + "anthropic" if 200 in statuses else "generic", + statuses, + cost=0.08 if _effective_dialog_model(model) == "ecnu-max" else 0.04, + headers_kind="anthropic", ) + ) + anthropic_vision_shape = { + "model": "ecnu-max", + "max_tokens": 32, + "messages": [ + { + "role": "user", + "content": [ + { + "type": "image", + "source": {"type": "base64", "media_type": "image/png", "data": "omitted"}, + } + ], + } + ], + } + cases.append( + _case( + "anthropic_max_vision_compatibility", + ("compatibility",), + "Anthropic-compatible", + ANTHROPIC_MESSAGES_URL, + "ecnu-max", + anthropic_vision_shape, + "Not documented as a stable contract; observe current ecnu-max image handling.", + "anthropic_vision_compat", + (200,), + cost=0.25, + headers_kind="anthropic", + payload_factory=_anthropic_vision_payload, + ) + ) + sdk_payload = { + "model": "ecnu-plus", + "max_tokens": 16, + "messages": [{"role": "user", "content": "只回复 ECNU_OK。"}], + } + cases.append( + _case( + "anthropic_sdk_plus", + ("compatibility",), + "Anthropic Python SDK", + ANTHROPIC_MESSAGES_URL, + "ecnu-plus", + sdk_payload, + "The optional SDK calls the compatibility root with retries disabled.", + "anthropic", + (200,), + cost=0.04, + custom_executor=anthropic_sdk_executor(sdk_payload), + ) + ) + return cases + +def _billable_cases() -> list[CaseSpec]: + cases: list[CaseSpec] = [] + endpoint = OPENAI_BASE + "/audio/speech" + # Required priority under the default budget: PCM, invalid voice, one image. + rows = [ + ("tts_xiayu_pcm", "xiayu", "pcm", (200,), "PCM returns binary audio and observable format headers.", True), + ("tts_invalid_voice", "definitely_invalid_voice", "mp3", (400,), "An invalid voice is documented as a 400 JSON client error.", True), + ] + for case_id, voice, response_format, statuses, expectation, required in rows: + payload = {"model": "ecnu-tts", "input": "你好。", "voice": voice, "response_format": response_format} + cases.append(_case(case_id, ("billable",), "OpenAI-compatible binary", endpoint, "ecnu-tts", payload, expectation, "tts" if 200 in statuses else "generic", statuses, cost=5.0, required=required)) + image_payload = { + "model": "ecnu-image", + "prompt": "白底蓝色圆形图标，简洁扁平，无文字", + "size": "512x512", + "response_format": "url", + } + cases.append( + _case( + "image_generation_documented", + ("billable",), + "OpenAI-compatible", + OPENAI_BASE + "/images/generations", + "ecnu-image", + image_payload, + "At most one documented 512x512 generation returns URL or base64 image data.", + "image", + (200,), + cost=30.0, + custom_executor=image_executor, + ) + ) + optional_rows = [ + ("tts_xiayu_mp3", "xiayu", "The default campus voice returns MP3 audio."), + ("tts_liwa_mp3", "liwa", "The second documented campus voice returns MP3 audio."), + ("tts_extended_voice_sample", "male_warm", "One extended voice is optional and runs only with budget remaining."), + ] + for case_id, voice, expectation in optional_rows: + payload = {"model": "ecnu-tts", "input": "你好。", "voice": voice, "response_format": "mp3"} + cases.append( + _case( + case_id, + ("billable",), + "OpenAI-compatible binary", + endpoint, + "ecnu-tts", + payload, + expectation, + "tts", + (200,), + cost=5.0, + required=False, + ) + ) + return cases + + +def build_cases() -> list[CaseSpec]: + return ( + _auth_cases() + + _chat_response_cases() + + _embedding_rerank_cases() + + _vision_structured_error_cases() + + _optional_sdk_cases() + + _compatibility_cases() + + _billable_cases() + ) + + +def _default_max_credits() -> float: + return DEFAULT_MAX_CREDITS + + +def build_parser() -> argparse.ArgumentParser: + parser = argparse.ArgumentParser(description="Run serial, sanitized ECNU API validation profiles.") + parser.add_argument("--profile", action="append", choices=PROFILES, help="Profile to run; repeat to combine (default: auth).") + parser.add_argument("--max-credits", type=float, default=_default_max_credits(), help="Conservative planned-credit cap (default: 50).") + parser.add_argument("--timeout", type=float, default=60.0) + parser.add_argument("--output", type=Path, help="Optional sanitized JSON path.") + parser.add_argument( + "--case", + dest="case_ids", + action="append", + help="Run only the named case within the selected profile; repeat as needed.", + ) + parser.add_argument("--strict", action="store_true") + parser.add_argument("--account-type", default="unspecified") + parser.add_argument("--network-environment", default="unspecified") + # Clean legacy compatibility: old flags map to the new non-billable profiles. + parser.add_argument("--low-cost", action="store_true", help=argparse.SUPPRESS) + parser.add_argument("--anthropic", action="store_true", help=argparse.SUPPRESS) + return parser + + +def selected_profiles(args: argparse.Namespace, parser: argparse.ArgumentParser) -> set[str]: + selected = set(args.profile or []) + if selected and (args.low_cost or args.anthropic): + parser.error("do not combine --profile with legacy --low-cost/--anthropic") + if not selected: + selected = {"auth"} + if args.low_cost: + selected = {"core"} + if args.anthropic: + selected.add("compatibility") + if "all" in selected: + return {"auth", "core", "compatibility", "billable"} + return selected + + +def _local_401_has_control(context: RunContext, spec: CaseSpec) -> bool: + if spec.case_id == "anthropic_max_1m": + successful_models = context.state.get("successful_models") + return isinstance(successful_models, set) and "ecnu-max" in successful_models + return bool(context.state.get("valid_auth_observed")) + + +def run_one(context: RunContext, spec: CaseSpec) -> dict[str, Any]: + if context.stop_reason: + return skipped_record(spec, context.stop_reason) + if spec.requires_valid_auth and context.auth_gate_reason: + return skipped_record(spec, context.auth_gate_reason, "unverified") + if not context.budget.reserve(spec.estimated_credits): + return skipped_record(spec, f"credit budget stop: {spec.estimated_credits:g} more credits would exceed {context.budget.limit:g}") + + executed = (spec.custom_executor or raw_executor)(context, spec) + if isinstance(executed, SkipExecution): + context.budget.release_unattempted(spec.estimated_credits) + return skipped_record(spec, executed.reason, executed.classification) + response = executed.response + shape = summarize_response(spec.response_kind, response, secrets=(context.api_key,)) + shape.update(sanitize_value(dict(executed.shape_updates), (context.api_key,))) + if spec.model and isinstance(shape.get("model"), str): + shape["response_model_matches_request"] = shape["model"] == spec.model + consumed, credit_basis = estimate_consumed_credits(spec, response.status, shape) + if executed.attempts > 1: + additional = spec.estimated_credits * (executed.attempts - 1) + consumed += additional + credit_basis += "; unexpected extra attempts charged at planned allowance" + shape["estimated_consumed_credits"] = round(consumed, 8) + shape["credit_estimate_basis"] = credit_basis + context.estimated_consumed_credits += consumed + if executed.attempts > 1: + shape["unexpected_retry_count"] = executed.attempts - 1 + + authenticated_success = ( + spec.requires_valid_auth + and response.status is not None + and 200 <= response.status < 300 + and (spec.case_id != "models_valid" or bool(shape.get("model_ids"))) + ) + if authenticated_success: + context.state["valid_auth_observed"] = True + if spec.model: + successful_models = context.state.setdefault("successful_models", set()) + if isinstance(successful_models, set): + successful_models.add(spec.model) + local_401_accepted = ( + response.status == 401 + and spec.case_id in CASE_LOCAL_401 + and _local_401_has_control(context, spec) + ) + + if response.transport_error: + result = "inconclusive" + notes = ["ambiguous transport outcome; the POST was not retried"] + elif shape.get("media_verification_inconclusive"): + result = "inconclusive" + notes = ["image generation returned, but bounded media verification was inconclusive"] + elif response.status not in spec.expected_statuses: + result = "mismatch" + notes = ["observed status differs from the documented expectation"] + elif spec.response_kind == "stream" and not response.headers.get( + "content-type", "" + ).startswith("text/event-stream"): + result = "mismatch" + notes = ["stream body was not returned with an SSE content type"] + elif not case_response_matches(spec, response.status, shape): + result = "mismatch" + notes = ["status matched but required response structure did not"] + elif executed.attempts > 1: + result = "mismatch" + notes = ["transport attempted the POST more than once"] + else: + result = "pass" + notes = ["structural check passed; generated content was omitted"] + + if local_401_accepted: + notes.append("case-specific 401 does not invalidate the already verified bearer token") + + if spec.capture_assistant_as and response.status == 200: + message = _chat_message(parse_json(response.body)) + if message: + context.state[spec.capture_assistant_as] = copy.deepcopy(dict(message)) + + if executed.attempts > 1: + context.stop_reason = "conservative global stop after a transport attempted more than once" + elif response.status == 429 and _official_api_endpoint(spec.endpoint): + context.stop_reason = "conservative global stop after ECNU API HTTP 429" + elif spec.case_id == "models_valid" and (response.status != 200 or not shape.get("model_ids")): + context.auth_gate_reason = "valid-token model discovery did not prove authentication; authenticated POSTs stopped" + elif spec.requires_valid_auth: + if response.status in {401, 403} and not local_401_accepted: + context.stop_reason = f"conservative stop after authenticated HTTP {response.status}" + elif response.transport_error and spec.method == "POST": + context.stop_reason = "conservative stop after an ambiguous authenticated POST" + + return make_case_record( + spec, + tested_at=utc_now(), + status=response.status, + content_type=response.headers.get("content-type", ""), + response_shape=shape, + headers=extract_important_headers(response.headers), + transport=executed.transport, + result=result, + classification=spec.evidence_classification, + notes=notes, + ) + + +def strict_failure_ids( + records: Sequence[Mapping[str, Any]], + specs: Mapping[str, CaseSpec], + explicit_case_ids: set[str], +) -> list[str]: + return [ + str(record["case_id"]) + for record in records + if record["result"] in {"mismatch", "inconclusive"} + or ( + record["result"] == "skipped" + and ( + specs[str(record["case_id"])].required + or str(record["case_id"]) in explicit_case_ids + ) + ) + ] + + +def main(argv: list[str] | None = None) -> int: + parser = build_parser() + args = parser.parse_args(argv) + if not math.isfinite(args.max_credits) or args.max_credits < 0: + parser.error("--max-credits must be finite and non-negative") + if not math.isfinite(args.timeout) or args.timeout <= 0: + parser.error("--timeout must be finite and positive") + profiles = selected_profiles(args, parser) + api_key = os.environ.get("ECNU_API_KEY") + if not api_key: + print("ECNU_API_KEY is required. Set it in the local environment; do not pass it on the command line.", file=sys.stderr) + return 2 + + selected = [case for case in build_cases() if case.profiles & profiles] + if args.case_ids: + requested = set(args.case_ids) + available = {case.case_id for case in selected} + unknown = requested - available + if unknown: + parser.error( + "--case is not in the selected profile: " + ", ".join(sorted(unknown)) + ) + selected = [case for case in selected if case.case_id in requested] + budget = CreditBudget(args.max_credits, planned=sum(case.estimated_credits for case in selected)) + with temporary_artifacts() as artifact_dir: + context = RunContext(api_key, args.timeout, artifact_dir, budget) + records = [run_one(context, case) for case in selected] + context.state.clear() + + report = { + "schema_version": 1, + "environment": { + "tested_at_utc": utc_now(), + "test_timezone": "UTC", + "os": platform.system(), + "os_release": platform.release(), + "python_version": platform.python_version(), + "openai_version": package_version("openai"), + "anthropic_version": package_version("anthropic"), + "langchain_openai_version": package_version("langchain-openai"), + "requests_version": package_version("requests"), + "httpx_version": package_version("httpx"), + "account_type": redact_text(args.account_type, (api_key,))[:100], + "network_environment": redact_text(args.network_environment, (api_key,))[:100], + }, + "profiles": sorted(profiles), + "official_hosts": {"openai_base": OPENAI_BASE, "anthropic_messages": ANTHROPIC_MESSAGES_URL, "service_status": STATUS_URL}, + "budget": { + "max_credits": budget.limit, + "planned_credits": round(budget.planned, 4), + "reserved_credits": round(budget.reserved, 4), + "estimated_consumed_credits": round(context.estimated_consumed_credits, 8), + "estimation_policy": "fixed prices plus conservative small-dialog allowances; reserve before request", + "budget_stop_triggered": budget.exhausted, + }, + "cases": records, + "notes": [ + "Requests were serial and POST retries were disabled.", + "Reports omit prompts, outputs, reasoning text, media, authorization, and one-time URLs.", + "A visible model is not proof of endpoint capability.", + ], + } output_text = json.dumps(report, ensure_ascii=False, indent=2) print(output_text) if args.output: @@ -428,23 +2716,11 @@ def main(argv: list[str] | None = None) -> int: args.output.write_text(output_text + "\n", encoding="utf-8") if args.strict: - failures = [ - name - for name, summary in tests.items() - if required_check_failed( - name, - summary, - enabled_low_cost=args.low_cost, - enabled_anthropic=args.anthropic, - ) - ] + specs = {case.case_id: case for case in selected} + failures = strict_failure_ids(records, specs, set(args.case_ids or [])) if failures: - print( - "Strict checks failed: " + ", ".join(failures), - file=sys.stderr, - ) + print("Strict checks failed: " + ", ".join(failures), file=sys.stderr) return 1 - return 0 diff --git a/scripts/validate_skill.py b/scripts/validate_skill.py index 9ff8c71..7c57fc5 100755 --- a/scripts/validate_skill.py +++ b/scripts/validate_skill.py @@ -9,6 +9,8 @@ ROOT = Path(__file__).resolve().parents[1] SKILL = ROOT / "SKILL.md" +LIVE_ARTIFACT_DIR = ".live-artifacts" +MAX_TEXT_SCAN_BYTES = 4 * 1024 * 1024 REQUIRED_FILES = [ "SKILL.md", @@ -22,13 +24,66 @@ "scripts/smoke_test.py", "scripts/validate_skill.py", "tests/test_smoke_test.py", + "tests/test_repository_contracts.py", ] NAME_RE = re.compile(r"^[a-z0-9]+(?:-[a-z0-9]+)*$") LINK_RE = re.compile(r"\[[^\]]+\]\(([^)]+)\)") -SECRET_RE = re.compile(r"\bsk-[A-Za-z0-9]{24,}\b") +SECRET_RE = re.compile( + r"(?]+)" +) +ROOT_USER_PATH_RE = re.compile(r"(?]+)" +) +CURRENT_STATE_RE = re.compile(r"(?im)^#{1,6}\s+current state\s*$") +RAW_REPORT_RE = re.compile( + r"(?i)^(?:smoke|live|raw)[-_]?(?:results?|responses?|reports?|evidence)" + r".*\.jsonl?$" +) +PROFILE_REPORT_RE = re.compile( + r"(?i)^(?:auth|core|compatibility|billable|all)(?:[-_].*)?\.jsonl?$" +) +MEDIA_SUFFIXES = { + ".aac", ".flac", ".gif", ".jpeg", ".jpg", ".mp3", ".mp4", + ".opus", ".pcm", ".png", ".wav", ".webp", +} +ALLOWED_DEVIATION_STATUSES = { + "active", "resolved", "inconclusive", "not-retested", +} +DEVIATION_FIELD_ALIASES = { + "tested at": "tested_at", + "test date": "tested_at", + "date": "tested_at", + "environment": "environment", + "test environment": "environment", + "protocol and endpoint": "protocol_endpoint", + "protocol endpoint": "protocol_endpoint", + "protocol": "protocol", + "endpoint": "endpoint", + "documented expectation": "documented_expectation", + "observed behavior": "observed_behavior", + "actual behavior": "observed_behavior", + "reproduction conditions": "reproduction_conditions", + "reproduction": "reproduction_conditions", + "impact": "impact", + "recommended fallback": "fallback", + "application fallback": "fallback", + "fallback": "fallback", + "status": "status", +} +REQUIRED_DEVIATION_FIELDS = { + "tested_at", "environment", "documented_expectation", "observed_behavior", + "reproduction_conditions", "impact", "fallback", "status", +} def parse_frontmatter(text: str) -> tuple[dict[str, str], str]: @@ -50,7 +105,6 @@ def parse_frontmatter(text: str) -> tuple[dict[str, str], str]: if ":" not in line: raise ValueError(f"invalid frontmatter line: {line!r}") key, value = line.split(":", 1) - key = key.strip() value = value.strip() if value in {">", "|"}: continuation: list[str] = [] @@ -61,97 +115,235 @@ def parse_frontmatter(text: str) -> tuple[dict[str, str], str]: break continuation.append(candidate.strip()) index += 1 - fields[key] = " ".join(part for part in continuation if part) + fields[key.strip()] = " ".join(part for part in continuation if part) continue - fields[key] = value.strip("\"'") + fields[key.strip()] = value.strip("\"'") index += 1 return fields, body -def iter_text_files() -> list[Path]: - paths: list[Path] = [] - for path in ROOT.rglob("*"): - if not path.is_file(): - continue - if any(part in {".git", ".venv", "__pycache__"} for part in path.parts): - continue - if path.suffix.lower() in { - ".md", - ".py", - ".yml", - ".yaml", - ".txt", - ".gitignore", - } or path.name == ".gitignore": - paths.append(path) - return paths +def iter_repository_files(root: Path = ROOT) -> list[Path]: + excluded = {".git", ".venv", "__pycache__", LIVE_ARTIFACT_DIR} + return [ + path for path in root.rglob("*") + if path.is_file() and not any(part in excluded for part in path.parts) + ] -def main() -> int: +def iter_text_files(root: Path = ROOT) -> list[Path]: + text_files: list[Path] = [] + for path in iter_repository_files(root): + with path.open("rb") as handle: + prefix = handle.read(8192) + if b"\x00" not in prefix: + text_files.append(path) + return text_files + + +def read_bounded_text(path: Path) -> tuple[str, bool]: + with path.open("rb") as handle: + data = handle.read(MAX_TEXT_SCAN_BYTES + 1) + exceeded = len(data) > MAX_TEXT_SCAN_BYTES + return data[:MAX_TEXT_SCAN_BYTES].decode("utf-8", errors="replace"), exceeded + + +def _is_placeholder(value: str) -> bool: + value = value.strip().rstrip(",;)").lower() + if value.startswith(("<", "$", "{", "[", "%")): + return True + markers = ( + "api_key", "api-key", "dummy", "example", "fake", "invalid", + "placeholder", "redacted", "test-", "token-here", "your-", + ) + return any(marker in value for marker in markers) + + +def find_literal_bearers(text: str) -> list[int]: + violations: list[int] = [] + for line_number, line in enumerate(text.splitlines(), start=1): + for match in AUTH_BEARER_RE.finditer(line): + if not _is_placeholder(match.group(1)): + violations.append(line_number) + return violations + + +def find_personal_paths(text: str) -> list[int]: + violations: list[int] = [] + for line_number, line in enumerate(text.splitlines(), start=1): + unix = UNIX_USER_PATH_RE.search(line) + windows = WINDOWS_USER_PATH_RE.search(line) + if ROOT_USER_PATH_RE.search(line): + violations.append(line_number) + elif unix and not _is_placeholder(unix.group(1)): + violations.append(line_number) + elif windows and not _is_placeholder(windows.group(1)): + violations.append(line_number) + return violations + + +def is_forbidden_artifact(relative: Path) -> bool: + if LIVE_ARTIFACT_DIR in relative.parts: + return False + name = relative.name.lower() + if name == ".env" or (name.startswith(".env.") and name != ".env.example"): + return True + if relative.suffix.lower() in MEDIA_SUFFIXES: + return True + raw_directories = { + "live-results", "raw-responses", "raw-results", "smoke-results", + } + if any(part.lower() in raw_directories for part in relative.parts[:-1]): + return True + return bool(RAW_REPORT_RE.fullmatch(name) or PROFILE_REPORT_RE.fullmatch(name)) + + +def validate_agents_text(text: str) -> list[str]: errors: list[str] = [] + if len(text.rstrip("\n").splitlines()) > 50: + errors.append("AGENTS.md exceeds the target maximum of 50 lines") + if CURRENT_STATE_RE.search(text): + errors.append("AGENTS.md must not contain a dated Current State section") + if find_personal_paths(text): + errors.append("AGENTS.md contains a machine-specific personal path") + return errors - for relative in REQUIRED_FILES: - if not (ROOT / relative).is_file(): - errors.append(f"missing required file: {relative}") - if not SKILL.is_file(): - for error in errors: - print(f"ERROR: {error}") - return 1 +def _normalize_label(label: str) -> str: + label = re.sub(r"[`*_]", "", label).strip().lower().replace("&", " and ") + label = re.sub(r"[/_-]+", " ", label) + return re.sub(r"\s+", " ", label) + + +def _deviation_entries(text: str) -> list[tuple[str, dict[str, str]]]: + headings = list(re.finditer(r"(?m)^#{2,3}\s+(.+?)\s*$", text)) + entries: list[tuple[str, dict[str, str]]] = [] + for index, heading in enumerate(headings): + end = headings[index + 1].start() if index + 1 < len(headings) else len(text) + fields: dict[str, str] = {} + for line in text[heading.end() : end].splitlines(): + match = re.match(r"^\s*[-*]\s+(.+?)\s*:\s*(.*?)\s*$", line) + if not match: + continue + alias = DEVIATION_FIELD_ALIASES.get(_normalize_label(match.group(1))) + if alias: + fields[alias] = match.group(2).strip(" *`") + if fields: + entries.append((heading.group(1).strip(), fields)) + return entries + - text = SKILL.read_text(encoding="utf-8") +def validate_known_deviations_text(text: str) -> list[str]: + entries = _deviation_entries(text) + if not entries: + return ["known_deviations.md has no field-based deviation entries"] + + errors: list[str] = [] + for title, fields in entries: + missing = REQUIRED_DEVIATION_FIELDS - fields.keys() + if "protocol_endpoint" not in fields and not {"protocol", "endpoint"} <= fields.keys(): + missing.add("protocol_endpoint") + empty = {name for name, value in fields.items() if not value} + if missing or empty: + labels = ", ".join(sorted(missing | empty)) + errors.append(f"deviation {title!r} is missing fields: {labels}") + status = fields.get("status", "").lower() + if status and status not in ALLOWED_DEVIATION_STATUSES: + errors.append(f"deviation {title!r} has invalid status: {status}") + return errors + + +def validate_gitignore_text(text: str) -> list[str]: + rules = { + line.strip() for line in text.splitlines() + if line.strip() and not line.lstrip().startswith("#") + } + if LIVE_ARTIFACT_DIR + "/" not in rules: + return [f".gitignore must ignore {LIVE_ARTIFACT_DIR}/"] + return [] + + +def collect_errors(root: Path = ROOT) -> list[str]: + errors = [ + f"missing required file: {relative}" + for relative in REQUIRED_FILES if not (root / relative).is_file() + ] + skill = root / "SKILL.md" + if not skill.is_file(): + return errors + + text = skill.read_text(encoding="utf-8") try: fields, body = parse_frontmatter(text) except ValueError as exc: errors.append(str(exc)) fields, body = {}, "" - name = fields.get("name", "") description = fields.get("description", "") - if not name: errors.append("frontmatter name is required") elif len(name) > 64 or not NAME_RE.fullmatch(name): errors.append("frontmatter name must be lowercase kebab-case and <=64 chars") - elif ROOT.name != name: - # Allow a checkout directory suffix in temporary or CI worktrees. - if not ROOT.name.startswith(name): - errors.append( - f"skill directory {ROOT.name!r} does not match name {name!r}" - ) - + elif root.name != name and not root.name.startswith(name): + errors.append(f"skill directory {root.name!r} does not match name {name!r}") if not description: errors.append("frontmatter description is required") elif len(description) > 1024: errors.append("frontmatter description exceeds 1024 characters") - if len(body.splitlines()) > 500: errors.append("SKILL.md body exceeds the recommended 500 lines") - for target in LINK_RE.findall(body): if "://" in target or target.startswith("#"): continue file_target = target.split("#", 1)[0] - if file_target and not (ROOT / file_target).exists(): + if file_target and not (root / file_target).exists(): errors.append(f"broken SKILL.md link: {target}") - for path in iter_text_files(): - content = path.read_text(encoding="utf-8", errors="replace") - relative = path.relative_to(ROOT) + for path in iter_text_files(root): + content, exceeded = read_bounded_text(path) + relative = path.relative_to(root) + if exceeded: + errors.append( + f"text file exceeds the {MAX_TEXT_SCAN_BYTES}-byte security scan limit: {relative}" + ) + continue if SECRET_RE.search(content): errors.append(f"possible committed API key in {relative}") if DIMENSION_ASSIGNMENT_RE.search(content): + errors.append(f"undocumented LangChain dimension request in {relative}") + for line in find_literal_bearers(content): + errors.append(f"literal Authorization bearer value in {relative}:{line}") + if relative != Path("AGENTS.md"): + for line in find_personal_paths(content): + errors.append(f"machine-specific personal path in {relative}:{line}") + + for path in iter_repository_files(root): + relative = path.relative_to(root) + if is_forbidden_artifact(relative): errors.append( - f"undocumented LangChain dimension request in {relative}" + f"generated or sensitive artifact outside {LIVE_ARTIFACT_DIR}/: {relative}" ) - if WINDOWS_USER_PATH_RE.search(content): - errors.append(f"machine-specific Windows user path in {relative}") + agents = root / "AGENTS.md" + if agents.is_file(): + errors.extend(validate_agents_text(agents.read_text(encoding="utf-8"))) + deviations = root / "references/known_deviations.md" + if deviations.is_file(): + errors.extend( + validate_known_deviations_text(deviations.read_text(encoding="utf-8")) + ) + gitignore = root / ".gitignore" + if gitignore.is_file(): + errors.extend(validate_gitignore_text(gitignore.read_text(encoding="utf-8"))) + else: + errors.append("missing required file: .gitignore") + return errors + +def main() -> int: + errors = collect_errors() if errors: for error in errors: print(f"ERROR: {error}") return 1 - print("Skill validation passed.") return 0 diff --git a/tests/test_repository_contracts.py b/tests/test_repository_contracts.py new file mode 100644 index 0000000..da63608 --- /dev/null +++ b/tests/test_repository_contracts.py @@ -0,0 +1,114 @@ +from __future__ import annotations + +import sys +import tempfile +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "scripts")) + +import validate_skill # noqa: E402 + + +class RepositoryValidatorHelpersTest(unittest.TestCase): + def test_secret_regex_covers_hyphen_and_underscore(self) -> None: + candidate = "sk-" + ("abc_DEF-123_" * 2) + self.assertIsNotNone(validate_skill.SECRET_RE.search(candidate)) + + def test_bearer_check_allows_placeholders_and_test_values(self) -> None: + text = "\n".join( + [ + "Authorization: Bearer ", + '"Authorization": f"Bearer {api_key}"', + "Authorization: Bearer test-secret-value", + "Authorization: Bearer invalid-smoke-test-token", + ] + ) + self.assertEqual(validate_skill.find_literal_bearers(text), []) + + def test_bearer_check_flags_literal_without_echoing_it(self) -> None: + text = "Authorization:" + " Bearer live-value-1234567890" + self.assertEqual(validate_skill.find_literal_bearers(text), [1]) + + def test_personal_path_check_is_cross_platform(self) -> None: + text = "\n".join( + [ + "/" + "Users/alice/project", + "/" + "home/alice/project", + "C:" + "\\Users\\alice\\project", + ] + ) + self.assertEqual(validate_skill.find_personal_paths(text), [1, 2, 3]) + placeholders = "/Users//project\nC:\\Users\\\\project" + self.assertEqual(validate_skill.find_personal_paths(placeholders), []) + + def test_artifact_policy_uses_dedicated_ignored_directory(self) -> None: + forbidden = [ + Path(".env"), + Path(".env.local"), + Path("voice.mp3"), + Path("smoke-results.json"), + Path("smoke-results") / "case.json", + Path("auth.json"), + Path("core-2026-08-23.json"), + Path("compatibility_alias.jsonl"), + ] + self.assertTrue(all(validate_skill.is_forbidden_artifact(p) for p in forbidden)) + allowed = [ + Path(".env.example"), + Path("references/example.json"), + Path(validate_skill.LIVE_ARTIFACT_DIR) / "voice.mp3", + ] + self.assertTrue(all(not validate_skill.is_forbidden_artifact(p) for p in allowed)) + + def test_security_scan_covers_all_nonbinary_file_types(self) -> None: + with tempfile.TemporaryDirectory() as directory: + root = Path(directory) + for name in (".env.example", "config.json", "check.sh", "pyproject.toml", "NOTICE"): + (root / name).write_text("safe text", encoding="utf-8") + (root / "binary.dat").write_bytes(b"text\x00binary") + names = {path.name for path in validate_skill.iter_text_files(root)} + self.assertTrue( + {".env.example", "config.json", "check.sh", "pyproject.toml", "NOTICE"} + <= names + ) + self.assertNotIn("binary.dat", names) + + def test_known_deviation_schema_and_status(self) -> None: + valid = """## Invalid token model list + +- Tested at: 2026-08-23T00:00:00Z +- Environment: Python 3.11, direct HTTP, personal token +- Protocol and endpoint: OpenAI-compatible `GET /models` +- Documented expectation: authentication error +- Observed behavior: empty list +- Reproduction conditions: use an invalid synthetic token +- Impact: empty data can be mistaken for success +- Recommended fallback: reject an empty list as inconclusive +- Status: active +""" + self.assertEqual(validate_skill.validate_known_deviations_text(valid), []) + for status in validate_skill.ALLOWED_DEVIATION_STATUSES: + candidate = valid.replace("Status: active", f"Status: {status}") + self.assertEqual(validate_skill.validate_known_deviations_text(candidate), []) + invalid = valid.replace("Status: active", "Status: permanent") + self.assertTrue(validate_skill.validate_known_deviations_text(invalid)) + + +class CurrentRepositoryContractsTest(unittest.TestCase): + def test_agents_repository_guide_contract(self) -> None: + text = (ROOT / "AGENTS.md").read_text(encoding="utf-8") + self.assertEqual(validate_skill.validate_agents_text(text), []) + + def test_known_deviations_repository_contract(self) -> None: + text = (ROOT / "references/known_deviations.md").read_text(encoding="utf-8") + self.assertEqual(validate_skill.validate_known_deviations_text(text), []) + + def test_live_artifact_directory_is_ignored(self) -> None: + text = (ROOT / ".gitignore").read_text(encoding="utf-8") + self.assertEqual(validate_skill.validate_gitignore_text(text), []) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_smoke_test.py b/tests/test_smoke_test.py index 0af9002..ec1dec9 100644 --- a/tests/test_smoke_test.py +++ b/tests/test_smoke_test.py @@ -1,8 +1,14 @@ from __future__ import annotations +import importlib.util +import io +import json import sys import unittest +from dataclasses import replace from pathlib import Path +from unittest.mock import patch +from urllib.error import URLError ROOT = Path(__file__).resolve().parents[1] sys.path.insert(0, str(ROOT / "scripts")) @@ -10,61 +16,986 @@ import smoke_test # noqa: E402 -class SmokeTestHelpersTest(unittest.TestCase): - def test_redact_text_removes_exact_and_key_shaped_values(self) -> None: +class FakeResponse: + def __init__(self, body: bytes = b"", lines: list[bytes] | None = None, headers=None): + self.status = 200 + self.headers = headers or {"content-type": "application/json"} + self._body = io.BytesIO(body) + self._lines = list(lines or []) + self.readline_calls = 0 + + def read(self, size: int = -1) -> bytes: + return self._body.read(size) + + def readline(self, size: int = -1) -> bytes: + self.readline_calls += 1 + return self._lines.pop(0) if self._lines else b"" + + def __enter__(self): # type: ignore[no-untyped-def] + return self + + def __exit__(self, *args): # type: ignore[no-untyped-def] + return None + + +class RedactionTest(unittest.TestCase): + def test_redacts_exact_key_shaped_bearer_base64_and_url(self) -> None: secret = "test-secret-value" key_shaped = "sk-" + ("a" * 32) + encoded = "A" * 200 text = ( - "Authorization: Bearer test-secret-value " - f"and {key_shaped}" + "Authori" + "zation: " + "Bear" + f"er {secret}; key={key_shaped}; " + f"data:image/png;base64,{encoded}; " + "url=https://download.example.test/once?ticket=secret" ) redacted = smoke_test.redact_text(text, (secret,)) - self.assertNotIn(secret, redacted) - self.assertNotIn(key_shaped, redacted) - self.assertGreaterEqual(redacted.count("[REDACTED_API_KEY]"), 2) + for forbidden in (secret, key_shaped, encoded, "ticket=secret"): + self.assertNotIn(forbidden, redacted) + self.assertIn("[REDACTED_API_KEY]", redacted) + self.assertIn("[REDACTED_BASE64_DATA]", redacted) + self.assertIn("[REDACTED_URL]", redacted) + + def test_reasoning_is_presence_and_length_only(self) -> None: + reasoning = "private hidden reasoning" + sanitized = smoke_test.sanitize_value( + {"reasoning_content": reasoning, "content": "bounded error"} + ) + self.assertNotIn("reasoning_content", sanitized) + self.assertTrue(sanitized["reasoning_content_present"]) + self.assertEqual(sanitized["reasoning_content_length"], len(reasoning)) + self.assertNotIn(reasoning, json.dumps(sanitized)) - def test_embedding_payload_uses_only_documented_fields(self) -> None: - payload = smoke_test.build_embedding_payload(["one", "two"]) + def test_allowlisted_headers_exclude_auth_and_cookies(self) -> None: + headers = smoke_test.extract_important_headers( + { + "Content-Type": "application/json", + "X-Request-ID": "request-1", + "Content-Disposition": 'attachment; filename="private prompt.mp3"', + "Authorization": "Bearer " + "test-secret", + "Set-Cookie": "session=secret", + "Server": "private-proxy", + } + ) self.assertEqual( - payload, + headers, + {"content-type": "application/json", "x-request-id": "request-1"}, + ) + + def test_error_echoes_are_reduced_to_presence_and_length(self) -> None: + private = "private prompt text" + body = json.dumps( + { + "detail": { + "messages": [{"content": private}], + "input": private, + "query": private, + "documents": [private], + "image_url": "data:image/png;base64," + "A" * 200, + "data": private, + "body": private, + "payload": private, + "arguments": private, + } + } + ).encode() + sample = smoke_test.bounded_error_sample(body) + self.assertNotIn(private, sample) + for field in ( + "messages", + "input", + "query", + "documents", + "image_url", + "data", + "body", + "payload", + "arguments", + ): + self.assertIn(field + "_length", sample) + def test_error_sample_is_bounded_and_redacted_for_text_and_json(self) -> None: + secret = "test-secret-value" + text_sample = smoke_test.bounded_error_sample( + (secret + " " + "x" * 5000).encode(), (secret,) + ) + self.assertNotIn(secret, text_sample) + self.assertLessEqual(len(text_sample), smoke_test.MAX_ERROR_TEXT) + + reasoning = "do not persist this reasoning" + json_sample = smoke_test.bounded_error_sample( + json.dumps( + { + "detail": { + "reasoning_content": reasoning, + "Authorization": "Bearer " + "test-hidden", + } + } + ).encode() + ) + self.assertNotIn(reasoning, json_sample) + self.assertNotIn("Bearer " + "test-hidden", json_sample) + self.assertIn("reasoning_content_length", json_sample) + + def test_success_chat_summary_omits_content_and_reasoning_text(self) -> None: + content = "ECNU_OK secret output" + reasoning = "hidden chain" + response = smoke_test.HttpResult( + 200, + {"content-type": "application/json"}, + json.dumps( + { + "choices": [ + { + "message": { + "content": content, + "reasoning_content": reasoning, + } + } + ], + "usage": {"prompt_tokens": 1}, + } + ).encode(), + ) + summary = smoke_test.summarize_response("chat", response) + rendered = json.dumps(summary) + self.assertNotIn(content, rendered) + self.assertNotIn(reasoning, rendered) + self.assertEqual(summary["content_length"], len(content)) + self.assertEqual(summary["reasoning_content_length"], len(reasoning)) + + def test_backend_model_path_is_not_retained(self) -> None: + summary = smoke_test._summarize_embedding( { - "model": "ecnu-embedding-small", - "input": ["one", "two"], - }, + "model": "/" + "root/cache/backend-model", + "data": [{"index": 0, "embedding": [0.0] * 1024}], + } ) + self.assertEqual(summary["model"], "[REDACTED_BACKEND_PATH]") + + +class DefaultsTest(unittest.TestCase): + def test_environment_does_not_override_credit_cap(self) -> None: + with patch.dict(smoke_test.os.environ, {"MAX_TEST_CREDITS": "999"}): + self.assertEqual(smoke_test._default_max_credits(), smoke_test.DEFAULT_MAX_CREDITS) + - def test_embedding_payload_rejects_token_ids(self) -> None: +class EmbeddingContractTest(unittest.TestCase): + def test_embedding_payload_has_only_model_and_string_input(self) -> None: + for input_value in ("one", ["one", "two"]): + payload = smoke_test.build_embedding_payload(input_value) + self.assertEqual(set(payload), {"model", "input"}) + self.assertNotIn("dimensions", payload) + if isinstance(payload["input"], list): + self.assertTrue(all(isinstance(item, str) for item in payload["input"])) + else: + self.assertIsInstance(payload["input"], str) + + def test_embedding_payload_rejects_empty_and_token_ids(self) -> None: + with self.assertRaises(ValueError): + smoke_test.build_embedding_payload("") + with self.assertRaises(TypeError): + smoke_test.build_embedding_payload([]) with self.assertRaises(TypeError): smoke_test.build_embedding_payload([1, 2]) # type: ignore[arg-type] - def test_models_summary_keeps_structure_not_raw_body(self) -> None: - result = smoke_test.HttpResult( - status=200, - headers={"content-type": "application/json"}, - body=( - b'{"object":"list","data":[' - b'{"id":"ecnu-plus"},{"id":"ecnu-max"}]}' + @unittest.skipUnless( + importlib.util.find_spec("httpx") + and importlib.util.find_spec("langchain_openai"), + "optional LangChain dependencies are not installed", + ) + def test_langchain_mock_transport_sends_strings_without_dimensions(self) -> None: + shape, attempts = smoke_test.run_langchain_mock_capture() + self.assertEqual(attempts, 1) + self.assertTrue(shape["input_is_string_array"]) + self.assertFalse(shape["dimensions_present"]) + self.assertEqual(shape["vector_lengths"], [1024, 1024]) + + +class TransportBudgetAndCleanupTest(unittest.TestCase): + @unittest.skipUnless(importlib.util.find_spec("httpx"), "httpx is not installed") + def test_sdk_transport_bounds_response_before_client_parse(self) -> None: + import httpx + + inner = httpx.MockTransport( + lambda request: httpx.Response(200, stream=httpx.ByteStream(b"x" * 11)) + ) + transport = smoke_test.RecordingHttpxTransport(inner) + client = httpx.Client(transport=transport) + try: + with patch.object(smoke_test, "MAX_RESPONSE_BYTES", 10): + response = client.get("https://example.test/test") + finally: + client.close() + self.assertEqual(response.content, b"x" * 10) + self.assertTrue(transport.response_limit_exceeded) + + def test_post_transport_failure_is_not_retried(self) -> None: + with patch.object( + smoke_test, "_open_api_request", side_effect=URLError("offline") + ) as mocked: + result = smoke_test.request( + "POST", + smoke_test.OPENAI_BASE + "/chat/completions", + payload={"model": "ecnu-plus"}, + ) + self.assertEqual(mocked.call_count, 1) + self.assertIsNone(result.status) + self.assertIn("URLError", result.transport_error or "") + + def test_stream_stops_at_done_without_reading_to_eof(self) -> None: + response = FakeResponse( + lines=[ + b'data: {"choices":[{"delta":{"content":"ok"}}]}\n', + b"data: [DONE]\n", + b"data: must-not-be-read\n", + ], + headers={"content-type": "text/event-stream"}, + ) + with patch.object(smoke_test, "_open_api_request", return_value=response): + result = smoke_test.stream_request( + smoke_test.OPENAI_BASE + "/chat/completions", + headers={}, + payload={"stream": True}, + timeout=1.0, + ) + self.assertEqual(response.readline_calls, 2) + self.assertTrue(result.body.endswith(b"[DONE]\n")) + self.assertNotIn(b"must-not-be-read", result.body) + + def test_stream_event_and_wall_caps_are_inconclusive(self) -> None: + response = FakeResponse( + lines=[b"data: {}\n", b"data: {}\n"], + headers={"content-type": "text/event-stream"}, + ) + with patch.object(smoke_test, "MAX_STREAM_EVENTS", 1), patch.object( + smoke_test, "_open_api_request", return_value=response + ): + result = smoke_test.stream_request( + smoke_test.OPENAI_BASE + "/chat/completions", + headers={}, + payload={"stream": True}, + timeout=1.0, + ) + self.assertIn("event limit", result.transport_error or "") + + wall_response = FakeResponse(lines=[b"data: {}\n"]) + with patch.object(smoke_test.time, "monotonic", side_effect=[0.0, 2.0]), patch.object( + smoke_test, "_open_api_request", return_value=wall_response + ): + result = smoke_test.stream_request( + smoke_test.OPENAI_BASE + "/chat/completions", + headers={}, + payload={"stream": True}, + timeout=1.0, + ) + self.assertIn("wall-time", result.transport_error or "") + + def test_non_finite_credit_limits_are_rejected(self) -> None: + for value in (float("nan"), float("inf"), float("-inf")): + with self.assertRaises(ValueError): + smoke_test.CreditBudget(value) + for value in ("nan", "inf", "-inf"): + with patch.object(sys, "stderr", new=io.StringIO()), self.assertRaises(SystemExit): + smoke_test.main(["--max-credits=" + value]) + with patch.object(sys, "stderr", new=io.StringIO()), self.assertRaises(SystemExit): + smoke_test.main(["--timeout=" + value]) + + def test_api_redirect_handler_never_forwards_request(self) -> None: + handler = smoke_test._NoRedirect() + self.assertIsNone( + handler.redirect_request(None, None, 307, "redirect", {}, "https://other.test") + ) + + def test_budget_reserves_before_calls_and_stops_at_limit(self) -> None: + budget = smoke_test.CreditBudget(limit=5.0, planned=10.0) + self.assertTrue(budget.reserve(3.0)) + self.assertFalse(budget.reserve(3.0)) + self.assertEqual(budget.reserved, 3.0) + self.assertTrue(budget.exhausted) + self.assertTrue(budget.reserve(0.0)) + + exact = smoke_test.CreditBudget(limit=5.0) + self.assertTrue(exact.reserve(5.0)) + self.assertFalse(exact.exhausted) + self.assertFalse(exact.reserve(0.1)) + self.assertTrue(exact.exhausted) + + def test_authenticated_429_stops_later_cases(self) -> None: + first = smoke_test._case( + "first", + ("core",), + "test", + smoke_test.OPENAI_BASE + "/test", + "ecnu-plus", + {}, + "test", + "generic", + (200,), + custom_executor=lambda context, spec: smoke_test.Execution( + smoke_test.HttpResult(429, {"content-type": "application/json"}, b"{}"), + "mock", ), ) - summary = smoke_test.summarize_response("models_valid", result) - self.assertEqual(summary["model_ids"], ["ecnu-plus", "ecnu-max"]) - self.assertNotIn("body", summary) + second_called = False - def test_error_summary_redacts_and_bounds_text(self) -> None: - secret = "test-secret-value" - result = smoke_test.HttpResult( - status=500, - headers={"content-type": "text/plain"}, - body=(secret + " " + ("x" * 5000)).encode(), - ) - summary = smoke_test.summarize_response( - "other", - result, - secrets=(secret,), - ) - error_text = summary["error_text"] - self.assertNotIn(secret, error_text) - self.assertLessEqual(len(error_text), smoke_test.MAX_ERROR_TEXT) + def second_executor(context, spec): # type: ignore[no-untyped-def] + nonlocal second_called + second_called = True + return smoke_test.Execution(smoke_test.HttpResult(200, {}, b"{}"), "mock") + + second = smoke_test._case( + "second", + ("core",), + "test", + smoke_test.OPENAI_BASE + "/test", + "ecnu-plus", + {}, + "test", + "generic", + (200,), + custom_executor=second_executor, + ) + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(10.0) + ) + smoke_test.run_one(context, first) + record = smoke_test.run_one(context, second) + self.assertFalse(second_called) + self.assertEqual(record["result"], "skipped") + self.assertIn("HTTP 429", record["notes"][0]) + + def test_429_from_invalid_probe_stops_every_later_case(self) -> None: + invalid = next( + case for case in smoke_test.build_cases() if case.case_id == "models_invalid_token" + ) + invalid = replace( + invalid, + custom_executor=lambda context, spec: smoke_test.Execution( + smoke_test.HttpResult(429, {"content-type": "application/json"}, b"{}"), + "mock", + ), + ) + later = next( + case for case in smoke_test.build_cases() if case.case_id == "models_missing_auth" + ) + called = False + + def execute_later(context, spec): # type: ignore[no-untyped-def] + nonlocal called + called = True + return smoke_test.Execution(smoke_test.HttpResult(200, {}, b"{}"), "mock") + + later = replace(later, custom_executor=execute_later) + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(10.0) + ) + smoke_test.run_one(context, invalid) + record = smoke_test.run_one(context, later) + self.assertFalse(called) + self.assertEqual(record["result"], "skipped") + self.assertIn("global stop", record["notes"][0]) + + def test_case_specific_401_requires_prior_valid_auth(self) -> None: + spec = next( + case + for case in smoke_test.build_cases() + if case.case_id == "error_unsupported_model" + ) + response = smoke_test.Execution( + smoke_test.HttpResult( + 401, + {"content-type": "application/json"}, + b'{"detail":"metadata failure"}', + ), + "mock", + ) + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(10.0) + ) + with patch.object(smoke_test, "raw_executor", return_value=response): + record = smoke_test.run_one(context, spec) + self.assertEqual(record["result"], "mismatch") + self.assertIn("HTTP 401", context.stop_reason or "") + + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(10.0) + ) + context.state["valid_auth_observed"] = True + with patch.object(smoke_test, "raw_executor", return_value=response): + smoke_test.run_one(context, spec) + self.assertIsNone(context.stop_reason) + + def test_unexpected_transport_attempt_stops_and_counts_allowance(self) -> None: + first = smoke_test._case( + "retrying", + ("core",), + "test", + smoke_test.OPENAI_BASE + "/test", + "ecnu-plus", + {}, + "one attempt", + "generic", + (200,), + cost=0.25, + custom_executor=lambda context, spec: smoke_test.Execution( + smoke_test.HttpResult(200, {"content-type": "application/json"}, b"{}"), + "mock", + attempts=2, + ), + ) + second = replace(first, case_id="later") + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(10.0) + ) + first_record = smoke_test.run_one(context, first) + second_record = smoke_test.run_one(context, second) + self.assertEqual(first_record["result"], "mismatch") + self.assertEqual( + first_record["actual_response_shape"]["estimated_consumed_credits"], + 0.5, + ) + self.assertEqual(second_record["result"], "skipped") + self.assertIn("more than once", second_record["notes"][0]) + + def test_unattempted_preflight_skip_releases_reservation(self) -> None: + skipped = smoke_test._case( + "unavailable", + ("core",), + "test", + smoke_test.OPENAI_BASE + "/test", + "ecnu-plus", + {}, + "optional dependency", + "generic", + (200,), + cost=1.0, + custom_executor=lambda context, spec: smoke_test.SkipExecution( + "dependency missing" + ), + ) + executed = replace( + skipped, + case_id="available", + custom_executor=lambda context, spec: smoke_test.Execution( + smoke_test.HttpResult(200, {"content-type": "application/json"}, b"{}"), + "mock", + ), + ) + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(1.0) + ) + skipped_record = smoke_test.run_one(context, skipped) + executed_record = smoke_test.run_one(context, executed) + self.assertEqual(skipped_record["result"], "skipped") + self.assertEqual(executed_record["result"], "pass") + self.assertEqual(context.budget.reserved, 1.0) + + def test_temporary_artifacts_are_removed_on_exception(self) -> None: + directory = None + with self.assertRaises(RuntimeError): + with smoke_test.temporary_artifacts() as temporary: + directory = temporary + (temporary / "media.bin").write_bytes(b"media") + raise RuntimeError("stop") + self.assertIsNotNone(directory) + self.assertFalse(directory.exists()) # type: ignore[union-attr] + + +class PreflightMatcherTest(unittest.TestCase): + @staticmethod + def case(case_id: str): # type: ignore[no-untyped-def] + return next(case for case in smoke_test.build_cases() if case.case_id == case_id) + + def test_models_valid_rejects_empty_ids(self) -> None: + spec = self.case("models_valid") + self.assertFalse( + smoke_test.case_response_matches( + spec, 200, {"valid_json": True, "model_ids": []} + ) + ) + + def test_expected_errors_require_json_and_tts_error_structure(self) -> None: + generic = self.case("error_missing_model") + text_shape = smoke_test.summarize_response( + "generic", + smoke_test.HttpResult(422, {"content-type": "text/plain"}, b"invalid"), + ) + self.assertFalse(smoke_test.case_response_matches(generic, 422, text_shape)) + + tts = self.case("tts_invalid_voice") + empty_json = smoke_test.summarize_response( + "generic", + smoke_test.HttpResult(400, {"content-type": "application/json"}, b"{}"), + ) + detail_json = smoke_test.summarize_response( + "generic", + smoke_test.HttpResult( + 400, + {"content-type": "application/json"}, + b'{"detail":"invalid voice"}', + ), + ) + documented_json = smoke_test.summarize_response( + "generic", + smoke_test.HttpResult( + 400, + {"content-type": "application/json"}, + b'{"error":"invalid voice","request_id":"test","details":{}}', + ), + ) + self.assertFalse(smoke_test.case_response_matches(tts, 400, empty_json)) + self.assertFalse(smoke_test.case_response_matches(tts, 400, detail_json)) + self.assertTrue(smoke_test.case_response_matches(tts, 400, documented_json)) + + def test_max_1m_401_is_local_and_plain_max_fallback_runs(self) -> None: + suffix = replace( + self.case("anthropic_max_1m"), + custom_executor=lambda context, spec: smoke_test.Execution( + smoke_test.HttpResult( + 401, {"content-type": "application/json"}, b'{"detail":"suffix"}' + ), + "mock", + ), + ) + fallback_body = json.dumps( + { + "model": "ecnu-max", + "content": [{"type": "text", "text": "ok"}], + "usage": {"input_tokens": 1, "output_tokens": 1}, + } + ).encode() + fallback = replace( + self.case("anthropic_max_1m_fallback_plain_max"), + custom_executor=lambda context, spec: smoke_test.Execution( + smoke_test.HttpResult( + 200, {"content-type": "application/json"}, fallback_body + ), + "mock", + ), + ) + with smoke_test.temporary_artifacts() as directory: + context = smoke_test.RunContext( + "test-key", 1.0, directory, smoke_test.CreditBudget(10.0) + ) + fallback_record = smoke_test.run_one(context, fallback) + suffix_record = smoke_test.run_one(context, suffix) + self.assertEqual(fallback_record["result"], "pass") + self.assertEqual(suffix_record["result"], "mismatch") + self.assertIsNone(context.stop_reason) + + def test_tts_requires_audio_disposition_and_pcm_headers(self) -> None: + pcm = self.case("tts_xiayu_pcm") + missing = smoke_test.summarize_response( + "tts", + smoke_test.HttpResult(200, {"content-type": "audio/pcm"}, b"pcm"), + ) + complete = smoke_test.summarize_response( + "tts", + smoke_test.HttpResult( + 200, + { + "content-type": "audio/pcm", + "content-disposition": "attachment", + "content-rate": "24000", + "content-channels": "1", + "content-bits": "16", + }, + b"pcm", + ), + ) + self.assertFalse(smoke_test.case_response_matches(pcm, 200, missing)) + self.assertTrue(smoke_test.case_response_matches(pcm, 200, complete)) + + def test_image_requires_verified_mime_dimensions_and_hash(self) -> None: + spec = self.case("image_generation_documented") + good = { + "data_count": 1, + "media_verified": True, + "media_content_type": "image/png", + "pixel_dimensions": [512, 512], + "sha256": "a" * 64, + } + self.assertTrue(smoke_test.case_response_matches(spec, 200, good)) + for key in ("media_verified", "media_content_type", "pixel_dimensions", "sha256"): + broken = dict(good) + broken.pop(key) + self.assertFalse(smoke_test.case_response_matches(spec, 200, broken)) + + def test_private_dns_and_redirect_targets_are_rejected(self) -> None: + private_info = [(2, 1, 6, "", ("127.0.0.1", 443))] + with patch.object(smoke_test.socket, "getaddrinfo", return_value=private_info): + self.assertIsNone( + smoke_test._resolve_public_https("https://private.test/image.png") + ) + self.assertFalse(smoke_test._public_ip("224.0.0.1")) + self.assertFalse(smoke_test._public_ip("64:ff9b::7f00:1")) + + public = "https://public.test/image.png" + redirect = smoke_test.HttpResult( + 302, {"location": "https://private.test/image.png"}, b"" + ) + with patch.object( + smoke_test, + "_resolve_public_https", + side_effect=[(smoke_test.urlsplit(public), "93.184.216.34"), None], + ), patch.object(smoke_test, "_pinned_https_get", return_value=redirect) as get: + result = smoke_test.fetch_public_image( + public, 1.0 + ) + self.assertEqual(get.call_count, 1) + self.assertIn("public HTTPS", result.transport_error or "") + + def test_image_download_bytes_are_bounded(self) -> None: + body, exceeded = smoke_test._read_bounded(io.BytesIO(b"x" * 11), 10) + self.assertEqual(len(body), 10) + self.assertTrue(exceeded) + + def test_image_integrity_requires_complete_valid_png(self) -> None: + valid = smoke_test.make_test_png() + self.assertEqual(smoke_test._image_dimensions(valid), [16, 16]) + self.assertEqual(smoke_test._image_mime(valid), "image/png") + self.assertIsNone(smoke_test._image_dimensions(valid[:24])) + corrupted = bytearray(valid) + corrupted[-5] ^= 1 + self.assertIsNone(smoke_test._image_mime(bytes(corrupted))) + header = smoke_test.struct.pack( + ">IIBBBBB", 512, 512, 8, 6, 0, 0, 0 + ) + forged = ( + b"\x89PNG\r\n\x1a\n" + + smoke_test._png_chunk(b"IHDR", header) + + smoke_test._png_chunk(b"IDAT", smoke_test.zlib.compress(b"x")) + + smoke_test._png_chunk(b"IEND", b"") + ) + self.assertIsNone(smoke_test._image_dimensions(forged)) + + def test_usage_counters_and_official_credit_formula_are_retained(self) -> None: + spec = self.case("chat_basic_ecnu_max") + shape = { + "usage_counters": { + "prompt_tokens": 100, + "completion_tokens": 10, + "prompt_tokens_details": {"cached_tokens": 20}, + } + } + credits, basis = smoke_test.estimate_consumed_credits(spec, 200, shape) + self.assertAlmostEqual(credits, 0.0372) + self.assertIn("ecnu-max", basis) + + sdk_embedding = self.case("openai_sdk_embedding") + fixed, fixed_basis = smoke_test.estimate_consumed_credits( + sdk_embedding, 200, {"usage_counters": {"prompt_tokens": 5}} + ) + self.assertEqual(fixed, 0.05) + self.assertIn("embedding", fixed_basis) + + response = smoke_test.HttpResult( + 200, + {"content-type": "application/json"}, + json.dumps( + { + "choices": [{"message": {"content": "ok"}}], + "usage": shape["usage_counters"], + } + ).encode(), + ) + summary = smoke_test.summarize_response("chat", response) + self.assertEqual(summary["usage_counters"], shape["usage_counters"]) + + def test_compatibility_vision_is_unverified_and_behavior_checked(self) -> None: + for case_id in ( + "responses_max_vision_compatibility", + "anthropic_max_vision_compatibility", + ): + spec = self.case(case_id) + self.assertTrue(spec.documented_expectation.startswith("Not documented")) + base = { + "output_count": 1, + "output_text_present": True, + "usage_keys": ["input_tokens"], + } if case_id.startswith("responses") else { + "content_count": 1, + "text_present": True, + } + self.assertFalse(smoke_test.case_response_matches(spec, 200, base)) + self.assertTrue( + smoke_test.case_response_matches( + spec, 200, {**base, "vision_behavior": "strip-image"} + ) + ) + + def test_thinking_tool_and_anthropic_alias_invariants(self) -> None: + tool = self.case("chat_thinking_tool_first") + shape = { + "tool_call_count": 1, + "tool_names": ["echo"], + "tool_arguments_json_valid": [True], + "tool_ping_argument": [True], + } + self.assertFalse(smoke_test.case_response_matches(tool, 200, shape)) + self.assertTrue( + smoke_test.case_response_matches( + tool, 200, {**shape, "reasoning_content_present": True} + ) + ) + alias = self.case("anthropic_sonnet_mapping") + alias_shape = {"content_count": 1, "text_present": True, "model": "ecnu-max"} + self.assertFalse(smoke_test.case_response_matches(alias, 200, alias_shape)) + self.assertTrue( + smoke_test.case_response_matches( + alias, 200, {**alias_shape, "model": "claude-sonnet-4-20250514"} + ) + ) + self.assertTrue( + smoke_test.case_response_matches( + alias, 200, {**alias_shape, "model": "ecnu-plus"} + ) + ) + opus = self.case("anthropic_opus_mapping") + self.assertFalse( + smoke_test.case_response_matches( + opus, 200, {**alias_shape, "model": "ecnu-plus"} + ) + ) + self.assertTrue( + smoke_test.case_response_matches( + opus, 200, {**alias_shape, "model": "ecnu-max"} + ) + ) + + def test_effort_invariants_and_anthropic_bearer_only(self) -> None: + none = self.case("responses_max_effort_none") + low = self.case("responses_max_effort_low") + base = { + "output_count": 1, + "output_text_present": True, + "usage_keys": ["input_tokens"], + } + self.assertTrue(smoke_test.case_response_matches(none, 200, base)) + self.assertFalse(smoke_test.case_response_matches(low, 200, base)) + self.assertTrue( + smoke_test.case_response_matches( + low, 200, {**base, "reasoning_content_present": True} + ) + ) + headers = smoke_test._headers("anthropic", "test-key") or {} + self.assertIn("Authorization", headers) + self.assertNotIn("x-api-key", headers) + + def test_selected_sdk_cases_are_strict_required(self) -> None: + selected = [ + case + for case in smoke_test.build_cases() + if "SDK" in case.protocol or "LangChain" in case.protocol + ] + self.assertTrue(selected) + self.assertTrue(all(case.required for case in selected)) + + def test_langchain_cases_require_exact_wire_and_vector_shapes(self) -> None: + wire = self.case("langchain_embedding_wire_capture") + valid_wire = { + "input_is_string_array": True, + "dimensions_present": False, + "vector_count": 2, + "vector_lengths": [1024, 1024], + } + self.assertTrue(smoke_test.case_response_matches(wire, 200, valid_wire)) + self.assertFalse( + smoke_test.case_response_matches( + wire, 200, {**valid_wire, "dimensions_present": True} + ) + ) + live = self.case("langchain_embedding_live") + valid_live = { + "count": 1, + "dimensions_present": False, + "vector_lengths": [1024], + } + self.assertTrue(smoke_test.case_response_matches(live, 200, valid_live)) + self.assertFalse( + smoke_test.case_response_matches( + live, 200, {**valid_live, "vector_lengths": []} + ) + ) + + def test_rerank_requires_nonempty_indexed_numeric_results(self) -> None: + spec = self.case("rerank_default") + valid = { + "result_count": 3, + "indexes": [0, 1, 2], + "score_types": ["float", "float", "float"], + } + self.assertTrue(smoke_test.case_response_matches(spec, 200, valid)) + for invalid in ( + {"result_count": 0, "indexes": [], "score_types": []}, + {**valid, "indexes": []}, + {**valid, "score_types": ["str", "str", "str"]}, + ): + self.assertFalse(smoke_test.case_response_matches(spec, 200, invalid)) + + def test_direct_plus_vision_requires_image_understanding(self) -> None: + spec = self.case("vision_direct_ecnu_plus") + base = {"choice_count": 1, "content_present": True} + self.assertFalse(smoke_test.case_response_matches(spec, 200, base)) + self.assertFalse( + smoke_test.case_response_matches( + spec, 200, {**base, "vision_behavior": "ignore-image"} + ) + ) + self.assertTrue( + smoke_test.case_response_matches( + spec, 200, {**base, "vision_behavior": "accept"} + ) + ) + + +class EvidenceAndProfileTest(unittest.TestCase): + def test_case_record_has_exact_required_schema(self) -> None: + spec = smoke_test.build_cases()[0] + record = smoke_test.skipped_record(spec, "offline") + self.assertEqual(tuple(record), smoke_test.CASE_FIELDS) + self.assertEqual(record["result"], "skipped") + self.assertEqual(record["classification"], "application-policy") + + def test_explicit_optional_skip_fails_strict_mode(self) -> None: + spec = replace(smoke_test.build_cases()[-1], required=False) + record = smoke_test.skipped_record(spec, "budget") + specs = {spec.case_id: spec} + self.assertEqual(smoke_test.strict_failure_ids([record], specs, set()), []) + self.assertEqual( + smoke_test.strict_failure_ids([record], specs, {spec.case_id}), + [spec.case_id], + ) + + def test_backend_model_label_is_recorded_without_failing_chat(self) -> None: + spec = next( + case + for case in smoke_test.build_cases() + if case.case_id == "chat_basic_ecnu_plus" + ) + shape = { + "choice_count": 1, + "content_present": True, + "usage_keys": ["prompt_tokens"], + "model": "backend-model-label", + } + self.assertTrue(smoke_test.case_response_matches(spec, 200, shape)) + + def test_negative_embedding_and_rerank_keep_success_shapes(self) -> None: + cases = {case.case_id: case for case in smoke_test.build_cases()} + self.assertEqual(cases["embedding_8193_chars"].response_kind, "embedding") + self.assertEqual(cases["rerank_document_8193"].response_kind, "rerank") + self.assertEqual( + cases["langchain_embedding_wire_capture"].evidence_classification, + "application-policy", + ) + + def test_anthropic_alias_budget_uses_effective_model(self) -> None: + cases = {case.case_id: case for case in smoke_test.build_cases()} + self.assertEqual(cases["anthropic_sonnet_mapping"].estimated_credits, 0.04) + self.assertEqual(cases["anthropic_opus_mapping"].estimated_credits, 0.08) + self.assertEqual(smoke_test._effective_dialog_model("ecnu-reasoner"), "ecnu-max") + self.assertEqual( + smoke_test._effective_dialog_model("ecnu-reasoner-lite"), "ecnu-plus" + ) + for case in cases.values(): + payload = None + if case.payload_factory and case.case_id not in { + "chat_thinking_tool_continue", + "chat_thinking_tool_omit_reasoning", + }: + try: + payload = case.payload_factory(None) # type: ignore[arg-type] + except (AttributeError, smoke_test.CaseUnavailable): + payload = None + self.assertGreaterEqual( + case.estimated_credits + 1e-9, + smoke_test._minimum_output_credit(case.model, payload), + case.case_id, + ) + + def test_thinking_tool_cases_have_room_for_reasoning_and_tool_output(self) -> None: + cases = {case.case_id: case for case in smoke_test.build_cases()} + first = cases["chat_thinking_tool_first"].payload_factory(None) + self.assertEqual(first["max_tokens"], 256) + + def test_matrix_contains_high_value_cases(self) -> None: + cases = smoke_test.build_cases() + ids = {case.case_id for case in cases} + expected = { + "models_valid", + "models_invalid_token", + "chat_stream_ecnu_plus", + "chat_thinking_tool_continue", + "responses_max_effort_low", + "embedding_token_ids", + "embedding_8193_chars", + "rerank_top_n_over_count", + "vision_direct_ecnu_max", + "structured_output_ecnu_plus", + "anthropic_max_1m", + "anthropic_invalid_effort", + "tts_xiayu_pcm", + "tts_invalid_voice", + "image_generation_documented", + } + self.assertTrue(expected <= ids, expected - ids) + self.assertEqual(len(ids), len(cases), "case IDs must be unique") + for case in cases: + self.assertTrue(case.endpoint.startswith("https://chat.ecnu.edu.cn/")) + + def test_default_auth_profile_has_no_image_or_tts(self) -> None: + parser = smoke_test.build_parser() + args = parser.parse_args([]) + profiles = smoke_test.selected_profiles(args, parser) + selected = [case.case_id for case in smoke_test.build_cases() if case.profiles & profiles] + self.assertEqual(profiles, {"auth"}) + self.assertNotIn("image_generation_documented", selected) + self.assertFalse(any(case_id.startswith("tts_") for case_id in selected)) + + def test_billable_plan_prioritizes_pcm_invalid_and_one_image(self) -> None: + selected = [ + case + for case in smoke_test.build_cases() + if "billable" in case.profiles and case.estimated_credits + ] + self.assertEqual(sum(case.estimated_credits for case in selected), 55.0) + self.assertEqual( + [case.case_id for case in selected[:3]], + ["tts_xiayu_pcm", "tts_invalid_voice", "image_generation_documented"], + ) + self.assertTrue(all(case.required for case in selected[:3])) + self.assertTrue(all(not case.required for case in selected[3:])) + self.assertEqual(selected[-1].case_id, "tts_extended_voice_sample") + + def test_all_profile_budget_keeps_required_billable_cases(self) -> None: + budget = smoke_test.CreditBudget(limit=50.0) + allowed: list[str] = [] + skipped: list[str] = [] + for case in smoke_test.build_cases(): + if budget.reserve(case.estimated_credits): + allowed.append(case.case_id) + elif case.estimated_credits: + skipped.append(case.case_id) + self.assertIn("tts_xiayu_pcm", allowed) + self.assertIn("tts_invalid_voice", allowed) + self.assertIn("image_generation_documented", allowed) + self.assertIn("tts_liwa_mp3", skipped) + self.assertIn("tts_extended_voice_sample", skipped) + self.assertLessEqual(budget.reserved, budget.limit) + + def test_legacy_flags_map_without_enabling_billable_profile(self) -> None: + parser = smoke_test.build_parser() + args = parser.parse_args(["--low-cost", "--anthropic"]) + self.assertEqual( + smoke_test.selected_profiles(args, parser), + {"core", "compatibility"}, + ) if __name__ == "__main__": From f1199458ae66f1335d5d9020a8fbe04fc2771f77 Mon Sep 17 00:00:00 2001 From: "J.Jason" <130959319+JJasonSun@users.noreply.github.com> Date: Mon, 24 Aug 2026 00:19:00 +0800 Subject: [PATCH 2/2] Document revalidation after ECNU platform updates --- README.md | 14 ++++++++++++++ references/known_deviations.md | 6 ++++++ 2 files changed, 20 insertions(+) diff --git a/README.md b/README.md index 8ff4f63..4e6c219 100644 --- a/README.md +++ b/README.md @@ -149,6 +149,20 @@ contract matches live behavior. - Do not add local absolute paths or machine-specific deployment instructions. - Run repository validation before opening a pull request. +## Revalidate after ECNU platform updates + +An ECNU release, model rollout, endpoint change, quota change, or announced fix +is a reason to consider a new targeted validation; it is not evidence that an +active deviation has been resolved. Review the updated official contract, +recalculate the credit allowance, and run only the affected `--case` probes +serially with fresh sanitized evidence. Billable TTS or image probes require +new account-owner authorization and must never run automatically. + +Update an observation date or mark a deviation `resolved` only after the same +behavior has been exercised again with the current runner. Preserve the prior +entry when the new run is inconclusive, and record both the changed contract +and the new observed result when the platform update changes expectations. + ## Official documentation API details can change. Verify production-critical behavior against the current diff --git a/references/known_deviations.md b/references/known_deviations.md index 123262d..d08df27 100644 --- a/references/known_deviations.md +++ b/references/known_deviations.md @@ -234,3 +234,9 @@ proves or disproves internal routing. 2. Use only `active`, `resolved`, `inconclusive`, or `not-retested` as status. 3. Remove keys, prompts, generated content, one-time URLs, and reasoning text. 4. Do not mark an item resolved from an unrelated success or an unexecuted case. +5. Treat an ECNU release, model rollout, endpoint change, quota change, or + announced fix as a revalidation trigger, not as resolution evidence. +6. Recheck the current contract and pricing, then rerun only the affected cases + with the current runner before changing dates, expectations, or statuses. +7. Require new account-owner authorization for billable revalidation; never + schedule TTS or image-generation probes automatically.