diff --git a/README.md b/README.md index c1f3854..113bb60 100644 --- a/README.md +++ b/README.md @@ -33,6 +33,12 @@ The official Python SDK & CLI for [ModelScope Hub](https://modelscope.cn) — do ## News +**v0.6.1** (2026-10-30) +- **Feature**: `ms-hub download --total-timeout` (and `HubApi(download_total_timeout=...)`) sets a per-file time budget that overrides the environment variable; `0` means no limit +- **Behaviour**: a transfer that keeps making progress is no longer capped by default — `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` now defaults to `0` (was `3600`), while idle timeouts, the low-speed watchdog and bounded retries still apply; set a positive value to restore a cap +- **Fix**: a stop caused by an expired budget keeps its partial data and is reported as pending, so a plain re-run resumes it instead of requiring `--retry-failed`; trickling response headers stay bounded by the idle timeout when no overall budget is set +- **Quality**: budget and cancellation messages are English and name both the budget and the variable that relaxes it + **v0.4.0** (2026-09-01) - **Feature**: complete OpenAPI coverage for Agent-IDP, MCP, and Studios — Agent Ed25519 identities, OIDC discovery/JWKS and signed JWT issuance (`HubApi`, `ms-hub agent-idp`); Studio lists, variables and configuration options; hosted MCP discovery; protected visibility and runtime metadata; read-only tokens can log in and rejected writes name the required tier. Agent private JWKs are only written to an explicitly requested owner-only file. - **Fix**: Studio compat calls no longer leak connection options or API tokens, or drop cover images; errors distinguish permission, quota and conflicts; anonymous Studio info and log pagination work @@ -232,7 +238,7 @@ print(report.status, report.succeeded, report.failed, report.deferred, report.re report.close() # Release temporary per-file results after consumption. ``` -- Recovery defaults: at most **2 extra retries per file**, shared by range parts and integrity recovery; failed-request inactivity and waits have a **60-second cumulative budget**. Idle/header waits are limited to **15 seconds**, low-speed checks use a **10-second window**, and cooldowns longer than **5 seconds** defer the file instead of tying up a worker. Size/hash mismatches get at most one full re-download. Healthy transfer time still uses `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT`; large files are not capped at 60 seconds. +- Recovery defaults: at most **2 extra retries per file**, shared by range parts and integrity recovery; failed-request inactivity and waits have a **60-second cumulative budget**. Idle/header waits are limited to **15 seconds**, low-speed checks use a **10-second window**, and cooldowns longer than **5 seconds** defer the file instead of tying up a worker. Size/hash mismatches get at most one full re-download. Healthy transfer time is **not** capped by default: a transfer that keeps making progress runs to completion, and `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` only applies when set to a positive value. - Each task uses **one hidden JSON file** outside downloaded contents (`cache_dir/.download_state/.ms_download_.json` by default), with no embedded database dependency. It combines sparse failures, recovery budgets, failed pages, and the latest call's summary under `report`; `report.report_path` equals `report.tracker_path`. Reruns and batches replace the same file, without extra report JSON or JSONL files. Successful files have no persistent per-file records; a healthy run writes only its scope header and final summary, not an update per file. Its scope is endpoint/repository/revision/output, independent of batch filters. Standard reusable lock files remain separate from JSON metadata. - State is advisory, never the sole evidence of success. Every rerun validates actual files against current size/hash metadata, even if hints say `FAILED` or `DEFERRED`; a repaired valid file clears its stale hint without network IO. Otherwise failures remain isolated until metadata changes or `retry_failed=True`. Missing or corrupt hints allow bounded fresh recovery; load/write problems are logged and exposed in `report.state_warnings`, not treated as data corruption or success. A crash during recovery preserves its budget if hints were saved; healthy in-flight files need no markers and resume from actual partial files. Old database-format trackers are not read or migrated: select a new JSON path; existing correct data is reused through validation. - Dataset pages are streamed with bounded queues. Failed pages are recorded and isolated across calls; later pages/roots can continue. Two consecutive failed pages stop that root, and `listing_complete=False` prevents false success. The next run can use `retry_failed=True` or an authoritative `file_list` iterable to recover incomplete enumeration. Model listings that hit an unrecoverable server cap also return a partial report. @@ -243,8 +249,12 @@ report.close() # Release temporary per-file results after consumption. ms-hub download owner/dataset --repo-type dataset --local-dir ./data --continue-on-error # Explicit repair run after the source is fixed ms-hub download owner/dataset --repo-type dataset --local-dir ./data --continue-on-error --retry-failed +# Cap each file at one hour, e.g. for a scheduled job with a fixed window +ms-hub download owner/dataset --repo-type dataset --local-dir ./data --total-timeout 3600 ``` +A file stopped by `--total-timeout` keeps its partial data and is reported as pending rather than failed, so a plain re-run resumes it; `--retry-failed` stays reserved for genuinely suspect files. + CLI exit codes: `0` complete, `3` partial after remaining known files have been processed, `1` global failure, `130` interrupted. Schedulers must treat `3` as partial completion, not an instruction to retry the whole task indefinitely. The SDK's `modelscope_hub.compat.snapshot_download(..., continue_on_error=True, tracker_path=...)` preserves a string return path and logs the report location; use `HubApi.download_repo_report` to read results directly. Older `modelscope.hub.snapshot_download` wrappers may not accept the new options: import the compat function from `modelscope_hub` directly until the wrapper is updated. ### Upload @@ -1060,7 +1070,7 @@ Token is persisted locally after `ms-hub login` and auto-loaded in subsequent se | `MODELSCOPE_DOWNLOAD_PART_SIZE_MB` | `160` | Parallel range chunk size (MB) | | `MODELSCOPE_DOWNLOAD_MAX_RETRIES` | `5` | Single-layer transport retries; `0` disables them; integrity retries are separately bounded | | `MODELSCOPE_DOWNLOAD_TIMEOUT` | `60` | Idle read timeout for streams and range parts, not the whole file (seconds) | -| `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` | `3600` | Per-file time budget, including locks, probes, transfers, retries, and validation (seconds) | +| `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` | `0` | Per-file time budget, including locks, probes, transfers, retries, and validation (seconds); `0` imposes no overall limit | | `MODELSCOPE_DOWNLOAD_MIN_BYTES_PER_SECOND` | `1024` | Minimum rate per connection (B/s); `0` disables low-speed detection | | `MODELSCOPE_DOWNLOAD_LOW_SPEED_WINDOW_SECONDS` | `60` | Minimum-rate observation window (seconds) | | `MODELSCOPE_DOWNLOAD_RETRY_MAX_DELAY_SECONDS` | `60` | Maximum download Retry-After or backoff delay (seconds) | diff --git a/src/modelscope_hub/_download.py b/src/modelscope_hub/_download.py index 900e9ea..8384692 100644 --- a/src/modelscope_hub/_download.py +++ b/src/modelscope_hub/_download.py @@ -56,6 +56,7 @@ DOWNLOAD_PART_SIZE, DOWNLOAD_RETRY_MAX_DELAY_SECONDS, DOWNLOAD_RETRY_TIMES, + DOWNLOAD_TIMEOUT, DOWNLOAD_TOTAL_TIMEOUT, ENV_FILE_LOCK, ENV_INTER_CLOUD_REGIONS, @@ -356,7 +357,9 @@ def _wait_download_retry(retry: Retry, exc: Exception, path: str, control: _Down except MaxRetryError: if isinstance(exc, HubError): raise exc - raise NetworkError(f"下载 {path} 在 {DOWNLOAD_RETRY_TIMES} 次重试后失败：{type(exc).__name__}") from exc + raise NetworkError( + f"Download of {path} failed after {DOWNLOAD_RETRY_TIMES} retries: {type(exc).__name__}" + ) from exc requested = getattr(exc, "retry_after", None) if control.recovery is not None: delay = max(0, requested) if requested is not None else min(retry.get_backoff_time(), 2) @@ -436,7 +439,7 @@ def _download_part_with_retry(params: tuple) -> None: try: control.check() if part_stop.is_set(): - raise _DownloadCancelledError("分片组已取消") + raise _DownloadCancelledError("Range part group cancelled") partial_length = os.path.getsize(part_file_name) if os.path.exists(part_file_name) else 0 if partial_length > expected: # Left over from a different split of the file; it cannot be resumed. @@ -467,7 +470,7 @@ def _download_part_with_retry(params: tuple) -> None: raise with control.response(r): if part_stop.is_set(): - raise _DownloadCancelledError("分片组已取消") + raise _DownloadCancelledError("Range part group cancelled") content_range = _parse_content_range(r.headers.get("Content-Range")) if content_range is not None and content_range[2] is not None and content_range[2] != file_size: raise FileSizeMismatchError( @@ -634,12 +637,21 @@ class DownloadManager: Dependencies are injected via constructor to keep this class testable. """ - def __init__(self, legacy_client: LegacyClient, config: HubConfig, *, user_agent: dict | str | None = None) -> None: + def __init__( + self, + legacy_client: LegacyClient, + config: HubConfig, + *, + user_agent: dict | str | None = None, + total_timeout: float | None = None, + ) -> None: from .utils import build_user_agent self._client = legacy_client self._config = config self._user_agent = build_user_agent(config=config, extra=user_agent) + # None defers to MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT; 0 or less means no limit. + self._total_timeout = total_timeout self._cached_region: str | None = None self._region_lock = threading.Lock() # Cache only confirmed OSS-internal resolutions. A repository can contain @@ -875,13 +887,16 @@ def download_file( ) -> Path: """Download one file with cache-first lookup and a shared budget for locks, probes, and retries.""" with _DownloadControl( - total_timeout=DOWNLOAD_TOTAL_TIMEOUT, + total_timeout=DOWNLOAD_TOTAL_TIMEOUT if self._total_timeout is None else self._total_timeout, cancel_event=_cancel_event, min_rate=DOWNLOAD_MIN_BYTES_PER_SECOND, speed_window=_recovery.policy.low_speed_window if _recovery is not None else DOWNLOAD_LOW_SPEED_WINDOW_SECONDS, + header_timeout=DOWNLOAD_TIMEOUT, recovery=_recovery, + budget_label="Per-file download", + budget_env="MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT", ) as control: with self._client.read_context(control): return self._download_file( @@ -1201,6 +1216,8 @@ def download_repo( with _DownloadControl( total_timeout=REPO_TREE_LIST_TIMEOUT_SECONDS, cancel_event=cancel_event, + budget_label="Repository listing", + budget_env="MODELSCOPE_REPO_TREE_LIST_TIMEOUT_SECONDS", ) as listing_control: with self._client.read_context(listing_control): for listing_root in _listing_roots(allow_patterns): @@ -1341,9 +1358,12 @@ def _download_archive( try: with ( _DownloadControl( - total_timeout=DOWNLOAD_TOTAL_TIMEOUT, + total_timeout=DOWNLOAD_TOTAL_TIMEOUT if self._total_timeout is None else self._total_timeout, min_rate=DOWNLOAD_MIN_BYTES_PER_SECOND, speed_window=DOWNLOAD_LOW_SPEED_WINDOW_SECONDS, + header_timeout=DOWNLOAD_TIMEOUT, + budget_label="Archive download", + budget_env="MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT", ) as control, self._client.read_context(control), ): @@ -1361,7 +1381,9 @@ def _download_archive( tmp.write(chunk) announced = _checkable_content_length(resp) if announced is not None and tmp_path.stat().st_size != announced: - raise _IncompleteBodyError("归档内容长度与响应头不一致") + raise _IncompleteBodyError( + "Archive body length does not match the announced Content-Length" + ) break except (_DownloadCancelledError, _DownloadDeadlineError): raise @@ -1936,7 +1958,9 @@ def request(headers: dict[str, str]) -> requests.Response: if resp.status_code == 206 and (content_range is None or content_range[0] != offset): tmp_path.unlink(missing_ok=True) - raise _IncompleteBodyError("服务器返回了不匹配请求起点的分段响应") + raise _IncompleteBodyError( + "Server returned a partial response that does not match the requested start offset" + ) announced: int | None = None if content_range is not None and content_range[2] is not None: announced = content_range[2] diff --git a/src/modelscope_hub/_download_control.py b/src/modelscope_hub/_download_control.py index 8557ce2..5866994 100644 --- a/src/modelscope_hub/_download_control.py +++ b/src/modelscope_hub/_download_control.py @@ -3,6 +3,7 @@ from __future__ import annotations import contextlib +import math import socket import threading import time @@ -66,9 +67,18 @@ class _DownloadCancelledError(Exception): class _DownloadDeadlineError(RequestTimeoutError): - """The total download time budget has been exhausted.""" + """A configured time budget has been exhausted. + + Stays non-retryable so the current run never loops on an expired budget. + Callers that track per-file state treat it as a resumable stop rather than + a rejected file, because the partial file on disk is still valid. + """ retryable = False + suggestion = ( + "A time budget expired rather than the transfer failing. Partial files are kept, so re-running " + "resumes them; raise MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT or set it to 0 to remove the per-file limit." + ) class _DownloadTooSlowError(NetworkError): @@ -83,6 +93,7 @@ class _WatchedResponse: error: Exception | None = None monitor_speed: bool = True header_deadline: float | None = None + header_limit: float = 0 def _response_socket(response: requests.Response) -> Any: @@ -107,13 +118,23 @@ def __init__( cancel_event: threading.Event | None = None, min_rate: int = 0, speed_window: float = 60, + header_timeout: float = 0, recovery: _RecoveryBudget | None = None, + budget_label: str = "Download", + budget_env: str = "", ) -> None: self.cancel_event = cancel_event if cancel_event is not None else threading.Event() - self.deadline = time.monotonic() + total_timeout + self.total_timeout = total_timeout + # A non-positive budget means "no overall limit": idle timeouts, the + # low-speed watchdog, bounded retries and cancellation still apply, so + # an infinite deadline removes only the wall-clock cap on healthy work. + self.deadline = math.inf if total_timeout <= 0 else time.monotonic() + total_timeout self.min_rate = min_rate self.speed_window = speed_window + self.header_timeout = header_timeout self.recovery = recovery + self.budget_label = budget_label + self.budget_env = budget_env self._responses: dict[requests.Response, _WatchedResponse] = {} self._transports: dict[Any, _WatchedResponse] = {} self._lock = threading.Lock() @@ -133,11 +154,15 @@ def __exit__(self, *args: Any) -> None: for response in responses: response.close() + def _deadline_error(self) -> _DownloadDeadlineError: + hint = f"; raise {self.budget_env} or set it to 0 to disable the limit" if self.budget_env else "" + return _DownloadDeadlineError(f"{self.budget_label} time budget of {self.total_timeout:g}s is exhausted{hint}") + def check(self) -> None: if self.cancel_event.is_set(): - raise _DownloadCancelledError("下载已取消") + raise _DownloadCancelledError("Download cancelled") if time.monotonic() >= self.deadline: - raise _DownloadDeadlineError("下载总时限已耗尽；已保存可继续下载的临时文件") + raise self._deadline_error() def timeouts(self, connect: float, read: float) -> tuple[float, float]: self.check() @@ -168,12 +193,22 @@ def track(self, response: requests.Response) -> None: @contextlib.contextmanager def transport(self, sock: Any) -> Iterator[None]: - """Interrupt sockets while reading headers so trickling headers cannot bypass deadlines.""" + """Interrupt sockets while reading headers so trickling headers cannot bypass deadlines. + + ``requests`` cannot apply its read timeout before the status line + arrives, so a server that emits one header byte at a time never trips + an idle timeout. The header budget is therefore the only bound on that + fault when no overall deadline is configured. + """ self.check() state = _WatchedResponse(sock, time.monotonic(), monitor_speed=False) if self.recovery is not None: started = getattr(self.recovery.activity, "last", time.monotonic()) - state.header_deadline = min(self.deadline, started + self.recovery.policy.idle_timeout) + state.header_limit = self.recovery.policy.idle_timeout + state.header_deadline = min(self.deadline, started + state.header_limit) + elif self.header_timeout > 0: + state.header_limit = self.header_timeout + state.header_deadline = min(self.deadline, state.started + state.header_limit) with self._lock: self._transports[sock] = state self._start_monitor() @@ -220,16 +255,19 @@ def _watch(self) -> None: if state.error is not None: continue if self.cancel_event.is_set(): - state.error = _DownloadCancelledError("下载已取消") + state.error = _DownloadCancelledError("Download cancelled") elif now >= self.deadline: - state.error = _DownloadDeadlineError("下载总时限已耗尽") + state.error = self._deadline_error() elif state.header_deadline is not None and now >= state.header_deadline: - state.error = RequestTimeoutError("Response headers exceeded the short recovery time limit") + state.error = RequestTimeoutError( + f"No response headers within the {state.header_limit:g}s idle time limit" + ) elif state.monitor_speed and self.min_rate and now - state.started >= self.speed_window: rate = state.received / (now - state.started) if rate < self.min_rate: state.error = _DownloadTooSlowError( - f"下载速率 {rate:.0f} B/s 低于 {self.min_rate} B/s，观察窗口 {self.speed_window:g}s" + f"Download rate {rate:.0f} B/s stayed below {self.min_rate} B/s " + f"over a {self.speed_window:g}s window" ) else: state.started, state.received = now, 0 @@ -241,7 +279,7 @@ def abort_responses(self) -> None: """Abort active responses without cancelling the parent, allowing single-stream fallback.""" with self._lock: for state in [*self._responses.values(), *self._transports.values()]: - state.error = _DownloadCancelledError("分片组已取消") + state.error = _DownloadCancelledError("Range part group cancelled") if state.sock is not None: with contextlib.suppress(OSError): state.sock.shutdown(socket.SHUT_RDWR) @@ -274,7 +312,7 @@ def chunks(self, response: requests.Response, chunk_size: int) -> Iterator[bytes yield chunk except (Urllib3HTTPError, OSError) as exc: self._check_response(response) - raise requests.ConnectionError(f"下载流读取失败：{type(exc).__name__}") from exc + raise requests.ConnectionError(f"Download stream read failed: {type(exc).__name__}") from exc finally: self.release(response) diff --git a/src/modelscope_hub/_legacy_api.py b/src/modelscope_hub/_legacy_api.py index 369387e..4cc38c9 100644 --- a/src/modelscope_hub/_legacy_api.py +++ b/src/modelscope_hub/_legacy_api.py @@ -252,7 +252,11 @@ def _request( control = getattr(self._read_state, "control", None) if single_attempt and control is None: # Bound both headers and body reads for single-page calls without an external controller. - with _DownloadControl(total_timeout=REPO_TREE_PAGE_TIMEOUT_SECONDS) as page_control: + with _DownloadControl( + total_timeout=REPO_TREE_PAGE_TIMEOUT_SECONDS, + budget_label="Repository listing page", + budget_env="MODELSCOPE_REPO_TREE_PAGE_TIMEOUT_SECONDS", + ) as page_control: with self.read_context(page_control): return self._request( method, @@ -817,6 +821,8 @@ def _fetch_tree_page(self, path: str, params: dict[str, Any]) -> requests.Respon with _DownloadControl( total_timeout=budget, cancel_event=parent.cancel_event if parent is not None else None, + budget_label="Repository listing page", + budget_env="MODELSCOPE_REPO_TREE_PAGE_TIMEOUT_SECONDS", ) as page_control: with self.read_context(page_control): return self._request( @@ -829,7 +835,9 @@ def _fetch_tree_page(self, path: str, params: dict[str, Any]) -> requests.Respon except _DownloadDeadlineError as exc: if parent is not None: parent.check() - raise RequestTimeoutError(f"列表单页请求超过 {REPO_TREE_PAGE_TIMEOUT_SECONDS}s") from exc + raise RequestTimeoutError( + f"Repository listing page request exceeded {REPO_TREE_PAGE_TIMEOUT_SECONDS}s" + ) from exc def _request_repo_tree( self, diff --git a/src/modelscope_hub/_resilient_download.py b/src/modelscope_hub/_resilient_download.py index dbc9f36..11d993d 100644 --- a/src/modelscope_hub/_resilient_download.py +++ b/src/modelscope_hub/_resilient_download.py @@ -16,7 +16,13 @@ from filelock import FileLock, Timeout from ._download import _listing_roots, _matches_patterns -from ._download_control import _DownloadCancelledError, _DownloadControl, _DownloadDeferredError, _RecoveryBudget +from ._download_control import ( + _DownloadCancelledError, + _DownloadControl, + _DownloadDeadlineError, + _DownloadDeferredError, + _RecoveryBudget, +) from ._download_state import _DownloadState, safe_error from .constants import DOWNLOAD_TOTAL_TIMEOUT, REPO_TREE_LIST_TIMEOUT_SECONDS from .errors import AuthenticationError, FileIntegrityError, InvalidParameter, NotSupportedError @@ -180,7 +186,12 @@ def produce() -> None: break put(entry) else: - with _DownloadControl(total_timeout=REPO_TREE_LIST_TIMEOUT_SECONDS, cancel_event=stop) as control: + with _DownloadControl( + total_timeout=REPO_TREE_LIST_TIMEOUT_SECONDS, + cancel_event=stop, + budget_label="Repository listing", + budget_env="MODELSCOPE_REPO_TREE_LIST_TIMEOUT_SECONDS", + ) as control: with manager._client.read_context(control): for root in _listing_roots(include): if repo_type == "dataset": @@ -216,7 +227,12 @@ def transfer(result: DownloadFileResult) -> None: _validate_entry({"Path": result.path, "Size": result.size, "Sha256": result.sha256}, output) if result.status in ("FAILED", "DEFERRED") and not retry_failed: target = output / result.path - with _DownloadControl(total_timeout=DOWNLOAD_TOTAL_TIMEOUT, cancel_event=stop) as control: + with _DownloadControl( + total_timeout=DOWNLOAD_TOTAL_TIMEOUT, + cancel_event=stop, + budget_label="Cached file validation", + budget_env="MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT", + ) as control: valid = target.is_file() and manager._cache_hit( target, result.sha256, @@ -261,6 +277,13 @@ def transfer(result: DownloadFileResult) -> None: except _DownloadCancelledError: state.finish(result.path, result.status if result.status in ("FAILED", "DEFERRED") else "PENDING") raise + except _DownloadDeadlineError as exc: + # An expired time budget is an externally imposed stop, not a verdict on + # the file: the partial bytes stay valid, so record it as unfinished work + # rather than isolating it. A plain re-run then resumes it without + # needing retry_failed, which only exists for genuinely suspect files. + state.finish(result.path, "PENDING", exc) + logger.warning("Time budget stopped %s; re-run to resume: %s", result.path, safe_error(exc)) except Exception as exc: origin = exc.__cause__ if isinstance(exc, _DownloadDeferredError) and exc.__cause__ else exc terminal = isinstance(origin, (FileIntegrityError, InvalidParameter)) or not getattr( diff --git a/src/modelscope_hub/api.py b/src/modelscope_hub/api.py index 15e38d4..1763e13 100644 --- a/src/modelscope_hub/api.py +++ b/src/modelscope_hub/api.py @@ -208,6 +208,12 @@ class HubApi: user_agent : dict, str or None, optional Application or channel information appended to the SDK UA and propagated to OpenAPI, legacy, probe and download requests. Must not contain credentials. + download_total_timeout : float or None, optional + Per-file download time budget in seconds, covering lock waits, probes, + transfers, retries and validation. ``None`` uses + ``MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT``; ``0`` or less imposes no overall + limit, leaving idle timeouts, the low-speed watchdog and bounded + retries as the only stop conditions. Examples -------- @@ -231,8 +237,10 @@ def __init__( endpoint: str | None = None, token: str | None = None, user_agent: dict | str | None = None, + download_total_timeout: float | None = None, ) -> None: self._user_agent_extra = dict(user_agent) if isinstance(user_agent, dict) else user_agent + self._download_total_timeout = download_total_timeout base = config or get_default_config() if config is None and (endpoint is not None or token is not None): from dataclasses import replace @@ -285,7 +293,12 @@ def legacy(self) -> LegacyClient: def downloader(self) -> DownloadManager: """Lazily-constructed :class:`DownloadManager`.""" if self._downloader is None: - self._downloader = DownloadManager(self.legacy, self._config, user_agent=self._user_agent_extra) + self._downloader = DownloadManager( + self.legacy, + self._config, + user_agent=self._user_agent_extra, + total_timeout=self._download_total_timeout, + ) return self._downloader @property @@ -2040,7 +2053,11 @@ def exists(endpoint: str) -> bool: user_agent=self._user_agent_extra, ) try: - with _DownloadControl(total_timeout=REPO_TREE_PAGE_TIMEOUT_SECONDS) as control: + with _DownloadControl( + total_timeout=REPO_TREE_PAGE_TIMEOUT_SECONDS, + budget_label="Endpoint probe", + budget_env="MODELSCOPE_REPO_TREE_PAGE_TIMEOUT_SECONDS", + ) as control: with probe.openapi.read_context(control): return probe.repo_exists(repo_id, repo_type) finally: diff --git a/src/modelscope_hub/cli/base.py b/src/modelscope_hub/cli/base.py index cf66030..9abd8ea 100644 --- a/src/modelscope_hub/cli/base.py +++ b/src/modelscope_hub/cli/base.py @@ -76,6 +76,7 @@ def make_api(args: Namespace) -> HubApi: token=getattr(args, "token", None), endpoint=getattr(args, "endpoint", None), user_agent={"invoked_by": "cli"}, + download_total_timeout=getattr(args, "total_timeout", None), ) diff --git a/src/modelscope_hub/cli/download.py b/src/modelscope_hub/cli/download.py index 74dc750..1bcebb3 100644 --- a/src/modelscope_hub/cli/download.py +++ b/src/modelscope_hub/cli/download.py @@ -95,6 +95,17 @@ def register(subparsers: SubParsers) -> None: help="Single hidden JSON for recovery hints and summary, outside downloaded contents.", ) p.add_argument("--retry-failed", action="store_true", help="Explicitly reset isolated files' recovery budgets.") + p.add_argument( + "--total-timeout", + dest="total_timeout", + type=float, + default=None, + help=( + "Per-file time budget in seconds, covering lock waits, probes, transfers, retries and " + "validation. 0 means no limit (the default); idle timeouts and low-speed detection still " + "apply. Overrides MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT." + ), + ) # Legacy compat (hidden from --help) add_legacy_download_args(p) @@ -178,10 +189,16 @@ def _make_api_with_endpoint(self) -> HubApi: """ endpoint = getattr(self.args, "endpoint", None) token = getattr(self.args, "token", None) + total_timeout = getattr(self.args, "total_timeout", None) if endpoint: endpoint = _resolve_cli_endpoint(endpoint) - return HubApi(token=token, endpoint=endpoint, user_agent={"invoked_by": "cli"}) + return HubApi( + token=token, + endpoint=endpoint, + user_agent={"invoked_by": "cli"}, + download_total_timeout=total_timeout, + ) api = make_api(self.args) try: @@ -189,7 +206,12 @@ def _make_api_with_endpoint(self) -> HubApi: self.args.repo_id, repo_type=self.args.repo_type, ) - return HubApi(token=token, endpoint=resolved, user_agent={"invoked_by": "cli"}) + return HubApi( + token=token, + endpoint=resolved, + user_agent={"invoked_by": "cli"}, + download_total_timeout=total_timeout, + ) except Exception: return api diff --git a/src/modelscope_hub/constants.py b/src/modelscope_hub/constants.py index b494d67..7599c68 100644 --- a/src/modelscope_hub/constants.py +++ b/src/modelscope_hub/constants.py @@ -921,9 +921,10 @@ def _env_register( DOWNLOAD_TOTAL_TIMEOUT: int = _env_int( "MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT", - 3600, - "Total time budget per file including retries, probes and lock wait (seconds)", + 0, + "Per-file total time budget incl. retries, probes and lock wait (seconds; 0 = no limit)", "Download", + allow_zero=True, ) DOWNLOAD_MIN_BYTES_PER_SECOND: int = _env_int( "MODELSCOPE_DOWNLOAD_MIN_BYTES_PER_SECOND", diff --git a/src/modelscope_hub/types.py b/src/modelscope_hub/types.py index bbdb3ca..6d9d4c9 100644 --- a/src/modelscope_hub/types.py +++ b/src/modelscope_hub/types.py @@ -58,9 +58,11 @@ def from_dict(cls: type[_TDataclass], data: Mapping[str, Any] | None) -> _TDatac class DownloadRetryPolicy: """Short recovery limits for report-mode downloads, shared by all file parts. - Healthy transfers keep the regular total file timeout. Only failed-request - inactivity and retry waits consume recovery_timeout. Longer Retry-After - values defer the file instead of occupying a worker or retrying early. + Healthy transfers are not bounded by these limits, nor by an overall time + cap unless MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT is set to a positive value. + Only failed-request inactivity and retry waits consume recovery_timeout. + Longer Retry-After values defer the file instead of occupying a worker or + retrying early. """ max_retries: int = 2 diff --git a/src/modelscope_hub/version.py b/src/modelscope_hub/version.py index b22f9da..02d16d8 100644 --- a/src/modelscope_hub/version.py +++ b/src/modelscope_hub/version.py @@ -1,3 +1,3 @@ """Version information for modelscope_hub.""" -__version__ = "0.6.0+main" +__version__ = "0.6.1+main" diff --git a/tests/cli/test_base.py b/tests/cli/test_base.py index 0ef7da3..a668cc0 100644 --- a/tests/cli/test_base.py +++ b/tests/cli/test_base.py @@ -97,14 +97,18 @@ def test_make_api_with_token(self): args = Namespace(token="my_token", endpoint=None) with patch("modelscope_hub.cli.base.HubApi") as mock_hub: make_api(args) - mock_hub.assert_called_once_with(token="my_token", endpoint=None, user_agent={"invoked_by": "cli"}) + mock_hub.assert_called_once_with( + token="my_token", endpoint=None, user_agent={"invoked_by": "cli"}, download_total_timeout=None + ) def test_make_api_without_token(self): """make_api passes None when no token.""" args = Namespace(token=None, endpoint=None) with patch("modelscope_hub.cli.base.HubApi") as mock_hub: make_api(args) - mock_hub.assert_called_once_with(token=None, endpoint=None, user_agent={"invoked_by": "cli"}) + mock_hub.assert_called_once_with( + token=None, endpoint=None, user_agent={"invoked_by": "cli"}, download_total_timeout=None + ) def test_make_api_with_endpoint(self): """make_api passes endpoint to HubApi.""" @@ -112,7 +116,19 @@ def test_make_api_with_endpoint(self): with patch("modelscope_hub.cli.base.HubApi") as mock_hub: make_api(args) mock_hub.assert_called_once_with( - token=None, endpoint="https://custom.endpoint.com", user_agent={"invoked_by": "cli"} + token=None, + endpoint="https://custom.endpoint.com", + user_agent={"invoked_by": "cli"}, + download_total_timeout=None, + ) + + def test_make_api_forwards_a_download_time_budget(self): + """A subcommand's --total-timeout reaches HubApi without extra plumbing.""" + args = Namespace(token=None, endpoint=None, total_timeout=1800.0) + with patch("modelscope_hub.cli.base.HubApi") as mock_hub: + make_api(args) + mock_hub.assert_called_once_with( + token=None, endpoint=None, user_agent={"invoked_by": "cli"}, download_total_timeout=1800.0 ) diff --git a/tests/test_download_integrity.py b/tests/test_download_integrity.py index 9c7a6d7..61c394c 100644 --- a/tests/test_download_integrity.py +++ b/tests/test_download_integrity.py @@ -562,6 +562,65 @@ def test_deadline_stops_trickling_headers_and_body(self, hub, api, dirs, monkeyp assert len(hub.seen("slow.bin", "cdn")) == 1 assert not (dirs[0] / "slow.bin").exists() + def test_trickling_headers_are_bounded_without_an_overall_deadline(self, hub, api, dirs, monkeypatch): + """requests cannot time out before the status line, so the header budget is the only bound. + + The fixture trickles response headers for about 1.9s and then stops + without a body. Failing before the headers even finish proves the + header budget interrupted it; urllib3 re-wraps the reason as its own + read timeout, so only timing and type are asserted. + """ + monkeypatch.setattr(_download, "DOWNLOAD_TOTAL_TIMEOUT", 0) + monkeypatch.setattr(_download, "DOWNLOAD_TIMEOUT", 0.3) + monkeypatch.setattr(_download, "DOWNLOAD_RETRY_TIMES", 0) + hub.add("drip.bin", b"x" * 100, header_trickle=True) + started = time.monotonic() + with pytest.raises(NetworkError): + api.download_file(REPO, "dataset", "drip.bin", local_dir=dirs[0], cache_dir=dirs[1], file_size=100) + assert time.monotonic() - started < 1.5 + + def test_a_steady_transfer_is_not_capped_by_default(self, hub, api, dirs): + """A healthy transfer slower than any fixed wall clock must still complete.""" + assert _download.DOWNLOAD_TOTAL_TIMEOUT == 0 + content = hub.add("steady.bin", os.urandom(4000), trickle_interval=0.0002) + path = api.download_file( + REPO, "dataset", "steady.bin", local_dir=dirs[0], cache_dir=dirs[1], file_size=len(content) + ) + assert path.read_bytes() == content + + def test_an_explicit_budget_names_the_variable_that_relaxes_it(self, hub, api, dirs, monkeypatch): + from modelscope_hub._download_control import _DownloadDeadlineError + + monkeypatch.setattr(_download, "DOWNLOAD_TOTAL_TIMEOUT", 0.3) + hub.add("capped.bin", b"x" * 100, trickle_interval=0.05) + with pytest.raises(_DownloadDeadlineError) as excinfo: + api.download_file(REPO, "dataset", "capped.bin", local_dir=dirs[0], cache_dir=dirs[1], file_size=100) + message = str(excinfo.value) + assert "Per-file download time budget of 0.3s" in message + assert "MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT" in message + + def test_cli_total_timeout_overrides_the_environment_default(self, hub, dirs, monkeypatch): + monkeypatch.setenv("MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT", "0") + hub.add("capped.bin", b"x" * 100, trickle_interval=0.05) + code = run_cmd( + [ + "--endpoint", + hub.url, + "download", + REPO, + "--repo-type", + "dataset", + "--local-dir", + str(dirs[0]), + "--cache-dir", + str(dirs[1]), + "--total-timeout", + "0.3", + ] + ) + assert code != 0 + assert not (dirs[0] / "capped.bin").exists() + def test_minimum_rate_interrupts_a_connection_that_never_goes_idle(self, hub, api, dirs, monkeypatch): from modelscope_hub._download_control import _DownloadTooSlowError @@ -734,7 +793,7 @@ def test_defaults_exports_registry_and_readme_are_aligned(self): import modelscope_hub.constants as constants expected = { - "DOWNLOAD_TOTAL_TIMEOUT": 3600, + "DOWNLOAD_TOTAL_TIMEOUT": 0, "DOWNLOAD_MIN_BYTES_PER_SECOND": 1024, "DOWNLOAD_LOW_SPEED_WINDOW_SECONDS": 60, "DOWNLOAD_RETRY_MAX_DELAY_SECONDS": 60, diff --git a/tests/test_download_report.py b/tests/test_download_report.py index efd31ae..c9ac430 100644 --- a/tests/test_download_report.py +++ b/tests/test_download_report.py @@ -57,6 +57,27 @@ def test_new_api_instances_and_batches_share_failure_isolation(hub, api, dirs): assert [file.path for file in first.iter_files()] == ["bad.bin"] +def test_time_budget_stop_is_resumable_without_retry_failed(hub, api, dirs, monkeypatch): + """An expired budget is an external stop, so a plain re-run must continue the partial file.""" + from modelscope_hub import _download + + content = hub.add("big.bin", b"x" * 400, trickle_interval=0.01) + hub.add("small.bin", b"ok") + monkeypatch.setattr(_download, "DOWNLOAD_TOTAL_TIMEOUT", 0.4) + first = run(api, dirs, max_workers=1) + assert first.status == "PARTIAL" + assert first.pending == 1 and first.failed == 0 and first.deferred == 0 + assert (dirs[0] / "big.bin.incomplete").exists() + + # No retry_failed: an externally imposed stop must not isolate the file. + monkeypatch.setattr(_download, "DOWNLOAD_TOTAL_TIMEOUT", 0) + hub.files["big.bin"].trickle_interval = 0 + second = run(api, dirs, max_workers=1) + assert second.complete + assert second.skipped_failures == 0 + assert (dirs[0] / "big.bin").read_bytes() == content + + def test_failed_file_is_only_reopened_explicitly_or_on_metadata_change(hub, api, dirs): hub.add("bad.bin", b"data", served=b"") first = run(api, dirs)