Download Time Budgets Become Opt-in

README.md

@@ -33,6 +33,12 @@ The official Python SDK & CLI for [ModelScope Hub](https://modelscope.cn) — do
 
 ## News
 
+**v0.6.1** (2026-10-30)
+- **Feature**: `ms-hub download --total-timeout` (and `HubApi(download_total_timeout=...)`) sets a per-file time budget that overrides the environment variable; `0` means no limit
+- **Behaviour**: a transfer that keeps making progress is no longer capped by default — `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` now defaults to `0` (was `3600`), while idle timeouts, the low-speed watchdog and bounded retries still apply; set a positive value to restore a cap
+- **Fix**: a stop caused by an expired budget keeps its partial data and is reported as pending, so a plain re-run resumes it instead of requiring `--retry-failed`; trickling response headers stay bounded by the idle timeout when no overall budget is set
+- **Quality**: budget and cancellation messages are English and name both the budget and the variable that relaxes it
+
 **v0.4.0** (2026-09-01)
 - **Feature**: complete OpenAPI coverage for Agent-IDP, MCP, and Studios — Agent Ed25519 identities, OIDC discovery/JWKS and signed JWT issuance (`HubApi`, `ms-hub agent-idp`); Studio lists, variables and configuration options; hosted MCP discovery; protected visibility and runtime metadata; read-only tokens can log in and rejected writes name the required tier. Agent private JWKs are only written to an explicitly requested owner-only file.
 - **Fix**: Studio compat calls no longer leak connection options or API tokens, or drop cover images; errors distinguish permission, quota and conflicts; anonymous Studio info and log pagination work
@@ -232,7 +238,7 @@ print(report.status, report.succeeded, report.failed, report.deferred, report.re
 report.close()  # Release temporary per-file results after consumption.
 ```
 
-- Recovery defaults: at most **2 extra retries per file**, shared by range parts and integrity recovery; failed-request inactivity and waits have a **60-second cumulative budget**. Idle/header waits are limited to **15 seconds**, low-speed checks use a **10-second window**, and cooldowns longer than **5 seconds** defer the file instead of tying up a worker. Size/hash mismatches get at most one full re-download. Healthy transfer time still uses `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT`; large files are not capped at 60 seconds.
+- Recovery defaults: at most **2 extra retries per file**, shared by range parts and integrity recovery; failed-request inactivity and waits have a **60-second cumulative budget**. Idle/header waits are limited to **15 seconds**, low-speed checks use a **10-second window**, and cooldowns longer than **5 seconds** defer the file instead of tying up a worker. Size/hash mismatches get at most one full re-download. Healthy transfer time is **not** capped by default: a transfer that keeps making progress runs to completion, and `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` only applies when set to a positive value.
 - Each task uses **one hidden JSON file** outside downloaded contents (`cache_dir/.download_state/.ms_download_<key>.json` by default), with no embedded database dependency. It combines sparse failures, recovery budgets, failed pages, and the latest call's summary under `report`; `report.report_path` equals `report.tracker_path`. Reruns and batches replace the same file, without extra report JSON or JSONL files. Successful files have no persistent per-file records; a healthy run writes only its scope header and final summary, not an update per file. Its scope is endpoint/repository/revision/output, independent of batch filters. Standard reusable lock files remain separate from JSON metadata.
 - State is advisory, never the sole evidence of success. Every rerun validates actual files against current size/hash metadata, even if hints say `FAILED` or `DEFERRED`; a repaired valid file clears its stale hint without network IO. Otherwise failures remain isolated until metadata changes or `retry_failed=True`. Missing or corrupt hints allow bounded fresh recovery; load/write problems are logged and exposed in `report.state_warnings`, not treated as data corruption or success. A crash during recovery preserves its budget if hints were saved; healthy in-flight files need no markers and resume from actual partial files. Old database-format trackers are not read or migrated: select a new JSON path; existing correct data is reused through validation.
 - Dataset pages are streamed with bounded queues. Failed pages are recorded and isolated across calls; later pages/roots can continue. Two consecutive failed pages stop that root, and `listing_complete=False` prevents false success. The next run can use `retry_failed=True` or an authoritative `file_list` iterable to recover incomplete enumeration. Model listings that hit an unrecoverable server cap also return a partial report.
@@ -243,8 +249,12 @@ report.close()  # Release temporary per-file results after consumption.
 ms-hub download owner/dataset --repo-type dataset --local-dir ./data --continue-on-error
 # Explicit repair run after the source is fixed
 ms-hub download owner/dataset --repo-type dataset --local-dir ./data --continue-on-error --retry-failed
+# Cap each file at one hour, e.g. for a scheduled job with a fixed window
+ms-hub download owner/dataset --repo-type dataset --local-dir ./data --total-timeout 3600
 ```
 
+A file stopped by `--total-timeout` keeps its partial data and is reported as pending rather than failed, so a plain re-run resumes it; `--retry-failed` stays reserved for genuinely suspect files.
+
 CLI exit codes: `0` complete, `3` partial after remaining known files have been processed, `1` global failure, `130` interrupted. Schedulers must treat `3` as partial completion, not an instruction to retry the whole task indefinitely. The SDK's `modelscope_hub.compat.snapshot_download(..., continue_on_error=True, tracker_path=...)` preserves a string return path and logs the report location; use `HubApi.download_repo_report` to read results directly. Older `modelscope.hub.snapshot_download` wrappers may not accept the new options: import the compat function from `modelscope_hub` directly until the wrapper is updated.
 
 ### Upload
@@ -1060,7 +1070,7 @@ Token is persisted locally after `ms-hub login` and auto-loaded in subsequent se
 | `MODELSCOPE_DOWNLOAD_PART_SIZE_MB` | `160` | Parallel range chunk size (MB) |
 | `MODELSCOPE_DOWNLOAD_MAX_RETRIES` | `5` | Single-layer transport retries; `0` disables them; integrity retries are separately bounded |
 | `MODELSCOPE_DOWNLOAD_TIMEOUT` | `60` | Idle read timeout for streams and range parts, not the whole file (seconds) |
-| `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` | `3600` | Per-file time budget, including locks, probes, transfers, retries, and validation (seconds) |
+| `MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT` | `0` | Per-file time budget, including locks, probes, transfers, retries, and validation (seconds); `0` imposes no overall limit |
 | `MODELSCOPE_DOWNLOAD_MIN_BYTES_PER_SECOND` | `1024` | Minimum rate per connection (B/s); `0` disables low-speed detection |
 | `MODELSCOPE_DOWNLOAD_LOW_SPEED_WINDOW_SECONDS` | `60` | Minimum-rate observation window (seconds) |
 | `MODELSCOPE_DOWNLOAD_RETRY_MAX_DELAY_SECONDS` | `60` | Maximum download Retry-After or backoff delay (seconds) |

src/modelscope_hub/_download.py

@@ -56,6 +56,7 @@
     DOWNLOAD_PART_SIZE,
     DOWNLOAD_RETRY_MAX_DELAY_SECONDS,
     DOWNLOAD_RETRY_TIMES,
+    DOWNLOAD_TIMEOUT,
     DOWNLOAD_TOTAL_TIMEOUT,
     ENV_FILE_LOCK,
     ENV_INTER_CLOUD_REGIONS,
@@ -356,7 +357,9 @@ def _wait_download_retry(retry: Retry, exc: Exception, path: str, control: _Down
     except MaxRetryError:
         if isinstance(exc, HubError):
             raise exc
-        raise NetworkError(f"下载 {path} 在 {DOWNLOAD_RETRY_TIMES} 次重试后失败：{type(exc).__name__}") from exc
+        raise NetworkError(
+            f"Download of {path} failed after {DOWNLOAD_RETRY_TIMES} retries: {type(exc).__name__}"
+        ) from exc
     requested = getattr(exc, "retry_after", None)
     if control.recovery is not None:
         delay = max(0, requested) if requested is not None else min(retry.get_backoff_time(), 2)
@@ -436,7 +439,7 @@ def _download_part_with_retry(params: tuple) -> None:
         try:
             control.check()
             if part_stop.is_set():
-                raise _DownloadCancelledError("分片组已取消")
+                raise _DownloadCancelledError("Range part group cancelled")
             partial_length = os.path.getsize(part_file_name) if os.path.exists(part_file_name) else 0
             if partial_length > expected:
                 # Left over from a different split of the file; it cannot be resumed.
@@ -467,7 +470,7 @@ def _download_part_with_retry(params: tuple) -> None:
                 raise
             with control.response(r):
                 if part_stop.is_set():
-                    raise _DownloadCancelledError("分片组已取消")
+                    raise _DownloadCancelledError("Range part group cancelled")
                 content_range = _parse_content_range(r.headers.get("Content-Range"))
                 if content_range is not None and content_range[2] is not None and content_range[2] != file_size:
                     raise FileSizeMismatchError(
@@ -634,12 +637,21 @@ class DownloadManager:
     Dependencies are injected via constructor to keep this class testable.
     """
 
-    def __init__(self, legacy_client: LegacyClient, config: HubConfig, *, user_agent: dict | str | None = None) -> None:
+    def __init__(
+        self,
+        legacy_client: LegacyClient,
+        config: HubConfig,
+        *,
+        user_agent: dict | str | None = None,
+        total_timeout: float | None = None,
+    ) -> None:
         from .utils import build_user_agent
 
         self._client = legacy_client
         self._config = config
         self._user_agent = build_user_agent(config=config, extra=user_agent)
+        # None defers to MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT; 0 or less means no limit.
+        self._total_timeout = total_timeout
         self._cached_region: str | None = None
         self._region_lock = threading.Lock()
         # Cache only confirmed OSS-internal resolutions. A repository can contain
@@ -875,13 +887,16 @@ def download_file(
     ) -> Path:
         """Download one file with cache-first lookup and a shared budget for locks, probes, and retries."""
         with _DownloadControl(
-            total_timeout=DOWNLOAD_TOTAL_TIMEOUT,
+            total_timeout=DOWNLOAD_TOTAL_TIMEOUT if self._total_timeout is None else self._total_timeout,
             cancel_event=_cancel_event,
             min_rate=DOWNLOAD_MIN_BYTES_PER_SECOND,
             speed_window=_recovery.policy.low_speed_window
             if _recovery is not None
             else DOWNLOAD_LOW_SPEED_WINDOW_SECONDS,
+            header_timeout=DOWNLOAD_TIMEOUT,
             recovery=_recovery,
+            budget_label="Per-file download",
+            budget_env="MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT",
         ) as control:
             with self._client.read_context(control):
                 return self._download_file(
@@ -1201,6 +1216,8 @@ def download_repo(
             with _DownloadControl(
                 total_timeout=REPO_TREE_LIST_TIMEOUT_SECONDS,
                 cancel_event=cancel_event,
+                budget_label="Repository listing",
+                budget_env="MODELSCOPE_REPO_TREE_LIST_TIMEOUT_SECONDS",
             ) as listing_control:
                 with self._client.read_context(listing_control):
                     for listing_root in _listing_roots(allow_patterns):
@@ -1341,9 +1358,12 @@ def _download_archive(
         try:
             with (
                 _DownloadControl(
-                    total_timeout=DOWNLOAD_TOTAL_TIMEOUT,
+                    total_timeout=DOWNLOAD_TOTAL_TIMEOUT if self._total_timeout is None else self._total_timeout,
                     min_rate=DOWNLOAD_MIN_BYTES_PER_SECOND,
                     speed_window=DOWNLOAD_LOW_SPEED_WINDOW_SECONDS,
+                    header_timeout=DOWNLOAD_TIMEOUT,
+                    budget_label="Archive download",
+                    budget_env="MODELSCOPE_DOWNLOAD_TOTAL_TIMEOUT",
                 ) as control,
                 self._client.read_context(control),
             ):
@@ -1361,7 +1381,9 @@ def _download_archive(
                                 tmp.write(chunk)
                         announced = _checkable_content_length(resp)
                         if announced is not None and tmp_path.stat().st_size != announced:
-                            raise _IncompleteBodyError("归档内容长度与响应头不一致")
+                            raise _IncompleteBodyError(
+                                "Archive body length does not match the announced Content-Length"
+                            )
                         break
                     except (_DownloadCancelledError, _DownloadDeadlineError):
                         raise
@@ -1936,7 +1958,9 @@ def request(headers: dict[str, str]) -> requests.Response:
 
                     if resp.status_code == 206 and (content_range is None or content_range[0] != offset):
                         tmp_path.unlink(missing_ok=True)
-                        raise _IncompleteBodyError("服务器返回了不匹配请求起点的分段响应")
+                        raise _IncompleteBodyError(
+                            "Server returned a partial response that does not match the requested start offset"
+                        )
                     announced: int | None = None
                     if content_range is not None and content_range[2] is not None:
                         announced = content_range[2]