Keep telemetry running after an unreadable saved login or a malformed collector answer, and refuse an unsendable collector token

README.md

@@ -40,7 +40,7 @@ The adapter bounds socket calls and reports telemetry failures without failing t
 
 `block()` finishes the current stage and records that validation checks refused the build, including the phase, check identifiers, failure count and optional check statuses. `fail(error_code=...)` preserves the caller's failure code alongside the exception type and failure class. Microcosm classifies interruptions, process termination and memory failures; this package forwards those classifications rather than duplicating them. Both outcomes finish telemetry, so later heartbeat or parent-exit handling cannot replace them.
 
-The service uses the existing Hugging Face credential from the environment or local login cache. The collector validates PolicyEngine membership before issuing a short-lived write credential. Missing or rejected credentials permanently mark that producer's events as local-only. The emitter sends a credential only when it is visible ASCII (letters, digits and punctuation), as every Hugging Face credential and collector token is. It refuses any other value, such as one with a pasted space, a tab, a line break or a non-ASCII character, before sending it: a bearer token (RFC 6750) cannot contain those characters, so the collector would refuse it, and Python's HTTP client raises an error that quotes the whole header value for a line break. Such a credential counts as rejected: the telemetry module marks the producer's events local-only with a warning that the credential contains characters that cannot be sent, collector submissions end `failed` with `CREDENTIAL_REJECTED`, and the Orrery module's graph jobs stay queued with `credential_unsendable`. Permanent HTTP client errors when registering a run or uploading events also keep only the affected producer's records local; HTTP 408, 429, server errors and network failures remain retryable. HTTP 401 invalidates the short-lived credential for renewal. The production destination belongs to the package: by default the collector's production hostname (`PRODUCTION_COLLECTOR_URL`). A module's `collector_origin` setting can choose only among the package's trusted collector origins, `TRUSTED_COLLECTOR_ORIGINS`: the hostname and the Cloud Run address of the same production service (`CLOUD_RUN_COLLECTOR_URL`, the earlier default). No interface release passes `collector_origin` today, so rolling back the hostname needs a new emitter release. Development overrides accept loopback addresses only, and no environment variable changes the destination.
+The service uses the existing Hugging Face credential from the environment or local login cache. The collector validates PolicyEngine membership before issuing a short-lived write credential. Missing or rejected credentials permanently mark that producer's events as local-only. A login saved by `hf auth login` that this user may not read, or that is not text, when neither `HF_TOKEN` nor `HUGGINGFACE_TOKEN` is set, does not make runs local-only: the login exists and can be repaired. Every pending run stays pending with the reason `huggingface_login_unreadable` (in the run's `local_only_reason` column), and one warning names only the error's type, never the file or its contents. The login is read again with the same backoff as after a collector outage, and the shutdown drain ends at once. As soon as this emitter can read the login again, whatever the collector then answers, it clears the reason of every run that waited for it, including runs it had already registered, and its shutdown drain retries a collector outage as usual; it then delivers the runs. A later emitter that can read the login delivers them too, and clears the reason when it registers each run. Other errors from huggingface_hub, such as a failed OIDC token exchange while `HF_OIDC_RESOURCE` is set, do not count as an unreadable login. The emitter sends a credential only when it is visible ASCII (letters, digits and punctuation), as every Hugging Face credential and collector token is. It refuses any other value, such as one with a pasted space, a tab, a line break or a non-ASCII character, before sending it: a bearer token (RFC 6750) cannot contain those characters, so the collector would refuse it, and Python's HTTP client raises an error that quotes the whole header value for a line break. Such a credential counts as rejected: the telemetry module marks the producer's events local-only with a warning that the credential contains characters that cannot be sent, collector submissions end `failed` with `CREDENTIAL_REJECTED`, and the Orrery module's graph jobs stay queued with `credential_unsendable`. Permanent HTTP client errors when registering a run or uploading events also keep only the affected producer's records local; HTTP 408, 429, server errors and network failures remain retryable, with backoff; a network failure includes an answer that is not valid HTTP or ends early, such as a truncated chunked answer. HTTP 401 invalidates the short-lived credential for renewal. The production destination belongs to the package: by default the collector's production hostname (`PRODUCTION_COLLECTOR_URL`). A module's `collector_origin` setting can choose only among the package's trusted collector origins, `TRUSTED_COLLECTOR_ORIGINS`: the hostname and the Cloud Run address of the same production service (`CLOUD_RUN_COLLECTOR_URL`, the earlier default). No interface release passes `collector_origin` today, so rolling back the hostname needs a new emitter release. Development overrides accept loopback addresses only, and no environment variable changes the destination.
 
 The queue upgrades the existing `20261007_01` Alembic revision to `20261008_02`, adding graph-job records without changing producer sequences, event IDs or upload eligibility. Unversioned databases are rejected without schema modification. There is no raw SQL, automatic schema adoption, or parallel legacy implementation. The hosted telemetry collector and calibration dashboard do not change.
 
@@ -56,9 +56,9 @@ A build on Modal runs without the operator's Hugging Face credential, so its emi
   | 2 | A usage error, such as a lifetime below 1 or a `--development-collector-url` that is not a loopback address; nothing was sent |
   | 3 | No Hugging Face credential (`HF_TOKEN`, `HUGGINGFACE_TOKEN` or `hf auth login`), or a saved login this user cannot read; nothing was sent |
   | 4 | The collector refused the credential (401, or 403 for a user outside the PolicyEngine organization), or the credential contains characters other than visible ASCII, such as a space or a line break, which cannot be sent as a bearer token, and was not sent |
-  | 5 | The collector was unreachable, its answer was not valid HTTP, or its answer had no non-empty `access_token` and positive integer `expires_in` |
+  | 5 | The collector was unreachable, its answer was not valid HTTP, or its answer had no `access_token` of visible ASCII characters, without spaces or line breaks, and positive integer `expires_in` |
   | 6 | The collector answered 422: it does not accept `lifetime_seconds`, so it predates tokens for other machines |
-  | 7 | An unexpected error; only the error's type is printed |
+  | 7 | An unexpected error; only the error's type is printed. This includes a failed OIDC token exchange while `HF_OIDC_RESOURCE` is set, such as an `HfHubHTTPError` from the Hub or an `OIDCError`: it fails before the collector is contacted, so it is not 5, and it is not a missing or unreadable login, so not 3 |
 
 - **On the other machine:** the launcher sets `MICROCOSM_COLLECTOR_TOKEN`, `MICROCOSM_COLLECTOR_TOKEN_EXPIRES_AT` and `MICROCOSM_COLLECTOR_ORIGIN` from that output. With them, `CollectorSession` sends that token and never reads or exchanges a Hugging Face credential; a session for another scope, such as collector submissions, has no credential. None of the variables chooses where the emitter sends: `MICROCOSM_COLLECTOR_ORIGIN` is only compared with the origin the module settings resolve.
 - **Where the token goes:** only to that collector origin. The telemetry module sends it with run registrations and events. The Orrery module gets its credential through the same session (`shared_session`), but sends it to its `publication_origin`; unless that is the collector origin, which it is not by default, the session withholds a given token from Orrery on purpose, and Orrery's jobs stay queued with `credential_withheld_from_destination`.

changelog.d/telemetry-keeps-running.fixed.md

@@ -0,0 +1 @@
+The telemetry module no longer stops in two cases. A login saved by `hf auth login` that this user may not read, or that is not text, with neither `HF_TOKEN` nor `HUGGINGFACE_TOKEN` set, keeps runs pending with the reason `huggingface_login_unreadable`, with one warning that names only the error's type; the login is read again with backoff, and once it can be read the runs are delivered. As soon as the login can be read again, the reason is cleared, including for runs already registered, and the shutdown drain retries a collector outage as usual. Other errors from huggingface_hub, such as a failed OIDC token exchange while `HF_OIDC_RESOURCE` is set, are not reported as an unreadable login: the telemetry module handles them as before, and `microcosm-emitter telemetry-token` now exits 7 (an unexpected error, naming its type) instead of 3, because the exchange fails before the collector is contacted. A collector answer that is not valid HTTP or ends early, such as a truncated chunked answer, during run registration or event upload is retried with backoff like other network failures, and the run's state does not change. The token exchange also refuses a collector `access_token` with characters other than visible ASCII, such as a space, as an answer without a usable token, so `microcosm-emitter telemetry-token` exits 5 instead of printing a token that the emitter given it would refuse, and a workstation's telemetry module retries the exchange instead of stopping.

src/microcosm_emitter/cli.py

@@ -33,7 +33,9 @@
 
 #: Exit codes of `telemetry-token`, one per kind of failure; 2 is a usage
 #: error, as for every command, and 7 an unexpected error, which the others do
-#: not name.
+#: not name. A failed OIDC token exchange while HF_OIDC_RESOURCE is set is 7:
+#: it fails at the CI provider or the Hub before the collector is contacted,
+#: so it is not 5, and it is not a missing or unreadable login, so not 3.
 EXIT_MISSING_CREDENTIAL: Final = 3
 EXIT_REJECTED_CREDENTIAL: Final = 4
 EXIT_COLLECTOR_UNAVAILABLE: Final = 5
@@ -71,7 +73,8 @@
     ),
 }
 #: `huggingface_hub` could not read the login `hf auth login` saved, such as a
-#: file this user may not read, and neither variable is set.
+#: file this user may not read, and neither variable is set
+#: (``UnreadableLoginError``).
 UNREADABLE_LOGIN_MESSAGE: Final = (
     "The Hugging Face login saved by `hf auth login` could not be read "
     "({error_type}), and neither HF_TOKEN nor HUGGINGFACE_TOKEN is set; nothing "
@@ -139,6 +142,7 @@ def telemetry_token(arguments: argparse.Namespace) -> int:
     """
     from microcosm_emitter.core.auth import (
         ExchangeFailure,
+        UnreadableLoginError,
         exchange_huggingface_credential,
         huggingface_token,
     )
@@ -147,9 +151,9 @@ def telemetry_token(arguments: argparse.Namespace) -> int:
     origin = arguments.collector_origin or _collector_origin(None)
     try:
         credential = huggingface_token()
-    except (OSError, UnicodeDecodeError) as error:
+    except UnreadableLoginError as error:
         print(
-            UNREADABLE_LOGIN_MESSAGE.format(error_type=type(error).__name__),
+            UNREADABLE_LOGIN_MESSAGE.format(error_type=error.error_type),
             file=sys.stderr,
         )
         return EXIT_MISSING_CREDENTIAL

src/microcosm_emitter/core/auth.py

@@ -56,6 +56,12 @@
 #: http.client raises for a line break with an error that quotes the whole
 #: header value, and for a character outside Latin-1 with one that holds it.
 _SENDABLE_CREDENTIAL = re.compile(r"[!-~]+")
+#: What ``_http_post`` raises when it gets no usable answer, which every caller
+#: handles as a network failure: no answer or a timeout (``OSError``,
+#: ``TimeoutError``), or an answer that is not valid HTTP or ends early, such
+#: as a truncated chunked answer (http.client's ``HTTPException``, which is not
+#: an ``OSError``).
+NETWORK_FAILURES = (OSError, TimeoutError, HTTPException)
 
 
 class _NoRedirectHandler(urllib.request.HTTPRedirectHandler):
@@ -132,14 +138,65 @@ def _http_post(
         return error.code, _decode_response(body)
 
 
+class UnreadableLoginError(OSError):
+    """The login saved by `hf auth login` could not be read, such as a file
+    this user may not read or one that is not text, and neither `HF_TOKEN`
+    nor `HUGGINGFACE_TOKEN` is set.
+
+    ``error_type`` is the name of the error ``huggingface_hub`` raised. That
+    error is not kept, not even as the context of this one: its text names
+    the file, and a ``UnicodeDecodeError`` holds the file's bytes. An
+    ``OSError``, as reading a file that fails is, so callers that handled the
+    ``PermissionError`` before still handle it.
+    """
+
+    def __init__(self, error_type: str) -> None:
+        super().__init__(
+            f"the saved Hugging Face login could not be read ({error_type})"
+        )
+        self.error_type = error_type
+
+
+def _saved_login_unreadable(error: OSError | UnicodeDecodeError) -> bool:
+    """Whether ``get_token()`` raised ``error`` reading the login files that
+    `hf auth login` saved: a ``PermissionError``, bytes that are not text, or
+    another ``OSError`` that names a file, such as ``IsADirectoryError``.
+
+    With ``HF_OIDC_RESOURCE`` set, huggingface_hub exchanges a CI OIDC token
+    instead of reading those files, so none of its errors count, such as the
+    ``HfHubHTTPError`` (an ``OSError`` that names no file) of a 503 from the
+    Hub, or an answer that is not text.
+    """
+    if os.environ.get("HF_OIDC_RESOURCE"):
+        return False
+    return isinstance(error, (PermissionError, UnicodeDecodeError)) or (
+        isinstance(error, OSError) and error.filename is not None
+    )
+
+
 def huggingface_token() -> str | None:
     """The build's Hugging Face token: `HF_TOKEN`, then `HUGGINGFACE_TOKEN`, then
-    the token saved by `hf auth login`; `None` without one."""
-    return (
+    the token saved by `hf auth login`; `None` without one.
+
+    Raises ``UnreadableLoginError`` when only the saved login is left and it
+    cannot be read (``_saved_login_unreadable``); every other error of
+    ``get_token()`` propagates unchanged.
+    """
+    variable_token = (
         os.environ.get("HF_TOKEN", "").strip()
         or os.environ.get("HUGGINGFACE_TOKEN", "").strip()
-        or get_token()
     )
+    if variable_token:
+        return variable_token
+    try:
+        return get_token()
+    except (OSError, UnicodeDecodeError) as error:
+        # huggingface_hub catches only FileNotFoundError when it reads the file.
+        if not _saved_login_unreadable(error):
+            raise
+        error_type = type(error).__name__
+    # Outside the except block, so the new error has no context to keep.
+    raise UnreadableLoginError(error_type)
 
 
 @dataclass(frozen=True)
@@ -201,9 +258,10 @@ def given_collector_token(
 
 @dataclass(frozen=True)
 class ExchangedToken:
-    """A usable answer to an exchange: a 200 with a non-empty ``access_token``
-    and a positive whole number of seconds, ``expires_in``. Its repr leaves
-    the token out."""
+    """A usable answer to an exchange: a 200 with an ``access_token`` of
+    visible ASCII characters, which can be sent as a bearer token, and a
+    positive whole number of seconds, ``expires_in``. Its repr leaves the
+    token out."""
 
     access_token: str = field(repr=False)
     expires_in: int
@@ -246,8 +304,11 @@ def exchange_huggingface_credential(
     Without ``scope`` the body is ``{}``, which the collector answers with a
     ``telemetry:write`` token. ``lifetime_seconds`` asks for a token that
     lasts that long, for a build on a machine without the credential; the
-    collector may grant less. A usable answer is a 200 with a non-empty
-    ``access_token`` and a positive integer ``expires_in``.
+    collector may grant less. A usable answer is a 200 with an
+    ``access_token`` that ``_SENDABLE_CREDENTIAL`` accepts, as the Hugging
+    Face credential must be, and a positive integer ``expires_in``. Any other
+    token could not be sent as a bearer token, so it counts as no usable
+    token: ``unavailable``.
     """
 
     if not hugging_face_token:
@@ -261,8 +322,7 @@ def exchange_huggingface_credential(
         status, answer = (post or _http_post)(
             origin + TOKEN_EXCHANGE_PATH, body, hugging_face_token
         )
-    except (OSError, TimeoutError, HTTPException):
-        # HTTPException: an answer that is not valid HTTP, or is truncated.
+    except NETWORK_FAILURES:
         return ExchangeFailure("unavailable")
     if status in {HTTPStatus.UNAUTHORIZED, HTTPStatus.FORBIDDEN}:
         return ExchangeFailure("rejected", status)
@@ -280,7 +340,7 @@ def exchange_huggingface_credential(
     if (
         status != HTTPStatus.OK
         or not isinstance(access_token, str)
-        or not access_token
+        or not _SENDABLE_CREDENTIAL.fullmatch(access_token)
         or type(expires_in) is not int
         or expires_in <= 0
     ):
@@ -354,6 +414,12 @@ class CollectorSession:
     ``token_expired`` or ``token_rejected``. A given token obtained from
     another origin than ``origin`` raises ``GivenCollectorTokenError``: its
     origin is only compared, and never chooses where anything is sent.
+
+    Without a given token, ``credential()`` raises the ``UnreadableLoginError``
+    of ``huggingface_token`` when the saved login cannot be read; the
+    telemetry module keeps its runs pending with the reason
+    ``huggingface_login_unreadable`` and reads the login again after a
+    backoff.
     """
 
     def __init__(

src/microcosm_emitter/core/constants.py

@@ -68,11 +68,23 @@
 # already has; a successful registration clears it.
 PENDING_GIVEN_TOKEN_EXPIRED: Final = "collector_token_expired"
 PENDING_GIVEN_TOKEN_REJECTED: Final = "collector_token_rejected"
+# Why a pending run waits: the login saved by `hf auth login` could not be read,
+# such as a file this user may not read, and neither HF_TOKEN nor
+# HUGGINGFACE_TOKEN is set. The login exists and can be repaired, so the run
+# stays pending for this emitter, or a later one, to deliver once it can be read.
+PENDING_HUGGINGFACE_LOGIN_UNREADABLE: Final = "huggingface_login_unreadable"
 
 NO_CREDENTIAL_MESSAGE: Final = (
     "Microcosm telemetry is local-only: no ambient Hugging Face credential was "
     "found. The dataset build will continue."
 )
+# Only the error's type, since its text names the file, or quotes its bytes.
+UNREADABLE_LOGIN_WAITING_MESSAGE: Final = (
+    "Microcosm telemetry is waiting: the Hugging Face login saved by `hf auth "
+    "login` could not be read ({error_type}), and neither HF_TOKEN nor "
+    "HUGGINGFACE_TOKEN is set. Queued events stay pending, and are delivered once "
+    "the login can be read. The dataset build will continue."
+)
 REJECTED_CREDENTIAL_MESSAGE: Final = (
     "Microcosm telemetry is local-only for this run: the ambient Hugging Face "
     "credential was not accepted as a PolicyEngine organization member. The "

src/microcosm_emitter/core/models.py

@@ -87,7 +87,8 @@ class TelemetryRunRecord(SpoolModel):
         default=UPLOAD_STATE_PENDING,
     )
     #: Why a local-only run is excluded from upload or, for a pending run,
-    #: why delivery waits (a given collector token expired or was refused).
+    #: why delivery waits: a given collector token expired or was refused, or
+    #: the saved Hugging Face login could not be read.
     local_only_reason: Mapped[str | None] = mapped_column(Text)
     updated_at: Mapped[str] = mapped_column(Text, nullable=False)
     events: Mapped[list[TelemetryEventRecord]] = relationship(