Fetch the CLI's token from cli.mapbox.com instead of building it in

AGENTS.md

@@ -106,7 +106,10 @@ may write to stdout, which must carry a request id into an error, that every
 telemetry marker is disclosed in README.md, that every end-to-end test file
 that runs the binary either sets `MAPBOX_CLI_NO_TELEMETRY=1` or points
 `MAPBOX_INTERNAL_TELEMETRY_URL` at loopback (so CI can't fire a real event by
-accident), and that the prose is American English.
+accident), that every such file points `MAPBOX_INTERNAL_CLI_TOKEN_URL` and
+`MAPBOX_INTERNAL_TOKENS_URL` at loopback (so CI can't fetch or check the
+CLI's token), and that the prose is American
+English.
 
 They read `src/` recursively and name a file by its path under it
 (`output/mod.rs`), so a module that moves into a directory stays covered —

CHANGELOG.md

@@ -19,6 +19,20 @@ that may never merge. They are not releases and are not listed here.
 
 ## Unreleased
 
+- The CLI's own token, used for telemetry when you have no token of your
+  own, is now fetched from `https://cli.mapbox.com/config/token.json` and
+  kept in `~/.mapbox/cli-token.json` instead of being built into the binary,
+  so it can be rotated without a release. What changes for an existing
+  setup:
+  - `MAPBOX_CLI_BUNDLED_TOKEN` is no longer read at build time. A build
+    from source fetches the token like a release does.
+  - Keeping the token creates `~/.mapbox` if it is missing, even on a run
+    whose only network use is telemetry.
+  - When an API refuses the token, the Tokens API is asked why. Only a
+    revoked, invalid, expired or malformed token is replaced, at once.
+    Three fetches in a row that find no usable token, or three refusals
+    in a row of a valid one, pause that for an hour.
+
 - `tilesets query` renders as a numbered list under `-o text` again, as
   documented. It had printed pretty JSON since 0.2.0, when the command moved
   from `tilequery get` and the renderer kept matching the old name.

README.md

@@ -302,10 +302,10 @@ It never includes a file path or free text you typed. It carries a random ID
 that is replaced on the first run a day after it was created. It is sent in
 the background, so a command never waits for it, and dropped if it can't be
 delivered. It is sent with your own token (`--token`, `MAPBOX_ACCESS_TOKEN`
-or your login) when you have one, and with a token built into the CLI
+or your login) when you have one, and with the CLI's own public token
 otherwise; Mapbox Events keeps the token an event was sent with, and the
-account it belongs to, alongside the event. A build from source has no
-built-in token, so with no token of your own the event is dropped.
+account it belongs to, alongside the event. With no token of your own and
+no way to fetch the CLI's, the event is dropped.
 `MAPBOX_CLI_NO_TELEMETRY=1` or `mapbox config set telemetry off` turns it off.
 
 ### Diagnostics and settings
@@ -479,8 +479,16 @@ and `--dry-run` show it alongside everything else on the request.
 Requests the CLI makes for itself rather than for your commands use your own
 token whenever you have one: `--token` or `MAPBOX_ACCESS_TOKEN`, then your
 login. Only when you have none does it use the CLI's token:
-`MAPBOX_CLI_TOKEN`, or one built into the binary. The CLI's token is only
-used for what `src/cli_token.rs` lists for it, today just telemetry.
+`MAPBOX_CLI_TOKEN`, or a public token fetched from
+`https://cli.mapbox.com/config/token.json` and kept in
+`~/.mapbox/cli-token.json`. It is replaced only when the Tokens API says it
+is revoked, invalid, expired or malformed, not when one API refuses it. A
+refusal of a valid token, or one the Tokens API cannot explain, counts; the
+third in a row pauses telemetry's sending with it for an hour, as three
+fetches in a row that find no usable token pause fetching. The token is
+restricted to `https://cli.mapbox.com/`, so every request with it carries
+that as its `Referer`. The CLI's token is only used for what
+`src/cli_token.rs` lists for it, today just telemetry.
 
 ### Proxies
 

build.rs

@@ -41,8 +41,6 @@
 use std::path::Path;
 
 fn main() {
-    check_bundled_token();
-
     let manifest = Path::new("../internal/openapi-command-config/PINNED_SOURCE");
 
     // The vendored specs and their provenance are ordinary build inputs now,
@@ -93,20 +91,6 @@ fn main() {
     warn_if_stale(committed_at);
 }
 
-/// `src/cli_token.rs` compiles `MAPBOX_CLI_BUNDLED_TOKEN` into the binary,
-/// where `strings` can read it. Only a public `pk.` token may go there, so
-/// anything else fails the build rather than shipping a secret.
-fn check_bundled_token() {
-    println!("cargo:rerun-if-env-changed=MAPBOX_CLI_BUNDLED_TOKEN");
-    let Ok(token) = std::env::var("MAPBOX_CLI_BUNDLED_TOKEN") else {
-        return;
-    };
-    let token = token.trim();
-    if !token.is_empty() && !token.starts_with("pk.") {
-        panic!("MAPBOX_CLI_BUNDLED_TOKEN must be a public pk. token");
-    }
-}
-
 /// New API surface lands in `openapi-specs` at something closer to a monthly
 /// rate, and the maintainer-only sync that regenerates `openapi/` from it
 /// runs weekly, so two weeks without one is already long enough to be worth a

docs/architecture.md

@@ -209,7 +209,9 @@ for the same `ps`-visibility reason Diagram 3 gives.
 
 Step ②'s token follows `src/cli_token.rs`'s fallback order: the user's own
 token first (flag, environment, or stored login), and only when none of
-those exist, a `pk.` token compiled into the binary at build time
-(`MAPBOX_CLI_BUNDLED_TOKEN`) as a last resort so usage from a user who has
-never logged in is still counted. A `cargo build` from source has no bundled
-token, so with no login either, the event is simply dropped.
+those exist, the CLI's `pk.` token as a last resort so usage from a user who
+has never logged in is still counted. Every build fetches that token from
+`https://cli.mapbox.com/config/token.json` and keeps it in
+`~/.mapbox/cli-token.json` until the Tokens API says it is dead
+(`src/cli_token/fetch.rs`). With no login and no way to fetch it, the event
+is simply dropped.

src/auth.rs

@@ -36,7 +36,7 @@ const TOKEN_ENDPOINT: &str = "https://api.mapbox.com/oauth/2.1/token";
 const REGISTRATION_ENDPOINT: &str = "https://api.mapbox.com/oauth/register";
 /// Where `--verify` sends the token. Reached with the token as a query
 /// parameter, so every failure path out of it has to redact.
-const VALIDATION_ENDPOINT: &str = "https://api.mapbox.com/tokens/v2";
+pub(crate) const VALIDATION_ENDPOINT: &str = "https://api.mapbox.com/tokens/v2";
 
 // Every scope here must be one the Accounts API's dynamic client
 // registration (DCR) will actually grant — a scope it doesn't recognize is
@@ -1317,36 +1317,72 @@ fn verify_token(token: &str, debug: bool, timeout: Option<Duration>) -> Result<V
         // pasted issue reports, and the token is a query parameter here.
         eprintln!("[debug] GET {VALIDATION_ENDPOINT}?access_token=<redacted>");
     }
-
-    let response = crate::http::send(
-        crate::http::client()?
-            .get(VALIDATION_ENDPOINT)
-            .query(&[("access_token", token)])
-            // The one request `auth` makes to a Mapbox API rather than to the
-            // authorization server, so it is the one `--timeout` has to reach.
-            // The other three — refresh, registration, code exchange — carry a
-            // few hundred bytes each and keep the client's own budget.
-            .timeout(crate::http::budget(timeout, crate::http::Payload::Bounded)),
+    // The one request `auth` makes to a Mapbox API rather than to the
+    // authorization server, so it is the one `--timeout` has to reach. The
+    // other three — refresh, registration, code exchange — carry a few
+    // hundred bytes each and keep the client's own budget.
+    check_token(
+        token,
+        crate::http::budget(timeout, crate::http::Payload::Bounded),
     )
-    // `reqwest::Error`'s `Display` appends the URL, and the token is in it.
-    .map_err(|e| crate::executor::transport_failure("Token check failed", e))?;
-
-    let status = response.status();
-    // Before `text()` consumes the response — see `executor::request_id`.
-    let request_id = crate::executor::request_id(response.headers());
-    let body = response
-        .text()
-        .map_err(|e| crate::executor::transport_failure("Failed to read the token check", e))?;
+}
 
+/// The request behind [`verify_token`]: the endpoint's answer,
+/// whose `code` is the verdict.
+fn check_token(token: &str, timeout: Duration) -> Result<Value> {
+    let (status, request_id, body) = send_token_check(VALIDATION_ENDPOINT, token, None, timeout)?;
     if !status.is_success() {
         return Err(CliError::http(status.as_u16(), &body)
             .with_request_id(request_id)
             .into());
     }
-
     serde_json::from_str(&body).context("Invalid JSON from the token check endpoint")
 }
 
+/// The verdict's `code` alone, read whatever the status, or `None` when there
+/// was no answer with one. How [`crate::cli_token`] tells a dead token from
+/// one an API merely refuses. Observed answering `200` for every verdict, but
+/// a dead token's `code` must not be lost if that ever changes, or the token
+/// would never be replaced.
+///
+/// `referer` is sent along, for a token restricted to a URL.
+pub(crate) fn token_code(
+    url: &str,
+    token: &str,
+    referer: Option<&str>,
+    timeout: Duration,
+) -> Option<String> {
+    let (_, _, body) = send_token_check(url, token, referer, timeout).ok()?;
+    let answer: Value = serde_json::from_str(&body).ok()?;
+    Some(answer.get("code")?.as_str()?.to_owned())
+}
+
+fn send_token_check(
+    url: &str,
+    token: &str,
+    referer: Option<&str>,
+    timeout: Duration,
+) -> Result<(reqwest::StatusCode, Option<String>, String)> {
+    let mut request = crate::http::client()?
+        .get(url)
+        .query(&[("access_token", token)])
+        .timeout(timeout);
+    if let Some(referer) = referer {
+        request = request.header(reqwest::header::REFERER, referer);
+    }
+    let response = crate::http::send(request)
+        // `reqwest::Error`'s `Display` appends the URL, and the token is in it.
+        .map_err(|e| crate::executor::transport_failure("Token check failed", e))?;
+
+    let status = response.status();
+    // Before `text()` consumes the response — see `executor::request_id`.
+    let request_id = crate::executor::request_id(response.headers());
+    let body = response
+        .text()
+        .map_err(|e| crate::executor::transport_failure("Failed to read the token check", e))?;
+    Ok((status, request_id, body))
+}
+
 /// A verdict from [`verify_token`] that is not `TokenValid`.
 ///
 /// Each code becomes its own error code: telling a revoked token from a typo