diff --git a/scripts/qcm/README.md b/scripts/qcm/README.md index 91af9c7..fd015a3 100644 --- a/scripts/qcm/README.md +++ b/scripts/qcm/README.md @@ -292,11 +292,11 @@ switch-off, an older decode landing after a newer ask, and an uninterrupted ask ## Remote reads (QQA-13) -`quiet-corners-maps/request.js` is the one way the map reads from the network. `fetchWithDeadline(url, { fetchImpl, signal, timeoutMs })` answers `{ ok, status: 'ok' | 'timeout' | 'aborted' | 'error' }` and never throws. Its timer is a plain `AbortController`, so it works without `AbortSignal.any`, it ends a fetch that ignores its signal, and it clears the timer and the listener on the caller's signal when the call settles. A caller's cancel is `aborted`, never `timeout`. `createLatest()` is latest-wins: a new call aborts the previous one and the older answer comes back `stale`. `source-terrain-names.js` re-exports `fetchJson` and `createSearchGuard` from it for the callers that already import them. +`quiet-corners-maps/request.js` is the one way the map reads from the network. `fetchWithDeadline(url, { fetchImpl, signal, timeoutMs, init, read, anyStatus })` (`read` swaps the body reader, e.g. `r => r.blob()`; `anyStatus` also reads the body of a non-2xx for services whose error answer carries the reason) answers `{ ok, status: 'ok' | 'timeout' | 'aborted' | 'error' }` and never throws. Its timer is a plain `AbortController`, so it works without `AbortSignal.any`, it ends a fetch that ignores its signal, and it clears the timer and the listener on the caller's signal when the call settles. A caller's cancel is `aborted`, never `timeout`. `orThrow(result, message)` is for the few adapters that throw on failure: the body of an ok result, else an Error carrying the result's `status` and `httpStatus` (a caller's cancel keeps the name `AbortError`). `createLatest()` is latest-wins: a new call aborts the previous one and the older answer comes back `stale`. `source-terrain-names.js` re-exports `fetchJson` and `createSearchGuard` from it for the callers that already import them. -The deadline is per source: `SOURCES..timeoutMs` in `source-registry.js`, `DEFAULT_SOURCE_TIMEOUT_MS` (10 s) unless the source cites a slower measurement in `timeoutMeasured`. GNIS place names keep 20 s because their latency is measured up to 31 s. Only the modules moved onto the helper honour it so far: GNIS place names, the quad index, and the NWS forecast and alerts loaders. Every other source (road names, hillshade, MVUM, stream names, topo tiles and the rest) carries the value but still runs at `REQUEST_POLICY.timeoutMs` (20 s) until it is converted. The quad index pages at most `MAX_PAGES` (20) times and stops on a page that adds no new cell, ending `coverage: 'partial'`; its `timeoutMs` bounds each attempt, including `searchByName`, and a timeout is not retried. +The deadline is per source: `SOURCES..timeoutMs` in `source-registry.js`, `DEFAULT_SOURCE_TIMEOUT_MS` (10 s) unless the source cites a slower measurement in `timeoutMeasured`. GNIS place names keep 20 s because their latency is measured up to 31 s, and the wetlands and government-units layers keep the 20 s page their measurements were taken against. Every module in that directory except `main.js` and the seven listed next reads through the helper at its own source's value (the paged boundary and site callers pass it to `pageThrough`; a module with no registry entry, such as the ecoregions export, uses the default). `main.js` still has raw reads until PR-B2. The seven that reached master after this work was written (`address-geocode.js`, `source-avalanche.js`, `source-nrhp.js`, `source-ntad-rail-airports.js`, `source-nws-observations.js`, `source-snotel.js`, `source-stream-gauges.js`) still make their own raw calls, with no deadline from the helper, and are on the `ALLOW` list until a follow-up moves them. `REQUEST_POLICY.timeoutMs` is no longer a request deadline. The quad index pages at most `MAX_PAGES` (20) times and stops on a page that adds no new cell, ending `coverage: 'partial'`; its `timeoutMs` bounds each attempt, including `searchByName`, and a timeout is not retried. -`request.test.mjs` covers the helper (timeout, abort, cleanup, no `AbortSignal.any`, latest-wins); `map-index.test.mjs`, `arcgis-pager.test.mjs` and `registry.test.mjs` hold the quad paging cap, the pager deadline and the per-source `timeoutMs` rule. +`request.test.mjs` covers the helper (timeout, abort, cleanup, no `AbortSignal.any`, `anyStatus`, `orThrow`, latest-wins); `no-raw-fetch.test.mjs` fails on any raw `fetch(` or `fetchImpl(` call in that directory outside `request.js` and its `ALLOW` list (`main.js`, removed in PR-B2, and the seven named above; it also fails if an entry stops holding a raw call, so the list cannot rot); `adapter-deadlines.test.mjs` hands every adapter that takes `fetchImpl` a fetch that never answers and asserts it settles at its deadline, and that a caller's abort ends it as a cancel, not a timeout (a new `source-*.js` module that takes `fetchImpl` must be added to its table); `build.test.mjs` holds that the build's cancel reaches the aerial, relief and elevation requests; `map-index.test.mjs`, `arcgis-pager.test.mjs` and `registry.test.mjs` hold the quad paging cap, the pager deadline and the per-source `timeoutMs` rule. ## Accessibility checks diff --git a/scripts/qcm/adapter-deadlines.test.mjs b/scripts/qcm/adapter-deadlines.test.mjs new file mode 100644 index 0000000..9047b53 --- /dev/null +++ b/scripts/qcm/adapter-deadlines.test.mjs @@ -0,0 +1,238 @@ +// Every adapter that reads from a remote service ends on its own deadline, and a caller's +// cancel ends it as a cancel. One table, one never-answering fetch: a fetch that hangs is the +// failure this guards (the page waits on a pending note forever), so no adapter is tested +// against a slow-but-answering one here. +// +// An adapter that takes a `timeoutMs` is run with a small one in real time. One that does +// not is run on node:test mock timers, with the clock moved by its registry timeoutMs, so +// the table stays fast and still pins the value the adapter really uses. `rounds` is how many +// answers in a row the adapter waits for before it can finish (a count, then a page). +import { test, mock } from 'node:test'; +import assert from 'node:assert/strict'; + +const M = '../../themes/headlessmode/static/js/quiet-corners-maps/'; +const mod = async (name) => import(`${M}${name}.js`); +const { SOURCES, DEFAULT_SOURCE_TIMEOUT_MS } = await mod('source-registry'); +const { pageThrough } = await mod('arcgis-pager'); +const { fetchCellsForBbox, searchByName } = await mod('source-map-index'); +const { fetchCaltransCctv } = await mod('camera-layer'); +const { fetchAerial } = await mod('source-aerial'); +const { fetchTerrain } = await mod('source-terrain'); +const { fetchElevations, profileFor } = await mod('source-elevation'); +const { fetchAqi } = await mod('source-aqi'); +const { fetchAurora } = await mod('source-aurora'); +const { loadEcoregions } = await mod('source-ecoregions'); +const { fetchFire } = await mod('source-fire'); +const { fetchSmoke } = await mod('source-smoke'); +const { fetchFloodView } = await mod('source-flood-zones'); +const { fetchJson: fetchGbif } = await mod('source-gbif'); +const { createHillshadeFetcher, hillshadeRequestFor } = await mod('source-hillshade'); +const landCover = await mod('source-land-cover'); +const terrainSurface = await mod('source-terrain-surface'); +const { fetchLayer: fetchMvum } = await mod('source-mvum'); +const { fetchAlerts, alertsFor } = await mod('source-nws-alerts'); +const { fetchFrameTime } = await mod('source-nws-radar'); +const { fetchAccess } = await mod('source-padus'); +const { createRoadNamesFetcher } = await mod('source-road-names'); +const { fetchTiles } = await mod('source-usgs-topo'); +const { fetchHourly } = await mod('source-weather'); +const { fetchWetlands } = await mod('source-wetlands'); +const { fetchStreamNames } = await mod('source-stream-names'); +const { fetchBlmRecSites } = await mod('source-blm-rec-sites'); +const { fetchCountyLines } = await mod('source-county-lines'); +const { fetchFederalLand } = await mod('source-federal-land'); +const { fetchRecSites } = await mod('source-rec-sites'); +const { fetchRoadQualityReport } = await mod('source-road-basic'); +const { fetchSurfaceOwnership } = await mod('source-surface-ownership'); +const { fetchTrails } = await mod('source-trails'); +const { fetchLandBoundaries } = await mod('source-land-boundaries'); +const stateSources = await mod('state-sources'); + +const SLACK_MS = 250; +const SMALL_MS = 50; +const never = () => new Promise(() => {}); +const flush = () => new Promise((resolve) => setImmediate(resolve)); + +const bbox = { minLon: -83.6, minLat: 35.0, maxLon: -83.4, maxLat: 35.2 }; +const sheet = { id: 'S1', pixel: { width: 800, height: 600 } }; +const feature = { id: 'r1', name: 'A trail', paths: [[[-83.5, 35.0], [-83.49, 35.01]]] }; +const landPair = (() => { + for (const state of stateSources.STATE_CODES) { + for (const family of stateSources.LAYER_IDS) { + const layer = stateSources.layerFor(state, family); + if (layer && stateSources.isBrowserFetchable(layer) && stateSources.isSemanticallyUniform(layer)) return { state, family }; + } + } + throw new Error('no state layer is fetchable from a browser'); +})(); + +// name, the registry source whose timeoutMs it must use (null: DEFAULT_SOURCE_TIMEOUT_MS), whether +// it takes `timeoutMs`, the sequential answers it waits for, and how to call it. +const ROWS = [ + { name: 'pageThrough', param: true, call: (o) => pageThrough({ url: 'x', params: {}, normalize: (r) => r, ...o }) }, + { name: 'fetchCellsForBbox', source: 'usgsQuadIndex', param: true, call: (o) => fetchCellsForBbox({ bboxLonLat: bbox, retries: 0, ...o }) }, + { name: 'searchByName', source: 'usgsQuadIndex', param: true, call: (o) => searchByName({ name: 'rainbow springs', ...o }) }, + { name: 'fetchStreamNames', source: 'nhdFlowlines', param: true, call: ({ fetchImpl, ...o }) => fetchStreamNames(bbox, { zoom: 13, fetch: fetchImpl, ...o }) }, + { name: 'fetchCaltransCctv', source: 'caltransCctv', call: (o) => fetchCaltransCctv(o) }, + { name: 'fetchAerial', source: 'usdaNaip', call: (o) => fetchAerial({ sheet, bboxLonLat: bbox, ...o }) }, + { name: 'fetchTerrain', source: 'usgs3dep', call: (o) => fetchTerrain({ sheet, bboxLonLat: bbox, ...o }) }, + { name: 'fetchElevations', source: 'usgs3dep', call: (o) => fetchElevations({ points: [[-83.5, 35.0]], ...o }) }, + { name: 'profileFor', source: 'usgs3dep', call: (o) => profileFor(feature, o) }, + { name: 'fetchAqi', source: 'epaAirNowAqi', call: ({ fetchImpl, signal }) => fetchAqi(bbox, { fetchImpl, signal }) }, + { name: 'fetchAurora', source: 'swpcAurora', call: (o) => fetchAurora(o) }, + { name: 'loadEcoregions', source: null, call: (o) => loadEcoregions(o) }, + { name: 'fetchFire', source: 'nifcFirePerimeters', call: ({ fetchImpl, signal }) => fetchFire(bbox, { fetchImpl, signal }) }, + { name: 'fetchSmoke', source: 'noaaHmsSmoke', call: ({ fetchImpl, signal }) => fetchSmoke(bbox, { fetchImpl, signal }) }, + { name: 'fetchFloodView', source: 'femaNfhl', call: ({ fetchImpl, signal }) => fetchFloodView(bbox, { fetchImpl, signal }) }, + { name: 'gbif fetchJson', source: 'gbifOccurrence', call: ({ fetchImpl, signal }) => fetchGbif('https://api.gbif.org/v1/x', { fetchImpl, signal }) }, + { name: 'createHillshadeFetcher', source: 'usgs3depHillshade', param: true, timeoutIn: 'factory', + call: ({ fetchImpl, signal, timeoutMs }) => createHillshadeFetcher({ fetchImpl, ...(timeoutMs ? { timeoutMs } : {}) }) + .fetchImage(hillshadeRequestFor({ bboxLonLat: bbox, width: 400, height: 300 }), { signal }), + verdict: (v) => v.status }, + { name: 'land-cover fetchSurface', source: 'nlcdLandCover', + call: ({ fetchImpl, signal }) => landCover.fetchSurface(landCover.surfaceRequestFor({ bboxLonLat: bbox, width: 400, height: 300 }), { fetchImpl, signal }) }, + { name: 'land-cover fetchPoint', source: 'nlcdLandCover', + call: ({ fetchImpl, signal }) => landCover.fetchPoint(landCover.pointRequestFor({ lon: -83.5, lat: 35.1 }), { fetchImpl, signal }) }, + { name: 'fetchMvum', source: 'usfsMvum', param: true, rounds: 2, + call: (o) => fetchMvum({ layer: 'roads', bboxLonLat: bbox, retries: 0, ...o }) }, + { name: 'fetchAlerts', source: 'nwsAlerts', call: (o) => fetchAlerts({ lat: 35.1, lon: -83.5, ...o }), verdict: (v) => v.detail }, + { name: 'alertsFor', source: 'nwsAlerts', param: true, call: ({ fetchImpl, signal, timeoutMs }) => alertsFor({ lat: 35.1, lon: -83.5, fetchImpl, timeoutMs, signal }), verdict: (v) => v.detail }, + { name: 'fetchFrameTime', source: 'nwsRadar', call: (o) => fetchFrameTime(o) }, + { name: 'fetchAccess', source: 'padusPublicAccess', call: ({ fetchImpl, signal }) => fetchAccess(bbox, { fetchImpl, signal }) }, + { name: 'fetchRoadNames', source: 'usgsRoadNames', call: ({ fetchImpl, signal }) => createRoadNamesFetcher().fetchRoadNames({ bbox, zoom: 14, fetchImpl, signal }) }, + { name: 'terrain-surface fetchSlopeImage', source: 'usgs3depSurface', + call: ({ fetchImpl, signal }) => terrainSurface.fetchSlopeImage(terrainSurface.slopeRequestFor({ bboxLonLat: bbox, width: 400, height: 300 }), { fetchImpl, signal }) }, + { name: 'terrain-surface fetchPoint', source: 'usgs3depSurface', + call: ({ fetchImpl, signal }) => terrainSurface.fetchPoint({ lon: -83.5, lat: 35.1 }, 'slope', { fetchImpl, signal }) }, + { name: 'fetchTiles', source: 'usgsTopo', param: true, + call: async (o) => { const r = await fetchTiles([{ key: '1/1/1', url: 'https://x.invalid/1' }], { retries: 0, ...o }); return r.results.get('1/1/1').error; } }, + { name: 'fetchHourly', source: 'nwsForecast', call: (o) => fetchHourly({ lat: 35.1, lon: -83.5, ...o }) }, + { name: 'fetchWetlands', source: 'fwsNwiWetlands', rounds: 2, call: ({ fetchImpl, signal }) => fetchWetlands({ bboxLonLat: bbox, fetchImpl, signal }) }, + // The pager callers have no timeoutMs of their own: the value they hand pageThrough is what is pinned. + { name: 'fetchBlmRecSites', source: 'blmRecSites', call: (o) => fetchBlmRecSites({ bboxLonLat: bbox, ...o }) }, + { name: 'fetchCountyLines', source: 'usgsGovUnits', call: (o) => fetchCountyLines({ bboxLonLat: bbox, ...o }) }, + { name: 'fetchFederalLand', source: 'usgsGovUnits', call: (o) => fetchFederalLand({ family: 'nationalPark', bboxLonLat: bbox, ...o }) }, + { name: 'fetchRecSites', source: null, call: (o) => fetchRecSites({ bboxLonLat: bbox, ...o }) }, + { name: 'fetchRoadQualityReport', source: null, call: (o) => fetchRoadQualityReport({ bboxLonLat: bbox, ...o }) }, + { name: 'fetchSurfaceOwnership', source: 'usfsSurfaceOwnership', call: (o) => fetchSurfaceOwnership({ bboxLonLat: bbox, ...o }) }, + { name: 'fetchTrails', source: null, call: (o) => fetchTrails({ bboxLonLat: bbox, ...o }) }, + { name: 'fetchLandBoundaries', source: null, call: (o) => fetchLandBoundaries({ ...landPair, bboxLonLat: bbox, ...o }) }, +]; + +const registryMs = (row) => (row.source ? SOURCES[row.source].timeoutMs : DEFAULT_SOURCE_TIMEOUT_MS); + +/** Settles to {value} or {error}, and reports when it has. */ +function watch(promise) { + const out = { done: false }; + promise.then((value) => { out.done = true; out.value = value; }, (error) => { out.done = true; out.error = error; }); + return out; +} + +/** Move the mock clock `ms`, then in 1 ms steps (the helper settles one macrotask after a + * deadline) until the call is done, for at most `steps` steps. */ +async function advance(out, ms, steps = 25) { + await flush(); + mock.timers.tick(ms); + for (let i = 0; i < steps && !out.done; i += 1) { await flush(); mock.timers.tick(1); } + await flush(); +} + +const looksLikeTimeout = (out, row) => { + if (out.error) return !(out.error.name === 'AbortError') && /within \d+ ms|did not answer|time(d)? ?out/i.test(String(out.error.message)); + const text = row.verdict ? row.verdict(out.value) : out.value; + return typeof text === 'string' && /within \d+ ms|did not answer|time(d)? ?out/i.test(text); +}; + +/** The call with no `timeoutMs` passed, on mock timers: not settled one tick short of the + * registry value, settled once it has passed. This is the value the adapter really uses. */ +async function assertRegistryDeadline(t, row) { + const ms = registryMs(row); + const rounds = row.rounds || 1; + mock.timers.enable({ apis: ['setTimeout'] }); + t.after(() => mock.timers.reset()); + const out = watch(row.call({ fetchImpl: never })); + await flush(); + mock.timers.tick(ms - 1); + await flush(); + assert.equal(out.done, false, `settled before its ${ms} ms deadline`); + for (let r = 0; r < rounds && !out.done; r += 1) await advance(out, r === 0 ? 1 : ms); + assert.ok(out.done, `still waiting ${rounds} x ${ms} ms in: the call has no deadline of its own`); +} + +for (const row of ROWS) { + const ms = registryMs(row); + const rounds = row.rounds || 1; + + test(`${row.name} ends on its deadline when the service never answers`, async (t) => { + if (row.param) { + const t0 = Date.now(); + const out = watch(row.call({ fetchImpl: never, timeoutMs: SMALL_MS })); + while (!out.done && Date.now() - t0 < 5000) await new Promise((r) => setTimeout(r, 5)); + assert.ok(out.done, 'still waiting after 5 s'); + assert.ok(Date.now() - t0 < SMALL_MS * rounds + SLACK_MS, `took ${Date.now() - t0} ms`); + return; + } + await assertRegistryDeadline(t, row); + }); + + // The default is the moved value, so a row that takes `timeoutMs` is run once more without it. + if (row.param) test(`${row.name} ends on its registry deadline when no timeoutMs is passed`, (t) => assertRegistryDeadline(t, row)); + + test(`${row.name} ends as a cancel, not a timeout, when the caller aborts`, async (t) => { + mock.timers.enable({ apis: ['setTimeout'] }); + t.after(() => mock.timers.reset()); + const caller = new AbortController(); + const out = watch(row.call({ fetchImpl: never, signal: caller.signal, ...(row.param ? { timeoutMs: ms } : {}) })); + await flush(); + mock.timers.tick(5); + await flush(); + assert.equal(out.done, false, 'settled with nobody asking'); + caller.abort(); + await advance(out, 0); + assert.ok(out.done, 'a caller abort did not end the call'); + assert.equal(looksLikeTimeout(out, row), false, `ended as a timeout: ${out.error?.message ?? JSON.stringify(out.value)}`); + }); +} + +// These four promise "never throws the map": a reader-asked layer whose area is missing or +// malformed reads as "no answer", it does not reject into the caller. Building the URL is part of +// the call, so it sits inside that promise too (B1 moved it out of the old try for three of them; +// fetchSmoke built it synchronously before any promise existed). +test('the never-throws adapters answer null for a missing or malformed area', async () => { + const fetchImpl = async () => { throw new Error('no network in this test'); }; + for (const [name, fn] of [['fetchAqi', fetchAqi], ['fetchFire', fetchFire], ['fetchAccess', fetchAccess], ['fetchSmoke', fetchSmoke]]) { + for (const bbox of [null, undefined, {}]) { + let got; + try { got = await fn(bbox, { fetchImpl }); } catch (e) { assert.fail(`${name}(${JSON.stringify(bbox)}) threw ${e.constructor.name}: ${e.message}`); } + assert.equal(got, null, `${name}(${JSON.stringify(bbox)})`); + } + } +}); + +test('the table covers every source module that reads from a service', async () => { + const { readdirSync } = await import('node:fs'); + const dir = new URL(M, import.meta.url); + // Modules whose adapters are tested above through their exports, or that only build + // requests, plus main.js (PR-B2) and the layer factories, which call these same adapters. + const covered = [ + 'arcgis-pager', 'source-map-index', 'source-terrain-names', 'source-stream-names', 'camera-layer', + 'source-aerial', 'source-terrain', 'source-elevation', 'source-aqi', 'source-aurora', 'source-ecoregions', + 'source-fire', 'source-smoke', 'source-flood-zones', 'source-gbif', 'source-hillshade', 'source-land-cover', + 'source-mvum', 'source-nws-alerts', 'source-nws-radar', 'source-padus', 'source-road-names', + 'source-terrain-surface', 'source-usgs-topo', 'source-weather', 'source-wetlands', 'source-blm-rec-sites', + 'source-county-lines', 'source-federal-land', 'source-rec-sites', 'source-road-basic', + 'source-surface-ownership', 'source-trails', 'source-land-boundaries', + ]; + // Source modules that reached master after QQA-13 and still make their own raw calls (no-raw-fetch.test.mjs + // lists them too). They have no deadline to pin yet; a row replaces the entry when each moves onto request.js. + const pending = [ + 'source-avalanche', 'source-nrhp', 'source-ntad-rail-airports', + 'source-nws-observations', 'source-snotel', 'source-stream-gauges', + ]; + const { readFileSync } = await import('node:fs'); + const reaching = readdirSync(dir).filter((f) => f.startsWith('source-') && f.endsWith('.js')) + .filter((f) => /\bfetchImpl\b|\bfetch:\s*fetchImpl/.test(readFileSync(new URL(f, dir), 'utf8'))) + .map((f) => f.replace(/\.js$/, '')); + assert.deepEqual(reaching.filter((n) => !covered.includes(n) && !pending.includes(n)), [], 'add a row for the new source module'); + assert.deepEqual(pending.filter((n) => !reaching.includes(n)), [], 'no longer reads from a service: drop it from pending'); +}); diff --git a/scripts/qcm/build.test.mjs b/scripts/qcm/build.test.mjs index 7377f9a..ba8bc81 100644 --- a/scripts/qcm/build.test.mjs +++ b/scripts/qcm/build.test.mjs @@ -62,12 +62,12 @@ function stubWorker() { } /** runBuild with every seam stubbed; returns the worker requests and what createPdf was handed. */ -async function build({ mvum = {}, roadQuality, layers, trails, land, sites, fetchImpl, names } = {}) { +async function build({ mvum = {}, roadQuality, layers, trails, land, sites, fetchImpl, names, signal } = {}) { const worker = stubWorker(); const seen = {}; const result = await runBuild({ recipe: layers ? { ...recipe, layers: { ...recipe.layers, ...layers } } : recipe, - plan, workerFactory: () => worker, PDFLib: {}, + plan, workerFactory: () => worker, PDFLib: {}, signal, adapters: { fetchImpl: fetchImpl || (async () => ({ ok: true, status: 200, arrayBuffer: async () => new Uint8Array([1, 2, 3]).buffer })), createImageBitmap: async () => ({ width: 256, height: 256, close() {} }), @@ -237,6 +237,39 @@ test('the aerial companion is built only when asked, and never fails the map', a assert.ok(broken.seen.sheets.length > 0, 'and the map still printed'); }); +// A cancel must reach the extras' own requests, and end the build instead of being read as +// "this extra failed" and carried on to the next one. +test('the build cancel reaches the aerial and elevation requests and ends the build', async () => { + const first = plan.sheets[0]; + const named = { ...featureOn(first, 'T-1', 'trail', 'yearlong'), symbolKey: 'trail-nst', name: 'Appalachian Trail' }; + const bytes = { ok: true, status: 200, arrayBuffer: async () => new Uint8Array([1, 2, 3]).buffer, json: async () => ({ samples: [] }) }; + // Cancel while one request is in flight, and record whether THAT request's signal followed. + const cancelDuring = async (needle, layers) => { + const cancel = new AbortController(); + const seen = { requestAborted: null, elevationAsked: 0 }; + await assert.rejects(build({ + layers, trails: { features: [named] }, signal: cancel.signal, + fetchImpl: async (url, init) => { + if (String(url).includes('getSamples')) seen.elevationAsked += 1; + if (String(url).includes(needle) && seen.requestAborted === null) { + cancel.abort(); + seen.requestAborted = init.signal.aborted; + throw new Error('cancelled mid-request'); + } + return bytes; + }, + }), { name: 'AbortError' }); + return seen; + }; + const aerial = await cancelDuring('imagery', { aerial: true, tripPage: true, trails: true }); + assert.equal(aerial.requestAborted, true, 'the aerial request never heard the cancel'); + assert.equal(aerial.elevationAsked, 0, 'a cancelled aerial sheet rolled on to the trip page'); + const relief = await cancelDuring('elevation.nationalmap.gov', { terrain: true, tripPage: true, trails: true }); + assert.equal(relief.requestAborted, true, 'the relief request never heard the cancel'); + const profile = await cancelDuring('getSamples', { tripPage: true, trails: true }); + assert.equal(profile.requestAborted, true, 'the elevation profile never heard the cancel'); +}); + // The two newest layers, at the seam the parity invariant actually broke at once: // a boundary reaches the SHEET through state.land and the preview through the same // list, so a federal boundary that prints must be a federal boundary that previews. diff --git a/scripts/qcm/no-raw-fetch.test.mjs b/scripts/qcm/no-raw-fetch.test.mjs new file mode 100644 index 0000000..4ba5de9 --- /dev/null +++ b/scripts/qcm/no-raw-fetch.test.mjs @@ -0,0 +1,106 @@ +// Every remote read in the map goes through request.js, so no module can forget its +// deadline or its cancel. This is the review-time half of that rule: a module that calls +// fetch( or fetchImpl( itself fails here, before a reader meets a request that never ends. +// It reads the source text only -- comments and string contents are blanked first, so +// prose that mentions fetch() is not a call. +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readdirSync, readFileSync } from 'node:fs'; +import { join } from 'node:path'; + +const DIR = new URL('../../themes/headlessmode/static/js/quiet-corners-maps/', import.meta.url); +const HELPER = 'request.js'; +// PR-B2 moves main.js onto request.js and removes this entry (and the list). Until then the +// page's own call sites (camera snapshot, the tile probe, the print adapters) are the only raw calls. +// +// The other seven entries are modules that reached master after QQA-13 was written (the E-P1..E-P6 +// field-map layers and the finder's street-address lookup). Each still owns a raw call and its own +// signal, so none has a deadline from request.js yet. Moving them onto fetchWithDeadline is a +// follow-up; until then they are listed so this test guards every other module, and the stale-entry +// test below drops an entry the day its raw call goes. +const ALLOW = [ + 'main.js', + 'address-geocode.js', + 'source-avalanche.js', 'source-nrhp.js', 'source-ntad-rail-airports.js', + 'source-nws-observations.js', 'source-snotel.js', 'source-stream-gauges.js', +]; + +// A call to fetch/fetchImpl that is not a method of something else (`x.fetch(` is not it) +// -- except the global spellings, which are the same call. +const RAW_CALL = /(?:^|[^A-Za-z0-9_$.])(?:globalThis\.|window\.|self\.)?(?:fetch|fetchImpl)\s*\(/; + +/** The source with comments, string contents and template text blanked to spaces, keeping + * `${ ... }` code and every newline, so a hit's line number is the file's. */ +function codeOnly(src) { + let out = ''; + const stack = []; // 'tpl' for an open template, 'expr' for ${ inside one + let i = 0; + const blank = (ch) => (ch === '\n' ? '\n' : ' '); + while (i < src.length) { + const ch = src[i], next = src[i + 1], top = stack[stack.length - 1]; + if (top === 'tpl') { + if (ch === '\\') { out += ' ' + blank(next); i += 2; continue; } + if (ch === '`') { stack.pop(); out += ' '; i += 1; continue; } + if (ch === '$' && next === '{') { stack.push('expr'); out += ' '; i += 2; continue; } + out += blank(ch); i += 1; continue; + } + if (ch === '/' && next === '/') { while (i < src.length && src[i] !== '\n') { out += ' '; i += 1; } continue; } + if (ch === '/' && next === '*') { + const end = src.indexOf('*/', i + 2), stop = end === -1 ? src.length : end + 2; + for (; i < stop; i += 1) out += blank(src[i]); + continue; + } + if (ch === "'" || ch === '"') { + out += ' '; i += 1; + while (i < src.length && src[i] !== ch && src[i] !== '\n') { if (src[i] === '\\') { out += ' '; i += 1; } out += blank(src[i]); i += 1; } + if (src[i] === ch) { out += ' '; i += 1; } + continue; + } + if (ch === '`') { stack.push('tpl'); out += ' '; i += 1; continue; } + if (top === 'expr' && ch === '}') { stack.pop(); out += ' '; i += 1; continue; } + if (top === 'expr' && ch === '{') { stack.push('expr'); out += ch; i += 1; continue; } + out += ch; i += 1; + } + return out; +} + +const rawCalls = (src) => codeOnly(src).split('\n') + .map((line, n) => (RAW_CALL.test(line) ? n + 1 : 0)).filter(Boolean); + +const files = readdirSync(DIR).filter((f) => f.endsWith('.js')).sort(); +const read = (f) => readFileSync(join(DIR.pathname, f), 'utf8'); + +test('the detector sees a call and ignores prose', () => { + assert.deepEqual(rawCalls('const r = await fetchImpl(url, { signal });'), [1]); + assert.deepEqual(rawCalls('x = fetch(u)\ny = globalThis.fetch(u)'), [1, 2]); + assert.deepEqual(rawCalls('await ok(`${fetch(a)}`)'), [1], 'code inside a template hole is code'); + assert.deepEqual(rawCalls('// fetch(u)\n/* fetchImpl(\n u) */\nconst s = "fetch(u)"; const t = `read via fetch() here`;'), []); + assert.deepEqual(rawCalls('const f = { fetchImpl, fetch: g }; thing.fetch(u); await fetchWithDeadline(u, {})'), []); + assert.deepEqual(rawCalls('fetchImpl = fetch, signal'), [], 'a default value is a reference, not a call'); +}); + +test('the scan covers the whole module directory', () => { + assert.ok(files.length > 60, `only ${files.length} files found`); + assert.ok(files.includes(HELPER) && files.includes('main.js')); +}); + +test('no module but request.js and the allow list calls fetch( or fetchImpl( itself', () => { + const offenders = []; + for (const f of files) { + if (f === HELPER || ALLOW.includes(f)) continue; + const lines = rawCalls(read(f)); + if (lines.length) offenders.push(`${f}:${lines.join(',')}`); + } + assert.deepEqual(offenders, [], 'route the read through fetchWithDeadline (request.js) with the source\'s registry timeoutMs'); +}); + +test('every allow list entry still holds a raw call, so a stale entry fails', () => { + for (const f of ALLOW) { + assert.ok(files.includes(f), `${f} is gone: drop it from ALLOW`); + assert.ok(rawCalls(read(f)).length > 0, `${f} no longer calls fetch( itself: drop it from ALLOW`); + } +}); + +test('request.js is where the network is reached', () => { + assert.match(read(HELPER), /await fetchImpl\(url,/); +}); diff --git a/scripts/qcm/registry.test.mjs b/scripts/qcm/registry.test.mjs index ddc135d..79a110d 100644 --- a/scripts/qcm/registry.test.mjs +++ b/scripts/qcm/registry.test.mjs @@ -270,6 +270,9 @@ test('every source carries a timeoutMs of 10000 or less unless it cites a measur } } assert.equal(SOURCES.gnisNames.timeoutMs, 20000); + // The paged reads were measured against a 20 s page, so they keep it rather than the 10 s default. + assert.equal(SOURCES.fwsNwiWetlands.timeoutMs, 20000); + assert.equal(SOURCES.usgsGovUnits.timeoutMs, 20000); assert.equal(SOURCES.nwsForecast.timeoutMs, 10000); assert.equal(SOURCES.nwsAlerts.timeoutMs, 10000); assert.equal(SOURCES.usgsQuadIndex.timeoutMs, 10000); diff --git a/scripts/qcm/request.test.mjs b/scripts/qcm/request.test.mjs index 478ffdc..55f867b 100644 --- a/scripts/qcm/request.test.mjs +++ b/scripts/qcm/request.test.mjs @@ -4,7 +4,7 @@ import { test, mock } from 'node:test'; import assert from 'node:assert/strict'; import { - fetchWithDeadline, withDeadline, createLatest, createSearchGuard, + fetchWithDeadline, withDeadline, createLatest, createSearchGuard, orThrow, } from '../../themes/headlessmode/static/js/quiet-corners-maps/request.js'; import { createForecastLoader } from '../../themes/headlessmode/static/js/quiet-corners-maps/source-weather.js'; import { SOURCES } from '../../themes/headlessmode/static/js/quiet-corners-maps/source-registry.js'; @@ -189,3 +189,44 @@ test('fetchWithDeadline with no timeoutMs returns an error rather than throwing' assert.match(result.error, /timeoutMs/); assert.equal(called, false); }); + +test('anyStatus reads the body of a non-2xx answer; without it the body is never read', async () => { + const problem = { detail: 'No forecast for this point.' }; + let reads = 0; + const notFound = { ok: false, status: 404, json: async () => { reads += 1; return problem; } }; + const plain = await fetchWithDeadline('https://x.invalid/a', { fetchImpl: async () => notFound, timeoutMs: 1000 }); + assert.equal(plain.ok, false); + assert.equal(plain.httpStatus, 404); + assert.equal(plain.body, undefined); + assert.equal(reads, 0); + const any = await fetchWithDeadline('https://x.invalid/a', { fetchImpl: async () => notFound, timeoutMs: 1000, anyStatus: true }); + assert.equal(any.ok, false); + assert.equal(any.status, 'error'); + assert.equal(any.httpStatus, 404); + assert.deepEqual(any.body, problem); + const unreadable = { ok: false, status: 502, json: async () => { throw new SyntaxError(''); } }; + const bad = await fetchWithDeadline('https://x.invalid/a', { fetchImpl: async () => unreadable, timeoutMs: 1000, anyStatus: true }); + assert.equal(bad.httpStatus, 502); + assert.equal(bad.body, undefined); + assert.equal(bad.malformed, undefined, 'a bad body on an error status is the error, not a malformed answer'); + const okBad = { ok: true, status: 200, json: async () => { throw new SyntaxError(''); } }; + const malformed = await fetchWithDeadline('https://x.invalid/a', { fetchImpl: async () => okBad, timeoutMs: 1000, anyStatus: true }); + assert.equal(malformed.malformed, true); +}); + +test('a custom reader that fails is not reported as bad JSON', async () => { + const res = { ok: true, status: 200, arrayBuffer: async () => { throw new Error('stream cut'); } }; + const got = await fetchWithDeadline('https://x.invalid/a', { fetchImpl: async () => res, timeoutMs: 1000, read: (r) => r.arrayBuffer() }); + assert.equal(got.malformed, true); + assert.doesNotMatch(got.error, /JSON/); +}); + +test('orThrow hands back an ok body and throws everything else with its status', async () => { + assert.deepEqual(orThrow({ ok: true, status: 'ok', body: { a: 1 } }), { a: 1 }); + assert.throws(() => orThrow({ ok: false, status: 'error', httpStatus: 503, error: 'HTTP 503' }), + (err) => err.message === 'HTTP 503' && err.status === 'error' && err.httpStatus === 503); + assert.throws(() => orThrow({ ok: false, status: 'timeout', error: 'No answer within 5 ms.' }, 'Elevation: slow.'), + (err) => err.message === 'Elevation: slow.' && err.status === 'timeout' && err.httpStatus === undefined); + assert.throws(() => orThrow({ ok: false, status: 'aborted', error: 'aborted' }), + (err) => err.name === 'AbortError' && err.status === 'aborted'); +}); diff --git a/themes/headlessmode/static/js/quiet-corners-maps/address-geocode.js b/themes/headlessmode/static/js/quiet-corners-maps/address-geocode.js index 2944006..692a748 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/address-geocode.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/address-geocode.js @@ -13,8 +13,8 @@ // logged, stored, or returned in a status; the caller keeps it out of the share URL, // saved views, the title and analytics. The request carries no cookies and no referrer. // -// There is no shared request helper on this base (no request.js / fetchWithDeadline), -// so the deadline is kept here, in this one function. +// request.js (fetchWithDeadline) is the shared helper now. This function still keeps its own +// deadline and cancel handling, so no-raw-fetch.test.mjs lists it until it moves onto it. export const GEOCODE_PATH = '/api/geocode'; /** Client-side deadline; the function's own upstream deadline is 8 s. */ diff --git a/themes/headlessmode/static/js/quiet-corners-maps/arcgis-pager.js b/themes/headlessmode/static/js/quiet-corners-maps/arcgis-pager.js index 1d4cce8..3273ee8 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/arcgis-pager.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/arcgis-pager.js @@ -6,7 +6,7 @@ // page (Delaware, 26,863 against a 2,000 cap). This existed once, inside the state // land fetcher; the federal layers need the same loop, and two hand-written copies // of a paging protocol is the drift this file exists to prevent. -import { REQUEST_POLICY } from './source-registry.js'; +import { DEFAULT_SOURCE_TIMEOUT_MS } from './source-registry.js'; import { fetchWithDeadline } from './request.js'; /** A page bigger than any measured service allows, and a hard stop on the loop. @@ -17,9 +17,8 @@ export const MAX_PAGES = 20; /** Read every page of one ArcGIS query. * - * Each page is its own request with its own `timeoutMs`. The default is the old - * REQUEST_POLICY ceiling: callers that have not yet passed their source's own timeout - * keep the 20 s the wetlands and federal-land measurements were taken against. + * Each page is its own request with its own `timeoutMs`. Callers pass their + * source's registry `timeoutMs`; one that does not gets the registry default. * A timeout or a caller cancel throws, like any other failed page. * * `normalize` turns a raw feature into this map's shape and may return null to @@ -29,7 +28,7 @@ export const MAX_PAGES = 20; */ export async function pageThrough({ url, params, normalize, label = 'the service', - fetchImpl = fetch, signal, pageSize = PAGE_CAP, maxPages = MAX_PAGES, timeoutMs = REQUEST_POLICY.timeoutMs, + fetchImpl = fetch, signal, pageSize = PAGE_CAP, maxPages = MAX_PAGES, timeoutMs = DEFAULT_SOURCE_TIMEOUT_MS, } = {}) { const size = Math.max(1, Math.min(pageSize, PAGE_CAP)); const features = [], seen = new Set(); diff --git a/themes/headlessmode/static/js/quiet-corners-maps/build.js b/themes/headlessmode/static/js/quiet-corners-maps/build.js index 1cbab56..605aef2 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/build.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/build.js @@ -70,12 +70,14 @@ const labelTextFor = (features) => { * sheet whose image fails is simply absent from the set: the map sheets are * already built and must still print. One function, not one per companion kind -- * the only thing that differs is which service is called. */ -async function companionSheets(plan, recipe, fetchImpl, check, fetchOne) { +async function companionSheets(plan, recipe, fetchImpl, check, fetchOne, signal) { const out = []; for (const sheet of plan.sheets) { check(); - try { out.push(await fetchOne({ sheet, bboxLonLat: recipe.bboxLonLat, fetchImpl })); } + try { out.push(await fetchOne({ sheet, bboxLonLat: recipe.bboxLonLat, fetchImpl, signal })); } catch { /* an extra page, never the map */ } + // A cancel ends the request as an error like any other; it must not read as "this extra failed". + check(); } return out; } @@ -84,15 +86,16 @@ async function companionSheets(plan, recipe, fetchImpl, check, fetchOne) { * Unnamed geometry is left out — a table of "unnamed" rows tells nobody anything. * Elevation is sampled per route, so this is capped rather than unbounded. */ const MAX_TRIP_ROUTES = 12; -async function tripRows(features, plan, sheets, fetchImpl, { zone, hemisphere, scale }) { +async function tripRows(features, plan, sheets, fetchImpl, { zone, hemisphere, scale, signal, check }) { const named = features.filter((f) => f.name && (f.paths || []).length).slice(0, MAX_TRIP_ROUTES); const rows = []; for (const f of named) { const on = plan.sheets.filter((sheet) => vectorsForSheet([f], sheet, { zone, hemisphere, scale }).vectors.length).map((s) => s.id); let profile = { lengthM: null, gainM: null, sampled: false }; // Climb is detail: a 3DEP outage costs the column, not the row or the page. - try { profile = { ...(await profileFor(f, { fetchImpl })), sampled: true }; } + try { profile = { ...(await profileFor(f, { fetchImpl, signal })), sampled: true }; } catch { /* leave the climb blank rather than guess it */ } + check(); rows.push({ id: f.id, name: f.name, useCode: f.useCode || null, sheets: on, ...profile }); } return rows; @@ -259,9 +262,9 @@ export async function runBuild({ recipe, plan, adapters, workerFactory, PDFLib, // Both extras are built HERE rather than handed in: this is the only place that // knows the sheets and the clipped features. Both are wrapped so a failure costs // the extra page and never the map — that is what makes them extras. - const aerials = recipe.layers?.aerial ? await companionSheets(plan, recipe, fetchImpl, check, fetchAerial) : []; - const companions = recipe.layers?.terrain ? await companionSheets(plan, recipe, fetchImpl, check, fetchTerrain) : []; - const trips = recipe.layers?.tripPage ? await tripRows(features, plan, sheets, fetchImpl, { zone, hemisphere, scale: recipe.scale }) : []; + const aerials = recipe.layers?.aerial ? await companionSheets(plan, recipe, fetchImpl, check, fetchAerial, signal) : []; + const companions = recipe.layers?.terrain ? await companionSheets(plan, recipe, fetchImpl, check, fetchTerrain, signal) : []; + const trips = recipe.layers?.tripPage ? await tripRows(features, plan, sheets, fetchImpl, { zone, hemisphere, scale: recipe.scale, signal, check }) : []; const pdfBytes = await createPdfImpl({ PDFLib, plan, sheets, meta, legend, manifest, trips, aerials, companions }); progress(PHASES.ASSEMBLE, pages, pages); return { pdfBytes, manifest, stats: { level, tileCount: tiles.count, sheets: sheets.length, ms: Date.now() - started, bytes: pdfBytes.byteLength } }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/camera-layer.js b/themes/headlessmode/static/js/quiet-corners-maps/camera-layer.js index 0d5451b..ff1f759 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/camera-layer.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/camera-layer.js @@ -8,6 +8,7 @@ // page no longer calls it: the road list is the build-time snapshot (cameras/index.json plus one // file per area), read by map area with filesInView; the function stays exported and tested. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; import { normalizeCaltransCctv, isLinkOnly } from './source-camera.js'; const CALTRANS_OPERATOR = 'Caltrans'; @@ -76,14 +77,13 @@ export function failureProvider(files, view, layer, fallback) { /** GETs the registry's own caltransCctv.serviceUrl and normalizes the result. Never throws -- * a network failure, a non-ok response or a JSON parse error all become { ok:false, reason }. */ export async function fetchCaltransCctv({ fetchImpl = fetch, signal, fetchedAt } = {}) { + const got = await fetchWithDeadline(SOURCES.caltransCctv.serviceUrl, { fetchImpl, signal, timeoutMs: SOURCES.caltransCctv.timeoutMs }); + if (!got.ok) { + if (got.httpStatus) return { ok: false, reason: `Caltrans's server returned ${got.httpStatus}.` }; + return { ok: false, reason: got.error || 'The request to Caltrans failed.' }; + } try { - const res = await fetchImpl(SOURCES.caltransCctv.serviceUrl, { signal }); - if (!res || !res.ok) { - const status = res && res.status; - return { ok: false, reason: status ? `Caltrans's server returned ${status}.` : "Caltrans's server did not respond." }; - } - const payload = await res.json(); - return { ok: true, cameras: normalizeCaltransCctv(payload, { fetchedAt }) }; + return { ok: true, cameras: normalizeCaltransCctv(got.body, { fetchedAt }) }; } catch (err) { return { ok: false, reason: (err && err.message) || 'The request to Caltrans failed.' }; } diff --git a/themes/headlessmode/static/js/quiet-corners-maps/request.js b/themes/headlessmode/static/js/quiet-corners-maps/request.js index 64269af..873497c 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/request.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/request.js @@ -40,8 +40,11 @@ const readJson = (res) => res.json(); * - `{ ok: false, status: 'aborted', error }` -- the caller's own `signal` ended it. * - `{ ok: false, status: 'error', error }` -- network failure, a non-2xx (`httpStatus` * is set), or a body that is not JSON (`malformed: true`). - * `init` is passed to fetch beside the deadline signal; `read` swaps the body reader. */ -export async function fetchWithDeadline(url, { fetchImpl = globalThis.fetch, signal, timeoutMs, init, read = readJson } = {}) { + * `init` is passed to fetch beside the deadline signal; `read` swaps the body reader. + * `anyStatus: true` is for a service whose error answer carries a reason worth showing: the + * body is read whatever the status, `ok` is the response's own, and a body that cannot be + * read on a non-2xx is `undefined` rather than malformed. */ +export async function fetchWithDeadline(url, { fetchImpl = globalThis.fetch, signal, timeoutMs, init, read = readJson, anyStatus = false } = {}) { if (signal && signal.aborted) return { ok: false, status: 'aborted', error: 'aborted' }; // A missing or bad deadline is a caller bug, but this call promises never to throw, so it // comes back as an error the panel can show instead of an exception that kills the panel. @@ -58,12 +61,14 @@ export async function fetchWithDeadline(url, { fetchImpl = globalThis.fetch, sig }); const work = (async () => { const res = await fetchImpl(url, { ...init, signal: deadline.signal }); - if (!res.ok) return { ok: false, status: 'error', httpStatus: res.status, error: `HTTP ${res.status}` }; + if (!res.ok && !anyStatus) return { ok: false, status: 'error', httpStatus: res.status, error: `HTTP ${res.status}` }; let body; try { body = await read(res); } catch (err) { if (deadline.signal.aborted) throw err; - return { ok: false, status: 'error', malformed: true, error: 'Malformed JSON response.' }; + if (!res.ok) return { ok: false, status: 'error', httpStatus: res.status, error: `HTTP ${res.status}` }; + return { ok: false, status: 'error', malformed: true, error: read === readJson ? 'Malformed JSON response.' : 'Unreadable response body.' }; } + if (!res.ok) return { ok: false, status: 'error', httpStatus: res.status, error: `HTTP ${res.status}`, body }; return { ok: true, status: 'ok', httpStatus: res.status, body }; })(); try { @@ -80,6 +85,18 @@ export async function fetchWithDeadline(url, { fetchImpl = globalThis.fetch, sig } } +/** For a caller that throws on failure: the body of an ok result, else an Error carrying the + * result's `status` and `httpStatus`, with `message` (default: the result's own error). A + * caller's cancel keeps the name `AbortError`, so it is never reported as a failure. */ +export function orThrow(result, message) { + if (result.ok) return result.body; + const err = new Error(message || result.error); + err.status = result.status; + if (result.httpStatus !== undefined) err.httpStatus = result.httpStatus; + if (result.status === 'aborted') err.name = 'AbortError'; + throw err; +} + /** Latest wins. `begin()` aborts whatever the previous call was doing and hands out a * token and a fresh signal; `isCurrent(token)` is false once a newer call has begun. * `run(task)` wraps both: `task(signal)` runs, and the answer is `{ stale, value }` -- diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-aerial.js b/themes/headlessmode/static/js/quiet-corners-maps/source-aerial.js index 99835d5..c429f00 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-aerial.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-aerial.js @@ -11,6 +11,9 @@ // hashes -- real, distinct imagery. That check exists because 3DEP's exportImage // returned a BYTE-IDENTICAL image for two different states, and a source that // answers every request with the same picture is worse than one that errors. +import { fetchWithDeadline, orThrow } from './request.js'; +import { SOURCES } from './source-registry.js'; + const SERVICE = 'https://imagery.nationalmap.gov/arcgis/rest/services/USGSNAIPPlus/ImageServer'; // A sheet's raster at 300 dpi is 2400x2850, which NAIP will happily render but @@ -45,9 +48,10 @@ export function aerialRequestFor(sheet, bboxLonLat) { * sheet is an extra, so a failure loses the companion page, not the map. */ export async function fetchAerial({ sheet, bboxLonLat, fetchImpl = fetch, signal } = {}) { const req = aerialRequestFor(sheet, bboxLonLat); - const res = await fetchImpl(`${req.url}?${new URLSearchParams(req.params)}`, { signal }); - if (!res.ok) throw new Error(`Aerial imagery: the service answered ${res.status}.`); - const bytes = new Uint8Array(await res.arrayBuffer()); + const got = await fetchWithDeadline(`${req.url}?${new URLSearchParams(req.params)}`, { + fetchImpl, signal, timeoutMs: SOURCES.usdaNaip.timeoutMs, read: (res) => res.arrayBuffer(), + }); + const bytes = new Uint8Array(orThrow(got, got.httpStatus ? `Aerial imagery: the service answered ${got.httpStatus}.` : undefined)); if (!bytes.length) throw new Error('Aerial imagery: the service returned an empty image.'); return { id: sheet.id, bytes, width: req.width, height: req.height, format: 'jpg' }; } diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-aqi.js b/themes/headlessmode/static/js/quiet-corners-maps/source-aqi.js index 6411fec..250f813 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-aqi.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-aqi.js @@ -16,6 +16,7 @@ // National AQI Forecast Updated Nightly", boilerplate from another service, and matching it // would be the aurora layer's mistake run backwards. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.epaAirNowAqi; export const SERVICE = SRC.serviceUrl; @@ -106,10 +107,9 @@ export function quietText() { /** Asked for only when a reader asks, and never throws the map. */ export async function fetchAqi(bbox, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(aqiQueryUrl(bbox), { signal }); - return res.ok ? await res.json() : null; - } catch (e) { return null; } + let url; try { url = aqiQueryUrl(bbox); } catch { return null; } // a bad area is "no answer" too + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + return got.ok ? got.body : null; } /** Every string this source can put in front of a reader, in one place, for the test. */ diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-aurora.js b/themes/headlessmode/static/js/quiet-corners-maps/source-aurora.js index 478d5d7..5f473f6 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-aurora.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-aurora.js @@ -17,6 +17,7 @@ // 65N. So nothing fetches it on first render, and "nothing to show" is the normal // answer rather than a failure to report. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; export const AURORA_SOURCE = SOURCES.swpcAurora; export const SERVICE = AURORA_SOURCE.serviceUrl; @@ -95,13 +96,8 @@ export function quietText() { /** Fetched only when a reader asks for it -- ~900 KB for the whole globe. Never throws * the map: a service that will not answer leaves the rest of the page alone. */ export async function fetchAurora({ fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(SERVICE, { signal }); - if (!res.ok) return null; - return await res.json(); - } catch (e) { - return null; - } + const got = await fetchWithDeadline(SERVICE, { fetchImpl, signal, timeoutMs: AURORA_SOURCE.timeoutMs }); + return got.ok ? got.body : null; } /** Every string this layer can put in front of a reader, in one place, so a test can diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-blm-rec-sites.js b/themes/headlessmode/static/js/quiet-corners-maps/source-blm-rec-sites.js index 4c53422..7be09b6 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-blm-rec-sites.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-blm-rec-sites.js @@ -117,7 +117,7 @@ export async function fetchBlmRecSites({ bboxLonLat, fetchImpl = fetch, signal, inSR: '4326', outSR: '4326', spatialRel: 'esriSpatialRelIntersects', outFields: FIELDS.join(','), returnGeometry: 'true', }, - normalize: normalizeBlmSite, fetchImpl, signal, pageSize, + normalize: normalizeBlmSite, fetchImpl, signal, timeoutMs: SOURCES.blmRecSites.timeoutMs, pageSize, }); const { features, ...rest } = page; return { sites: features, ...rest }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-county-lines.js b/themes/headlessmode/static/js/quiet-corners-maps/source-county-lines.js index d56afa1..7529ed3 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-county-lines.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-county-lines.js @@ -22,6 +22,7 @@ // of its own. It also obliges no access notice -- it makes no claim about who may be // on the ground, only about which county records that ground. import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; +import { SOURCES } from './source-registry.js'; import { SERVICE } from './source-federal-land.js'; export const LAYER_ID = 'countyLines'; @@ -83,6 +84,7 @@ export async function fetchCountyLines({ bboxLonLat, fetchImpl = fetch, signal, }, fetchImpl, signal, + timeoutMs: SOURCES.usgsGovUnits.timeoutMs, pageSize, }); return { ...page, skipped: null }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-ecoregions.js b/themes/headlessmode/static/js/quiet-corners-maps/source-ecoregions.js index 7b7a6e1..7082c23 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-ecoregions.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-ecoregions.js @@ -20,6 +20,9 @@ // to domestic copyright protection under 17 U.S.C. § 105." Redistributing this artifact // from our own hosting is what that sentence permits; TERMS_URL is the page it is on. +import { fetchWithDeadline } from './request.js'; +import { DEFAULT_SOURCE_TIMEOUT_MS } from './source-registry.js'; + export const LAYER_ID = 'epaEcoregions'; /** Same-origin, so no host reaches the policy. Fetched once, when a reader asks. */ export const EXPORT_URL = '/data/quiet-corners/ecoregions-l3.json'; @@ -124,11 +127,9 @@ export function areaReport(regions, point) { /** One national file, fetched when a reader asks and held. A pan is not a new question when * the answer is the whole country. Never throws the map. */ export async function loadEcoregions({ fetchImpl = fetch, url = EXPORT_URL, signal } = {}) { - try { - const res = await fetchImpl(url, { signal }); - if (!res.ok) return null; - return readExport(await res.json()); - } catch (e) { return null; } + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs: DEFAULT_SOURCE_TIMEOUT_MS }); + if (!got.ok) return null; + try { return readExport(got.body); } catch (e) { return null; } } /** Every string this source can put in front of a reader, in one place, for the test. */ diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-elevation.js b/themes/headlessmode/static/js/quiet-corners-maps/source-elevation.js index 782e971..64693fd 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-elevation.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-elevation.js @@ -17,6 +17,8 @@ // built on that would have printed nothing while appearing to work. The tint is // blocked on a spike, not shipped broken. See the plan's §4 note. import { measureLength } from './measure.js'; +import { fetchWithDeadline, orThrow } from './request.js'; +import { SOURCES } from './source-registry.js'; const SERVICE = 'https://elevation.nationalmap.gov/arcgis/rest/services/3DEPElevation/ImageServer'; @@ -82,9 +84,10 @@ export async function fetchElevations({ points, fetchImpl = fetch, signal } = {} returnFirstValueOnly: 'true', f: 'json', }); - const res = await fetchImpl(`${SERVICE}/getSamples`, { method: 'POST', body, signal }); - if (!res.ok) throw new Error(`Elevation: the service answered ${res.status}.`); - const json = await res.json(); + const got = await fetchWithDeadline(`${SERVICE}/getSamples`, { + fetchImpl, signal, timeoutMs: SOURCES.usgs3dep.timeoutMs, init: { method: 'POST', body }, + }); + const json = orThrow(got, got.httpStatus ? `Elevation: the service answered ${got.httpStatus}.` : undefined); if (json.error) throw new Error(`Elevation: ${json.error.message || 'the service refused the query.'}`); // getSamples may return fewer samples than points asked for (NoData is // omitted, not nulled), so place each by its own index rather than by order. diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-federal-land.js b/themes/headlessmode/static/js/quiet-corners-maps/source-federal-land.js index df8da1e..fa44129 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-federal-land.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-federal-land.js @@ -17,6 +17,7 @@ // lines too — a National Forest proclamation boundary in particular encloses // private inholdings that were never federal. import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; +import { SOURCES } from './source-registry.js'; export const SERVICE = 'https://carto.nationalmap.gov/arcgis/rest/services/govunits/MapServer'; @@ -108,6 +109,7 @@ export async function fetchFederalLand({ family, bboxLonLat, fetchImpl = fetch, }, fetchImpl, signal, + timeoutMs: SOURCES.usgsGovUnits.timeoutMs, pageSize, }); return { ...page, skipped: null }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-fire.js b/themes/headlessmode/static/js/quiet-corners-maps/source-fire.js index c0e9f63..df6cd46 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-fire.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-fire.js @@ -21,6 +21,7 @@ // contained -- yet 56 of 188 measured 100 per cent and 38 carried null. // `attr_ContainmentDateTime` was null on ALL 188 and is never read. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; import { serviceError, envelope, viewReader } from './arcgis-view.js'; const SRC = SOURCES.nifcFirePerimeters; @@ -184,10 +185,9 @@ export function fireLayer(L) { /** Asked for only when a reader asks, and never throws the map. */ export async function fetchFire(bbox, { fetchImpl = fetch, signal, widthPx } = {}) { - try { - const res = await fetchImpl(fireQueryUrl(bbox, widthPx), { signal }); - return res.ok ? await res.json() : null; - } catch { return null; } + let url; try { url = fireQueryUrl(bbox, widthPx); } catch { return null; } // a bad area is "no answer" too + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + return got.ok ? got.body : null; } /** One bbox question at a time, waiting out a drag — the same protocol as the other bounded surfaces, diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-flood-zones.js b/themes/headlessmode/static/js/quiet-corners-maps/source-flood-zones.js index 8e9abe8..4ae1140 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-flood-zones.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-flood-zones.js @@ -21,6 +21,7 @@ // `features` is simply absent, so an adapter that trusts `res.ok` reports a flood plain as // empty ground. Every body goes through readError first. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.femaNfhl; const LAYER = (id) => SRC.layers.find((l) => l.id === id); @@ -252,10 +253,8 @@ export function floodZonesLayer(L) { } const getJson = async (req, fetchImpl, signal) => { - try { - const res = await fetchImpl(queryUrl(req), { signal }); - return res.ok ? await res.json() : null; - } catch (e) { return null; } + const got = await fetchWithDeadline(queryUrl(req), { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + return got.ok ? got.body : null; }; /** One view's worth of questions, in the order that makes the expensive one conditional: diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-gbif.js b/themes/headlessmode/static/js/quiet-corners-maps/source-gbif.js index 784b452..e27e554 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-gbif.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-gbif.js @@ -19,6 +19,7 @@ // // Every reader-facing string is built here so gbif.test.mjs can scan the lot. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.gbifOccurrence; export const SERVICE = SRC.serviceUrl; @@ -239,10 +240,8 @@ export function gbifLayer(L) { } export async function fetchJson(url, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(url, { signal }); - return res.ok ? await res.json() : null; - } catch (e) { return null; } + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + return got.ok ? got.body : null; } /** One cell at a time, and only for the view the reader stopped on. The same shape as the AQI diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-hillshade.js b/themes/headlessmode/static/js/quiet-corners-maps/source-hillshade.js index 754fba6..54262fe 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-hillshade.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-hillshade.js @@ -14,8 +14,7 @@ // // Cancellation/staleness follows source-terrain-surface.js's `live` closure pattern (an // in-flight request for a view/light the reader has left is dropped, never drawn); the request -// timeout follows source-terrain-names.js's `fetchJson` 20s AbortController pattern -// (REQUEST_POLICY.timeoutMs). No automatic retry loop: a slow/failed request reports its +// timeout is request.js's deadline at the source's registry `timeoutMs`. No automatic retry loop: a slow/failed request reports its // status and stops. // // "Sun now" is a pure function of lat/lon/date -- quiet-corners-sky/sun-moon.js exports @@ -26,7 +25,8 @@ // // The one sentence this layer must never be read as contradicting: hillshade is not a // cast-shadow, visibility, or passability calculation. -import { SOURCES, REQUEST_POLICY } from './source-registry.js'; +import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; import { SYMBOLS } from './legend.js'; const SRC = SOURCES.usgs3depHillshade; @@ -93,32 +93,19 @@ export function hillshadeImageUrl(request) { /** A generation-guarded, timeout-guarded image fetcher: a response for a bounds/light the * reader has already left is reported `stale`, never drawn, and a request past `timeoutMs` - * (default `REQUEST_POLICY.timeoutMs`, the same 20s used across this directory) aborts and - * reports `timeout`. No automatic retry -- a caller that wants another attempt asks again. */ -export function createHillshadeFetcher({ fetchImpl = fetch, timeoutMs = REQUEST_POLICY.timeoutMs } = {}) { + * (default the source's registry `timeoutMs`) aborts and reports `timeout`. No automatic + * retry -- a caller that wants another attempt asks again. */ +export function createHillshadeFetcher({ fetchImpl = fetch, timeoutMs = SRC.timeoutMs } = {}) { let generation = 0; async function fetchImage(request, { signal } = {}) { generation += 1; const myGeneration = generation; - const controller = new AbortController(); - const onAbort = () => controller.abort(); - if (signal) { if (signal.aborted) onAbort(); else signal.addEventListener('abort', onAbort); } - const timer = setTimeout(() => controller.abort(), timeoutMs); - try { - const res = await fetchImpl(hillshadeImageUrl(request), { signal: controller.signal }); - if (generation !== myGeneration) return { status: 'stale' }; - if (!res.ok) return { status: 'error', error: `HTTP ${res.status}` }; - const blob = await res.blob(); - if (generation !== myGeneration) return { status: 'stale' }; - return { status: 'ready', blob }; - } catch (err) { - if (generation !== myGeneration) return { status: 'stale' }; - const timedOut = err?.name === 'AbortError' && !(signal && signal.aborted); - return { status: timedOut ? 'timeout' : 'error', error: err?.message || String(err) }; - } finally { - clearTimeout(timer); - if (signal) signal.removeEventListener('abort', onAbort); - } + const got = await fetchWithDeadline(hillshadeImageUrl(request), { + fetchImpl, signal, timeoutMs, read: (res) => res.blob(), + }); + if (generation !== myGeneration) return { status: 'stale' }; + if (got.ok) return { status: 'ready', blob: got.body }; + return { status: got.status === 'timeout' ? 'timeout' : 'error', error: got.error }; } return { fetchImage }; } diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-land-boundaries.js b/themes/headlessmode/static/js/quiet-corners-maps/source-land-boundaries.js index 68ad5ad..e5013ce 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-land-boundaries.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-land-boundaries.js @@ -18,6 +18,7 @@ // catalogue, which was measured against the live services. import { layerFor, whereFor, outFieldsFor, isBrowserFetchable, isSemanticallyUniform, isDrawable } from './state-sources.js'; import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; +import { DEFAULT_SOURCE_TIMEOUT_MS } from './source-registry.js'; // The notice a drawn boundary obliges lives with the symbols it attaches to. export { LAND_COVERAGE_SEMANTICS } from './legend.js'; @@ -165,6 +166,7 @@ export async function fetchLandBoundaries({ state, family, bboxLonLat, fetchImpl }, fetchImpl, signal, + timeoutMs: DEFAULT_SOURCE_TIMEOUT_MS, pageSize: Math.min(pageSize, layer.maxRecordCount || PAGE_CAP), }); return { ...page, skipped: null }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-land-cover.js b/themes/headlessmode/static/js/quiet-corners-maps/source-land-cover.js index bbe6179..b203949 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-land-cover.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-land-cover.js @@ -41,6 +41,7 @@ // Wisconsin; its colour is measured off the legend swatch, its NUMBER is read off the // legend's order, and it says so rather than passing as known. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; export const PRODUCTS = Object.freeze({ landCover: SOURCES.nlcdLandCover, canopy: SOURCES.nlcdTreeCanopy }); export const PRODUCT_IDS = Object.freeze(Object.keys(PRODUCTS)); @@ -91,6 +92,7 @@ export const LAND_COVER_CLASSES = Object.freeze({ /** The service echoes the request Origin and sets `access-control-allow-credentials: true` * rather than `*`. It is keyless and anonymous and nothing here has an account with it, so * every request says so explicitly instead of relying on a default. */ +const timeoutFor = (request) => (PRODUCTS[request && request.product] || PRODUCTS.landCover).timeoutMs; export const FETCH_OPTIONS = Object.freeze({ credentials: 'omit' }); /** The year, or a throw. Nothing in this file can build a request without going through it. */ @@ -323,19 +325,19 @@ export function readerStrings() { * URL: `img-src` already allows blob:, so the whole pair costs the policy one connect-src * host -- 24 characters -- instead of 48. Never throws the map. */ export async function fetchSurface(request, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(surfaceImageUrl(request), { ...FETCH_OPTIONS, signal }); - return res && res.ok ? await res.blob() : null; - } catch (e) { return null; } + const got = await fetchWithDeadline(surfaceImageUrl(request), { + fetchImpl, signal, timeoutMs: timeoutFor(request), init: FETCH_OPTIONS, read: (res) => res.blob(), + }); + return got.ok ? got.body : null; } /** A point read, as JSON. `res.ok` is deliberately not the test for success -- the caller * reads the body, because this service states its errors inside a 200. */ export async function fetchPoint(request, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(pointUrl(request), { ...FETCH_OPTIONS, signal }); - return await res.json(); - } catch (e) { return null; } + const got = await fetchWithDeadline(pointUrl(request), { + fetchImpl, signal, timeoutMs: timeoutFor(request), init: FETCH_OPTIONS, anyStatus: true, + }); + return got.body ?? null; } /** A Leaflet image overlay re-exported for the view it is asked about, the same shape as the diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-mvum.js b/themes/headlessmode/static/js/quiet-corners-maps/source-mvum.js index 0cfa161..61d977f 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-mvum.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-mvum.js @@ -1,7 +1,8 @@ /* Quiet Corners Field Maps — USFS Motor Vehicle Use Map adapter. Pure: bounded envelope queries, paging while * the service flags `exceededTransferLimit`, a count cross-check, vehicle/date normalization, honest coverage - * (complete / partial / unavailable / unknown). Imports the registry only; network, clock and sleep are injected. */ + * (complete / partial / unavailable / unknown). Imports the registry and request.js only; network, clock and sleep are injected. */ import { SOURCES, REQUEST_POLICY, FAILURE } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; export const LAYERS = Object.freeze({ roads: 1, trails: 2 }); export const OUT_FIELDS = Object.freeze([ @@ -88,25 +89,22 @@ async function requestJson(url, { fetchImpl, signal, timeoutMs, retries, sleep } for (let attempt = 0; attempt <= retries; attempt += 1) { if (signal && signal.aborted) return { ok: false, error: 'aborted' }; if (attempt > 0) await sleep(REQUEST_POLICY.retryBackoffMs[Math.min(attempt - 1, REQUEST_POLICY.retryBackoffMs.length - 1)]); - const ctl = new AbortController(); const onAbort = () => ctl.abort(); - if (signal) signal.addEventListener('abort', onAbort, { once: true }); - const timer = setTimeout(onAbort, timeoutMs); - try { - const res = await fetchImpl(url, { signal: ctl.signal }); - if (!res.ok) { last = `HTTP ${res.status}`; if (res.status === 429 || res.status >= 500) continue; return { ok: false, error: last }; } - let body; try { body = await res.json(); } catch { return { ok: false, error: 'malformed JSON' }; } - if (body && body.error) return { ok: false, error: `service error ${body.error.code ?? ''}`.trim() }; - return { ok: true, body: body || {} }; - } catch (e) { - if (signal && signal.aborted) return { ok: false, error: 'aborted' }; - last = ctl.signal.aborted ? `timeout after ${timeoutMs} ms` : `network error: ${e && e.message}`; - } finally { clearTimeout(timer); if (signal) signal.removeEventListener('abort', onAbort); } + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs }); + if (got.status === 'aborted') return { ok: false, error: 'aborted' }; + if (!got.ok) { + if (got.httpStatus) { last = `HTTP ${got.httpStatus}`; if (got.httpStatus === 429 || got.httpStatus >= 500) continue; return { ok: false, error: last }; } + if (got.malformed) return { ok: false, error: 'malformed JSON' }; + last = got.status === 'timeout' ? `timeout after ${timeoutMs} ms` : `network error: ${got.error}`; + continue; + } + if (got.body && got.body.error) return { ok: false, error: `service error ${got.body.error.code ?? ''}`.trim() }; + return { ok: true, body: got.body || {} }; } return { ok: false, error: `${last} after ${retries + 1} attempts` }; } export async function fetchLayer({ layer, bboxLonLat, fetchImpl = globalThis.fetch, signal, pageSize = 1000, - timeoutMs = REQUEST_POLICY.timeoutMs, retries = REQUEST_POLICY.retries, onProgress, now = () => new Date().toISOString(), sleep = defaultSleep }) { + timeoutMs = SOURCES.usfsMvum.timeoutMs, retries = REQUEST_POLICY.retries, onProgress, now = () => new Date().toISOString(), sleep = defaultSleep }) { const layerId = LAYERS[layer] ?? layer; const layerKey = LAYER_KEY[layerId]; const opts = { fetchImpl, signal, timeoutMs, retries, sleep }; const warnings = []; const features = []; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-nws-alerts.js b/themes/headlessmode/static/js/quiet-corners-maps/source-nws-alerts.js index 3e4bef1..9f2b158 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-nws-alerts.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-nws-alerts.js @@ -22,7 +22,7 @@ // product is in force -- never that the ground is free of hazards. Every reader-facing string // is built here so scripts/qcm/nws-alerts.test.mjs can scan the lot. import { SOURCES } from './source-registry.js'; -import { withDeadline } from './request.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.nwsAlerts; export const SERVICE = SRC.serviceUrl; @@ -170,15 +170,16 @@ export const UNREADABLE_TEXT = 'The National Weather Service alert feed answered /** One ask. A non-200 is a FAILURE, not an empty answer: the API answers 400 for a query * parameter it does not recognise, and rendering that as "nothing in force" is the one lie * this state is able to tell. */ -export async function fetchAlerts({ lat, lon, fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(alertsUrl(lat, lon), { headers: requestHeaders(), signal }); - if (res.ok) return { status: 'ok', httpStatus: res.status, body: await res.json() }; - const problem = await res.json().catch(() => ({})); - return { status: 'failed', httpStatus: res.status, detail: problem.detail || problem.title || '' }; - } catch (e) { - return { status: 'failed', httpStatus: null, detail: String((e && e.message) || e) }; +export async function fetchAlerts({ lat, lon, fetchImpl = fetch, signal, timeoutMs = SRC.timeoutMs } = {}) { + const got = await fetchWithDeadline(alertsUrl(lat, lon), { + fetchImpl, signal, timeoutMs, init: { headers: requestHeaders() }, anyStatus: true, + }); + if (got.ok) return { status: 'ok', httpStatus: got.httpStatus, body: got.body }; + if (got.httpStatus !== undefined) { + const problem = (got.body && typeof got.body === 'object') ? got.body : {}; + return { status: 'failed', httpStatus: got.httpStatus, detail: problem.detail || problem.title || '' }; } + return { status: 'failed', httpStatus: null, detail: got.error }; } // QUX-09b (D5): the registry's per-source deadline, as source-weather.js's loader carries -- a hung @@ -189,12 +190,15 @@ export const DEFAULT_TIMEOUT_MS = SRC.timeoutMs; * showing it puts A's warnings under B's name. The previous ask is aborted, and the page * drops any answer for a place it has moved off. */ let live = null; -export async function alertsFor({ lat, lon, fetchImpl = fetch, timeoutMs = DEFAULT_TIMEOUT_MS } = {}) { +export async function alertsFor({ lat, lon, fetchImpl = fetch, timeoutMs = DEFAULT_TIMEOUT_MS, signal } = {}) { if (live) live.abort(); const ctl = (live = new AbortController()); - const deadline = withDeadline(ctl.signal, timeoutMs); + const onAbort = () => ctl.abort(); + if (signal) { if (signal.aborted) onAbort(); else signal.addEventListener('abort', onAbort, { once: true }); } let got; - try { got = await fetchAlerts({ lat, lon, fetchImpl, signal: deadline.signal }); } finally { deadline.dispose(); } + try { got = await fetchAlerts({ lat, lon, fetchImpl, signal: ctl.signal, timeoutMs }); } finally { + if (signal) signal.removeEventListener('abort', onAbort); + } if (live === ctl) live = null; return got; } diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-nws-radar.js b/themes/headlessmode/static/js/quiet-corners-maps/source-nws-radar.js index c092620..800ba7a 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-nws-radar.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-nws-radar.js @@ -20,6 +20,7 @@ // already answers "what will it do"; this answers "what is falling now". // `readerStrings()` exists so the test can scan the lot for that promise. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.nwsRadar; @@ -98,15 +99,10 @@ export function frameVintage(validEndMs, nowMs = Date.now()) { /** The service's own rolling window, read off its description. Never throws the map: * an unknown age is a state the row can show, not a reason to lose the layer. */ export async function fetchFrameTime({ fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(`${SERVICE}?f=json`, { signal }); - if (!res.ok) return null; - const body = await res.json(); - const extent = body && body.timeInfo && body.timeInfo.timeExtent; - return Array.isArray(extent) && extent.length === 2 ? Number(extent[1]) : null; - } catch (e) { - return null; - } + const got = await fetchWithDeadline(`${SERVICE}?f=json`, { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + if (!got.ok) return null; + const extent = got.body && got.body.timeInfo && got.body.timeInfo.timeExtent; + return Array.isArray(extent) && extent.length === 2 ? Number(extent[1]) : null; } /** Every string this layer can put in front of a reader, in one place, so the test diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-padus.js b/themes/headlessmode/static/js/quiet-corners-maps/source-padus.js index d99145e..70054e2 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-padus.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-padus.js @@ -24,6 +24,7 @@ // 3,033,604. Nothing overlaps; the polygons run outside the box. Per unit, or not at all. // * `UK` is a fourth answer, not a shade of `OA` and not silence. import { SOURCES, FAILURE } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.padusPublicAccess; const LAYER = SRC.layers[0]; @@ -151,10 +152,9 @@ export function accessReport(body, retrievedOn = null) { /** Asked for only when a reader asks, and never throws the map. `res.ok` is deliberately * not consulted: this service answers a bad request 200. `accessReport` reads the body. */ export async function fetchAccess(bbox, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(padusQueryUrl(bbox), { signal }); - return await res.json(); - } catch (e) { return null; } + let url; try { url = padusQueryUrl(bbox); } catch { return null; } // a bad area is "no answer" too + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs: SRC.timeoutMs, anyStatus: true }); + return got.body ?? null; } /** Every string this source can put in front of a reader, in one place, for the test. */ diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-rec-sites.js b/themes/headlessmode/static/js/quiet-corners-maps/source-rec-sites.js index 6042182..1bbc69a 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-rec-sites.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-rec-sites.js @@ -17,6 +17,7 @@ // nothing at all and the layer silently disappears. Verified against the // service's own schema, not from memory. import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; +import { DEFAULT_SOURCE_TIMEOUT_MS } from './source-registry.js'; const SERVICE = 'https://apps.fs.usda.gov/arcx/rest/services/EDW/EDW_RecInfraRecreationSites_02/MapServer/0'; const FIELDS = ['site_name', 'site_type', 'fee_charged', 'seasonal_operational_status', 'open_season', 'total_capacity']; @@ -128,6 +129,7 @@ export async function fetchRecSites({ bboxLonLat, fetchImpl = fetch, signal, pag normalize: normalizeSite, fetchImpl, signal, + timeoutMs: DEFAULT_SOURCE_TIMEOUT_MS, pageSize, }); const { features, ...rest } = page; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-registry.js b/themes/headlessmode/static/js/quiet-corners-maps/source-registry.js index 35a4465..31db1fe 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-registry.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-registry.js @@ -199,8 +199,9 @@ const CALTRANS_COVERAGE_AREAS = Object.freeze([ /** The deadline for one request to a source that has not cited a slower measurement. * Per source, not one number for the app: the GNIS names path is measured at 31 s. - * Only the modules moved onto request.js read it so far (GNIS names, quad index, the NWS - * forecast and alerts loaders); the rest still run at REQUEST_POLICY.timeoutMs until PR-B. */ + * Every module in this directory except main.js reads its own source's value through + * request.js (scripts/qcm/no-raw-fetch.test.mjs holds that); main.js joins them in PR-B2. + * `REQUEST_POLICY.timeoutMs` is no longer a request deadline. */ export const DEFAULT_SOURCE_TIMEOUT_MS = 10000; const withTimeouts = (sources) => Object.fromEntries(Object.entries(sources) .map(([key, src]) => [key, { timeoutMs: DEFAULT_SOURCE_TIMEOUT_MS, ...src }])); @@ -424,6 +425,10 @@ export const SOURCES = deepFreeze(withTimeouts({ ], spatialReference: null, limits: { maxRecordCount: 2000, partialPageFlag: 'exceededTransferLimit' }, + // Keeps the 20 s every page of these layers was read against before the deadline moved + // per source; a flat 10 s would end a federal-land or county page that has always answered. + timeoutMs: 20000, + timeoutMeasured: '20 s per page (the shared pager default the federal-land measurements were taken against)', pagination: true, cors: 'star', attribution: 'U. S. Geological Survey - National Geospatial Program, Government Units.', @@ -1705,13 +1710,17 @@ export const SOURCES = deepFreeze(withTimeouts({ maxAllowableOffsetDegrees: 0.0001, // Refuse rather than page. 1,759 records measured in ONE desktop view over // Horicon, and 29,608 in a degree square -- ~90 minutes of wall clock against - // a 20 s timeout. Four pages is what fits REQUEST_POLICY. + // a 20 s timeout. Four pages is what fits this source's own `timeoutMs`. recordBudgetPerExtent: 4000, // The PAINTED route returns a fully transparent PNG, HTTP 200, no error, above // 1:100,000 -- blank at Leaflet z12, paints at z13. Recorded so nobody adds // `export` to img-src and ships a blank picture that reads as "no wetlands". paintedMinZoom: 13, }, + // 20 s, not the 10 s default: paging params re-cost a query 20x (source-wetlands.js), so a + // budget-sized page runs past 10 s. Not re-measured here; this is the value the budget was sized to. + timeoutMs: 20000, + timeoutMeasured: '8.71 s unpaged, 20x that with paging params (2026-09-15, source-wetlands.js)', pagination: true, cors: 'reflects-origin', attribution: 'U.S. Fish and Wildlife Service, National Wetlands Inventory, ' diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-road-basic.js b/themes/headlessmode/static/js/quiet-corners-maps/source-road-basic.js index fac67bd..7165727 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-road-basic.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-road-basic.js @@ -6,7 +6,7 @@ // keyed on route id. Measured 2026-09-06 over the Standing Indian quad: 30 of 31 // MVUM roads matched (97%), levels 2-5 and four surface types present. An // unmatched road keeps its plain width -- drawn, never dropped. -import { SOURCES } from './source-registry.js'; +import { SOURCES, DEFAULT_SOURCE_TIMEOUT_MS } from './source-registry.js'; import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; const SERVICE = 'https://apps.fs.usda.gov/arcx/rest/services/EDW/EDW_RoadBasic_01/MapServer/0'; @@ -121,6 +121,7 @@ export async function fetchRoadQualityReport({ bboxLonLat, fetchImpl = fetch, si normalize: (one) => one, fetchImpl, signal, + timeoutMs: DEFAULT_SOURCE_TIMEOUT_MS, pageSize, }); const byId = normalizeRoadQuality(page.features); diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-road-names.js b/themes/headlessmode/static/js/quiet-corners-maps/source-road-names.js index 6048b86..2d793bd 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-road-names.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-road-names.js @@ -14,6 +14,7 @@ // the caller ignores it (the same convention source-hillshade.js uses). import { SOURCES } from './source-registry.js'; import { serviceError, envelope } from './arcgis-view.js'; +import { fetchWithDeadline } from './request.js'; import { layersForZoom, groupRoads, roadLabelOrder, ROAD_CLASSES } from './road-names.js'; const SRC = SOURCES.usgsRoadNames; @@ -62,17 +63,14 @@ const cacheKey = (b, zoom) => `${round(b.minLon)},${round(b.minLat)},${round(b.m /** One layer's answer: {features, exceeded} or {failed}. Never throws, except to say aborted. */ async function askLayer({ layer, bbox, zoom, fetchImpl, signal }) { - try { - const res = await fetchImpl(buildRoadQuery({ layer, bbox, zoom }), { signal }); - if (!res.ok) return { failed: `HTTP ${res.status}` }; - const body = await res.json(); - const refused = serviceError(body); - if (refused) return { failed: refused }; - if (!Array.isArray(body?.features)) return { failed: 'the service answered without a feature list.' }; - return { features: body.features.map((f) => ({ ...f, layer })), exceeded: body.exceededTransferLimit === true }; - } catch (err) { - return signal.aborted ? { aborted: true } : { failed: err?.message || String(err) }; - } + const got = await fetchWithDeadline(buildRoadQuery({ layer, bbox, zoom }), { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + if (got.status === 'aborted') return { aborted: true }; + if (!got.ok) return { failed: got.httpStatus ? `HTTP ${got.httpStatus}` : got.error }; + const body = got.body; + const refused = serviceError(body); + if (refused) return { failed: refused }; + if (!Array.isArray(body?.features)) return { failed: 'the service answered without a feature list.' }; + return { features: body.features.map((f) => ({ ...f, layer })), exceeded: body.exceededTransferLimit === true }; } /** A fetcher with its own in-flight slot and cache. The module's `fetchRoadNames` is one shared diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-smoke.js b/themes/headlessmode/static/js/quiet-corners-maps/source-smoke.js index 0ca2d17..bdde940 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-smoke.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-smoke.js @@ -18,6 +18,7 @@ // the day has not been analysed, NOT that the sky is clear -- and the two are told apart by a // 1 KB `max(Start)` statistics query rather than by assumption. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; import { serviceError, envelope, viewReader } from './arcgis-view.js'; const SRC = SOURCES.noaaHmsSmoke; @@ -196,13 +197,13 @@ export function smokeLayer(L) { /** Asked for only when a reader asks, and never throws the map. */ async function getJson(url, fetchImpl, signal) { - try { - const res = await fetchImpl(url, { signal }); - return res.ok ? await res.json() : null; - } catch { return null; } + const got = await fetchWithDeadline(url, { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + return got.ok ? got.body : null; } -export const fetchSmoke = (bbox, { fetchImpl = fetch, signal } = {}) => - getJson(smokeQueryUrl(bbox), fetchImpl, signal); +export const fetchSmoke = async (bbox, { fetchImpl = fetch, signal } = {}) => { + let url; try { url = smokeQueryUrl(bbox); } catch { return null; } // a bad area is "no answer" too + return getJson(url, fetchImpl, signal); +}; /** One bbox question at a time, waiting out a drag. The day-state question rides with it, and only * when the view was empty, so the row never mixes one view's plumes with another's day-state. */ diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-stream-names.js b/themes/headlessmode/static/js/quiet-corners-maps/source-stream-names.js index 3a62626..d6705b5 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-stream-names.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-stream-names.js @@ -10,7 +10,7 @@ // Returns {status, features, omittedKinds} like source-terrain-names.js fetchBounded: // ready every name asked was answered capped more names (or reaches) than one answer carries // empty below the lowest zoom: no request timeout / error / aborted the fetchJson shapes -import { SOURCES, REQUEST_POLICY } from './source-registry.js'; +import { SOURCES } from './source-registry.js'; import { KINDS } from './terrain-names.js'; import { fetchJson } from './source-terrain-names.js'; import { envelope } from './arcgis-view.js'; @@ -150,7 +150,7 @@ const failure = (result, signal) => { /** Named rivers and creeks in a view at a zoom: the longest names per kind, one record each, * anchored on the reach in view. */ -export async function fetchStreamNames(bounds, { zoom, fetch: fetchImpl = globalThis.fetch, signal, timeoutMs = REQUEST_POLICY.timeoutMs } = {}) { +export async function fetchStreamNames(bounds, { zoom, fetch: fetchImpl = globalThis.fetch, signal, timeoutMs = SRC.timeoutMs } = {}) { checkBbox(bounds); const layer = nhdLayerForZoom(zoom); if (!layer) return { status: 'empty', features: [], omittedKinds: [] }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-surface-ownership.js b/themes/headlessmode/static/js/quiet-corners-maps/source-surface-ownership.js index ca860bb..8452218 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-surface-ownership.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-surface-ownership.js @@ -17,6 +17,7 @@ // requested, nor is `comments`, nor the tax fields. What is asked for is the // record id, the owner class, the record status and the forest's name. import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; +import { SOURCES } from './source-registry.js'; export const SERVICE = 'https://apps.fs.usda.gov/arcx/rest/services/EDW/EDW_SurfaceOwnership_01/MapServer'; export const LAYER_NUMBER = 0; @@ -72,6 +73,7 @@ export async function fetchSurfaceOwnership({ bboxLonLat, fetchImpl = fetch, sig }, fetchImpl, signal, + timeoutMs: SOURCES.usfsSurfaceOwnership.timeoutMs, pageSize, }); return { ...page, skipped: null }; diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-terrain-surface.js b/themes/headlessmode/static/js/quiet-corners-maps/source-terrain-surface.js index 437a073..0c43527 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-terrain-surface.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-terrain-surface.js @@ -36,6 +36,7 @@ // tan(degrees) x 100 -- 45 degrees is 100 per cent -- so the two are never interchangeable // and every number here carries its unit in the same string. import { SOURCES } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; const SRC = SOURCES.usgs3depSurface; @@ -253,17 +254,13 @@ export function coverageText({ covered, missing }) { * layer costs the CSP nothing -- and it is the only route that can read the alpha channel * the coverage answer lives in. Never throws the map. */ export async function fetchSlopeImage(request, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(slopeImageUrl(request), { signal }); - return res.ok ? await res.blob() : null; - } catch (e) { return null; } + const got = await fetchWithDeadline(slopeImageUrl(request), { fetchImpl, signal, timeoutMs: SRC.timeoutMs, read: (res) => res.blob() }); + return got.ok ? got.body : null; } export async function fetchPoint(lonLat, surface, { fetchImpl = fetch, signal } = {}) { - try { - const res = await fetchImpl(pointUrl(pointRequestFor(lonLat, surface)), { signal }); - return res.ok ? await res.json() : null; - } catch (e) { return null; } + const got = await fetchWithDeadline(pointUrl(pointRequestFor(lonLat, surface)), { fetchImpl, signal, timeoutMs: SRC.timeoutMs }); + return got.ok ? got.body : null; } /** A Leaflet image overlay whose picture is re-exported for the view it is asked about, the diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-terrain.js b/themes/headlessmode/static/js/quiet-corners-maps/source-terrain.js index f9add2d..b977af4 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-terrain.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-terrain.js @@ -16,6 +16,9 @@ // That check is not ceremony -- 3DEP's exportImage once returned a BYTE-IDENTICAL // image for two different states, and a source that answers every request with the // same picture is worse than one that errors. +import { fetchWithDeadline, orThrow } from './request.js'; +import { SOURCES } from './source-registry.js'; + const SERVICE = 'https://elevation.nationalmap.gov/arcgis/rest/services/3DEPElevation/ImageServer'; /** The named raster function. Not a chain: the service renders it server-side. */ @@ -54,9 +57,10 @@ export function terrainRequestFor(sheet, bboxLonLat) { * sheet is an extra, so a failure loses the companion page, not the map. */ export async function fetchTerrain({ sheet, bboxLonLat, fetchImpl = fetch, signal } = {}) { const req = terrainRequestFor(sheet, bboxLonLat); - const res = await fetchImpl(`${req.url}?${new URLSearchParams(req.params)}`, { signal }); - if (!res.ok) throw new Error(`Shaded relief: the service answered ${res.status}.`); - const bytes = new Uint8Array(await res.arrayBuffer()); + const got = await fetchWithDeadline(`${req.url}?${new URLSearchParams(req.params)}`, { + fetchImpl, signal, timeoutMs: SOURCES.usgs3dep.timeoutMs, read: (res) => res.arrayBuffer(), + }); + const bytes = new Uint8Array(orThrow(got, got.httpStatus ? `Shaded relief: the service answered ${got.httpStatus}.` : undefined)); if (!bytes.length) throw new Error('Shaded relief: the service returned an empty image.'); return { id: sheet.id, bytes, width: req.width, height: req.height, format: 'png' }; } diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-trails.js b/themes/headlessmode/static/js/quiet-corners-maps/source-trails.js index de965a4..be2472a 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-trails.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-trails.js @@ -12,6 +12,7 @@ // inventory, republished. 7 are `NST - Appalachian`. 6 of the 63 carry NO // allowed-use flags at all, which is the case the honesty rule below exists for. import { pageThrough, PAGE_CAP } from './arcgis-pager.js'; +import { DEFAULT_SOURCE_TIMEOUT_MS } from './source-registry.js'; const SERVICE = 'https://carto.nationalmap.gov/arcgis/rest/services/transportation/MapServer/37'; const USE_FIELDS = { hike: 'hikerpedestrian', bike: 'bicycle', horse: 'packsaddle', motor: 'motorcycle' }; @@ -92,6 +93,7 @@ export async function fetchTrails({ bboxLonLat, fetchImpl = fetch, signal, pageS normalize: (raw) => { const t = normalizeTrail(raw); return t.paths.length ? t : null; }, fetchImpl, signal, + timeoutMs: DEFAULT_SOURCE_TIMEOUT_MS, pageSize, }); return { diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-usgs-topo.js b/themes/headlessmode/static/js/quiet-corners-maps/source-usgs-topo.js index 9142d71..3ce26c1 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-usgs-topo.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-usgs-topo.js @@ -4,6 +4,7 @@ // top-left — the service's {z}/{y}/{x} addressing is the standard XYZ grid with y and x swapped in the path. import { inverse } from './geometry.js'; import { SOURCES, REQUEST_POLICY, FAILURE } from './source-registry.js'; +import { fetchWithDeadline } from './request.js'; import { mercatorPixel, TILE_SIZE, EARTH_RADIUS } from './raster-placement.js'; export { TILE_SIZE, EARTH_RADIUS }; @@ -89,22 +90,16 @@ const ABORTED = { ok: false, status: null, bytes: null, error: 'aborted' }; /** Fetch unique tiles with bounded concurrency, per-request timeout, retries on network/5xx/429 only, and cancel via `signal`. */ export async function fetchTiles(tiles, { - fetchImpl = globalThis.fetch, signal, concurrency = REQUEST_POLICY.maxConcurrent, timeoutMs = REQUEST_POLICY.timeoutMs, + fetchImpl = globalThis.fetch, signal, concurrency = REQUEST_POLICY.maxConcurrent, timeoutMs = TOPO.timeoutMs, retries = REQUEST_POLICY.retries, backoffMs = REQUEST_POLICY.retryBackoffMs, onProgress, } = {}) { const queue = [...new Map(tiles.map((t) => [t.key, t])).values()], results = new Map(), requested = queue.length; let received = 0, retriedCount = 0, next = 0; const attempt = async (tile) => { - const ctrl = new AbortController(), timer = setTimeout(() => ctrl.abort(new Error(`timeout after ${timeoutMs} ms`)), timeoutMs); - const onAbort = () => ctrl.abort(signal.reason); - signal?.addEventListener('abort', onAbort, { once: true }); - try { - const res = await fetchImpl(tile.url, { signal: ctrl.signal }); - if (!res.ok) return { ok: false, status: res.status, bytes: null, error: `HTTP ${res.status}` }; - return { ok: true, status: res.status, bytes: await res.arrayBuffer(), error: null }; - } catch (err) { - return { ok: false, status: null, bytes: null, error: String(err?.message ?? err) }; - } finally { clearTimeout(timer); signal?.removeEventListener('abort', onAbort); } + const got = await fetchWithDeadline(tile.url, { fetchImpl, signal, timeoutMs, read: (res) => res.arrayBuffer() }); + if (got.ok) return { ok: true, status: got.httpStatus, bytes: got.body, error: null }; + if (got.httpStatus) return { ok: false, status: got.httpStatus, bytes: null, error: `HTTP ${got.httpStatus}` }; + return { ok: false, status: null, bytes: null, error: got.status === 'timeout' ? `timeout after ${timeoutMs} ms` : got.error }; }; const retryable = (r) => r.status === null || r.status === 429 || r.status >= 500; const worker = async () => { diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-weather.js b/themes/headlessmode/static/js/quiet-corners-maps/source-weather.js index 1cb24dd..3803e3d 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-weather.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-weather.js @@ -4,7 +4,7 @@ // content/quiet-corners-maps.md. TWO STEPS: `/points/{lat},{lon}` maps a lat/lon to an office and grid cell and hands back the // forecast URL, used AS GIVEN -- hand-building `/gridpoints/BMX/96,76/...` hard-codes a grid the office may renumber. import { SOURCES } from './source-registry.js'; -import { withDeadline } from './request.js'; +import { withDeadline, fetchWithDeadline, orThrow } from './request.js'; const API = SOURCES.nwsForecast.origin; // The refresh intervals are the service's own cache-control, not taste: `max-age=3600` on the hourly product, `max-age=85988` on /points -- a grid cell never moves. @@ -77,11 +77,15 @@ export function normalizeHourly(json = {}) { // A 404 `application/problem+json` is the API holding nothing for this point: a state to // show, not an error to throw. Every other status is a fault, and loud. async function getJson(url, { fetchImpl, signal }) { - const res = await fetchImpl(url, { headers: requestHeaders(), signal }); - if (res.ok) return { json: await res.json() }; - if (res.status !== 404) throw new Error(`NWS answered ${res.status} for ${url}.`); - const problem = await res.json().catch(() => ({})); - return { status: NO_FORECAST, detail: problem.detail || problem.title || '', coverage: COVERAGE }; + const got = await fetchWithDeadline(url, { + fetchImpl, signal, timeoutMs: SOURCES.nwsForecast.timeoutMs, init: { headers: requestHeaders() }, anyStatus: true, + }); + if (got.ok) return { json: got.body }; + if (got.httpStatus === 404) { + const problem = (got.body && typeof got.body === 'object') ? got.body : {}; + return { status: NO_FORECAST, detail: problem.detail || problem.title || '', coverage: COVERAGE }; + } + return orThrow(got, got.httpStatus ? `NWS answered ${got.httpStatus} for ${url}.` : undefined); } // The gridpoints product, the OTHER shape on this origin -- `/gridpoints/{office}/{x},{y}`, named by diff --git a/themes/headlessmode/static/js/quiet-corners-maps/source-wetlands.js b/themes/headlessmode/static/js/quiet-corners-maps/source-wetlands.js index a3a3253..2c22663 100644 --- a/themes/headlessmode/static/js/quiet-corners-maps/source-wetlands.js +++ b/themes/headlessmode/static/js/quiet-corners-maps/source-wetlands.js @@ -3,10 +3,10 @@ // THE BINDING CONSTRAINT IS REQUEST TIME, NOT RECORD COUNT. Measured 2026-09-15: // merely SENDING resultOffset/resultRecordCount re-costs a query 20x, even at // offset 0 -- a Louisiana 1-degree extent took 8.71 s unpaged and 176.74 s with -// paging params, against REQUEST_POLICY.timeoutMs of 20,000. Thirty pages of that +// paging params, against this source's `timeoutMs` of 20,000. Thirty pages of that // is ninety minutes for one degree square. So this layer asks for a COUNT first and // REFUSES above `recordBudgetPerExtent`, and that refusal is the layer rather than a -// polish item: it is the only thing keeping the requests inside their own policy. +// polish item: it is the only thing keeping the requests inside their own deadline. // Adding `orderByFields` made it worse still -- that variant never returned inside a // 180 s cutoff -- and paging is stable without one (overlap 0, union exactly 1,759), // so no query here sorts. @@ -27,6 +27,7 @@ // empty answer honest. All 105 No_Data parts are Alaska and the Pacific territories. import { SOURCES } from './source-registry.js'; import { pageThrough } from './arcgis-pager.js'; +import { fetchWithDeadline } from './request.js'; import { WETLAND_SEMANTICS } from './legend.js'; const SRC = SOURCES.fwsNwiWetlands; @@ -210,10 +211,10 @@ export function normalizeWetland(raw) { } const post = async (q, fetchImpl, signal) => { - try { - const res = await fetchImpl(q.url, { method: 'POST', body: new URLSearchParams(q.params), signal }); - return res.ok ? await res.json() : null; - } catch { return null; } + const got = await fetchWithDeadline(q.url, { + fetchImpl, signal, timeoutMs: SRC.timeoutMs, init: { method: 'POST', body: new URLSearchParams(q.params) }, + }); + return got.ok ? got.body : null; }; /** Count, then decide, then fetch. The order is the design. @@ -232,7 +233,7 @@ export async function fetchWetlands({ bboxLonLat, fetchImpl = fetch, signal } = const q = featureQuery(bboxLonLat); const page = await pageThrough({ url: q.url, label: 'The National Wetlands Inventory', params: q.params, - normalize: normalizeWetland, fetchImpl, signal, pageSize: PAGE_SIZE, maxPages: MAX_PAGES, + normalize: normalizeWetland, fetchImpl, signal, timeoutMs: SRC.timeoutMs, pageSize: PAGE_SIZE, maxPages: MAX_PAGES, }); const partial = page.coverage !== 'complete'; // Two point queries at the box's centre, asked TOGETHER and only after the budget