Document direct querying, helper functions, reproducibility

[bbest]

Context

The docs Quarto book has a comprehensive maps.qmd and a good db.qmd, but api.qmd is a near-empty stub and there's no page on querying the GCS parquet data directly or with the new calcofi4r helpers.

Goal

Document how to query CalCOFI data three ways — direct SQL, calcofi4r helpers, and the int-app download — with a reproducibility story tying them together.

Tasks

• New data-access.qmd — direct DuckDB + GCS parquet querying: httpfs setup, single-file vs hive-partitioned read_parquet examples, ## Reproducibility section explaining the int-app download query/ folder
• New helpers.qmd — the calcofi4r bio↔env matching helpers with worked examples
• Expand api.qmd — add a "superseded by calcofi4r helpers + direct querying" callout
• _quarto.yml — insert data-access.qmd and helpers.qmd after db.qmd, before api.qmd
• Recurring worked example across all three pages: "Pacific sardine larvae + temperature, Q1 2023, relaxed matching" shown three ways — direct SQL, cc_match_ichthyo_by_name(), int-app download — all producing identical rows
• Verify: quarto render docs/; confirm the worked-example SQL matches attr(cc_match_ichthyo_by_name(...), "sql")

Blocked by: CalCOFI/workflows#51, CalCOFI/apps#40, CalCOFI/calcofi4r#10, CalCOFI/int-app#5. Final issue of a 5-issue epic.

[bbest]

Part 6 — docs: done (branch data-access-helpers-docs, docs@7230f60)

Two new chapters + an api.qmd expansion, all rendering clean to HTML.

• data-access.qmd — querying frozen-release Parquet directly with DuckDB over public GCS: httpfs (+spatial) setup, single-file (read_parquet('https://…/{table}.parquet')) vs hive-partitioned (ctd_thin / ctd_summary via the s3-style glob with anonymous s3 config), R / Python / calcofi4r::cc_get_db() entry points, a ## Reproducibility section explaining the Integrated App download query/ folder, and the worked example as direct SQL.
• helpers.qmd — the calcofi4r matching helpers: cc_match_bio_env() engine + the three wrappers, the worked example as a one-liner, the portable attr(x,"sql") / query_meta / return_sql contract, a parameters section, and an old-API → helper migration table.
• api.qmd — a .callout-important "superseded by direct querying + calcofi4r helpers" banner and an endpoint → replacement table; legacy endpoint notes retained for reference.
• _quarto.yml — chapter order is now … maps → db → data-access → helpers → api → portals … (api.qmd moved after db.qmd so the new pages sit "after db.qmd, before api.qmd" as specified).

Verification

• All three pages render clean to HTML (_book/{data-access,helpers,api}.html).
• The worked-example SQL in data-access.qmd is character-identical to attr(cc_match_ichthyo_by_name("Sardinops sagax", env_var="temperature", life_stage="larva", date_min="2018-01-01", date_max="2018-03-31", relax_matching=TRUE), "sql") — verified programmatically (identical() on the extracted ```sql block vs the helper's emitted SQL) — and re-running it returns the same 13 rows.

Decisions & concerns

• Worked example is Q1 2018, not Q1 2023. CTD-bottle environmental data in the current release ends 2021-05; net-tow biological data runs to 2023-01, so "Q1 2023" matches zero env rows. Q1 2018 has 13 sardine larvae + ample temperature bottles. A .callout-note on both pages explains why. (Same change made in Bio<->env matching helpers querying GCS parquet directly calcofi4r#10 and Reproducible downloads: original + summarized data + SQL metadata int-app#5.)
• "Three ways, identical rows" is two-ways-exact + one-way-same-engine. Direct SQL and cc_match_ichthyo_by_name() are byte-identical SQL → identical 13 rows. The Integrated App download is the third way but its filter set is taxa + quarters + dates (no life_stage selector), so a sardine / Q1 2018 bundle returns the egg + larva superset — same cc_match_bio_env() engine, same reproducibility, different filter. The pages state this precisely rather than overclaiming byte-identity for all three.
• Code blocks are non-executable fenced blocks (```sql / ```r / ```python), matching db.qmd's style — so quarto render needs no network or calcofi4r install. I verified the SQL/IDs out-of-band instead (incl. fixing genus Engraulis WoRMS id 125620 → 125724 and the Integrated App URL → https://app.calcofi.io/int, matching maps.qmd).
• Render note: quarto render of the full book attempts html + pdf + docx + epub; the pdf path stalled badly in this environment. I verified each new page with quarto render <file> --to html. The deployed _book is HTML (GHA → gh-pages), so this is sufficient — but a full multi-format quarto render docs/ should be sanity-checked in CI.
• Hive-partitioned access confirmed against live GCS: s3://…/ctd_thin/**/*.parquet with hive_partitioning = true → 5,551,571 rows / 97 cruises; plain https:// can't glob (documented).

Tasks

• New data-access.qmd — direct DuckDB + GCS parquet querying; httpfs setup; single-file vs hive-partitioned; ## Reproducibility
• New helpers.qmd — calcofi4r matching helpers with worked examples
• Expand api.qmd — "superseded" callout + endpoint→replacement table
• _quarto.yml — data-access.qmd + helpers.qmd after db.qmd, before api.qmd
• Recurring worked example (Q1 2018) across the pages — direct SQL ≡ cc_match_ichthyo_by_name(); Integrated App download is the third way
• Verified: pages render to HTML; worked-example SQL matches attr(cc_match_ichthyo_by_name(...), "sql")