Add an offline pinned FAQ adapter and exact feed projection

[alexeygrigorev]

Parent: #42

Normative authority:

• 03 — source ownership and FAQ adapter
• 03 — public provenance boundary
• 02 — FAQ paths, fragments, and JSON compatibility
• Application boundaries: source parsing belongs in content_sync; public readers in content must not import the adapter.
• Verification strategy

Product outcome

Provide one deterministic, network-free dtc-faq adapter and five-field JSON-feed projection from an already verified immutable FAQ checkout. It must prove both the accepted checked baseline and the current green source without changing the public FAQ reader, rendering a public edit link, or introducing any direct-sync/activation behavior.

The accepted target input is exactly:

• repository DataTalksClub/faq;
• branch main;
• commit 3a170c98ad6af394408595f697ce6624ffab155f;
• tree 6b00fa01f1bc9fee4147b4f6e30f685a429f98c8;
• source CI run 33236963075, successful;
• selected roots _questions/ and images/ only.

The adapter receives a caller-supplied local checkout plus the expected immutable commit/tree identity. It performs no clone, fetch, remote-head lookup, provider call, hook, filter, repository-code execution, database mutation, or public-reader switch.

Evidence baseline and target census

The checked website projection on origin/main@face8e4808d65afbf0374d1ced7a88079950d663 is content/faq_projection.json SHA-256 7b6e5723b2ab0cf453254c10fb06a08175ca2bee5b9c65d98cc0534acfe8f209. It pins DataTalksClub/faq@c8da1deea9e24945922702994de101dd90a5380a, tree f880b4b63cdd1b213bd1303d8874e4ce02e63965, with:

• 6 courses, 70 populated sections, 1,401 questions;
• per-course counts in public order: Data Engineering 404, Stock Markets Analytics 93, Machine Learning 471, LLM 139, AI Dev Tools 41, MLOps 253;
• 101 image declarations resolving to 99 unique referenced assets; and
• two management-only CSS assets plus all source images copied separately by the historical projection, outside this feed-only slice.

The target commit has:

• 6 course metadata files, 74 declared sections, 70 populated sections, 1,406 questions;
• per-course counts in the same public order: 404, 93, 471, 144, 41, 253;
• 1,406 non-empty questions and answers, 101 image declarations, 99 unique referenced image paths, and 136 source image files;
• five added LLM questions: 5cc511f85b, 2fed4b176c, 920bb7ead8, 8471c98e1f, and 670737acd7;
• answer-only changes to eae0fb50aa and 8fd8cd336d; no metadata or image change from the accepted baseline;
• four accepted equal-sort_order groups, resolved deterministically by source filename; and
• two YAML numeric-looking IDs, 6739977244 and 9506089527, normalized to ten-character strings as required by the fragment/feed contract.

Four declared sections currently contain no question: ai-dev-tools-zoomcamp/module-6, stock-markets-analytics-zoomcamp/misc, llm-zoomcamp/module-4-homework, and llm-zoomcamp/module-6-homework. Preserve all 74 declarations and their order in management candidate data, but the flattened feed and its populated-section count contain only records from the 70 populated sections.

Adapter and candidate contract

Add a content_sync-owned immutable FAQ contract, adapter, bounded diagnostics, and focused fixtures/tests. The result contains:

• source stable ID, repository, branch, commit SHA, tree SHA, adapter/parser schema versions, selected-input digest, counts, and deterministic bundle/feed digests;
• courses in this exact public order: data-engineering-zoomcamp, stock-markets-analytics-zoomcamp, machine-learning-zoomcamp, llm-zoomcamp, ai-dev-tools-zoomcamp, mlops-zoomcamp;
• each course’s exact slug, display name, ordered declared sections, and management-only Slack/Telegram/comment metadata where present;
• each section’s exact ID, name, declaration order, optional management comment, populated state, and ordered questions;
• each question’s normalized ten-character string ID, course and section identity/name, positive integer sort_order, exact question, exact raw Markdown answer, normalized source path, and management-only moving-main edit URL;
• each declared image’s ID, description, normalized source path, stable candidate public path, size, media type, and SHA-256; duplicate references to the same source asset share one asset identity; and
• feed projections containing only {id,course,section,question,answer} in section order and then (sort_order, source filename byte order).

The courses.json projection contains only {course,course_name,path,questions_count} in the exact course order above. Its legacy path remains /json/<course>.json even though the website serves the compatibility route beneath /faq/json/.

Question/answer bytes are decoded as strict UTF-8 and preserved as source text after frontmatter separation; Jinja/Liquid/dbt-looking text is data and is never evaluated. Valid Unicode content is preserved, not normalized or rewritten. Structural identifiers and paths remain bounded ASCII.

The current two numeric YAML ID scalars are a reviewed compatibility exception: accept a non-boolean integer only when its canonical decimal representation is exactly ten characters, then normalize it to a string. Every other ID type, length, or character set fails. Do not add general scalar coercion.

Validation and failure behavior

• Accept only the exact metadata/question/image schemas exercised by the target checkout. Unknown structural keys, unknown courses/sections, duplicate course/section/question identities, duplicate per-question image IDs, invalid/blank question or answer, invalid sort order, or conflicting asset identity fail closed.
• Metadata is authoritative for course/section membership and order. A question under an undeclared course/section fails. Equal positive sort_order values are allowed and use the filename tie-break above.
• Resolve images only below images/<same-course>/ with the accepted .png/.jpg media types. Missing, symlinked, traversal, absolute, cross-course, remote, wrong-type, oversized, or checksum-conflicting assets fail. Unreferenced source images are inventory evidence, not candidate assets.
• Use safe, bounded YAML event scanning/construction. Reject aliases, unsafe tags/types, duplicate mapping keys, invalid UTF-8, path/control-character violations, depth/node/file-count/byte/time overruns, symlinks, gitlinks, and unexpected selected-root file types.
• Bounds must cover the measured target (1,551 selected files, 10,472,655 selected bytes, largest file 862,721 bytes) with explicit reviewed headroom; they must not be unbounded or inferred from the checkout at runtime.
• Failures return stable code plus bounded normalized relative path only. Do not include source content, raw YAML, exception text, absolute paths, tokens, credentials, or provider payloads.
• A failed parse returns no partial bundle/feed and changes no checked projection, public response, model, or prior artifact.
• Two exact runs over the same checkout produce byte-identical canonical bundle/feed evidence and digests regardless of filesystem enumeration order.

Public provenance boundary

edit_url, repository, branch, revision, tree, checksum, source path, diagnostics, and source-status evidence are management/build data only. They must not occur in FAQ HTML, either JSON feed, public structured data, sitemap/search output, or other anonymous response.

This issue must retain the current public guarantee: /faq/ and /faq/<course>.html show no “Edit on GitHub” control and no source/revision disclosure. The public five-field question feed and four-field course feed contain no provenance field. Do not add an edit-link browser scenario: public edit-link behavior is forbidden by specification 03.

Acceptance criteria

• The network-free adapter rejects a mismatched repository/branch/commit/tree and never invokes network, repository hooks/filters, or source code.
• Exact replay of c8da1de... produces 6/70/1,401/99 and payload-equal course/question feed arrays to the checked projection, including ten-character string IDs and exact order.
• Exact replay of 3a170c... produces 6 courses, 74 declared/70 populated sections, 1,406 questions, 101 declarations/99 referenced assets, the exact per-course counts and seven-record delta above, and deterministic evidence on two runs.
• Course, metadata-section, question, equal-sort tie, raw Markdown, valid Unicode, image-reference, and feed-field/order contracts are covered by focused tests.
• The two exact numeric-ID exceptions normalize to strings; booleans, floats, nulls, sequences, mappings, other integers, malformed strings, and duplicate IDs fail.
• Traversal/symlink/gitlink, unknown or cross-course paths, unsafe YAML, duplicate keys, missing/invalid metadata, missing/conflicting assets, invalid UTF-8/structural controls, and every resource bound fail with content-free deterministic diagnostics and no partial result.
• Repository tests prove candidate management provenance is absent from both feed projections and existing anonymous FAQ HTML/JSON responses.
• A versioned selective-CI plan records exact base/head, graph/plan digests, all dispositions, and fresh focused/full evidence required by the graph.

Verification scenarios

1. Parse the accepted baseline and target immutable checkouts twice with deliberately shuffled filesystem enumeration; compare complete canonical evidence/digests and the exact census/delta.
2. Compare baseline feed arrays against content/faq_projection.json, including the fixed course order, raw answers, four equal-sort groups, and two normalized numeric IDs.
3. Use bounded fixtures for every schema/type/path/YAML/resource denial above and assert stable diagnostic code/path with no raw input leakage.
4. Inspect existing anonymous FAQ HTML, course feed, and course-index feed for absence of edit URLs, repository names, revisions, checksums, and source paths.

Browser

The graph-selected backend-only browser tier is required. Run make test-playwright-smoke against the exact frozen candidate and retain plan-bound evidence. This is regression coverage for the shared/test-infrastructure impact; it does not authorize a page, route, payload, or rendering change and adds no issue-specific public browser scenario.

Screenshots

not_applicable for this child: it adds an offline adapter/evidence surface and must not change a template, public route, response payload, CSS, asset route, or checked projection. Existing request-level provenance-denial regression tests are required. Any implementation that changes public rendering or feeds is out of scope and must return to PM for a separate render-impact issue with desktop/mobile screenshots.

Dependencies and downstream work

No open implementation dependency. The accepted checked projection and immutable source checkout are the inputs; this slice deliberately does not depend on #38’s direct-sync models, #273–#278, a provider credential, or a database/public-reader decision.

Downstream only:

• parent Deliver source-backed FAQ pages, stable anchors, and exact JSON feeds #42 retains the public-page/current-pin lifecycle and will need a separately groomed checked-projection/public-regression slice after this adapter is accepted;
• Implement deterministic direct-upsert runner and partial recovery #274 may later consume the adapter from the generic network-free direct-sync runner;
• Cut public content reads and projections over to direct-sync authority #276 owns any database-backed public-reader cutover;
• Build Django-owned PostgreSQL search and cross-source graph activation #44 owns search integration.

A merge for this child uses Refs #42 and Closes #293; it does not close #42 or claim source activation.

Non-goals

No source clone/fetch/webhook/reconciliation, credentials, durable jobs, source lock, direct database upsert, ContentRelease, SyncLog, activation/rollback, Studio/admin API, search/graph indexing, public-reader or checked-projection switch, public page/template/asset/CSS change, new public route or field, public edit link, source repository mutation, provider/production access, or execution of DataTalksClub/faq code.

[alexeygrigorev]

Grooming Complete

This child is engineer-ready as the decision-free offline portion of #42.

It pins DataTalksClub/faq@3a170c98ad6af394408595f697ce6624ffab155f / tree 6b00fa01f1bc9fee4147b4f6e30f685a429f98c8 and the successful source CI run, records the exact checked-versus-current census, specifies the immutable candidate/feed schemas, binds the four equal-sort groups and two numeric-ID normalization exceptions, and defines bounded YAML/path/asset/resource denials.

The public provenance rule is explicit: edit URLs and source identity stay management-only and cannot enter anonymous FAQ HTML, feeds, structured data, search/sitemap output, or other public responses. Browser/screenshots are not_applicable only because this child must not change any public render or payload; any such change returns to PM for a separate render-impact issue.

Dependencies are intentionally narrow: no open implementation prerequisite and no dependency on #38/#273–#278. The future checked-projection adoption, direct-sync reader cutover, and search work remain separately owned.

No implementation, test, browser, checked projection, source repository, provider, database, production, commit, push, merge, deployment, or rollback state was changed during grooming.

[alexeygrigorev]

Software engineer report — frozen uncommitted candidate

Implemented the groomed, network-free scope for #293 in an isolated issue worktree based on authoritative origin/main face8e4808d65afbf0374d1ced7a88079950d663. The candidate is intentionally uncommitted and has not been pushed, merged, accepted, or exposed through a public route.

Candidate

• Worktree: .tmp/issue-293-engineer; branch: issue-293-faq-adapter.
• Eight issue-owned files, 2,145 inserted lines: the immutable FAQ contract, bounded adapter, hardened repository verifier/materializer, management-only verification command, and three focused test modules.
• Frozen binary-diff SHA-256: 56aea7b21e4ca2997b0bea17c2fbb27feaedb35bd6e41e91c19aa161be9ffe1e.
• git diff --check: pass. HEAD and origin/main both remain face8e4808d65afbf0374d1ced7a88079950d663.

The adapter verifies the exact repository/origin/ref/tree identity and materializes only _questions and images through hardened Git plumbing. It disables prompts, credential helpers, lazy fetch, replacement refs, hooks, filters, external diff, global/system config, and the file protocol. It applies the groomed file/byte/depth/node/time bounds, regular-file-only rules, strict UTF-8 and bounded YAML parsing, exact schema and course membership, duplicate/control/path denials, same-course local image validation with media signatures, deterministic ASCII filename tie ordering, and the two narrowly allowed ten-digit integer-ID normalizations.

The management candidate retains source paths, edit URLs, checksums, and provenance. The public-shaped projections contain only the groomed course fields and five question fields. Tests prove that provenance does not enter anonymous/public-shaped data or literal Jinja data. No route, template, checked public projection, database model, activation, direct-sync reader, search behavior, source repository, provider, or production state was changed.

Exact accepted-source evidence

Both immutable source revisions were verified from local, no-network scratch clones under this issue worktree.

• Baseline c8da1de / tree f880b4: 6 courses, 74 declared sections, 70 populated sections, 1,401 questions, 101 image declarations, 99 referenced images, 136 source images, 1,546 selected files, 10,469,770 selected bytes. Selected-input digest 069dc27d77e23e468f00c297b014eaf1a0f536141b66a281c50194d2925095e5; bundle digest 54f1eda4e35918d7e8f8c06ce6875d4f7c6cfba1ed0abdd038c20b9814f91420; course-feed digest 7a5c406d6f5f9f10e6c057e59bc40b6ff99534f446d1c4081741ee7ac97187a5; question-feed digest 33a1ac1da0b499e1c07ec9e0529bef68cc779580a0711087f1fd172d0c3cf54b. The feed equals the accepted checked content/faq_projection.json byte-for-object.
• Target 3a170c98ad6af394408595f697ce6624ffab155f / tree 6b00fa01f1bc9fee4147b4f6e30f685a429f98c8: 6 courses, 74 declared sections, 70 populated sections, 1,406 questions, 101 image declarations, 99 referenced images, 136 source images, 1,551 selected files, 10,472,655 selected bytes. Per-course question counts: 404, 93, 471, 144, 41, 253 in the frozen six-course order. Selected-input digest b79a01e749516ad38a1444d5370eff267be90877be09067d2eaf2e5c5d83302f; bundle digest 9e200f49ec450d3c32a0a860680398f01060f311015b69a467f9d6f7a9e20f39; course-feed digest 84ad419898a9055d6dc450c26a3369a0d93537d0c0f15ad2276b53f7cf9ca84d; question-feed digest 079dc2d584f9e2c126aecb571e9a37eb26292a0c787d05d422085a8494f8f41a.
• The target delta is exactly the groomed five additions and two answer-only changes. All four equal-sort groups match the pinned ASCII filename order. IDs 6739977244 and 9506089527 normalize to strings; other non-string forms fail closed.
• Reversing filesystem enumeration produces byte-identical projections and digests.

Verification buckets

Rerun and passed:

• Focused exact-checkout/security suite: 20/20 tests, 0 skips, including all acceptance census/delta/digest/order/ID assertions and adversarial Git/YAML/path/asset/resource/provenance cases.
• make test-ci: 607 passed.
• Migration drift, OpenAPI drift, management-command parity, Django system check, deployment check, database-portability check, security check, and lint: pass.
• Targeted Ruff check/format check and mypy for all eight owned files: pass.
• Container verification: pass; artifact records all 13 assertions, including non-root runtime, static-manifest rejection cases, local migration graph, health identity, and unified route response.
• Verification selector generation and plan validation: pass.

Fresh selector evidence:

• Plan SHA-256 207efbebb95fbe7298a7b127d88ff01dd16d2ef70877e2162b3abc834b85b484; policy 5; full profile (test_infrastructure).
• Worktree source manifest 3e4a870484af8ea0bcbe0f3924e94fef1f5bb61401d2980f70c59087112ce989; impact graph 05ffeee99244c9becd5e98e40be08188f1c5e2a61204daece984c3beb1d1bce6.
• Direct node shared.content_sync; risk flags shared_runtime, test_infrastructure; render impact false.

Not applicable:

• Compatibility and content-invariant components are outside the computed impact closure.
• Browser screenshots are not_applicable: this issue changes no public render or public payload. No screenshots were captured.

Blocked by inherited authoritative-base failures (not modified here):

• Full quality stops on frozen terminology digests for content/public_projection/podcasts.json, wiki.json, and wiki_search.json; the base format check reports six unrelated files; base mypy reports four errors in content/event_speaker_bio_normalization.py.
• Full Django reaches the inherited collection-hub pagination failure CollectionHubPaginationTests.test_controls_are_one_labelled_landmark_with_one_current_page, followed by the parallel-runner traceback serialization cascade.
• The generic selector requests Playwright smoke because this is a full/test-infrastructure profile. It was not run: the groomed Add an offline pinned FAQ adapter and exact feed projection #293 boundary and engineer assignment explicitly make browser verification N/A because there is no public render/payload change. This is disclosed rather than represented as a fully valid aggregate verification report.

No evidence was reused. The candidate is frozen for an independent tester, subject to the tester treating the documented inherited base-red gates and browser disposition according to the issue lifecycle. I have not committed or changed the issue acceptance state.

[alexeygrigorev]

Product manager lifecycle review — HOLD

Verdict: HOLD. Return to engineering; do not hand this candidate to the independent tester and do not accept, commit, merge, or push it.

Candidate and evidence audit

• I independently observed the frozen uncommitted candidate at face8e4808d65afbf0374d1ced7a88079950d663: the same eight added content_sync files and no other tracked changes.
• The binary diff SHA-256 is still 56aea7b21e4ca2997b0bea17c2fbb27feaedb35bd6e41e91c19aa161be9ffe1e.
• The plan SHA-256 is still 207efbebb95fbe7298a7b127d88ff01dd16d2ef70877e2162b3abc834b85b484, policy 5 / graph 05ffeee99244c9becd5e98e40be08188f1c5e2a61204daece984c3beb1d1bce6.
• The plan classifies the change as full / test_infrastructure, with direct node shared.content_sync, risk flags shared_runtime and test_infrastructure, no render impact, and browser profile smoke.
• The reported focused 20/20 adapter/security tests, exact immutable-source census/delta/digests, make test-ci 607/607, targeted static checks, and container assertions are useful engineering evidence for the implementation contract. They are not a substitute for required plan components.

Blocking discrepancies

1. The current graph policy requires the Playwright component to rerun as make test-playwright-smoke. Backend-only means the smoke tier; it does not mean Playwright is not applicable. The engineer explicitly did not run this required component.
2. Screenshots alone are not_applicable: the plan has render impact false, no required captures, and correctly classifies screenshots as render_inputs_unchanged_without_reusable_visual. No unrelated screenshot should be manufactured.
3. Required full-profile quality and Django components are red. Their failures may be correctly attributed to the authoritative base, but PROCESS does not allow a required red component to become a pass or acceptable skip. The exact candidate therefore cannot produce a successful tester-final report on this base.
4. There is no complete plan-bound four-bucket verification report in the frozen worktree. The available verification directory contains the selector inputs/plan and container check, not successful evidence for every required component.
5. No independent tester report exists, so PM acceptance is not available in any event.

Grooming correction and next handoff

I corrected this issue's Browser/screenshots wording: the graph-selected smoke run is required, while screenshots remain explicitly not applicable. This is a correction to #293 itself, not a new product or render-impact requirement, so no separate issue is needed.

Engineering must resume against a base on which every required full-profile component can pass, recompute the exact plan/digests, run and retain the required smoke evidence, and post a complete four-bucket engineer report. If repairing/rebasing the base changes any plan input or candidate identity, all affected evidence must be regenerated. Only then may a separate tester independently recompute the plan, validate the exact candidate and all acceptance criteria, run the selected focused/full and smoke gates, and produce a tester-final success report. Screenshots remain N/A unless the source diff gains render impact, which would be out of scope and require PM review.

This is a lifecycle HOLD, not product rejection of the focused FAQ adapter behavior.