diff --git a/architecture/README.md b/architecture/README.md index 46f71c6..165a8f4 100644 --- a/architecture/README.md +++ b/architecture/README.md @@ -2,68 +2,64 @@ [Open the interactive system map](https://www.theodoreoy.com/architecture/). -The viewer maps the website's content, build, deployment, and browser interactions. -It supports label search, light and dark themes, relationship -tracing, and image export. A downloaded `index.html` also works offline. +The viewer maps content inputs, static rendering, validation, GitHub Pages delivery, +and browser interactions. It supports label search, light and dark themes, +relationship tracing, and image export. | File | Responsibility | | --- | --- | | `site.json` | Editable Archify definition of the current system | | `index.html` | Standalone interactive viewer | -| `README.md` | Viewer usage and maintenance | -| `LICENSE` | MIT license for the Archify viewer | +| `LICENSE` | MIT license for the viewer | -The primary navigation contains About and Experience. Twenty-three canonical content -routes cover these pages, three Selected Research manuscripts, and eighteen Experience -detail pages. Eight compatibility pages bring the content export to thirty-one routes. -This viewer at `/architecture/` makes thirty-two exported routes, excluding 404. -`website/app/lib/content/` defines identity, routes, content loaders, and research -Markdown and math rendering. `website/content/` contains approved Experience copy, -structured manuscripts, frozen source evidence, and integrity contracts. GitHub -Actions publishes the assembled `website/out/` export to GitHub Pages on the custom domain. +## System represented -About opens with its interactive Canvas2D name entrance before the profile. The navigation -menu, shared article contents, and legal-note previews run in the browser. The website uses no request-time API, database, -authentication, or analytics. The [maintenance guide](../docs/maintenance.md) -documents content ownership, output contracts, security, and publishing. +`website/app/lib/content/` owns identity, routing, content loading, and research +Markdown/math rendering. Experience presents five categories and eighteen entries; +About includes three Selected Research papers. The AI portfolio's single maintained +source is `website/content/sources/ai-portfolio/ai-research-and-engineering.md`; +its importer generates the AI showcase fields and four article JSON files. +Selected Research uses pinned Markdown/TeX inputs with provenance; legal manuscripts +use structured paragraph runs and footnotes. The [maintenance guide](../docs/maintenance.md) +describes content ownership, import commands, and extension checks. -About presents three Selected Research entries with links to on-site manuscripts -and research summaries. Critical Dependence Stress for Portfolio Diversification -appears first, followed by the Asian-options and Rough Heston papers. Article headers -contain the return link, title, GitHub link, and Full Manuscript link. Reading content -begins at Abstract; pre-abstract front matter stays in the protected source. The -articles include core chapters (dependence stress 1–5; Asian options 1–10; Rough -Heston 1–8) and references; appendices are linked on GitHub. Each source JSON records -its exact upstream commit, blob, and manuscript URL. Frozen manuscript evidence lives -in `website/content/sources/selected-research/`. The `dependence/` subdirectory contains -the dependence-stress paper's frozen TeX files and independent provenance record; -`website/scripts/import-dependence-research.mjs` imports that manuscript deterministically. +Next.js exports HTML, CSS, and JavaScript to `website/out/`. Build assembly includes +this viewer and applies production CSP before GitHub Actions deploys the artifact +to GitHub Pages. There is no request-time API, database, authentication, or analytics. +The main site's browser interactions are the Canvas2D entrance, navigation menu, +article contents, and legal-note previews. Reading and ordinary links remain +available without JavaScript. -The main website takes its grid, warm paper color, and fine rules from Cognition as -a design reference and uses self-hosted Inter. Page titles and section headings use -22px, entry headings use 18px, and body text uses 14px. Experience directory labels -and numbers use 18px; formulas use 1.15em. The architecture viewer has independent -diagram controls and geometry and shares the site's HTML type roles. +## Update the viewer -Update `site.json` from the current source and `.github/workflows/pages.yml`, then -validate and generate it with the installed Archify skill: +Update `site.json` against the current application and `.github/workflows/pages.yml`, +then run these commands from the repository root. `` is the installed +Archify skill directory. ```sh node /bin/archify.mjs validate architecture architecture/site.json --quality showcase --json node /bin/archify.mjs deliver architecture architecture/site.json architecture/index.html --quality showcase --json node website/scripts/integrate-architecture.mjs +npm --prefix website run check ``` -Run these commands from the repository root. `` is the local Archify skill -directory. The integration script applies the website's six HTML type roles, -major-section heading semantics, and normalized control labels while preserving -SVG geometry. The viewer loads -`../assets/fonts/inter.css`, preloads regular Inter, and inherits `--font-text`. -Visual exports embed the local regular and italic Inter files in SVG; PNG and WebM -render from the same prepared fonts. The integration is safe to run repeatedly. -Run `npm --prefix website run check` to verify the complete export, including the -viewer and its production content security policy. +The integration applies shared HTML type roles, responsive reader layout, +major-section heading semantics, control labels, and export-font settings while +preserving SVG geometry. It handles both freshly generated and already integrated +viewers; running it again should leave the current viewer unchanged. -The viewer uses the MIT license. Published pages use the website's self-hosted Inter -fonts under the SIL Open Font License, included with the font assets. A standalone -offline HTML file uses system sans fallbacks when those font assets are unavailable. +The viewer loads `../assets/fonts/inter.css`, preloads regular Inter, and inherits +the site's fixed optical axis of 14 and lining numerals. General page and module +headings use 22px, secondary headings 18px, and body/control text 14px at every +viewport. Diagram labels retain their own SVG coordinate sizes. On the main site, +inline formulas use 1em and displayed formulas 1.15em. + +SVG exports embed the local regular and italic Inter files and retain the same +fixed optical settings; PNG and WebM render from those prepared fonts. Check theme +switching, search, tracing, keyboard controls, responsive layout, and export after +regeneration. The complete build also verifies the viewer's production CSP and +local assets. + +The standalone HTML works offline with system sans fallbacks when font assets are +unavailable. The viewer uses the MIT license; the self-hosted Inter files use the +SIL Open Font License included beside the font assets. diff --git a/docs/design.md b/docs/design.md index 527ce26..5f6d41b 100644 --- a/docs/design.md +++ b/docs/design.md @@ -81,8 +81,7 @@ introductions are omitted. Summaries use upright black body text at 14px; project and organization titles use the 18px subheading role. No content is truncated to fit a card. -The black and blue in the approved Word document identify outer and detail content; -blue is not a prose color on the website. All body copy uses the existing ink colors. +All body copy uses the shared ink colors. Blue remains reserved for navigation, hover, and keyboard focus. All eighteen entries use real Read More links to dedicated pages. The eight AI and @@ -100,8 +99,8 @@ keyboard focus and touch targets. Do not repeat the outer summary in the detail AI pages include the scope disclosure, Key Contributions with any nested method lists, Architectural Context, and the numbered manuscript chapters. Thin rules -separate top-level contribution items across all detail pages. The frozen original -manuscript and explicit reviewed corrections retain the content's provenance. +separate top-level contribution items across all detail pages. A single authoritative +AI manuscript supplies the introductory copy, nested lists, and complete mathematical body. Legal pages place Key Contributions before the complete Abstract and manuscript. The shared contents rail includes the contribution block and source chapter links. Return links target the matching diff --git a/docs/maintenance.md b/docs/maintenance.md index 6244852..782292f 100644 --- a/docs/maintenance.md +++ b/docs/maintenance.md @@ -1,309 +1,184 @@ # Maintenance -The site is a static personal record published at [theodoreoy.com](https://www.theodoreoy.com/). +The site is a static portfolio published at [theodoreoy.com](https://www.theodoreoy.com/). +Next.js builds HTML, CSS, and JavaScript; GitHub Pages serves the assembled export. +There is no request-time API, database, authentication, form submission, or analytics. +Use Node.js 24 to match deployment. Run application commands from `website/`, or use +`npm --prefix website run ` from the repository root. ## Repository | Directory | Responsibility | | --- | --- | | `.github/` | Repository presentation and the Pages workflow | -| `architecture/` | Interactive system map and its editable specification | -| `docs/` | Design principles and maintenance guidance | -| `website/` | Application, content, assets, tools, and tests | +| `architecture/` | Editable system-map definition and standalone viewer | +| `docs/` | Design and maintenance guidance | +| `website/app/` | Routes, shared readers, content loaders, and browser interactions | +| `website/content/` | Maintained content, research source inputs, provenance, and integrity contracts | +| `website/public/` | Assets included in the static export | +| `website/source-assets/` | Editable brand masters | +| `website/scripts/`, `tests/`, `tooling/` | Import, build, verification, and lint utilities | + +`website/.gitignore` excludes installed dependencies, build output, caches, logs, +generated Next.js types, and TypeScript build metadata. Keep local screenshots, +audit logs, and original source documents outside the repository unless they are +explicitly maintained research inputs. Only visitor-facing assets belong in `public/`. + +## Content ownership and routes + +Paths below are relative to `website/`. + +| Content | Maintained input | Loader or importer | Route | +| --- | --- | --- | --- | +| About and coursework | `app/components/AboutContent.tsx`, `app/lib/content/about.ts` | Shared About component | `/` | +| Experience | `content/experience-showcase.json`; AI fields are generated from the manuscript below | `app/lib/content/experience-showcase.ts` | `/experience/`, `/experience/[slug]/` | +| AI research | `content/sources/ai-portfolio/ai-research-and-engineering.md` | `scripts/import-ai-portfolio.mjs`, `app/lib/content/ai-projects.ts` | `/past-experience/artificial-intelligence/[paper]/` | +| Selected Research | `content/sources/selected-research/` | Research import scripts, `app/lib/content/selected-research.ts` | `/education/[paper]/` | +| Legal research | `content/legal-papers/` structured manuscripts | `app/lib/content/legal-papers.ts` | `/past-experience/legal-research-and-policy-analysis/[paper]/` | +| Architecture | `../architecture/site.json` | Archify and `scripts/integrate-architecture.mjs` | `/architecture/` | + +`app/lib/content/site.ts` owns site identity and primary navigation; +`app/lib/content/routes.ts` supplies canonical sitemap routes. +`experience-domains.ts` defines the five Experience categories. All eighteen +Experience entries have dedicated reading pages: four AI manuscripts, four legal +manuscripts, and ten ordinary detail pages. + +Compatibility routes remain functional: `/education/` renders shared About content; +`/past-experience/` and `/now/` render Experience; the five +`/past-experience/[slug]/` category pages filter the same approved entries. Their +canonical URLs point to the corresponding primary page. Keep these shared renderers +and their return links intact when changing navigation. + +### AI research + +Edit `content/sources/ai-portfolio/ai-research-and-engineering.md` as the single +authority for all four AI projects. It contains the directory disclosure, project +metadata, summaries, scope disclosures, contributions and nested lists, +Architectural Context, and complete numbered manuscripts with citations, tables, +and TeX. Preserve the project's metadata labels, ordered section structure, and +chapter sequence so the importer can validate the complete input before writing. -Run application commands from `website/`, or use `npm --prefix website run ` -from the repository root. `website/.gitignore` excludes the static export, build and -package caches, installed dependencies, generated Next.js types, and TypeScript -build metadata. The tracked `architecture/index.html` and generated brand assets -in `website/public/assets/brand/` are deliverables maintained by their documented -generation scripts. Keep local screenshots and audit logs outside the repository. +```sh +node scripts/import-ai-portfolio.mjs +``` -## Routes and content +The importer updates the four AI entries in `content/experience-showcase.json` and +the four article JSON files in `content/artificial-intelligence/`. It records the +source path and LF-normalized SHA-256, preserves non-AI entries, and copies each +numbered manuscript directly. Edit the canonical manuscript rather than generated +AI fields. Citation links and mathematical definitions belong in that manuscript. -Paths in this table are relative to `website/`. +The list presents the disclosure and four briefs. Detail pages show the institution +and project title, Position, Location, Dates, scope disclosure, Key Contributions, +Architectural Context, and numbered body. Each article returns to its project +anchor on Experience. Tests compare the full generated fields and manuscript +blocks with the maintained source and verify mathematical rendering independently. + +### Selected Research -| Page | Route | Source | +The three papers use pinned research inputs and deterministic importers: + +| Paper | Input and importer | On-site scope | | --- | --- | --- | -| About | `/` | `app/page.tsx`, `app/home.module.css`, shared About content | -| Education compatibility alias | `/education/` | `app/education/page.tsx`, shared About content | -| Dependence-stress research | `/education/critical-dependence-stress-for-portfolio-diversification/` | `app/lib/content/selected-research.ts`, `content/selected-research/critical-dependence-stress-for-portfolio-diversification.json` | -| Asian-options research | `/education/certified-valuation-arithmetic-asian-options/` | `app/lib/content/selected-research.ts`, `content/selected-research/certified-valuation-arithmetic-asian-options.json` | -| Rough Heston research | `/education/certified-rough-heston-valuation/` | `app/lib/content/selected-research.ts`, `content/selected-research/certified-rough-heston-valuation.json` | -| Experience | `/experience/` | `app/components/experience/`, `content/experience-showcase.json` | -| Experience detail pages | `/experience/[slug]/` | Approved entries from `content/experience-showcase.json` | -| Experience compatibility aliases | `/past-experience/`, `/now/` | Shared Experience page | -| Experience category compatibility pages | `/past-experience/[slug]/` | Filtered approved Experience sections | -| AI research projects | `/past-experience/artificial-intelligence/[paper]/` | `app/lib/content/ai-projects.ts`, `content/artificial-intelligence/` | -| Legal articles | `/past-experience/legal-research-and-policy-analysis/[paper]/` | `app/past-experience/[slug]/[paper]/page.tsx`, `app/lib/content/legal-papers.ts`, `content/legal-papers/` | -| Architecture | `/architecture/` | `../architecture/index.html` | - -`content/experience-showcase.json` is the approved English presentation source, -derived from `experience-merged-bilingual.docx` with approved display capitalization. -Its source-document SHA-256 identifies the reviewed Word file. It contains five categories and eighteen -entries. The file contains public English copy only; full manuscripts have separate -protected sources. `app/lib/content/experience-domains.ts` defines the five categories, -and `app/lib/content/experience-showcase.ts` maps entries to their reading pages. - -The primary navigation contains About and Experience. Experience presents all five -numbered sections in one page, with a single-column anchor directory at every -screen width and eighteen summaries. All eighteen Read More actions are links to -real pages. The eight AI and legal entries use their manuscript URLs; -the other ten entries use `/experience/[slug]/`. Outer headings lead with the -institution or organization; legal headings use `Duke Law School: {research area} -Research`. Detail-page titles use `{institution}: {project title}`; internal title -colons render as em dashes, preserving only the institution separator. Position, -Location, and Dates form three stacked rows at every viewport, followed by the -approved Key Contributions with contextual notes appended as the final list items. -All items share the same typography and color. Do not repeat the outer summary. - -AI pages continue directly from Key Contributions into the numbered article body. -The renderer starts at the first numbered chapter; the full source JSON includes -the title, Project and Author lines, and introductory material for provenance. -Legal articles present their complete Abstract, text, Author Note, and Footnotes -after the contribution block. Contents starts with Key Contributions and uses the -source chapter anchors. - -`app/lib/content/routes.ts` supplies twenty-three canonical sitemap routes: About, -Experience, three Selected Research articles, eight Experience manuscripts, and ten -other Experience detail pages. -The static export includes `/education/`, `/past-experience/`, `/now/`, and the -five category URLs as compatibility pages. The category pages render the same -approved entries, use `/experience/` as their canonical URL, and link back to the -corresponding Experience section. -There are thirty-one exported content routes and the separate `/architecture/` -reference: thirty-two routes in total, excluding 404. - -Biographical, AI research, Selected Research, and legal source files are checked against `content/protected-sources.json`. -`content/visible-copy-manifest.json` independently records text, metadata, alt text, -and ARIA labels for all thirty-one content pages. Copy changes require the owner's explicit -request and an intentional update to these contracts. Presentation-only changes to -a protected component update its source hash while preserving the approved prose, -metadata, and formula source contracts. Tests also verify page counts and navigation -independently of the registry. - -## Application - -Next.js exports static HTML, CSS, and JavaScript. GitHub Pages serves these files; -there is no request-time backend, database, authentication, form submission, or analytics. - -- `app/components/SiteShell.tsx` owns the shared reading frame. -- `app/components/SiteNavigation.tsx` owns the mobile menu and focus handling. -- `app/page.tsx` presents the interactive Canvas2D name entrance before About; `app/components/dithered-entrance/` owns its pointer, keyboard, fallback, and reduced-motion behavior. Its scroll cue targets `#home-profile`. -- `app/components/article/` owns the shared reading layout, responsive contents navigation, and section tracking. -- `app/components/research/` owns the shared research article and mathematical presentation. -- `app/globals.css` and `app/home.module.css` implement the [design system](design.md). -- `app/lib/content/site.ts` owns site identity and metadata helpers. -- `app/lib/content/research-markdown.ts` parses research prose, headings, tables, and formulas. -- `app/lib/content/article-headings.ts` supplies Roman-numeral chapter labels, lettered subsections, and verified internal section links while preserving source anchors and reference numerals. -- `app/lib/content/research-math.ts` renders mathematical SVG and assistive MathML on the server. - -`app/components/AboutContent.tsx` supplies the shared About and Education-alias -content. It presents Duke University once above both degree records and omits the -fellowship from About. `app/lib/content/about.ts` supplies all 49 course names in four -categories. Categories stack vertically; each list flows down up to two independent -CSS columns with a 16rem target width, a 24px column gap, and 2px between items. -Course names stay together across column breaks, and lists use one column below -768px. `app/home.module.css` scopes the compact -record spacing and consistent role styles to About. Its reading frame uses the -shared Inter family with `font-optical-sizing: none` and `"opsz" 14`; heading and -prose tracking is zero. Keep these role sizes synchronized with Experience and research pages. - -Add a website page by updating its source, the route registry, sitemap expectations, -and output tests together. Keep browser interaction in focused client components. -Add a server only when a defined product requirement needs one. - -`tooling/directory-glob/` supplies Next.js lint root-directory discovery through -the pinned Tinyglobby implementation. Its adapter preserves literal, wildcard, -recursive, and platform-specific directory paths. Integration tests exercise the -Next.js consumer directly. The dependency override is scoped to the Next.js lint -plugin; application dependencies and lint rules retain their normal configuration. - -### Experience source updates - -The approved bilingual Word document is the current authority for showcase copy. -Read its English blocks only; preserve category and entry order, metadata, each -contribution, and the two contextual notes. Apply approved display-format changes -without changing the source-document hash; update that hash only when a replacement -Word file is approved. AI and legal manuscript sources are -independently protected and are not regenerated from the showcase document. - - -### AI research projects - -Domain `01`, AI Research and Engineering, links four project reading pages through -Read More actions following the outer summaries. Their manuscript URLs are stable. -Each JSON source stores its original Notion title, URL, edit timestamp, -and complete Markdown body. Preserve that source, including its original opening -Project and Author paragraph. The rendered page shows the institution and -project title, vertical metadata, and approved Key Contributions, followed by -the first numbered chapter. Keep the numbered body, emphasis, citations, table -headers and cells, and formulas in their -original order. Do not add an abstract, references section, publication details, -or a download. Each reading page returns to its project anchor on Experience. - -| Project page | Notion source | -| --- | --- | -| Statistical inference and resource allocation in expert data production | [Source](https://app.notion.com/p/3ed867f96db58171ad07d36248e8d05c?pvs=204) | -| Task validity in financial synthetic data | [Source](https://app.notion.com/p/3ed867f96db5810798b2f3668e2dd0a5?pvs=204) | -| Verification and supervision in scientific reasoning tasks | [Source](https://app.notion.com/p/3ed867f96db5814c91c0fd96d9348c2f?pvs=204) | -| Evidence uncertainty and decision guarantees in investment research | [Source](https://app.notion.com/p/3ed867f96db581cfaae7d40b93aaa080?pvs=204) | - -`ResearchArticle` renders both AI projects and Selected Research with a focused CSS -Module that gives prose and table cells the shared 14px reading size, and H3 headings -the shared 18px subheading size. -`ResearchMath` uses `app/lib/content/research-math.ts` to render TeX on the server with the pinned MathJax -4.1.3 engine and its TeX font package. Visible equations contain complete SVG glyph -paths, use the shared `--type-math` relative scale of 1.15em and ink color, and load no browser math -engine or font resources. `ResearchMath.module.css` owns formula spacing and -scrolling. Display rows omit sentence punctuation and use explicit multiline spacing; -mathematical terms and equation labels remain intact. Assistive MathML preserves -semantic formulas and the exact original TeX annotation. Invalid formulas fail the -build. Tables and wide equations expose focusable scrolling regions. - -AI project tests compare the numbered body with its source blocks, -verify the chapter boundary used for rendering, and check all 277 original formula annotations, -five tables, 27 citation links, and 26 second-level headings with the approved -documents. Update the project registry, independent route expectations, protected -source hashes, and visible-copy contracts together when synchronizing approved text. -The four project snapshots protect prose and formula positions independently from -the ordered TeX-and-display-mode hash and formula count. SVG geometry and assistive -MathML have independent regression checks for accents, fractions, aligned equations, -Greek glyphs, superscripts, and equation numbers. Formula presentation changes preserve -narrative text and source annotations. Approved navigation order or scrolling changes -update only the corresponding contract fields after independent verification. +| Critical Dependence Stress for Portfolio Diversification | `content/sources/selected-research/dependence/`; `scripts/import-dependence-research.mjs` | Abstract, chapters 1–5, references, and supporting-material links | +| Certified Valuation of Arithmetic Asian Options | `content/sources/selected-research/asian-source.md`; `scripts/import-selected-research.mjs` | Abstract, chapters 1–10, references, and supporting-material links | +| Certified Rough Heston Valuation | `content/sources/selected-research/rough-source.md`; `scripts/import-selected-research.mjs` | Abstract, chapters 1–8, references, and supporting-material links | -### Selected Research +```sh +node scripts/import-selected-research.mjs +node scripts/import-dependence-research.mjs +node scripts/import-dependence-research.mjs --check +``` -About presents three compact research entries before Education and Selected Coursework. -Critical Dependence Stress for Portfolio Diversification appears first, followed -by the Asian-options and Rough Heston papers. All three titles link to their on-site -manuscripts. Title links use black ink, turning blue on -hover or keyboard focus. A short paragraph states the mathematical method and the -achieved result within the paper's specified example. Keep numerical bounds and -improvements tied to their model, horizon, payoff, and comparison. Source links remain -on the manuscript pages; About has no separate -Paper, Repository, or Results action row. `app/lib/content/selected-research.ts` -loads three structured sources from `content/selected-research/`: - -| Source | On-site manuscript scope | -| --- | --- | -| `critical-dependence-stress-for-portfolio-diversification.json` | Abstract, chapters 1–5, references, and an appendix/supporting-material link | -| `certified-valuation-arithmetic-asian-options.json` | Abstract, chapters 1–10, references, and an appendix/supporting-material link | -| `certified-rough-heston-valuation.json` | Abstract, chapters 1–8, references, and an appendix/supporting-material link | - -Frozen Asian-options and Rough Heston Markdown sources and their shared provenance -record live in `content/sources/selected-research/`; `scripts/import-selected-research.mjs` -imports those two papers. The dependence-stress paper uses frozen TeX files and its -own provenance record in `content/sources/selected-research/dependence/`, with -deterministic conversion in `scripts/import-dependence-research.mjs`. The import -scripts and content tests verify structured manuscripts against their pinned sources. -The frozen source tree includes supporting TeX inputs used to resolve manuscript -cross-references. Its four PNG figures are source evidence; the copies in -`public/assets/research/dependence/` are served assets. Tests verify that the served -and exported images match the frozen bytes. - -The pages use the shared `ResearchArticle` and `ResearchMath` components, with a -contents list, source links, and return links to the corresponding About research -entry at `/#`. Their navigation marks About as current. They publish the complete -core chapters. The article header contains only the return link, title, GitHub link, -and Full Manuscript link. Reading content starts at Abstract; source front matter -before Abstract remains in the protected source. Appendices are available in the -GitHub manuscript. -Each JSON source stores `repositoryUrl`, `sourceUrl`, `sourceCommit`, and `sourceBlob`. -The manuscript link is pinned to that source commit. When refreshing a paper, compare -against the pinned file, preserve the complete core chapters and references, and -update the provenance only for the version actually imported. - -Preserve the Asian paper's distinction between the projected Euler pricing certificate -and its continuous-model comparison, including the assumptions for weak expansion and -posterior quantile results. Preserve the Rough Heston paper's stated configurations, -certificate scope, error definitions, and reproducibility limits. Approved content -changes update the source hash, visible-copy contract, independent output expectations, -and route registry together. Verify rendered formulas, internal manuscript links, and -appendix destinations before publishing. - -### Legal papers - -Domain `03`, Legal Research and Policy Analysis, links four selected papers: -`autonomous-authority-in-space`, `small-states-and-strategic-space-dependence`, -`solar-geoengineering-comparison-and-continuity`, and -`mangrove-restoration-and-compensatory-mitigation`. Preserve each paper's source -title, subtitle, author, date, abstract, numbered chapter hierarchy, paragraphs, -citations, and acknowledgment in its structured source. The manuscripts credit -Letao Ouyang and carry a December 2024 date. - -| Manuscript | Source PDF | +The source records identify the upstream repository, commit, blob, and file hashes. +Refresh these only for the version actually imported. Retain complete core chapters, +tables, references, theorem assumptions, error definitions, and reproducibility +limits. Distinguish the Asian paper's projected-Euler certificate from its +continuous-model comparison, and retain the Rough Heston paper's stated +configurations and finite-candidate scope. + +The dependence source tree includes supporting TeX inputs needed to resolve +cross-references. Its four PNG figures are source evidence; copies in +`public/assets/research/dependence/` are served assets. Tests check their byte +identity. The readers link appendices and supplementary proofs to the pinned full +manuscripts and return to the corresponding About research entry. + +### Legal research + +`content/legal-papers/` preserves source titles, authors, dates, Abstracts, chapter +hierarchies, paragraph runs, citations, acknowledgments, and numbered footnotes. +The approved source PDFs are external inputs, not served downloads: + +| Structured manuscript | Source PDF | | --- | --- | -| Who May Choose the Lesser Risk: Solar Geoengineering and the Legal Duties of Comparison and Continuity | `ESSAY1.pdf` | -| Small States and the Governance of Strategic Space Dependence | `ESSAY2.pdf` | -| Autonomous Authority in Space: Risk Tradeoffs and the Law of Delegation | `ESSAY3.pdf` | -| Mangrove Restoration and the Limits of Compensatory Mitigation: Lessons from Florida for the Greater Bay Area | `ESSAY4.pdf` | - -`PaperParagraph.text` equals the concatenated run text; runs preserve source italics, -bold, capitalization, superscripts, and numbered references. Citation text uses the -same body size and spacing, including source small-capital runs. Numbered notes remain -sequential and every chapter has a unique anchor. Keep original PDFs and extraction -records outside public assets and the static export. - -`LegalPaperPage` renders the institution-and-project heading, vertical metadata, -Key Contributions, complete original Abstract, article, and endnotes. The shared -`ArticleContents` provides responsive chapter navigation and section tracking. `LegalPaperReader` -adds native-dialog note previews, with source-link focus restoration and a route to -the complete endnote. Original anchors, contents links, and return links remain -available without JavaScript. Reading controls use the shared font and type tokens. - -To add an approved paper, update its structured source, the legal-paper slug registry, -its entry and route mapping in `experience-showcase.ts`, and the independent content/output -contracts together. Use the supplied manuscript as the wording and citation authority. - -Article validation checks all thirty-one exported content routes, eighteen Experience -reading links including four legal articles, chapter -anchors, complete numbered references, bidirectional footnote links, and the absence -of public PDF files. Validate the published text against the approved source before -refreshing visible-copy hashes. Read back the four live article pages after publishing. - -### Static export - -The `build` command runs Next.js and then `scripts/assemble-static.mjs`. Assembly -places the standalone architecture viewer and license under `/architecture/`, -normalizes Next.js segment payload names, and applies the production HTML security -policy. Payload normalization validates containment and destination collisions -before moving files; export entries may not be symbolic links. Preserve these -output-contract tests when updating Next.js. - -## Brand assets - -`website/source-assets/brand/lo-mark.svg` is the editable LO master. -`npm run generate:brand-icons` creates its public SVG, favicon, and Apple touch derivatives. -`website/source-assets/brand/og.png` is the social-preview master; -`npm run optimize:images` creates `public/assets/brand/og-1774.jpg`. -`.github/assets/readme-cover.jpg` is the repository cover. - -Editable masters and served derivatives have separate roles. Only assets needed by -visitors belong in `website/public/`. - -## Typography assets - -`website/public/assets/fonts/` contains the unmodified Inter 4.1 variable WOFF2 -files, their SIL Open Font License, and the shared font stylesheet. The source is -the [official Inter 4.1 release](https://github.com/rsms/inter/releases/tag/v4.1). -The site layout and architecture viewer load this same stylesheet and regular-face -preload. Keep text in the inherited font; reserve separate lettering for brand artwork. - -The visual reference is Cognition's grid, warm paper color, restrained rules, and -spacing. Inter remains the site's own self-hosted typeface. Research and other body -text share 14px with 1.6 leading; subsection headings and institution/project/paper -names use 18px, including detail-page H1s. General page titles and section headings -use 22px with 1.25 leading at every viewport. Inline math uses 1em and displayed -math uses 1.15em. Pure inline numeric values use Inter and retain accessible MathML. -Shared Inter settings fix the optical axis at 14 and use lining numerals. -Experience directory labels and numbers use 18px. Metadata, main -navigation, section indices, field labels, and footers use 14px. The semantic token -names remain distinct; mobile layouts retain the same 22px, 18px, and 14px hierarchy. -Keep these roles synchronized through `app/globals.css`; do not give a particular -article a larger prose size or change type size to fit a long title. - -## Validation and publishing +| `solar-geoengineering-comparison-and-continuity.json` | `ESSAY1.pdf` | +| `small-states-and-strategic-space-dependence.json` | `ESSAY2.pdf` | +| `autonomous-authority-in-space.json` | `ESSAY3.pdf` | +| `mangrove-restoration-and-compensatory-mitigation.json` | `ESSAY4.pdf` | + +`PaperParagraph.text` must equal the concatenated run text. Preserve emphasis, +capitalization, superscripts, unique chapter anchors, and sequential notes. +`LegalPaperPage` presents metadata and contributions before the complete Abstract, +manuscript, and endnotes. `LegalPaperReader` adds native-dialog note previews with +focus restoration; chapter and endnote links also work without JavaScript. + +## Shared rendering and extension + +`SiteShell` owns the reading frame, `SiteNavigation` the mobile menu, and +`ArticleReadingLayout`/`ArticleContents` the article structure and responsive +navigation. `ExperienceSection` and `ExperienceDetails` keep all categories and +detail pages consistent. Browser interaction stays in focused client components; +the About entrance is isolated in `components/dithered-entrance/`. + +`ResearchArticle` serves AI and Selected Research. The Markdown parser supports +typed prose, emphasis, links, headings, tables, and formulas rather than arbitrary +HTML. `ResearchMath` renders TeX to self-contained SVG and assistive MathML during +the build; no browser math engine is loaded. Original TeX remains in accessible +annotations. Invalid formulas fail the build. Wide equations and tables scroll +locally without shrinking the reading text. + +Use the shared Inter and typography tokens: 22px general titles and module headings, +18px institution/project/paper names and Experience directory items, and 14px prose, +metadata, and auxiliary navigation. Inline formulas use 1em; displayed formulas +use 1.15em. Pure inline numbers inherit Inter. These roles remain unchanged on +mobile. See [design](design.md) for spacing and mathematical geometry. + +When adding content, update its source, loader/slug registry, route mapping, sitemap +expectations, and output tests together. Experience routes are keyed by stable +entry IDs; add the corresponding mapping when adding an entry. The loader checks +for missing routes, duplicate slugs, and mismatched AI or legal manuscript routes. + +`content/protected-sources.json` checks approved source bytes with LF normalization; +`content/visible-copy-manifest.json` independently checks rendered text, metadata, +alt text, and ARIA labels. Review intended content changes before updating either +contract. Presentation changes preserve approved prose and formula annotations. +Keep independent route, link, figure, and mathematical checks when updating a loader. + +`tooling/directory-glob/` adapts pinned Tinyglobby for Next.js lint directory +discovery. Its override is scoped to the lint plugin; preserve its directory-path +behavior and consumer tests when changing dependencies. + +## Assets and static assembly + +`source-assets/brand/lo-mark.svg` is the editable mark; +`npm run generate:brand-icons` produces the public SVG and icon derivatives. +`source-assets/brand/og.png` is the social-preview master; +`npm run optimize:images` produces `public/assets/brand/og-1774.jpg`. +The repository cover lives in `.github/assets/readme-cover.jpg`. + +The Inter variable fonts and SIL Open Font License live in `public/assets/fonts/`. +The site and architecture viewer load the same stylesheet, with automatic optical +sizing disabled, the optical axis fixed at 14, and lining numerals. Retain font +licenses and use separate lettering only for brand artwork. + +`npm run build` runs Next.js, then `scripts/assemble-static.mjs`. Assembly includes +the architecture viewer and its license, normalizes segment payload filenames, and +applies production CSP. Payload moves validate containment and destination +collisions; export entries may not be symbolic links. Preserve these output checks +when upgrading Next.js. See [architecture](../architecture/README.md) for viewer +generation and export-font integration. + +## Validation, security, and publishing ```sh cd website @@ -314,29 +189,27 @@ npm audit --audit-level=low npm run preview:static ``` -Use Node.js 24 to match the deployment workflow. The local static preview binds to -`127.0.0.1` by default and serves the assembled export; use it for production-policy -verification. `npm run dev` uses the development security policy required by Next.js -development tooling. - -Review the export at desktop and phone widths, including 320px. Check keyboard -navigation, menu dismissal and resizing, reduced motion, the JavaScript-free fallback, -local links, and the architecture viewer. Run `git diff --check` and inspect staged -files for credentials, temporary files, and generated build output. - -The SHA-pinned GitHub Actions workflow validates pull requests. A push to `main` -validates, audits dependencies, and uploads `website/out/` to GitHub Pages. Build jobs -have read-only repository access; only deployment receives Pages write and OIDC access. -After publishing, verify the workflow for the exact commit and the live domain separately. - -The build computes SHA-256 permissions for the exact inline scripts in each exported -HTML page, including the architecture viewer. The policy appears before scripts and -blocks arbitrary inline scripts, evaluation, frames, objects, and form submission. -Fonts load from the site's own origin. Inline event handlers are blocked on every -page. Inline styles support the framework and standalone viewer. -GitHub Pages controls HTTP response headers. The HTML policy cannot set response -headers or the `frame-ancestors` directive. Keep dependency versions and the lockfile -aligned, and preserve the major-scoped dependency overrides. The dependency audit -checks the advisory data available at run time. - -See [architecture](../architecture/README.md) for viewer editing and validation. +`check` verifies protected sources, lints, builds, tests the artifact, and compares +visible copy. The static preview binds to `127.0.0.1:8123` by default and serves +`out/`; use it to inspect production behavior. `npm run dev` uses the separate +development security policy required by Next.js tooling. + +Review desktop and phone layouts, including 320px, keyboard navigation, menu +dismissal, reduced motion, JavaScript-free content, local links, equations, tables, +note previews, and architecture exports. Run `git diff --check` and inspect staged +files for credentials, temporary files, and unintended generated output. + +The workflow validates pull requests with read-only repository access. A push to +`main` runs validation and the dependency audit, uploads `website/out/`, and deploys +with Pages write and OIDC permissions confined to the deployment job. Every action +is SHA-pinned. After publishing, verify the workflow for the exact commit and read +back the live routes separately. + +Each exported HTML page receives hashes for its exact inline scripts before any +script executes. Production CSP blocks arbitrary inline scripts, eval, inline event +handlers, embedded frames/objects, and form submission; fonts and network fetches +use the same origin. Inline styles support framework and viewer geometry. +GitHub Pages controls HTTP response headers: a meta policy cannot enforce +`frame-ancestors` or supply host-level security headers. Keep the lockfile, pinned +dependencies, and scoped overrides aligned. An advisory check is current only when +the npm endpoint returns a successful result. diff --git a/website/README.md b/website/README.md index 52e4275..966eda2 100644 --- a/website/README.md +++ b/website/README.md @@ -29,7 +29,7 @@ npm run dev | Directory | Responsibility | | --- | --- | | `app/` | Pages, shared shell and research readers, content registry, and client interactions | -| `content/` | Approved Experience copy, research manuscripts, frozen source evidence, and integrity manifests | +| `content/` | Experience copy, research manuscripts, authoritative source material, and integrity manifests | | `public/` | Assets served directly by GitHub Pages | | `source-assets/` | Editable brand masters | | `scripts/` | Content checks, asset generation, build assembly, and local preview | @@ -64,16 +64,13 @@ Experience entries. `app/lib/content/experience-domains.ts` defines the five cat starts with Position, Location, Dates, and Key Contributions. The four AI and four legal pages continue into their protected manuscripts. -The four AI entries use the approved portfolio copy frozen in -`content/sources/ai-portfolio/`. Run `node scripts/import-ai-portfolio.mjs` from -`website/` to reproduce their briefs, disclosures, contributions, architectural -contexts, nested contribution lists, and mathematical bodies. The import restores -the earlier citation URLs, carries forward the two nonlinear Taylor definitions -from the previous article, and applies the exact scoped corrections recorded in -`editorial-revisions.json`. The original manuscript remains unchanged. Each revision -records its original text, replacement, and reason; a missing or repeated match -stops the import. The source file, citation map, and revision record are protected; -tests check the complete original-to-reviewed text and mathematical rendering. +The four AI entries share one authoritative manuscript: +`content/sources/ai-portfolio/ai-research-and-engineering.md`. Edit that file and run +`node scripts/import-ai-portfolio.mjs` from `website/` to generate their briefs, +disclosures, contributions, architectural contexts, nested method lists, and +mathematical bodies. The importer validates the document structure before writing +the generated JSON. Tests independently compare all text, citation links, tables, +and TeX with this source and check the exported mathematical rendering. The build includes the standalone [architecture viewer](../architecture/). See [maintenance](../docs/maintenance.md) for routes, publishing, and extension diff --git a/website/app/components/experience/Experience.module.css b/website/app/components/experience/Experience.module.css index e96aebd..e54f566 100644 --- a/website/app/components/experience/Experience.module.css +++ b/website/app/components/experience/Experience.module.css @@ -123,10 +123,9 @@ line-height: var(--reading-leading); } .position { grid-column: 1 / -1; } -.metadata dt { color: var(--muted); font-size: var(--type-label); line-height: var(--label-leading); margin-bottom: 3px; } +.metadata dt { color: var(--muted); font-size: var(--type-label); line-height: var(--label-leading); margin-bottom: 0; } .metadata dd { margin: 0; } -.metadata > div { display: grid; grid-template-columns: 90px minmax(0, 1fr); align-items: baseline; gap: 8px 16px; } -.metadata dt { margin-bottom: 0; } +.metadata > div { display: grid; grid-template-columns: 6.5em minmax(0, 1fr); align-items: baseline; gap: 8px 16px; } .contributions h2, .contributions h5 { margin: 0 0 16px; font-size: var(--type-heading); diff --git a/website/app/lib/content/experience-showcase.ts b/website/app/lib/content/experience-showcase.ts index 5de74da..42692b3 100644 --- a/website/app/lib/content/experience-showcase.ts +++ b/website/app/lib/content/experience-showcase.ts @@ -34,11 +34,26 @@ const source = JSON.parse(readFileSync(join(process.cwd(), "content", "experienc categories: { id: string; disclosure?: CopySection }[]; entries: SourceEntry[]; }; -const entrySlugs = [ - ...aiResearchProjectSlugs, "jiritsu-network", "euromonitor-international", "china-construction-bank", - ...legalPaperSlugs, "jones-lang-lasalle", "hubble-technology-investment", "saif-partners", - "citic-securities", "ey-parthenon", "independent-stem-olympiad-training", "stem-olympiad-competitions", -]; +const entrySlugs: Readonly> = { + 1: "statistical-inference-and-resource-allocation-in-expert-data-production", + 2: "task-validity-in-financial-synthetic-data", + 3: "verification-and-supervision-in-scientific-reasoning-tasks", + 4: "evidence-uncertainty-and-decision-guarantees-in-investment-research", + 5: "jiritsu-network", + 6: "euromonitor-international", + 7: "china-construction-bank", + 8: "autonomous-authority-in-space", + 9: "small-states-and-strategic-space-dependence", + 10: "solar-geoengineering-comparison-and-continuity", + 11: "mangrove-restoration-and-compensatory-mitigation", + 12: "jones-lang-lasalle", + 13: "hubble-technology-investment", + 14: "saif-partners", + 15: "citic-securities", + 16: "ey-parthenon", + 17: "independent-stem-olympiad-training", + 18: "stem-olympiad-competitions", +}; // Presentation labels are separate from the approved Experience copy. const legalResearchAreas: Record = { 8: "Space Law and Autonomous Systems Research", @@ -52,7 +67,12 @@ export const showcaseEntries: readonly ShowcaseEntry[] = source.entries.map((ent !entry.metadata.position || !entry.metadata.location || !entry.metadata.dates) { throw new Error(`Incomplete approved Experience entry: ${entry.id}`); } - const slug = entrySlugs[index]; + const slug = entrySlugs[entry.id]; + if (!slug || + (entry.categoryId === "01" && !aiResearchProjectSlugs.some((project) => project === slug)) || + (entry.categoryId === "03" && !legalPaperSlugs.some((paper) => paper === slug))) { + throw new Error(`Experience entry has no matching route: ${entry.id}`); + } const articlePath = entry.categoryId === "01" ? aiResearchProjectPath(slug) : entry.categoryId === "03" ? legalPaperPath(slug) : undefined; const organization = entry.categoryId === "03" ? "Duke Law School" : entry.organization; @@ -65,6 +85,9 @@ export const showcaseEntries: readonly ShowcaseEntry[] = source.entries.map((ent }; }); if (showcaseEntries.length !== 18) throw new Error("The approved Experience document contains 18 entries."); +if (new Set(showcaseEntries.map(({ slug }) => slug)).size !== showcaseEntries.length) { + throw new Error("Experience entries must have unique routes."); +} export const showcaseDomains: readonly ShowcaseDomain[] = experienceDomainDefinitions.map((domain) => ({ ...domain, entries: showcaseEntries.filter((entry) => entry.categoryId === domain.number), diff --git a/website/app/past-experience/components/LegalPaperPage.module.css b/website/app/past-experience/components/LegalPaperPage.module.css index 89e47d5..108235c 100644 --- a/website/app/past-experience/components/LegalPaperPage.module.css +++ b/website/app/past-experience/components/LegalPaperPage.module.css @@ -81,11 +81,6 @@ border-top: 1px solid var(--rule-strong); } -.overview { - min-width: 0; - max-width: var(--reading-measure); -} - .contents { margin-bottom: 2rem; padding: 0.5rem 0; diff --git a/website/content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json b/website/content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json index 5521e14..541e185 100644 --- a/website/content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json +++ b/website/content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json @@ -1,12 +1,12 @@ { "slug": "evidence-uncertainty-and-decision-guarantees-in-investment-research", "title": "Evidence uncertainty and decision guarantees in investment research", - "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", + "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/ai-research-and-engineering.md", "sourceEditedAt": "2026-10-27", "markdown": "## 1 Where existing methods leave the decision unresolved\n\n[SHAP](https://arxiv.org/abs/1705.07874)'s local accuracy property decomposes a given model prediction into additive feature contributions, while conditional Shapley methods address feature dependence and Rashomon-set methods examine explanation stability across near-optimal models (*[SHAP](https://arxiv.org/abs/1705.07874); [Aas et al.](https://arxiv.org/html/1903.10464v3); [Marx et al.](https://proceedings.mlr.press/v206/marx23a.html)*). However, **these explainability methods impose zero requirement that the underlying corporate input facts $`x`$ be true**. In institutional investment diligence—where self-reported disclosures may contain AI-polished greenwashing—a further mathematical layer is required to determine *which factual correction to the evidence could flip the current investment decision*.\n\nFor a linear scoring model under a fixed marginal background $`Q`$, the [SHAP](https://arxiv.org/abs/1705.07874) attribution assigned to feature $`j`$ is $`\\phi_j = \\beta_j(x_j - E_Q X_j)`$. Suppose feature $`j`$ carries an unresolved factual or greenwashing uncertainty bounded by $`|\\delta_j| \\le d_j`$. Its true sensitivity to factual verification is $`|\\beta_j|d_j`$, and over a box of admissible evidence errors, the exact worst-case swing in the company's score is:\n\n$$\n\\sup_{|\\delta_j| \\le d_j} |f(x + \\delta) - f(x)| = \\sum_j |\\beta_j| d_j\n$$\n\nThe triangle inequality gives the upper bound, which is attained by choosing the endpoint of each error interval so that all perturbations push the score in the same direction. This identity exposes a fundamental flaw in standard XAI-driven diligence:\n• A core financial metric with a massive [SHAP](https://arxiv.org/abs/1705.07874) bar $`|\\phi_j|`$ may come from audited statutory filings that have already been thoroughly verified ($`d_j \\approx 0`$); re-auditing it wastes diligence budget.\n• Conversely, a self-reported ESG or operational metric whose current claimed value sits near the sector average ($`x_j \\approx E_Q X_j \\implies \\phi_j \\approx 0`$) appears completely unimportant on a [SHAP](https://arxiv.org/abs/1705.07874) chart, yet may harbor substantial unresolved greenwashing error ($`d_j \\gg 0`$) capable of reversing the deal ranking!\n\nNaive bootstrap resampling of existing records fails for a parallel reason: it cannot detect a shared reporting or AI-marketing bias. Suppose $`n`$ reported observations satisfy $`X_i = \\mu + b + \\varepsilon_i`$, where $`\\varepsilon_i \\sim \\mathcal{N}(0, \\sigma^2)`$ are independent random errors and $`b \\neq 0`$ is an unmodeled systemic bias shared across records (such as sector-wide self-reporting inflation or coordinated AI-PR sentiment manipulation). A conventional nominal 95% normal confidence interval centered on the sample mean $`\\bar{X}`$ covers the true fundamental parameter $`\\mu`$ with probability:\n\n$$\n\\Pr\\left\\{ \\mu \\in \\left[ \\bar{X} - z\\frac{\\sigma}{\\sqrt{n}}, \\ \\bar{X} + z\\frac{\\sigma}{\\sqrt{n}} \\right] \\right\\} = \\Phi\\left( z - \\frac{b\\sqrt{n}}{\\sigma} \\right) - \\Phi\\left( -z - \\frac{b\\sqrt{n}}{\\sigma} \\right) \\longrightarrow 0 \\quad (\\text{as } n \\to \\infty)\n$$\n\nwhere $`z = \\Phi^{-1}(0.975) \\approx 1.96`$. **Collecting more biased documents ($`n \\to \\infty`$) shrinks random variance while driving true confidence coverage to zero!** A bootstrap that resamples the same biased disclosures has no statistical leverage to identify $`b`$. Active evidence verification must therefore bound input uncertainty before stability analysis can provide valid decision guarantees.\n\n## 2 Joint uncertainty over models and evidence\n\nTo evaluate whether an ESG or investment ranking is genuinely trustworthy, our engine represents **model-selection multiplicity** and **unresolved evidence uncertainty** jointly.\n\nFix the prediction target, input definitions, and output scale. Consider $`M`$ candidate scoring models $`\\mathcal{F}_0`$ specified before examining a validation set of $`n`$ i.i.d. observations from the target evaluation distribution, independent of model construction, with loss bounded in $`[0, 1]`$. For failure probability $`\\alpha \\in (0, 1)`$, Hoeffding's inequality and a union bound guarantee:\n\n$$\n\\Pr\\left\\{ \\max_{f \\in \\mathcal{F}_0} |\\hat{L}_n(f) - L(f)| \\le t_n \\right\\} \\ge 1 - \\alpha, \\qquad t_n = \\sqrt{\\frac{\\log(2M / \\alpha)}{2n}}\n$$\n\nLet $`f^*`$ minimize true population loss within $`\\mathcal{F}_0`$, and retain the empirical **Rashomon set** of all near-optimal models within $`2t_n`$ of the empirical validation minimum:\n\n$$\n\\mathcal{F}_\\alpha = \\left\\{ f \\in \\mathcal{F}_0 : \\hat{L}_n(f) \\le \\min_{g \\in \\mathcal{F}_0} \\hat{L}_n(g) + 2t_n \\right\\}\n$$\n\nOn the simultaneous concentration event, $`\\hat{L}_n(f^*) \\le L(f^*) + t_n \\le \\min_g \\hat{L}_n(g) + 2t_n`$, guaranteeing $`f^* \\in \\mathcal{F}_\\alpha`$ while every retained model has population excess loss at most $`4t_n`$ (*[Marx et al.](https://proceedings.mlr.press/v206/marx23a.html)*).\n\nSimultaneously, the **evidence uncertainty set** $`\\mathcal{X}(E)`$ encodes all unresolved questions regarding entity boundaries, reporting periods, physical units, measurement definitions, shared third-party data sources, and suspected greenwashing claims under current diligence state $`E`$. Crucially, each element $`X \\in \\mathcal{X}(E)`$ specifies a *joint* input configuration across all candidate companies—preserving cross-company error dependencies rather than treating each company field as an isolated error bar. Defining the joint uncertainty space $`\\Omega(E) = \\mathcal{F}_\\alpha \\times \\mathcal{X}(E)`$, we compute the exact worst-case pairwise comparison bounds for Companies $`i`$ and $`k`$:\n\n$$\n\\underline{\\Delta}_{ik} = \\inf_{(f, X) \\in \\Omega(E)} [f(x_i) - f(x_k)], \\qquad \\overline{\\Delta}_{ik} = \\sup_{(f, X) \\in \\Omega(E)} [f(x_i) - f(x_k)]\n$$\n\nWhenever $`\\underline{\\Delta}_{ik} > 0`$, the strict ordering $`i \\succ k`$ is certified across every defensible model in $`\\mathcal{F}_\\alpha`$ and every admissible factual resolution in $`\\mathcal{X}(E)`$. If calibration establishes that $`\\mathcal{X}(E)`$ contains the true ground-truth inputs with probability at least $`1 - \\beta`$ at the specified certification stage, a union bound guarantees joint inclusion of $`f^*`$ and the true inputs with probability at least $`1 - \\alpha - \\beta`$. Thus the unconditional probability of issuing a ranking certificate that is false for $`f^*`$ evaluated at the true inputs is at most $`\\alpha + \\beta`$, without requiring independence between model validation and evidence calibration. This guarantee concerns the best model in the prespecified class; it does not certify model-class adequacy or realized investment outcomes. Repeated adaptive certification requires calibration valid across the stages being reported.\n\n## 3 Computing comparison bounds with support functions\n\nOur joint evidence representation also makes worst-case comparison bounds computationally tractable via **convex support functions** (*[Ben-Tal and Nemirovski](https://www2.isye.gatech.edu/~nemirovs/MOR_RCO_1998.pdf), Robust Optimization*). Let $`u \\in U`$ collect shared and company-specific unresolved factual quantities (including both shared benchmark errors and company-specific greenwashing tolerances), parameterizing company inputs as $`x_i(u) = \\bar{x}_i + B_i u`$. Under affine scoring models, the pairwise score difference between Company $`i`$ and Company $`k`$ takes the form $`\\Delta_f(u) = m_f + a_f^\\top u`$, where $`a_f = (B_i - B_k)^\\top \\beta_f`$. Introducing the support function $`h_U(a) = \\sup_{u \\in U} a^\\top u`$ yields exact closed-form comparison bounds:\n\n$$\n\\inf_{u \\in U} \\Delta_f(u) = m_f - h_U(-a_f), \\qquad \\sup_{u \\in U} \\Delta_f(u) = m_f + h_U(a_f)\n$$\n\nFor an ellipsoidal uncertainty set $`U = \\{Cz : \\|z\\|_2 \\le r\\}`$, the Cauchy-Schwarz inequality yields $`h_U(a_f) = r\\|C^\\top a_f\\|_2`$; when $`U`$ is an $`\\ell_\\infty`$ box centered at zero, $`h_U(a_f)`$ reduces to a weighted sum of absolute values.\n\nThis formulation delivers a major operational advantage over naive company-by-company confidence intervals: **Exact Cancellation of Shared-Source Errors**. Let $`S`$ index shared-source coordinates on which the two companies have identical loadings, so $`(B_i-B_k)_{:,S}=0`$. Under the same linear scoring model:\n\n$$\n(a_f)_S=0,\\qquad \\Delta_f(u)=m_f+(a_f)_{S^c}^{\\top}u_{S^c}\n$$\n\nThe shared-source contribution cancels identically in the pairwise comparison; company-specific uncertainty remains in the support-function bound. If all uncertain directions cancel, then $`a_f=0`$ and $`h_U(\\pm a_f)=0`$. Constructing separate uncertainty intervals for Company $`i`$ and Company $`k`$ and subtracting them falsely allows the shared source to vary in opposite directions simultaneously, manufacturing a spurious worst case that paralyzes investment decisions.\n\nSet $`m_f=\\Delta_f(0)`$ and $`a_f=\\nabla\\Delta_f(0)`$.\n\nFor smooth nonlinear models where $`\\Delta_f(u)`$ is twice continuously differentiable on $`\\|u\\|_2 \\le r`$ with Hessian operator norm bounded by $`K_f`$, Taylor's remainder theorem gives the certified lower bound:\n\n$$\n|\\Delta_f(u) - m_f - a_f^\\top u| \\le \\kappa_f, \\qquad \\kappa_f = \\frac{1}{2}K_f r^2 \\implies \\underline{\\Delta}_f \\ge m_f - r\\|a_f\\|_2 - \\kappa_f\n$$\n\nwhile non-smooth tree ensembles are bounded via leaf-region partitioning or mixed-integer linear programming, concentrating compute strictly on company pairs near the zero decision boundary.\n\n## 4 Choosing verification queries by the regret they can remove\n\nWhen $`\\underline{\\Delta}_{ik} \\le 0 < \\overline{\\Delta}_{ik}`$, the ranking between candidate investments remains unresolved—indicating that suspected greenwashing or data ambiguity could flip the decision. Rather than telling the investor which company to pick, our system recommends **which specific verification query $`q`$ (e.g., auditing a supplier contract, requesting raw utility bills, or cross-checking a regulatory filing) to execute next**.\n\nLet $`A`$ be the finite set of institutional actions (such as advancing Company $`i`$ to final diligence, advancing Company $`k`$, or deferring rating) and $`U(a, \\omega)`$ be the pre-specified decision utility under scenario $`\\omega \\in \\Omega`$, with nonempty $`\\Omega`$ and uniformly bounded utilities. The baseline **minimax decision regret** over deterministic actions is:\n\n$$\nR^*(\\Omega) = \\min_{a \\in A} \\sup_{\\omega \\in \\Omega} \\left[ \\max_{b \\in A} U(b, \\omega) - U(a, \\omega) \\right]\n$$\n\nExecuting a verification query $`q`$ produces an observation $`o \\in O_q`$ that restricts the compatible scenario set to a nonempty subset $`\\Omega_{q,o} \\subseteq \\Omega`$. The **guaranteed worst-case reduction in decision regret** provided by query $`q`$ is:\n\n$$\nV(q) = R^*(\\Omega) - \\sup_{o \\in O_q} R^*(\\Omega_{q,o}) \\ge 0\n$$\n\nThe non-negativity of $`V(q)`$ follows immediately from set inclusion $`\\Omega_{q,o} \\subseteq \\Omega`$: shrinking the set of plausible greenwashing/evidence scenarios cannot increase worst-case regret for any fixed action, and minimizing over actions $`a \\in A`$ preserves the inequality (*[Robust Active Preference Elicitation](https://arxiv.org/html/2003.01899)*). Crucially, **$`V(q)`$ requires no subjective probability prior over whether a corporate disclosure is exaggerated**. At a fixed evidence state, maximizing $`V(q)`$ among affordable queries minimizes worst-case regret after one query. With positive unequal query costs $`c_q`$, $`V(q)/c_q`$ is a one-step efficiency heuristic; complementary or sequential queries require joint or adaptive optimization under the total diligence budget $`B`$.\n\n## 5 Checking that corrections propagate through attribution\n\nOnce diligence resolves a query and updates a company's verified evidence vector from $`x`$ to $`x'`$, our pipeline guarantees that the factual correction propagates deterministically through all downstream scores, peer rankings, and [SHAP](https://arxiv.org/abs/1705.07874) explanations. Holding the scoring function $`f`$, background reference $`Q`$, and rubric version fixed, **Shapley efficiency** enforces the exact conservation identity (*[SHAP local accuracy property](https://arxiv.org/pdf/1705.07874)*):\n\n$$\nf(x) = \\phi_0 + \\sum_j \\phi_j(f, Q, x), \\qquad \\sum_j [\\phi_j(f, Q, x') - \\phi_j(f, Q, x)] = f(x') - f(x)\n$$\n\nIf approximate Shapley outputs have efficiency residuals $`|\\phi_0+\\sum_j\\phi_j-f(x)|\\le\\eta`$ and $`|\\phi_0+\\sum_j\\phi'_j-f(x')|\\le\\eta'`$ under the same baseline $`\\phi_0`$, the triangle inequality gives $`|\\sum_j \\Delta \\phi_j - \\Delta f| \\le \\eta + \\eta'`$ for the recomputation gate. This checks aggregate conservation, not the accuracy of each attribution or the truth of the input evidence. Every database update logs explicit version identifiers for `(Field Version, Evidence Document ID, Model Version, Background Q, Rubric Version)` so that changes driven by factual diligence corrections are never conflated with model retraining.\n\n## 6 Reproducible quantitative comparisons\n\nThree reproducible analytical experiments verify the mathematical mechanisms derived above:\n\n1. **Collapse of Conventional Confidence Intervals Under Shared Greenwashing / Reporting Bias**:\n• Set shared reporting bias $`b = 0.2`$ and independent noise $`\\sigma = 1`$. Ignoring the common bias $`b`$, a nominal 95% confidence interval achieves actual coverage of only **48.40% at $`n = 100`$ records**, collapsing to **0.000637% at $`n = 1,000`$ records**. When targeted verification establishes the bias bound $`|b| \\le 0.2`$, expanding the interval half-width to $`0.2 + 1.96/\\sqrt{n}`$ restores **97.50% actual coverage**.\n2. **Shared-Source Error Cancellation in Pairwise Company Ranking**:\n• Two target companies have a nominal score difference of $`0.6`$ points and share an additive sector-benchmark error in $`[-1, 1]`$. Subtracting independently constructed company intervals gives **$`[-1.4, 2.6]`$**, failing to certify an ordering. Our joint support-function model cancels the incompatible error combinations identically, yielding **$`[0.6, 0.6]`$** and certifying the ranking.\n3. **Minimax-Regret Verification Routing vs. Naive [SHAP](https://arxiv.org/abs/1705.07874) Attribution Ranking**:\n• Consider one verification query of equal cost across 8 scenarios formed from 2 Rashomon-set candidate models ($`s \\in \\{-1, 1\\}`$) and 4 unresolved evidence configurations ($`z_1 \\in \\{-0.4, 0.4\\}`$, $`z_2 \\in \\{-0.2, 0.2\\}`$), with pairwise score difference $`\\Delta_s = 0.15s + z_1 + 0.3s z_2`$. A further shared feature has a large [SHAP](https://arxiv.org/abs/1705.07874) attribution but has already been verified ($`d_0 = 0`$). The decision action is to select Company $`i`$ or Company $`k`$, with regret measured in model-score units:\n\n\n\n\n\n\n
Verification QueryRemaining Regret After Worst Possible ObservationGuaranteed Reduction in Regret $`V(q)`$
**Recheck the already-resolved field with a large [SHAP](https://arxiv.org/abs/1705.07874) attribution**0.61**0**
**Verify the smaller unresolved variable $`z_2`$**0.61**0**
**Verify $`z_1`$, which determines the sign of the score difference****0****0.61**
\n\nEnumerating all eight scenarios proves that ranking diligence work by raw [SHAP](https://arxiv.org/abs/1705.07874) magnitude wastes 100% of the query budget ($`V(q) = 0`$), whereas our minimax-regret criterion isolates $`z_1`$ and reduces worst-case decision regret from **0.61 to zero** in a single query. By uniting **Rashomon model multiplicity**, **support-function evidence bounds**, and **minimax-regret verification routing**, this architecture provides institutional investors with a rigorous, auditable defense against AI-era greenwashing.", "sourceDocument": { - "fileName": "theodore_ouyang_ai_portfolio_optimized.md", - "path": "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "sha256": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283" + "fileName": "ai-research-and-engineering.md", + "path": "content/sources/ai-portfolio/ai-research-and-engineering.md", + "sha256": "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d" } } diff --git a/website/content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json b/website/content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json index 236557a..1642336 100644 --- a/website/content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json +++ b/website/content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json @@ -1,12 +1,12 @@ { "slug": "statistical-inference-and-resource-allocation-in-expert-data-production", "title": "Statistical inference and resource allocation in expert data production", - "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", + "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/ai-research-and-engineering.md", "sourceEditedAt": "2026-10-27", "markdown": "## 1 The observation mechanism creates its own failure modes\n\nIn production RLHF pipelines, the routing policy used to inspect submissions alters the statistical properties of the observed sample. Let $`X`$ denote observable task features (encoding domain vertical, prompt complexity, and contributor history) and $`Y \\in \\{0, 1\\}`$ indicate an error in the original submission under fixed acceptance criteria. The true conditional error probability is $`q(x) = P(Y = 1 \\mid X = x)`$. Let $`S = 1`$ indicate selection for an independent reference audit. Conditional on $`X`$, selection is randomized with probability $`\\pi(x) > 0`$, independently of the reference label that has yet to be observed.\n\nWhen operational dashboards use the error fraction among verified submissions to assess the full population, they actually target:\n\n$$\nE[Y \\mid S = 1] = \\frac{E[\\pi(X)q(X)]}{E[\\pi(X)]}\n$$\n\n$$\nE[Y \\mid S = 1] - E[Y] = \\frac{\\operatorname{Cov}(\\pi(X), q(X))}{E[\\pi(X)]}\n$$\n\nWhen review priorities successfully target high-risk tasks, the covariance between audit probability $`\\pi(X)`$ and true defect probability $`q(X)`$ is strictly positive. A higher error detection rate in the reviewed sample cannot then be read directly as a deterioration in overall production quality; it often reflects sharper risk targeting. This identity describes the distribution of a selected item. In a finite batch, the ratio of observed errors to selected items also has a random denominator, so the two quantities should not be conflated.\n\nExpert rankings are affected by the exact same mechanism. Let $`a_{jh}`$ be the probability that expert $`j`$ is correct on task stratum $`h`$, and let $`p_{jht}`$ describe the mix of tasks assigned to that expert in period $`t`$. Even with unchanged underlying ability, observed accuracy shifts whenever task routing changes:\n\n$$\nA_{jt} = \\sum_h p_{jht} a_{jh}, \\qquad A_{j,t+1} - A_{jt} = \\sum_h (p_{jh,t+1} - p_{jht}) a_{jh}\n$$\n\nValid cross-period comparisons therefore require a fixed target task mix $`w_h`$, reporting standardized accuracy:\n\n$$\nA_j^{\\text{std}} = \\sum_h w_h a_{jh}\n$$\n\nwith overlapping assignments or shared anchor cases to support estimation within each stratum. In our production routing engine, we model each contributor's stratum-level reliability via a conjugate Beta-Binomial scheme: given prior $`\\operatorname{Beta}(\\alpha_{0h}, \\beta_{0h})`$ and $`n_{jht}`$ audited tasks with $`k_{jht}`$ verified successes, the posterior reliability distribution $`\\operatorname{Beta}(\\alpha_{0h} + k_{jht}, \\beta_{0h} + n_{jht} - k_{jht})`$ yields shrinkage-regularized point estimates and credible intervals that account for unequal sample sizes across comparable audits. Pooling different difficulty levels, criteria versions, and selection mechanisms into a single unstratified binomial sample leaves the preceding selection bias unresolved.\n\nThese distinctions also guide algorithmic method selection. **[Dawid-Skene](https://doi.org/10.2307/2346806)** models represent differences in reviewer confusion matrices across latent ground truth, whereas **[DeCCaF](https://arxiv.org/html/2403.06906v1)** incorporates task conditions, error costs, and reviewer capacity into human-AI deferral and highlights how assigning difficult cases distorts naive evaluations of expert performance. The architecture developed here addresses the critical interface between selective operational deferral and unbiased population quality inference in ongoing post-training production (*[Dawid-Skene](https://doi.org/10.2307/2346806); [DeCCaF](https://arxiv.org/html/2403.06906v1)*).\n\n## 2 Correct predictions with audits to recover the population target\n\nFix a production batch of $`N`$ original submissions, its acceptance criteria, and its latent reference labels $`Y_1, \\dots, Y_N`$. The inferential target is the finite-batch error rate $`\\mu = N^{-1} \\sum_{i=1}^N Y_i`$. Let $`f_i \\in [0, 1]`$ be a fixed, inexpensive risk prediction (synthesized from contributor posterior reliability, task complexity features, and automated heuristic checks) and $`\\pi_i \\in [\\pi_{\\min}, 1]`$ be the assigned audit probability, with independent draws $`S_i \\sim \\operatorname{Bernoulli}(\\pi_i)`$. Both predictions and sampling probabilities are locked before the current batch's reference labels are revealed.\n\nWe deploy a prediction-assisted, inverse-probability-weighted (IPW) residual-corrected estimator:\n\n$$\n\\hat{\\mu} = \\frac{1}{N} \\sum_{i=1}^N \\left[ f_i + \\frac{S_i}{\\pi_i} (Y_i - f_i) \\right]\n$$\n\nIts design-based validity follows directly. Write the prediction residual as $`e_i = Y_i - f_i`$ and condition on the fixed batch, predictions, and sampling probabilities:\n\n$$\n\\hat{\\mu} - \\mu = \\frac{1}{N} \\sum_{i=1}^N \\left( \\frac{S_i}{\\pi_i} - 1 \\right) e_i, \\qquad E_S[\\hat{\\mu} - \\mu] = 0\n$$\n\nInverse-probability weighting restores the population representation altered by selective verification, while predictions reduce the residual variation that expensive human expert audits must resolve. Crucially for production robustness, **this design-based unbiasedness does not require $`f_i = q(X_i)`$**: inaccurate predictions can reduce statistical efficiency, but they never introduce selection bias under the specified Bernoulli sampling design. While the estimator builds on established active statistical inference principles (*[Active Statistical Inference: mean estimation and sampling design](https://arxiv.org/html/2403.03208v3)*), our production architecture specifies the exact immutable objects being audited, the rubric version lineage, and the telemetry records required for downstream optimization.\n\nSpecifically, original and revised submissions must be retained as separate database entities. Estimating $`\\hat{\\mu}_{\\text{raw}}`$ on frozen originals and $`\\hat{\\mu}_{\\text{final}}`$ on post-review deliverables cleanly distinguishes *defects generated in initial production* from *defects corrected by review*. Furthermore, whenever post-training criteria evolve, a shared set of anchor cases is adjudicated under both rubric versions; submissions from different periods are then compared under a common reference version, preventing stricter acceptance standards from being mistaken for declining contributor capability.\n\n## 3 Allocate audit effort according to residuals and cost\n\nIndependent Bernoulli sampling eliminates cross-item covariance terms, yielding the exact design variance:\n\n$$\n\\operatorname{Var}_S(\\hat{\\mu}) = \\frac{1}{N^2} \\sum_{i=1}^N \\frac{1 - \\pi_i}{\\pi_i} e_i^2\n$$\n\nThis expression identifies the true mathematical objective of audit allocation: **audit effort should address what the automated risk predictions have not resolved**. For a binary error label with conditional error probability $`q_i`$, the expected squared residual decomposes into two components:\n\n$$\nm_i := E[(Y_i - f_i)^2 \\mid X_i] = q_i(1 - q_i) + (q_i - f_i)^2\n$$\n\nThe two terms are intrinsic conditional Bernoulli variance $`q_i(1 - q_i)`$ and squared prediction calibration error $`(q_i - f_i)^2`$. This identity establishes a counter-intuitive operational principle: **a high error probability $`q_i`$ does not necessarily imply high inferential value**. If a task category is almost certain to contain an error ($`q_i \\approx 1`$) and the predictor correctly recognizes that fact ($`f_i \\approx 1`$), then $`m_i \\approx 0`$—spending an independent reference audit on that item yields less statistical information about population quality than auditing a moderately risky task whose outcome is genuinely uncertain ($`q_i \\approx 0.5`$). The high-risk task still merits operational correction, which is precisely why the *value of measurement* and the *value of remediation* require separate mathematical models.\n\nLet $`c_i > 0`$ be the empirical cost of an independent reference audit (calibrated by reconciling platform review logs with Hubstaff active-work telemetry) and $`B`$ be the expected audit-cost budget. First consider the optimal design when the true residual second moments $`m_i > 0`$ are known:\n\n$$\n\\min_{\\pi} \\sum_{i=1}^N \\frac{m_i}{\\pi_i} \\quad \\text{s.t.} \\quad \\sum_{i=1}^N c_i \\pi_i \\le B, \\quad \\pi_{\\min} \\le \\pi_i \\le 1\n$$\n\nThe omitted constant $`-\\sum_i m_i`$ in the numerator of the variance does not depend on the allocation $`\\pi`$. For probabilities strictly between their lower and upper bounds, the Lagrangian first-order condition yields the clipped square-root allocation law:\n\n$$\n-\\frac{m_i}{\\pi_i^2} + \\lambda c_i = 0 \\implies \\pi_i^* = \\operatorname{clip}_{[\\pi_{\\min}, 1]} \\left( \\sqrt{\\frac{m_i}{\\lambda c_i}} \\right)\n$$\n\nfor budgets below the cost of a full audit, $`\\lambda > 0`$ is chosen to satisfy the budget constraint (the multiplier need not be unique when clipping is active), and feasibility requires $`B \\ge \\pi_{\\min} \\sum_i c_i`$. If the budget covers a full audit, every item is audited ($`\\pi_i^* = 1`$). The square-root law increases sampling intensity for larger expected residuals while penalizing expensive audits, and the positive probability floor $`\\pi_{\\min}`$ preserves observation coverage and bounded IPW weights across the entire task space.\n\nProduction implementation substitutes regularized stratum estimates $`\\hat{m}_i > 0`$ obtained from earlier independent audits or a pilot calibration batch. True $`m_i`$ defines the optimality analysis; estimated values $`\\hat{m}_i`$ determine the operational allocation. By estimating residuals at the level of `(task stratum × criteria version × cost stratum)`, the pipeline avoids fitting unstable parameters to individual submissions. (Note that $`B`$ constrains *expected* audit cost per batch. Workflows requiring an exact integer count must use a feasible fixed-size sampling design: its first-order inclusion probabilities sum to that integer, and its variance depends on joint inclusion probabilities. These probabilities need not equal the Bernoulli optimum $`\\pi_i^*`$.)\n\n## 4 Derive the gain and identify when it disappears\n\nWe now derive the exact analytical efficiency gain of cost-aware residual allocation over uniform sampling using the same predictor, and characterize when active allocation fails. Let $`C = \\sum_{i=1}^N c_i`$, so that the budget-equivalent uniform sampling probability is $`p = B / C`$. When true residual second moments $`m_i`$ are available and no optimal probability reaches a clipping boundary, averaging design variance over the conditional label distribution gives:\n\n$$\nV_{\\text{uniform}} = \\frac{1}{N^2} \\left[ \\frac{C \\sum_i m_i}{B} - \\sum_i m_i \\right]\n$$\n\n$$\nV_{\\text{optimal}} = \\frac{1}{N^2} \\left[ \\frac{\\left(\\sum_i \\sqrt{m_i c_i}\\right)^2}{B} - \\sum_i m_i \\right]\n$$\n\n(If a common sampling probability is used within each fixed task stratum, substituting that stratum's finite-batch mean squared residual for $`m_i`$ makes these expressions exact finite-batch identities.) By the Cauchy-Schwarz inequality:\n\n$$\nV_{\\text{uniform}} - V_{\\text{optimal}} = \\frac{C \\sum_i m_i - \\left(\\sum_i \\sqrt{m_i c_i}\\right)^2}{N^2 B} \\ge 0\n$$\n\nEquality holds **if and only if $`m_i / c_i`$ is constant across all items**. Efficiency gains therefore depend strictly on identifiable heterogeneity in residual second moment per unit audit cost; with homogeneous residuals and costs, optimal allocation reduces to uniform sampling. When clipping is active at $`\\pi_{\\min}`$ or $`1`$, constrained optimization is still no worse than feasible uniform policies, with the exact gain evaluated directly at the clipped primal objective.\n\nCrucially, the same analysis exposes the exact variance penalty of model misspecification. If the allocation uses inaccurate residual estimates $`\\hat{m}_i`$ and neither allocation is clipped, the excess variance relative to the true optimum is:\n\n$$\nV(\\hat{\\pi}) - V(\\pi^*) = \\frac{1}{N^2 B} \\left[ \\left( \\sum_i \\sqrt{\\hat{m}_i c_i} \\right) \\left( \\sum_i \\frac{m_i \\sqrt{c_i}}{\\sqrt{\\hat{m}_i}} \\right) - \\left( \\sum_i \\sqrt{m_i c_i} \\right)^2 \\right] \\ge 0\n$$\n\nSeverely underestimating $`\\hat{m}_i`$ in any region of the task space inflates the penalty through the $`\\sqrt{\\hat{m}_i}`$ denominator. Enforcing a positive sampling floor $`\\pi_{\\min}`$ caps this worst-case blowup, while mixing active and uniform probabilities via the budget-preserving defensive convex combination:\n\n$$\n\\pi_i(\\rho) = (1 - \\rho)\\hat{\\pi}_i + \\rho p\n$$\n\nallows the shrinkage weight $`\\rho \\in [0, 1]`$ to be tuned across independent calibration batches (*[Robust Sampling for Active Statistical Inference](https://arxiv.org/html/2511.08991v1)*).\n\n## 5 Allocate review by expected loss reduction\n\nWhereas statistical auditing seeks to *measure* population quality with minimum variance, operational review seeks to *remove defects* from the current batch before training handoff. For task $`i`$ under handling action $`a`$ (e.g., accept directly, route to peer review, escalate to expert adjudication, or request contributor revision), let $`r_{ia}`$ be the expected error loss remaining *after* the action, $`d_{ia}`$ its operational cost, and $`t_{iaj}`$ the time required from specialist $`j`$ (measured via platform and Hubstaff telemetry). Binary assignment decisions $`z_{ia} \\in \\{0, 1\\}`$ solve:\n\n$$\n\\min_{z} F(z) = \\sum_{i,a} z_{ia}(r_{ia} + \\eta d_{ia}) \\quad \\text{s.t.} \\quad \\sum_a z_{ia} = 1, \\quad \\sum_{i,a} z_{ia} t_{iaj} \\le H_j \\quad \\forall j\n$$\n\nwhere $`H_j`$ is available specialist time and $`\\eta \\ge 0`$ converts resource cost to the scale of error loss (*[DeCCaF: capacity constraints and global assignment](https://arxiv.org/html/2403.06906v1)*).\n\nWhen review costs are equal and each reviewed task consumes one slot, **the optimal priority ranking is by marginal remediation gain $`\\Delta_i = r_{i0} - r_{i1}`$, not by initial risk $`r_{i0}`$**. Missing external evidence may make a high-risk task difficult to repair through one additional review ($`\\Delta_i`$ small), whereas a self-contained calculation error in a moderate-risk task can be completely eliminated ($`\\Delta_k`$ large). Moving a single review slot from task $`i`$ to task $`k`$ changes total expected loss by:\n\n$$\n\\Delta F = (r_{i0} + r_{k1}) - (r_{i1} + r_{k0}) = \\Delta_i - \\Delta_k\n$$\n\nWhenever $`\\Delta_k > \\Delta_i`$, the exchange strictly reduces delivered defect loss, regardless of the tasks' initial risk ordering.\n\nWe can also state the exact calibration precision required of the action-risk estimators. Suppose $`|\\hat{r}_{ia} - r_{ia}| \\le \\delta`$ for every feasible action, costs and capacities are known, and $`\\hat{z}`$ minimizes the predicted objective $`\\hat{F}(z)`$. Relative to the true optimal feasible assignment $`z^*`$:\n\n$$\nF(\\hat{z}) \\le \\hat{F}(\\hat{z}) + N\\delta \\le \\hat{F}(z^*) + N\\delta \\le F(z^*) + 2N\\delta\n$$\n\nThis bound identifies *post-action residual loss* $`r_{ia}`$ as the critical empirical calibration target, motivating randomized pilot trials across permitted review actions.\n\nFinally, both budgets—remediation and statistical measurement—combine into a single batch-level **Mixed-Integer Convex Program (MICP)**. Let $`\\tau^2`$ be the target variance ceiling for the quality estimate, $`B_{\\text{total}}`$ the combined cost budget, and $`V_{\\hat{m}}(\\pi) = N^{-2}\\sum_i (1 - \\pi_i)\\hat{m}_i / \\pi_i`$ the planning variance under the calibrated residual model:\n\n$$\n\\min_{z, \\pi} \\sum_{i,a} z_{ia}\\hat{r}_{ia} \\quad \\text{s.t.} \\quad \\sum_{i,a} z_{ia} d_{ia} + \\sum_i c_i \\pi_i \\le B_{\\text{total}}, \\quad V_{\\hat{m}}(\\pi) \\le \\tau^2\n$$\n\n$$\n\\sum_a z_{ia} = 1, \\quad z_{ia} \\in \\{0, 1\\}, \\quad \\sum_{i,a} z_{ia} t_{iaj} \\le H_j^{\\text{review}} \\ \\forall j, \\quad \\pi_{\\min} \\le \\pi_i \\le 1\n$$\n\nHere $`H_j^{\\text{review}}`$ is hard capacity allocated to remediation actions while reference audits are scheduled independently on frozen original submissions. The precision ceiling is model-based: independent audits must check residual calibration, and expected audit cost does not impose a hard limit on realized audit hours. Infeasibility exposes a conflict between the planning precision target, feasible review assignments, and available engineering budget.\n\n## 6 Reproducible experiments at equal cost\n\nThe following self-contained synthetic experiments test the mathematical mechanisms derived above (they validate algorithmic behavior on specified public parameters rather than reporting proprietary client telemetry). A fixed batch contains $`N = 10,000`$ submissions partitioned into three strata of $`6,000`$, $`3,000`$, and $`1,000`$ items, containing $`60`$, $`300`$, and $`350`$ errors respectively (true batch defect rate $`\\mu = 7.10\\%`$). Unit audit costs are $`1`$, $`1.5`$, and $`3`$ units across the three strata. Every policy operates under an expected audit budget of $`B = 2,000`$ units and a minimum sampling floor of $`\\pi_{\\min} = 3\\%`$.\n\nRisk predictions for the three strata are $`2\\%`$, $`8\\%`$, and $`30\\%`$ (batch average $`6.60\\%`$, differing from the true defect rate by $`0.50`$ percentage points). Cost-aware allocation uses the true within-stratum mean squared residuals (ideal calibration), while the misspecified policy reverses the residual ordering across strata. Holding predictions, budget, and submissions fixed, $`200,000`$ independent Monte Carlo repetitions verify the analytical variance formulas:\n\n\n\n\n\n\n\n\n
Audit PolicySampling Probabilities Across the Three StrataStd. Dev. of Corrected Estimate (pct. points)Variance Change Relative to Uniform Sampling
**Uniform sampling with the same residual correction**14.81% / 14.81% / 14.81%0.568Baseline
**Prioritize predicted error probability ($`f_i`$)**3.00% / 11.56% / 43.33%0.657+33.77% increase
**Residual- and cost-aware allocation with ideal calibration**7.89% / 19.37% / 21.84%**0.515****-17.79% reduction**
**Misspecified residual ordering (reversed)**23.00% / 11.77% / 3.00%0.983+199.66% increase
**Misspecified allocation with 70% weight on uniform sampling**17.27% / 13.90% / 11.27%0.615+17.07% increase
\n\nUnder ideal residual calibration, the standard deviation of the population quality estimate falls by **9.33%** (a **17.79% variance reduction**). Notice that for the naive high-risk-first policy, the uncorrected raw error detection rate averages **19.60%** (wildly distorting true quality), whereas our IPW residual-corrected estimator recovers **7.1015%** while exposing that naive risk-prioritization actually *inflates* estimation variance by **33.77%** due to over-sampling the expensive third stratum. Across all five policies, Monte Carlo variances match analytical formulas to within $`0.5\\%`$.\n\nA separate, exact allocation experiment isolates remediation gains. Two task groups contain $`100`$ items each, with initial error risks of $`0.40`$ and $`0.15`$. Additional review reduces those risks to $`0.30`$ ($`\\Delta_1 = 0.10`$) and $`0.01`$ ($`\\Delta_2 = 0.14`$), respectively. Review costs are equal and capacity is $`100`$ items. Ranking by initial risk leaves $`100(0.30) + 100(0.15) = 45`$ expected errors; ranking by expected loss reduction ($`\\Delta_i`$) leaves $`100(0.40) + 100(0.01) = 41`$ expected errors—cutting expected defect loss by **8.89%** at the exact same review volume.\n\n## 7 Establish whether the design improves production\n\nProduction deployment tests statistical quality inference and operational defect control separately, recognizing that *detecting more errors in a targeted queue is never, by itself, evidence of better delivered quality*:\n1. **Inferential Validation on Frozen Adjudicated Batches**: Compare uniform sampling, legacy high-risk routing, cost-aware residual allocation, and defensive mixtures against complete independent adjudication—evaluating bias, MSE, and interval coverage across individual task strata and rubric versions.\n2. **Capacity-Matched Remediation Trials**: Compare legacy tiered rules against marginal-gain ($`\\Delta_i`$) action allocation at equal total specialist hours, measuring severe defects surviving into final post-training corpora and total fully-loaded cost per accepted submission.\n\nThe contributor reliability records, tiered review workflows, and random reference audits I established from September 2023 provided the operational foundation for this architecture. The mathematical analysis above states the sampling, calibration, and capacity assumptions under which quality estimation is unbiased and audit allocation is optimal when residual second moments are correctly specified; the synthetic benchmarks test those mechanisms under their stated conditions.", "sourceDocument": { - "fileName": "theodore_ouyang_ai_portfolio_optimized.md", - "path": "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "sha256": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283" + "fileName": "ai-research-and-engineering.md", + "path": "content/sources/ai-portfolio/ai-research-and-engineering.md", + "sha256": "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d" } } diff --git a/website/content/artificial-intelligence/task-validity-in-financial-synthetic-data.json b/website/content/artificial-intelligence/task-validity-in-financial-synthetic-data.json index b456580..0503122 100644 --- a/website/content/artificial-intelligence/task-validity-in-financial-synthetic-data.json +++ b/website/content/artificial-intelligence/task-validity-in-financial-synthetic-data.json @@ -1,12 +1,12 @@ { "slug": "task-validity-in-financial-synthetic-data", "title": "Task validity in financial synthetic data", - "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", + "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/ai-research-and-engineering.md", "sourceEditedAt": "2026-10-27", "markdown": "## 1 Why better distributional fit may leave task validity unchanged\n\nModern tabular generative models address distinct distributional challenges: **[CTGAN](https://arxiv.org/abs/1907.00503)** handles mixed discrete-continuous types, multimodality, and class imbalance via mode-specific normalization and conditional vectors, while **[TabDDPM](https://proceedings.mlr.press/v202/kotelnikov23a.html)** applies Gaussian and multinomial diffusion to numerical and categorical fields (*[CTGAN](https://arxiv.org/abs/1907.00503); [TabDDPM](https://proceedings.mlr.press/v202/kotelnikov23a.html)*). However, while both learn empirical distributions effectively, **financial task validity requires strict structural invariants that global distributional similarity metrics cannot enforce**: exact accounting identities, chronological ordering of fiscal periods, covenant applicability conditions, and logical consistency between numerical inputs and reference judgments.\n\nConsider a failure mode that admits an exact mathematical proof. Valid financial inputs $`x \\in \\mathbb{R}^d`$ must satisfy $`m`$ linear accounting or ratio identities $`Ax = b`$, where $`A \\in \\mathbb{R}^{m \\times d}`$ has full row rank. The valid population distribution $`P`$ is supported entirely on the affine subspace $`M = \\{x \\in \\mathbb{R}^d : Ax = b\\}`$ and has finite second moments. Let the columns of $`N \\in \\mathbb{R}^{d \\times m}`$ form an orthonormal basis for the normal space orthogonal to $`M`$. Suppose a generative model captures $`P`$ tangential to the manifold but adds a tiny amount of independent noise in the normal directions: $`X_\\sigma = X + \\sigma N \\varepsilon`$, where $`\\varepsilon \\sim \\mathcal{N}(0, I_m)`$ is independent of $`X \\sim P`$. Writing the generated distribution as $`Q_\\sigma`$, we have for every $`\\sigma > 0`$:\n\n$$\nW_2(P, Q_\\sigma) \\le \\sqrt{E\\|X_\\sigma - X\\|^2} = \\sigma \\sqrt{m} \\longrightarrow 0 \\quad (\\text{as } \\sigma \\to 0^+)\n$$\n\n$$\nP(M) = 1, \\qquad Q_\\sigma(M) = 0\n$$\n\n$$\n\\|P - Q_\\sigma\\|_{\\text{TV}} := \\sup_B |P(B) - Q_\\sigma(B)| = 1\n$$\n\nThe first line follows from the coupling $`(X, X_\\sigma)`$ constructed on the same probability space. The second follows because $`AN`$ is invertible and a continuous $`m`$-dimensional Gaussian vector equals zero with probability zero. The third follows by taking the Borel set $`B = M`$. Thus, **the 2-Wasserstein distance $`W_2(P, Q_\\sigma)`$ can converge to zero while the acceptance rate under exact business constraints remains identically zero ($`Q_\\sigma(M) = 0`$) and Total Variation distance remains at its maximum of $`1`$**.\n\nThis counterexample dictates how responsibilities must be partitioned in a production synthetic data architecture (*[SDV constraint-augmented generation](https://docs.sdv.dev/sdv/modeling/constraint-augmented-generation-cag)*):\n• **Copula models and conditional sampling** capture cross-sectional rank and tail dependencies across continuous corporate indicators (company scale, operating history, leverage, and margin ratios) so columns are never generated independently.\n• **[CTGAN](https://arxiv.org/abs/1907.00503) and [TabDDPM](https://proceedings.mlr.press/v202/kotelnikov23a.html)** model complex mixed-type and multimodal empirical distributions, followed by explicit business-rule validation.\n• **Deterministic variable transformations and manifold projections** enforce hard accounting equalities, ratio bounds, and chronological sequences exactly.\n\n## 2 Enforce business constraints while preserving the correct conditional distribution\n\nEnforcing hard constraints by generating unconstrained candidates and discarding violations (post-hoc rejection sampling) becomes computationally intractable as multiple financial constraints tighten simultaneously. Suppose a candidate draw has $`m`$ standardized constraint residuals that behave as independent standard normal variables, and each must lie within a tight tolerance $`[-\\epsilon, \\epsilon]`$. The acceptance probability and expected candidate draws required per valid input are:\n\n$$\n\\alpha_m(\\epsilon) = [2\\Phi(\\epsilon) - 1]^m \\sim \\left( \\sqrt{\\frac{2}{\\pi}} \\epsilon \\right)^m \\quad (\\epsilon \\to 0^+), \\qquad E[N_{\\text{attempts}}] = \\frac{1}{\\alpha_m(\\epsilon)}\n$$\n\nEach additional accounting or ratio constraint multiplies the expected candidate count. Setting each tolerance to $`\\epsilon = 0.05`$ residual standard deviations yields the following exact analytical scaling:\n\n\n\n\n\n\n
Simultaneous Constraints ($`m`$)Rejection Sampling Acceptance Rate $`\\alpha_m(\\epsilon)`$Expected Candidate Draws per Valid Input
**1**3.9878%25.08
**2**0.1590%628.84
**4****0.0002529%****395,442.59**
\n\nGenerating directly through the degrees of freedom permitted by the constraints eliminates these wasted draws. In the linear Gaussian regime, covariance-weighted affine projection also preserves the exact theoretical conditional distribution. For a Gaussian copula, this result applies only when the constraints are affine in its latent Gaussian coordinates; nonlinear marginal transformations do not generally preserve affine constraints in the observed financial variables. Let the initial unconstrained draw be $`Z \\sim \\mathcal{N}(\\mu, \\Sigma)`$ with $`\\Sigma \\succ 0`$. Rather than overwriting a single balance-sheet line item (which corrupts covariance structure), we apply a **covariance-weighted Mahalanobis projection**:\n\n$$\nX = \\arg\\min_{x : Ax = b} \\frac{1}{2}(x - Z)^\\top \\Sigma^{-1}(x - Z) = Z + K(b - AZ), \\qquad K = \\Sigma A^\\top (A \\Sigma A^\\top)^{-1}\n$$\n\nThe first-order Lagrangian stationarity condition is $`\\Sigma^{-1}(X - Z) + A^\\top \\lambda = 0`$; substituting $`AX = b`$ and eliminating the multiplier $`\\lambda`$ yields the closed-form update. Since $`AK = I_m`$, every draw satisfies $`AX = b`$ exactly, with mean and covariance:\n\n$$\nE[X] = \\mu + K(b - A\\mu)\n$$\n\n$$\n\\operatorname{Cov}(X) = (I - KA)\\Sigma(I - KA)^\\top = \\Sigma - \\Sigma A^\\top (A \\Sigma A^\\top)^{-1} A\\Sigma\n$$\n\nBecause an affine transformation preserves Gaussianity, these moments match the exact regular conditional Gaussian law of $`Z \\mid AZ = b`$ (*[Majumdar and Majumdar on Gaussian distributions under linear conditioning](https://arxiv.org/abs/1710.09285)*).\n\nIn our synthetic verification benchmark, generating **200,000 synthetic 12-dimensional financial vectors under 4 simultaneous linear constraints** achieved a **maximum absolute constraint residual of $`1.21 \\times 10^{-14}`$** and a **relative Frobenius covariance error of 0.662%** against the theoretical conditional covariance—using direct constrained draws without post-hoc rejection. The nearly 400,000 candidate count above belongs to the separate standardized-residual rejection model. Fields fully determined by accounting definitions are computed analytically; range bounds and chronological date orderings that admit bijective transformations (such as log-transforms or positive increments $`t_{k} = t_{k-1} + \\exp(\\xi_k)`$) are enforced in latent generation space.\n\n## 3 Use decision margins to determine when reference judgments must be recomputed\n\nSynthetic expansion of a golden financial case often preserves the qualitative business narrative while perturbing decisive financial figures, covenant thresholds, or macroeconomic inputs. **Retaining the original golden-case reference answer after perturbing the numbers injects systematic label corruption into post-training**, even when every synthetic input satisfies all accounting constraints. Just as *[FinQA](https://aclanthology.org/2021.emnlp-main.300/)* pairs financial questions with executable reasoning programs, synthetic preference engineering requires the link between inputs, reference conclusions, and expert preference explanations to update dynamically (*[FinQA](https://aclanthology.org/2021.emnlp-main.300/)*).\n\nConsider a formal decision rule for choosing between two corporate actions. Let $`d(x)`$ denote the net valuation or metric advantage of Action A over Action B, so the true reference decision is $`\\operatorname{sign} d(x)`$. If $`d`$ is $`L`$-Lipschitz on a feasible neighborhood, a sufficient condition for retaining the original reference decision without recomputation is that the perturbation $`\\Delta`$ remains strictly within the decision margin:\n\n$$\n|d(x + \\Delta) - d(x)| \\le L\\|\\Delta\\|, \\qquad |d(x)| > L\\|\\Delta\\| \\implies \\operatorname{sign} d(x + \\Delta) = \\operatorname{sign} d(x)\n$$\n\nBecause our diagnostic tasks deliberately target **contested decision boundaries** where models are most prone to error, $`|d(x)|`$ is intentionally small! We can derive the exact closed-form rate of **stale-label error** for a linear comparison rule $`d(x) = w^\\top x - c`$ with initial margin $`\\gamma = d(x) > 0`$. Under a feasible manifold perturbation $`\\Delta \\sim \\mathcal{N}(0, \\Omega)`$ with projected variance $`s^2 = w^\\top \\Omega w > 0`$, the probability that retaining the original decision reverses the true ground-truth label is:\n\n$$\n\\Pr\\{d(x + \\Delta) < 0\\} = \\Pr\\{w^\\top \\Delta < -\\gamma\\} = \\Phi\\left(-\\frac{\\gamma}{s}\\right)\n$$\n\nVerified across $`1,000,000`$ Monte Carlo draws:\n• When the margin-to-noise ratio $`\\gamma / s = 2`$, the inherited-label error rate is **2.28%**.\n• When $`\\gamma / s = 1`$, the inherited-label error rate jumps to **15.87%**.\n• Near diagnostic decision boundaries where $`\\gamma / s = 0.25`$, **40.13% of retained labels are false!**\n\nThis analytical result explains why our pipeline links case expansion directly to reference recomputation and fresh **five-level pairwise preference grading** (see the later methodological comparison in *[HelpSteer2-Preference](https://arxiv.org/abs/2410.01257)*). Whenever a synthetic perturbation crosses or approaches a decision boundary, the reference solution is recomputed, fresh candidate model trajectories are sampled, and domain experts evaluate the pair on our five-level preference scale—explicitly annotating which trajectory identified the flipped decisive variable and supported its judgment with primary evidence.\n\n## 4 Define the information available for each temporal task\n\nWhile cross-sectional corporate comparisons do not require continuous-time stochastic processes, dynamic financial tasks—such as derivatives valuation, interest-rate term-structure interpolation, and multi-period liquidity stress testing—require path generators. In these temporal regimes, we deploy **stochastic differential equations (SDEs)** and **neural SDEs** to model path distributions beyond single-time marginals (*[Neural SDEs as Infinite-Dimensional GANs](https://proceedings.mlr.press/v139/kidger21b.html)*), alongside **Brownian bridges** and **stochastic interpolation** (aligning with classical MathWorks stochastic interest-rate path constructions) when conditioning on known intermediate observations or endpoints.\n\nHowever, path conditioning introduces a severe **temporal evaluation leakage hazard** whose magnitude can be derived in closed form. Consider a scaled Brownian motion $`X_u = \\sigma W_u`$ over $`s < t < T`$. Under squared error loss, the optimal squared-error predictor acting at time $`s`$ with access only to the historical filtration $`\\mathcal{F}_s`$ has conditional mean squared error $`\\sigma^2(t - s)`$. If a synthetic pipeline conditions path generation on the future terminal endpoint $`X_T`$ and inadvertently exposes that conditioned structure to a forecasting task, Gaussian conditioning gives (*[Pitman on the Brownian bridge](https://www.stat.berkeley.edu/~pitman/s205s03/b_bridge.pdf)*):\n\n$$\n\\operatorname{Var}(X_t \\mid \\mathcal{F}_s) = \\sigma^2(t - s)\n$$\n\n$$\n\\operatorname{Var}(X_t \\mid \\mathcal{F}_s, X_T) = \\sigma^2(t - s) - \\frac{[\\sigma^2(t - s)]^2}{\\sigma^2(T - s)} = \\sigma^2(t - s)\\frac{T - t}{T - s}\n$$\n\nWhen $`t = (s + T)/2`$ is the midpoint of the forecasting horizon, conditioning on $`X_T`$ halves the conditional variance, producing an **apparent 29.29% reduction in RMSE** ($`1 - 1/\\sqrt{2}`$) that is 100% attributable to future information leakage rather than model capability! (By contrast, if the task explicitly supplies both endpoints $`X_s`$ and $`X_T`$ and asks the model to audit an *interpolation* or back-cast an intervening accounting period, Brownian bridge conditioning is mathematically appropriate.)\n\nOur task environment specification therefore enforces a strict separation across three information sets:\n1. **Construction Information ($`\\mathcal{F}_{\\text{gen}}`$)**: Complete paths, terminal endpoints, or latent parameters used by the generator to synthesize the scenario.\n2. **Permitted Predictor Filtration ($`\\mathcal{F}_s`$)**: The exact historical information legally visible to the model at decision time $`s`$.\n3. **Evaluation & Oracle Information ($`\\mathcal{F}_{\\text{eval}}`$)**: Realized post-$`s`$ outcomes reserved strictly for scoring or explicitly disclosed oracle baselines.\n\n## 5 Test independent execution through Batch Zero and retain the task family as a statistical unit\n\nEven when constraint projections, decision-margin recomputations, and filtration rules are mathematically sound, a post-training specification must remain reproducible when executed by hundreds of external specialists. To solve this organizational scaling bottleneck, I created **Batch Zero**:\n• **The Batch Zero Protocol**: Prior to launching scaled vendor production (*Batch One*), a select cohort of doctoral-level domain specialists with proven delivery records independently executed the entire specification—authoring seed cases, running synthetic expansions, verifying reference solutions, and assigning five-level pairwise preference scores. Every failed case or inter-expert disagreement in Batch Zero triggered surgical revisions to prompt instructions, boundary definitions, and calibration rubrics. Following the pilot’s use in preparing external vendor workflows, **the partner foundation model company adopted Batch Zero as a standard pre-scaling gate across other new post-training tracks**.\n\nSimultaneously, our statistical governance prevents **synthetic sample-size overstatement**. Suppose $`G`$ independent golden-case families are each expanded into $`k`$ synthetic variants ($`Gk`$ nominal records), where any two variants within a family share correlation $`\\rho`$ and common variance $`\\sigma_Y^2`$. Expanding the covariance sum for the sample mean yields the exact **effective sample size** $`N_{\\text{eff}}`$:\n\n$$\n\\operatorname{Var}(\\bar{Y}) = \\frac{Gk\\sigma_Y^2 + Gk(k - 1)\\rho\\sigma_Y^2}{(Gk)^2} = \\frac{\\sigma_Y^2}{Gk}[1 + (k - 1)\\rho] \\implies N_{\\text{eff}} = \\frac{Gk}{1 + (k - 1)\\rho}\n$$\n\nWith $`G = 100`$ independent families, $`k = 20`$ synthetic variants per family ($`2,000`$ nominal tasks), and within-family correlation $`\\rho = 0.5`$, **the nominal 2,000 records provide the exact same statistical precision for the mean as just $`190.48`$ independent records**—and ignoring within-family correlation understates standard errors by a factor of **$`3.24\\times`$**! Consequently, our production pipeline enforces strict family-level train/evaluation splits and computes all confidence intervals with family-level and expert-level hierarchical clustering.\n\n## 6 Establish whether the complete method justifies scaled production\n\nTo evaluate whether our constraint-preserving synthesis and preference architecture justifies scaled production, we hold the task mix, generator training budget, expert review budget, and final acceptance criteria fixed, introducing each methodological layer in staged blind comparisons evaluated across independent task families:\n\n\n\n\n\n\n\n
Comparison StageMethodological Upgrade IntroducedCore Research & Production Question
**Existing production baseline**Unconstrained generation + manual expert review & reworkWhat is the true fully-loaded cost per finally accepted, valid case?
**Add manifold-constrained generation**Fix generator; enforce affine projections & variable transformsDo invalid candidates and manual expert repairs drop while preserving conditional distributions?
**Add margin-aware reference recomputation**Recompute reference judgments when the sufficient Lipschitz margin check fails; use $`\\gamma/s`$ to quantify Gaussian label-flip riskDo stale-label errors decline, producing valid supervision on both sides of decision boundaries?
**Apply complete Batch Zero & filtration criteria**Jointly check $`\\mathcal{F}_s`$ timing conditions and independent executionDo post-training gains persist after excluding leakage and clustering by source family?
\n\nBecause raw acceptance rates reward generating trivial variants of easy templates, and raw generation speed ignores manual expert repair costs, we govern production decisions via **Scenario-Weighted Family Coverage Efficiency**. Let $`j`$ index predefined cells in the financial capability matrix with strategic weights $`w_j \\ge 0`$ ($`\\sum_j w_j = 1`$) and required independent valid family counts $`n_j^* > 0`$:\n\n$$\n\\operatorname{Coverage} = \\sum_j w_j \\min\\left(1, \\frac{n_j^{\\text{valid}}}{n_j^*}\\right), \\qquad \\operatorname{Efficiency} = \\frac{\\operatorname{Coverage}}{C_{\\text{generation}} + C_{\\text{review}} + C_{\\text{rework}} + C_{\\text{adjudication}}}\n$$\n\nOnce a cell reaches its target $`n_j^*`$ independent families, repeatedly expanding easy templates yields zero marginal coverage gain (*[Alaa et al. on synthetic data evaluation](https://proceedings.mlr.press/v162/alaa22a.html)*). Together, manifold constraint enforcement, decision-margin label recomputation, temporal filtration isolation, and **Batch Zero** calibration establish a rigorous engineering standard for synthetic preference data in frontier model post-training.", "sourceDocument": { - "fileName": "theodore_ouyang_ai_portfolio_optimized.md", - "path": "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "sha256": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283" + "fileName": "ai-research-and-engineering.md", + "path": "content/sources/ai-portfolio/ai-research-and-engineering.md", + "sha256": "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d" } } diff --git a/website/content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json b/website/content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json index c1ba347..cf95a70 100644 --- a/website/content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json +++ b/website/content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json @@ -1,12 +1,12 @@ { "slug": "verification-and-supervision-in-scientific-reasoning-tasks", "title": "Verification and supervision in scientific reasoning tasks", - "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", + "sourceUrl": "https://github.com/130U/130U.github.io/blob/main/website/content/sources/ai-portfolio/ai-research-and-engineering.md", "sourceEditedAt": "2026-10-27", "markdown": "## 1 Correct final answers can conceal invalid reasoning and reverse model rankings\n\nChecking only the final answer is appropriate only when that final value constitutes the entire deliverable. In scientific reasoning, quantitative finance, and risk engineering, the model formulation, theorem assumptions, compliant tool invocation, intermediate derivations, and conclusion must hold together. In the OlympiadBench and Omni-MATH samples later analyzed to construct *ProcessBench*, **32.2% and 51.8% of solutions with correct final answers still contained process errors** (*[ProcessBench, Section 3.3](https://arxiv.org/html/2412.06559v2)*). Answer checking and trajectory validity measure fundamentally different things.\n\nSuppose problems and model trajectories are drawn from a target capability distribution. Let $`Z = 1`$ indicate that the complete solution (including reasoning steps and compliant tool execution) is valid, and let $`V = 1`$ indicate that the evaluation protocol passes it. For model $`m`$, let $`\\theta_m = \\Pr_m(Z = 1)`$ be its true validity rate, $`\\alpha_m = \\Pr_m(V = 1 \\mid Z = 0)`$ its **false acceptance rate** on invalid solutions (e.g., passing a flawed proof, a brute-force guess, or a non-compliant Python tool call that yields the right number), and $`\\beta_m = \\Pr_m(V = 0 \\mid Z = 1)`$ its **false rejection rate** on valid solutions (e.g., rejecting a valid alternative derivation). The law of total probability gives:\n\n$$\n(1) \\qquad s_m = E_m[V] = \\alpha_m + (1 - \\alpha_m - \\beta_m)\\theta_m, \\qquad s_m - \\theta_m = \\alpha_m(1 - \\theta_m) - \\beta_m\\theta_m\n$$\n\nThe first bias term $`\\alpha_m(1 - \\theta_m)`$ credits invalid trajectories; the second term $`-\\beta_m\\theta_m`$ penalizes valid ones. **Adding more benchmark problems reduces sampling variance in $`\\hat{s}_m`$, but never removes the structural gap $`s_m - \\theta_m`$**. Because different models exhibit different failure modes (e.g., one model adheres strictly to compliant tools and rigorous deduction, while another aggressively invokes unauthorized code interpreters or heuristic shortcuts), their $`\\alpha_m`$ and $`\\beta_m`$ can differ enough to **strictly reverse their leaderboard ranking**:\n• In a concrete arithmetic scenario, suppose Model A and Model B have true validity rates of $`\\theta_A = 72\\%`$ and $`\\theta_B = 64\\%`$, respectively.\n• If their invalid trajectories pass a final-answer check with probabilities $`\\alpha_A = 4\\%`$ and $`\\alpha_B = 36\\%`$ (while $`\\beta_A = \\beta_B = 0\\%`$), their observed benchmark scores become **$`s_A = 73.12\\%`$** and **$`s_B = 76.96\\%`$**—ranking the less capable, shortcut-prone model higher!\n\nWhile process supervision extends evaluation to intermediate steps (*[Let's Verify Step by Step, Sections 2 and 3](https://arxiv.org/html/2305.20050v1)*), the architecture formalized here addresses three practical requirements in scientific post-training: omissions in reference solutions, valid alternative proof methods, and strict enforcement of tool-compliance and structural-insight obligations defined *before* model responses are generated.\n\n## 2 Define acceptance through proof obligations and allow valid alternative solutions\n\nA complete task deliverable specifies its domain assumptions and environmental compliance constraints $`\\Gamma_q`$, the target conclusion $`C_q`$, and the proof obligations required to establish it. A model trajectory $`\\tau`$ may choose its own intermediate lemmas, coordinate representations, or order of deduction; acceptance depends strictly on whether it fulfills the obligations while obeying tool governance. Organizing the response as a directed acyclic dependency graph (DAG) $`G`$ yields the local soundness check for every node $`v_j`$ alongside the global tool-compliance predicate $`\\mathcal{T}_{\\text{comp}}(\\tau) \\in \\{0, 1\\}`$:\n\n$$\n(2) \\qquad \\Gamma_q \\cup \\{v_k : k \\in \\operatorname{pa}(j)\\} \\vdash v_j, \\qquad \\Gamma_q \\cup \\{v_j : j \\in \\operatorname{sink}(G)\\} \\vdash C_q, \\qquad \\mathcal{T}_{\\text{comp}}(\\tau) = 1\n$$\n\nHere $`\\vdash`$ denotes valid derivability within the sound domain inference system, and $`\\mathcal{T}_{\\text{comp}}(\\tau) = 1`$ certifies that every external tool invocation $`t \\in \\tau`$ belongs to the task's permitted compliance sandbox $`\\mathcal{A}_q`$ (for instance, $`\\mathcal{A}_q = \\{\\text{Excel Audited Workbook}\\}`$ and $`\\text{Python VM} \\notin \\mathcal{A}_q`$). If the root nodes follow from verified assumptions $`\\Gamma_q`$, each dependency edge is sound, and $`\\mathcal{T}_{\\text{comp}}(\\tau) = 1`$, structural induction over a topological ordering of $`G`$ establishes the complete conclusion from local checks.\n\nIn production implementation, I organized the central obligations around five verifiable gates:\n1. **Valid Model Formulation**: Checking that physical, financial, or probabilistic state spaces are well-defined.\n2. **Compliant Tool Selection ($`\\mathcal{T}_{\\text{comp}}(\\tau) = 1`$)**: When a task’s specified data-governance policy permits only the approved Excel environment, trajectories that invoke an unauthorized Python virtual machine are rejected immediately ($`V = 0`$), even if the final numerical calculation is exact.\n3. **Method Applicability & Uniform Error Control**: In physics tasks, pointwise convergence at a fixed parameter value does not establish that an approximation remains valid inside an optimization, an integral, or an exchange of limits. Authors and models must specify the approximation parameter, its domain of validity, and uniform error bounds before claiming an order of accuracy. In risk engineering, the corresponding obligations govern distributional and tail-dependence assumptions behind extreme-value bounds.\n4. **Structural Insight vs. Brute-Force Penalization**: Obligations require identifying the governing mathematical invariant rather than exhausting finite grids via raw compute.\n5. **First-Error Localization in Feedback**: Algebra that is locally correct under a false premise does not restore proof validity. Recording the first failed obligation node $`v_j`$ and marking its descendant subgraph $`\\operatorname{desc}(v_j)`$ as unverified resolves the post-error ambiguity highlighted in *[ProcessBench, Section 3.1](https://arxiv.org/html/2412.06559v2)*.\n\n## 3 Produce an error certificate as part of acceptance\n\nAn executable optimization subfamily illustrates how formal mathematical certificates replace brittle string matching. Consider the equality-constrained strongly convex quadratic problem:\n\n$$\n(3) \\qquad v^* = \\min_{Ax = b} f(x), \\qquad f(x) = \\frac{1}{2}x^\\top H x + c^\\top x, \\qquad H \\succeq \\mu I, \\quad \\mu > 0\n$$\n\nwhere $`H`$ is symmetric, $`Ax = b`$ is feasible, and $`\\mu > 0`$ is a verified lower bound on curvature. For any feasible candidate $`x`$ ($`Ax = b`$), choose any dual multiplier vector $`\\nu`$ and define the stationarity residual $`r = Hx + c + A^\\top \\nu`$. Strong convexity gives the first inequality below for every feasible displacement $`d \\in \\ker A`$; completing the square in $`d`$ yields an exact **optimality error certificate**:\n\n$$\n(4) \\qquad f(x + d) \\ge f(x) + r^\\top d + \\frac{\\mu}{2}\\|d\\|^2\n$$\n\n$$\n\\inf_{d \\in \\ker A} \\left( r^\\top d + \\frac{\\mu}{2}\\|d\\|^2 \\right) = -\\frac{\\|P_{\\ker A} r\\|_2^2}{2\\mu}\n$$\n\n$$\n0 \\le f(x) - v^* \\le \\frac{\\|P_{\\ker A} r\\|_2^2}{2\\mu}\n$$\n\nThe operator $`P_{\\ker A}`$ is the orthogonal projection onto the null space of feasible directions $`\\ker A`$, removing residual components normal to the constraints that do not affect optimality along the feasible manifold (*[Boyd and Vandenberghe, Convex Optimization, Sections 5.5 and 9.1](https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf)*). A zero projected residual ($`P_{\\ker A} r = 0`$) certifies exact optimality; otherwise, the certificate bounds the suboptimality gap in objective units:\n• Solutions obtained via **Lagrange multipliers** or by **first eliminating the equality constraints** map seamlessly to the exact same semantic obligations and pass verification.\n• Conversely, merely reporting the correct optimal value $`v^*`$ does not satisfy the certificate obligations: feasibility and the claimed objective value must be checked independently. If a submission asserts a KKT multiplier witness, the full stationarity equation $`Hx+c+A^\\top\\nu=0`$ must also be verified. The projected-residual bound alone cannot validate the claimed multiplier, because $`P_{\\ker A}A^\\top\\nu=0`$ for every $`\\nu`$; a valid equality-elimination proof need not submit such a witness.\n\nOptimization certificates are one instance of our broader obligation design. In an original kinematics benchmark I created for the project, I used the **path invariance of vertical travel time** to reduce a 2D trajectory problem to 1D horizontal scheduling—separately proving an **attained minimum completion time**, an **unattained supremum**, and **harmonic-number ($`H_n = \\sum_{k=1}^n k^{-1}`$) asymptotics** for a staircase path construction approaching the supremum. This benchmark demonstrates why feasibility, optimal bounds, structural reduction, and attainment require separate proof obligations even when a model guesses the final limiting value correctly.\n\n## 4 Task expansion must propagate mathematical error\n\nWhile symbolic templates (*[GSM-Symbolic, Section 3.1](https://arxiv.org/html/2410.05229v1)*) parameterize variable domains and answer formulas, scientific and engineering benchmarks also require verifiable laws governing how a transformation changes feasible sets, objective values, and acceptance tolerances.\n\nLet the original problem and its transformed variant have feasible sets $`F`$ and $`F'`$ with objectives $`J`$ and $`J'`$ and finite infima $`v`$ and $`v'`$. Suppose the task designer verifies a bijection $`\\Phi : F \\to F'`$ satisfying the uniform bound for every feasible $`u \\in F`$:\n\n$$\n(5) \\qquad |J'(\\Phi(u)) - aJ(u) - b| \\le \\varepsilon, \\quad a > 0\n$$\n\n$$\nav + b - \\varepsilon \\le v' \\le av + b + \\varepsilon\n$$\n\n$$\nJ(u) - v \\le \\delta \\implies J'(\\Phi(u)) - v' \\le a\\delta + 2\\varepsilon\n$$\n\nTaking infima across the first line yields the second; subtracting the lower bound on $`v'`$ from the upper bound on $`J'(\\Phi(u))`$ yields the third. This theorem answers an essential engineering question: after a unit conversion, parameter rescaling, or numerical transformation with bounded error $`\\varepsilon`$, what acceptance tolerance must apply to a previously $`\\delta`$-optimal solution? Retaining the raw tolerance $`\\delta`$ can misclassify valid transformed solutions as model failures, while relaxing tolerances ad-hoc conceals genuine capability gaps. Moreover, when $`\\varepsilon = 0`$, the bijection and positive scaling $`a > 0`$ preserve whether an optimum is attained; variants that alter boundary compactness or admissible control classes require fresh attainment proofs.\n\n## 5 Label quality determines what post-training research optimizes\n\nGranular step-level process labels improve post-training only if they measure true logical and compliance validity; systematic grading errors are directly absorbed by downstream Process Reward Models (PRMs) and RLHF policies. Let $`h`$ denote the problem and reasoning prefix through the current step, $`Z = 1`$ indicate true step validity under the problem's assumptions, and $`V \\in \\{0, 1\\}`$ be the noisy acceptance label produced by the grader with conditional false acceptance $`\\alpha(h)`$ and false rejection $`\\beta(h)`$. Writing $`p(h) = \\Pr(Z = 1 \\mid h)`$, a process evaluator with logit $`s`$ trained via binary cross-entropy optimizes toward the corrupted conditional target:\n\n$$\n(6) \\qquad \\tilde{p}(h) = \\alpha(h) + [1 - \\alpha(h) - \\beta(h)]p(h)\n$$\n\n$$\n\\frac{\\partial \\ell(s, V)}{\\partial s} = \\sigma(s) - V\n$$\n\n$$\nE[\\nabla_s \\ell(s, V) - \\nabla_s \\ell(s, Z) \\mid h] = p(h) - \\tilde{p}(h)\n$$\n\nThis expected gradient bias formula proves mathematically why naive evaluation pipelines damage post-training:\n• **When false rejection dominates ($`\\beta(h)p(h) > \\alpha(h)[1-p(h)]`$)**: $`\\tilde{p}(h) < p(h)`$, shifting the learned validity target downward and penalizing valid alternative proofs.\n• **When false acceptance dominates ($`\\alpha(h)[1-p(h)] > \\beta(h)p(h)`$)**: $`\\tilde{p}(h) > p(h)`$, shifting the learned validity target upward and rewarding flawed proofs or non-compliant tool calls. The two effects cancel in the conditional mean when their weighted contributions are equal.\n\nFurthermore, while class-conditional noise correction (*[Learning with Noisy Labels, Sections 2 and 3](https://papers.nips.cc/paper_files/paper/2013/file/3871bd64012152bfb53fdf04b401193f-Paper.pdf)*) can unbias the mean estimate when $`\\alpha, \\beta`$ are known constants ($`\\alpha + \\beta < 1`$), it inflates estimation variance quadratically:\n\n$$\n(7) \\qquad \\hat{\\theta}_{\\text{corr}} = \\frac{\\bar{V} - \\alpha}{1 - \\alpha - \\beta}, \\qquad \\operatorname{Var}(\\hat{\\theta}_{\\text{corr}}) = \\frac{\\operatorname{Var}(\\bar{V})}{(1 - \\alpha - \\beta)^2}\n$$\n\nFor task families whose validity is exactly decidable by a sound and complete certificate verifier, $`\\alpha=\\beta=0`$ removes this noise-correction variance penalty. The zero observed error rates in the constructed stress test below do not establish zero population error on unrestricted scientific reasoning.\n\n## 6 Reproducible stress tests for false acceptance and false rejection\n\nTo validate our certificate verifier reproducibly, I constructed **600 strongly convex quadratic task families** and generated **6 algebraic submissions per family (3,600 total submissions)** using **exact rational arithmetic**:\n• **1,200 Valid Certificates (2 per family)**: One derived in Lagrangian multiplier form and one in equality-elimination form.\n• **2,400 Controlled-Defect Submissions (4 per family)**: (1) An infeasible construction ($`Ax \\neq b`$) yielding the exact optimal value $`v^*`$; (2) failed stationarity along a feasible direction in $`\\ker A`$ while reporting $`v^*`$; (3) a claimed multiplier that fails the stationarity equation while reporting $`v^*`$; and (4) an incorrect final objective value. Crucially, the first three defect classes (1,800 submissions, or **75% of all invalid submissions**) report the exact right final answer $`v^*`$!\n\nWe benchmarked three deterministic acceptance protocols across all 3,600 submissions:\n\n\n\n\n\n\n
Acceptance ProtocolFalse Acceptance Rate on Invalid Certificates ($`2,400`$)False Rejection Rate on Valid Certificates ($`1,200`$)Valid Share of Accepted Submissions
**Final-answer matching only****75.0%** (admits 1,800 flawed solutions)0.0%40.0%
**Strict reference path (ablation)**0.0%**50.0%** (rejects 600 valid elimination proofs)100.0%
**Semantic certificate verification (Ours)****0.0%****0.0%****100.0%**
\n\nWithin this exact-arithmetic benchmark, semantic certificate verification admitted **1,800 fewer invalid proofs** than final-answer matching and retained **600 more valid alternative solutions** than strict reference-path matching, while passing all 600 feasible-perturbation checks on the projected-residual bound in Equation (4).\n\n## 7 Account for dependence within task families and the full cost of expert review\n\nGenerating surface variants of a single problem template does not yield independent observations of scientific capability. Across $`F`$ independent task families with $`K`$ variants per family, common marginal variance $`\\sigma^2`$, and common pairwise within-family correlation $`\\rho`$, the variance of the sample mean and effective sample size $`n_{\\text{eff}}`$ are:\n\n$$\n(8) \\qquad \\operatorname{Var}(\\bar{Z}) = \\frac{\\sigma^2}{FK}[1 + (K - 1)\\rho], \\qquad n_{\\text{eff}} = \\frac{FK}{1 + (K - 1)\\rho}\n$$\n\nIf $`F = 50`$ task families each have $`K = 8`$ variants ($`400`$ total responses) with $`\\rho = 0.6`$, the $`400`$ responses provide the same precision as just **$`n_{\\text{eff}} \\approx 77`$ independent observations**—mandating family-level train/test partitioning and new combinations of physical and mathematical principles rather than superficial numerical templating.\n\nFinally, full production cost accounting determines when automated certificate verification outperforms manual PhD review. Let $`C_0`$ be the upfront engineering cost of building the proof-obligation certificates and verifier, $`c_e`$ the legacy workflow's expert review cost per submission, $`c_v`$ the automated verifier's unit execution cost, and $`u`$ the fraction of submissions escalated to human adjudication. Once false acceptance and false rejection standards are met, the certificate architecture reduces total cost whenever submission volume $`N`$ exceeds:\n\n$$\n(9) \\qquad C_0 + N(c_v + uc_e) < Nc_e \\iff N > \\frac{C_0}{(1 - u)c_e - c_v}, \\qquad (1 - u)c_e > c_v\n$$\n\nBy connecting capability specifications, CoT proof obligations, tool-compliance sandboxes, and exact optimality certificates, the framework developed from this collaboration provides a rigorous blueprint for verifiable reasoning evaluation and post-training supervision.", "sourceDocument": { - "fileName": "theodore_ouyang_ai_portfolio_optimized.md", - "path": "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "sha256": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283" + "fileName": "ai-research-and-engineering.md", + "path": "content/sources/ai-portfolio/ai-research-and-engineering.md", + "sha256": "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d" } } diff --git a/website/content/experience-showcase.json b/website/content/experience-showcase.json index ee68153..30ccd73 100644 --- a/website/content/experience-showcase.json +++ b/website/content/experience-showcase.json @@ -67,7 +67,7 @@ "id": 2, "categoryId": "01", "organization": "Duke University × Top-Tier Foundation Model Company", - "summary": "Initiated in September 2023 for a frontier foundation model program *(partner identity anonymized under NDA)*, this project defined what constitutes high-signal financial post-training data and established an end-to-end paradigm for adversarial task design, constraint-preserving synthetic data engineering, and structured preference supervision. To expose subtle reasoning blind spots where base models and practitioners systematically stumble, I engineered diagnostic, error-inducing financial decision tasks and a five-level pairwise preference protocol that translates doctoral- and CFA-level expert judgments into evidence-grounded training signals for RLHF and direct preference optimization. I paired these task specifications with a four-regime mathematical synthesis architecture—Gaussian copulas for cross-sectional corporate fundamentals, covariance-weighted affine projections enforcing exact accounting identities ($1.21 \\times 10^{-14}$ maximum residual across 200,000 12-dimensional draws), CTGAN and TabDDPM with Lipschitz margin checks and Gaussian margin-to-noise ($\\gamma/s$) analysis for label recomputation for multimodal tables, and SDEs, neural SDEs, and Brownian bridges with strict temporal filtration isolation ($\\mathcal{F}_s$) for continuous-time price and interest-rate trajectories. To bridge research design and massive vendor scale-up, I created **Batch Zero**—an independent execution and calibration gate subsequently adopted across the company for new post-training tracks.", + "summary": "Initiated in September 2023 for a frontier foundation model program *(partner identity anonymized under NDA)*, this project defined what constitutes high-signal financial post-training data and established an end-to-end paradigm for adversarial task design, constraint-preserving synthetic data engineering, and structured preference supervision. To expose subtle reasoning blind spots where base models and practitioners systematically stumble, I engineered diagnostic, error-inducing financial decision tasks and a five-level pairwise preference protocol that translates doctoral- and CFA-level expert judgments into evidence-grounded training signals for RLHF and direct preference optimization. I paired these task specifications with a four-regime mathematical synthesis architecture—Gaussian copulas for cross-sectional corporate fundamentals, covariance-weighted affine projections enforcing exact accounting identities ($1.21 \\times 10^{-14}$ maximum residual across 200,000 12-dimensional draws), CTGAN and TabDDPM for multimodal tables, with Lipschitz margin checks and Gaussian margin-to-noise ($\\gamma/s$) analysis to guide label recomputation, and SDEs, neural SDEs, and Brownian bridges with strict temporal filtration isolation ($\\mathcal{F}_s$) for continuous-time price and interest-rate trajectories. To bridge research design and massive vendor scale-up, I created **Batch Zero**—an independent execution and calibration gate subsequently adopted across the company for new post-training tracks.", "readMoreLabel": "Read More", "detailTitle": "Financial Task Design and Synthetic Data Engineering", "metadata": { @@ -486,8 +486,8 @@ } ], "aiSourceDocument": { - "fileName": "theodore_ouyang_ai_portfolio_optimized.md", - "path": "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "sha256": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283" + "fileName": "ai-research-and-engineering.md", + "path": "content/sources/ai-portfolio/ai-research-and-engineering.md", + "sha256": "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d" } } diff --git a/website/content/legal-papers/autonomous-authority-in-space.json b/website/content/legal-papers/autonomous-authority-in-space.json index 6b9fd34..c7f0ae1 100644 --- a/website/content/legal-papers/autonomous-authority-in-space.json +++ b/website/content/legal-papers/autonomous-authority-in-space.json @@ -34,24 +34,13 @@ "type": "heading", "level": 1, "id": "section-1", - "text": "I. INTRODUCTION", + "text": "I. Introduction", "sourcePages": [ 2 ], "runs": [ { - "text": "I." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "I" - }, - { - "text": "NTRODUCTION", - "smallCaps": true + "text": "I. Introduction" } ] }, @@ -153,24 +142,13 @@ "type": "heading", "level": 1, "id": "section-9", - "text": "II. FROM RISK CLASSIFICATION TO A COMPARATIVE DECISION PROCEDURE", + "text": "II. From risk classification to a comparative decision procedure", "sourcePages": [ 4 ], "runs": [ { - "text": "II." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "F" - }, - { - "text": "ROM RISK CLASSIFICATION TO A COMPARATIVE DECISION PROCEDURE", - "smallCaps": true + "text": "II. From risk classification to a comparative decision procedure" } ] }, @@ -362,24 +340,13 @@ "type": "heading", "level": 1, "id": "section-23", - "text": "III. EXISTING LAW GOVERNS THE ACTORS BUT LEAVES OPERATIONAL CHOICES", + "text": "III. Existing law governs the actors but leaves operational choices", "sourcePages": [ 7 ], "runs": [ { - "text": "III." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "E" - }, - { - "text": "XISTING LAW GOVERNS THE ACTORS BUT LEAVES OPERATIONAL CHOICES", - "smallCaps": true + "text": "III. Existing law governs the actors but leaves operational choices" } ] }, @@ -617,24 +584,13 @@ "type": "heading", "level": 1, "id": "section-36", - "text": "IV. DELAYED SPACECRAFT OPERATIONS AND THE CASE FOR ADVANCE PERMISSION", + "text": "IV. Delayed spacecraft operations and the case for advance permission", "sourcePages": [ 11 ], "runs": [ { - "text": "IV." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "D" - }, - { - "text": "ELAYED SPACECRAFT OPERATIONS AND THE CASE FOR ADVANCE PERMISSION", - "smallCaps": true + "text": "IV. Delayed spacecraft operations and the case for advance permission" } ] }, @@ -883,24 +839,13 @@ "type": "heading", "level": 1, "id": "section-53", - "text": "V. ORBITAL SAFETY AND RISKS IMPOSED ON OTHER OPERATORS", + "text": "V. Orbital safety and risks imposed on other operators", "sourcePages": [ 15 ], "runs": [ { - "text": "V." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "O" - }, - { - "text": "RBITAL SAFETY AND RISKS IMPOSED ON OTHER OPERATORS", - "smallCaps": true + "text": "V. Orbital safety and risks imposed on other operators" } ] }, @@ -1217,24 +1162,13 @@ "type": "heading", "level": 1, "id": "section-75", - "text": "VI. SETI AND THE DIFFERENCE BETWEEN A CANDIDATE AND A CONCLUSION", + "text": "VI. SETI and the difference between a candidate and a conclusion", "sourcePages": [ 20 ], "runs": [ { - "text": "VI." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "SETI" - }, - { - "text": " AND THE DIFFERENCE BETWEEN A CANDIDATE AND A CONCLUSION", - "smallCaps": true + "text": "VI. SETI and the difference between a candidate and a conclusion" } ] }, @@ -1406,24 +1340,13 @@ "type": "heading", "level": 1, "id": "section-86", - "text": "VII. IMPLEMENTING AUTHORITY THAT CAN BE REVISED", + "text": "VII. Implementing authority that can be revised", "sourcePages": [ 24 ], "runs": [ { - "text": "VII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "I" - }, - { - "text": "MPLEMENTING AUTHORITY THAT CAN BE REVISED", - "smallCaps": true + "text": "VII. Implementing authority that can be revised" } ] }, @@ -1646,24 +1569,13 @@ "type": "heading", "level": 1, "id": "section-103", - "text": "VIII. OBJECTIONS AND THE RISKS OF REGULATION", + "text": "VIII. Objections and the risks of regulation", "sourcePages": [ 27 ], "runs": [ { - "text": "VIII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "O" - }, - { - "text": "BJECTIONS AND THE RISKS OF REGULATION", - "smallCaps": true + "text": "VIII. Objections and the risks of regulation" } ] }, @@ -1875,24 +1787,13 @@ "type": "heading", "level": 1, "id": "section-119", - "text": "IX. CONCLUSION", + "text": "IX. Conclusion", "sourcePages": [ 31 ], "runs": [ { - "text": "IX." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "ONCLUSION", - "smallCaps": true + "text": "IX. Conclusion" } ] }, diff --git a/website/content/legal-papers/mangrove-restoration-and-compensatory-mitigation.json b/website/content/legal-papers/mangrove-restoration-and-compensatory-mitigation.json index 07ea891..e452f13 100644 --- a/website/content/legal-papers/mangrove-restoration-and-compensatory-mitigation.json +++ b/website/content/legal-papers/mangrove-restoration-and-compensatory-mitigation.json @@ -35,24 +35,13 @@ "type": "heading", "level": 1, "id": "section-1", - "text": "I. INTRODUCTION", + "text": "I. Introduction", "sourcePages": [ 2 ], "runs": [ { - "text": "I." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "I" - }, - { - "text": "NTRODUCTION", - "smallCaps": true + "text": "I. Introduction" } ] }, @@ -129,24 +118,13 @@ "type": "heading", "level": 1, "id": "section-7", - "text": "II. COMPARING LEGAL DECISIONS AND ECOLOGICAL OUTCOMES", + "text": "II. Comparing legal decisions and ecological outcomes", "sourcePages": [ 3 ], "runs": [ { - "text": "II." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "OMPARING LEGAL DECISIONS AND ECOLOGICAL OUTCOMES", - "smallCaps": true + "text": "II. Comparing legal decisions and ecological outcomes" } ] }, @@ -273,24 +251,13 @@ "type": "heading", "level": 1, "id": "section-16", - "text": "III. THE GOVERNING LEGAL FRAMEWORKS", + "text": "III. The governing legal frameworks", "sourcePages": [ 5 ], "runs": [ { - "text": "III." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "T" - }, - { - "text": "HE GOVERNING LEGAL FRAMEWORKS", - "smallCaps": true + "text": "III. The governing legal frameworks" } ] }, @@ -771,24 +738,13 @@ "type": "heading", "level": 1, "id": "section-42", - "text": "IV. RESTORATION EVIDENCE AND ITS LIMITS", + "text": "IV. Restoration evidence and its limits", "sourcePages": [ 13 ], "runs": [ { - "text": "IV." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "R" - }, - { - "text": "ESTORATION EVIDENCE AND ITS LIMITS", - "smallCaps": true + "text": "IV. Restoration evidence and its limits" } ] }, @@ -1015,31 +971,13 @@ "type": "heading", "level": 1, "id": "section-55", - "text": "V. SAN TIN AND RELIANCE ON FUTURE WETLAND FUNCTION", + "text": "V. San Tin and reliance on future wetland function", "sourcePages": [ 17 ], "runs": [ { - "text": "V." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "S" - }, - { - "text": "AN ", - "smallCaps": true - }, - { - "text": "T" - }, - { - "text": "IN AND RELIANCE ON FUTURE WETLAND FUNCTION", - "smallCaps": true + "text": "V. San Tin and reliance on future wetland function" } ] }, @@ -1205,24 +1143,13 @@ "type": "heading", "level": 1, "id": "section-65", - "text": "VI. CONDITIONS FOR ACCOUNTABLE RESTORATION", + "text": "VI. Conditions for accountable restoration", "sourcePages": [ 20 ], "runs": [ { - "text": "VI." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "ONDITIONS FOR ACCOUNTABLE RESTORATION", - "smallCaps": true + "text": "VI. Conditions for accountable restoration" } ] }, @@ -1538,38 +1465,13 @@ "type": "heading", "level": 1, "id": "section-88", - "text": "VII. COSTS, MEASUREMENT, AND ENFORCEMENT", + "text": "VII. Costs, measurement, and enforcement", "sourcePages": [ 25 ], "runs": [ { - "text": "VII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "OSTS", - "smallCaps": true - }, - { - "text": "," - }, - { - "text": " MEASUREMENT", - "smallCaps": true - }, - { - "text": "," - }, - { - "text": " AND ENFORCEMENT", - "smallCaps": true + "text": "VII. Costs, measurement, and enforcement" } ] }, @@ -1647,24 +1549,13 @@ "type": "heading", "level": 1, "id": "section-94", - "text": "VIII. CONCLUSION", + "text": "VIII. Conclusion", "sourcePages": [ 27 ], "runs": [ { - "text": "VIII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "ONCLUSION", - "smallCaps": true + "text": "VIII. Conclusion" } ] }, diff --git a/website/content/legal-papers/small-states-and-strategic-space-dependence.json b/website/content/legal-papers/small-states-and-strategic-space-dependence.json index 8e364a7..27e54aa 100644 --- a/website/content/legal-papers/small-states-and-strategic-space-dependence.json +++ b/website/content/legal-papers/small-states-and-strategic-space-dependence.json @@ -34,24 +34,13 @@ "type": "heading", "level": 1, "id": "section-1", - "text": "I. INTRODUCTION", + "text": "I. Introduction", "sourcePages": [ 2 ], "runs": [ { - "text": "I." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "I" - }, - { - "text": "NTRODUCTION", - "smallCaps": true + "text": "I. Introduction" } ] }, @@ -140,24 +129,13 @@ "type": "heading", "level": 1, "id": "section-8", - "text": "II. DEFINING THE COMPARISON AND THE MECHANISMS", + "text": "II. Defining the comparison and the mechanisms", "sourcePages": [ 4 ], "runs": [ { - "text": "II." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "D" - }, - { - "text": "EFINING THE COMPARISON AND THE MECHANISMS", - "smallCaps": true + "text": "II. Defining the comparison and the mechanisms" } ] }, @@ -276,24 +254,13 @@ "type": "heading", "level": 1, "id": "section-15", - "text": "III. WHAT INTERNATIONAL LAW SUPPLIES AND WHAT IT LEAVES OPEN", + "text": "III. What international law supplies and what it leaves open", "sourcePages": [ 6 ], "runs": [ { - "text": "III." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "W" - }, - { - "text": "HAT INTERNATIONAL LAW SUPPLIES AND WHAT IT LEAVES OPEN", - "smallCaps": true + "text": "III. What international law supplies and what it leaves open" } ] }, @@ -491,24 +458,13 @@ "type": "heading", "level": 1, "id": "section-27", - "text": "IV. THREE ARRANGEMENTS AND THEIR LIMITS", + "text": "IV. Three arrangements and their limits", "sourcePages": [ 9 ], "runs": [ { - "text": "IV." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "T" - }, - { - "text": "HREE ARRANGEMENTS AND THEIR LIMITS", - "smallCaps": true + "text": "IV. Three arrangements and their limits" } ] }, @@ -910,24 +866,13 @@ "type": "heading", "level": 1, "id": "section-52", - "text": "V. PARTICIPATION AND INFLUENCE IN MULTILATERAL GOVERNANCE", + "text": "V. Participation and influence in multilateral governance", "sourcePages": [ 16 ], "runs": [ { - "text": "V." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "P" - }, - { - "text": "ARTICIPATION AND INFLUENCE IN MULTILATERAL GOVERNANCE", - "smallCaps": true + "text": "V. Participation and influence in multilateral governance" } ] }, @@ -1056,24 +1001,13 @@ "type": "heading", "level": 1, "id": "section-59", - "text": "VI. SAFEGUARDS FOR ACCOUNTABLE CHOICE", + "text": "VI. Safeguards for accountable choice", "sourcePages": [ 18 ], "runs": [ { - "text": "VI." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "S" - }, - { - "text": "AFEGUARDS FOR ACCOUNTABLE CHOICE", - "smallCaps": true + "text": "VI. Safeguards for accountable choice" } ] }, @@ -1199,24 +1133,13 @@ "type": "heading", "level": 1, "id": "section-69", - "text": "VII. CONCLUSION", + "text": "VII. Conclusion", "sourcePages": [ 23 ], "runs": [ { - "text": "VII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "ONCLUSION", - "smallCaps": true + "text": "VII. Conclusion" } ] }, diff --git a/website/content/legal-papers/solar-geoengineering-comparison-and-continuity.json b/website/content/legal-papers/solar-geoengineering-comparison-and-continuity.json index 1b6b361..c87beba 100644 --- a/website/content/legal-papers/solar-geoengineering-comparison-and-continuity.json +++ b/website/content/legal-papers/solar-geoengineering-comparison-and-continuity.json @@ -35,24 +35,13 @@ "type": "heading", "level": 1, "id": "section-1", - "text": "I. THE AUTHORIZATION PROBLEM", + "text": "I. The authorization problem", "sourcePages": [ 2 ], "runs": [ { - "text": "I." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "T" - }, - { - "text": "HE AUTHORIZATION PROBLEM", - "smallCaps": true + "text": "I. The authorization problem" } ] }, @@ -145,31 +134,13 @@ "type": "heading", "level": 1, "id": "section-8", - "text": "II. WHAT RISK-RISK ANALYSIS CAN ESTABLISH", + "text": "II. What risk-risk analysis can establish", "sourcePages": [ 3 ], "runs": [ { - "text": "II." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "W" - }, - { - "text": "HAT RISK", - "smallCaps": true - }, - { - "text": "-" - }, - { - "text": "RISK ANALYSIS CAN ESTABLISH", - "smallCaps": true + "text": "II. What risk-risk analysis can establish" } ] }, @@ -269,24 +240,13 @@ "type": "heading", "level": 1, "id": "section-15", - "text": "III. COMPARING POLICIES WITHOUT CHOOSING THE ANSWER IN ADVANCE", + "text": "III. Comparing policies without choosing the answer in advance", "sourcePages": [ 6 ], "runs": [ { - "text": "III." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "OMPARING POLICIES WITHOUT CHOOSING THE ANSWER IN ADVANCE", - "smallCaps": true + "text": "III. Comparing policies without choosing the answer in advance" } ] }, @@ -500,24 +460,13 @@ "type": "heading", "level": 1, "id": "section-31", - "text": "IV. RESEARCH PERMISSION AND THE LIMITS OF EXISTING LAW", + "text": "IV. Research permission and the limits of existing law", "sourcePages": [ 10 ], "runs": [ { - "text": "IV." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "R" - }, - { - "text": "ESEARCH PERMISSION AND THE LIMITS OF EXISTING LAW", - "smallCaps": true + "text": "IV. Research permission and the limits of existing law" } ] }, @@ -683,24 +632,13 @@ "type": "heading", "level": 1, "id": "section-41", - "text": "V. A COMPARATIVE JUSTIFICATION DUTY", + "text": "V. A comparative justification duty", "sourcePages": [ 13 ], "runs": [ { - "text": "V." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "A" - }, - { - "text": " COMPARATIVE JUSTIFICATION DUTY", - "smallCaps": true + "text": "V. A comparative justification duty" } ] }, @@ -945,24 +883,13 @@ "type": "heading", "level": 1, "id": "section-59", - "text": "VI. THE PROBLEM OF EXIT", + "text": "VI. The problem of exit", "sourcePages": [ 18 ], "runs": [ { - "text": "VI." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "T" - }, - { - "text": "HE PROBLEM OF EXIT", - "smallCaps": true + "text": "VI. The problem of exit" } ] }, @@ -1093,24 +1020,13 @@ "type": "heading", "level": 1, "id": "section-69", - "text": "VII. OBJECTIONS AND THE LIMITS OF THE ARGUMENT", + "text": "VII. Objections and the limits of the argument", "sourcePages": [ 20 ], "runs": [ { - "text": "VII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "O" - }, - { - "text": "BJECTIONS AND THE LIMITS OF THE ARGUMENT", - "smallCaps": true + "text": "VII. Objections and the limits of the argument" } ] }, @@ -1202,24 +1118,13 @@ "type": "heading", "level": 1, "id": "section-75", - "text": "VIII. CONCLUSION", + "text": "VIII. Conclusion", "sourcePages": [ 23 ], "runs": [ { - "text": "VIII." - }, - { - "text": " ", - "smallCaps": true - }, - { - "text": "C" - }, - { - "text": "ONCLUSION", - "smallCaps": true + "text": "VIII. Conclusion" } ] }, diff --git a/website/content/protected-sources.json b/website/content/protected-sources.json index d28f6e1..2bf8ebc 100644 --- a/website/content/protected-sources.json +++ b/website/content/protected-sources.json @@ -5,8 +5,8 @@ "app/experience/page.tsx": "993f5aa641588d4cd2d5ea7ef7d7826e1b5cafbd06b1bc2bc704f802e23df777", "app/components/experience/ExperienceContent.tsx": "4fa27f8b87ffc9e798846da715e10f72aa7c18947786aad113fb1b1260d4e7f0", "app/components/experience/ExperienceDetails.tsx": "fc0d3ca896ab636dcadcb5370cb176065f70cd748b2cfbdfb89271fe83b8e261", - "app/lib/content/experience-showcase.ts": "14462c94181c1cbe375a23d74a6de1e10c216a05341cde1968601251c65b3b45", - "content/experience-showcase.json": "8d19a7cadecc6ac5a5af93fcde88e30a898cf3cbde35304e2c6579aea37b117b", + "app/lib/content/experience-showcase.ts": "c769e21ec5ba92d66d439746d1090eee5c83ea129e69b30b21943d18325512dd", + "content/experience-showcase.json": "5003dd0195067ee81adff9cc80ec70f7b35d474f1d717e6da0ba9e1eff8f33f3", "app/components/AboutContent.tsx": "6f5861525912ac8a692478b4d7f442f7308b6cecb566c66a36d407ea1f28f843", "app/lib/content/about.ts": "8d79e72133dcbcae9523771a099f53f0aecb879cea384769ff6bb0144e05626f", "app/components/research/ResearchArticle.tsx": "4abb9ad9ee42d0c71ead3b0f43846320cc06a4c79454d91cb9e32ade0fa356f5", @@ -27,25 +27,23 @@ "app/lib/content/metadata-excerpt.ts": "567139663925462cc48584a53dabb91e50e3727acd68afe7ddf61fa2818f0e71", "app/past-experience/[slug]/[paper]/page.tsx": "0a54d7899422b36ca794920d635fc93e4771183105041d9504479e4333f444b0", "app/past-experience/components/LegalPaperPage.tsx": "d227289591733176ebe9a6c6421f267cdc4475c23a83757169b26532ab88fb67", - "content/legal-papers/autonomous-authority-in-space.json": "087d4708bdaf4b487e47438681b3f79941ba08ac2ddfe08a5dd4830fd8a4f2d9", - "content/legal-papers/small-states-and-strategic-space-dependence.json": "1ea90829b39b0b56923641c84ed6af46d57cc6de5c13047ced495cc1f0726acb", - "content/legal-papers/solar-geoengineering-comparison-and-continuity.json": "f6734b9a29d6ffdcc1c0e612f0ba1917ce9dea2fc9ec5132c81b12fe1e7b54f0", - "content/legal-papers/mangrove-restoration-and-compensatory-mitigation.json": "be8ad403b6504ad18cac1b6bdf55925214b389ea6f3c0f325f0fac1bea1e9cd9", + "content/legal-papers/autonomous-authority-in-space.json": "ee5c9a0bd42264a26a7cd221ad8341ae2c91d7e0b959fb2d1a5fe69ceb5e5ced", + "content/legal-papers/small-states-and-strategic-space-dependence.json": "5d7ef044d762e243e061ff65de12a8c6ab23513b7ba0e9ee01b352d18ac3a0f5", + "content/legal-papers/solar-geoengineering-comparison-and-continuity.json": "eb104d73d6eaa1893a08726df869c628ed18ed2071499bbf45dbb33421e3461f", + "content/legal-papers/mangrove-restoration-and-compensatory-mitigation.json": "1f8657ff433586221e4e15b4d26c33f72b025becc4704865c102402bfdfefbb0", "app/lib/content/ai-projects.ts": "acf07dd865754e90b1b75c89bb65e6f93e26cd41f3f267c2a80a7dad185c9b5f", "app/lib/content/routes.ts": "610d99093fe37e69b6603570a8d55f0d058fc44571ba4aed8f2f6c4c6a306cea", - "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json": "ad84ce64dcd9cc2d1e1affa61f46c34e5461c6bfdede23ec15fbec110fcb4643", - "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json": "0cafd69ca15623e0ef593a13becd439f7d0f5a6c252cba94ec1ab4c453652cdd", - "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json": "02727352f1487759c9b003067a0c02a11d8af028128cb80c0c6d3538717a027b", - "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json": "0f264a90815ab5c65f1b52665d0392e4c1e27800695421f5f66a9094fa78d4c2", + "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json": "47c5aaee6f6fb629e102ac28a3fa8ec8aecd0e9cc7363b4ab6393429ed783f2e", + "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json": "ce413f7a748a6eb33714ab1709df4daed4a738ac74ff9eba1042a946c5c911dd", + "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json": "6cf298145c2e4bf20a53623b40d0a5e8fe231fcda9589fc38821dae45e1af50f", + "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json": "6473726c418f185e8606c89c07cebd37b689668478c93f3be58281977a8ec373", "app/experience/[slug]/page.tsx": "997cadef95f145e2abf55cd64506e278425b5c30d4cfccb9b835453b811c755e", "app/lib/content/experience-domains.ts": "7362085d4857daa83ae476f176163378dde550572d5a1932d2f142d4d1a00c7d", "app/education/critical-dependence-stress-for-portfolio-diversification/page.tsx": "b0dbb4c593e681c702908a707b8dc31ac1800472f749a8897bc3c7d2550f1f90", "content/selected-research/critical-dependence-stress-for-portfolio-diversification.json": "66dae1b9bdbf757d46992632bb69c409e24079d157cbd155852c7e48e2ebe26a", "app/components/research/ResearchInline.tsx": "ad5cae237a294ce103f6edd3aa71ba7beadc972f587861bd39a0d157ddfb4315", - "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283", - "content/sources/ai-portfolio/citation-links.json": "40eeff369104f887512f2acd183136338b127744420ca88e2d3329e30ea14093", - "scripts/import-ai-portfolio.mjs": "cfc2e092fcedcdf487662fd10aaae9aba9f58d0036ebc568561f7e9f83c187d1", + "scripts/import-ai-portfolio.mjs": "2279e6dfe815b467d59effe6c8b333a61a6bbf605e9d5bdfa2c01d9fefc1498c", "app/lib/content/research-number.ts": "2d29d06ffde33580bd5e655b547b28c953d64b0d2b0fa7fe31663da68dd1276b", - "content/sources/ai-portfolio/editorial-revisions.json": "6171190332fee0deae11d95e60f3278f885bfac7cb75093ab3608a299e9cf3c5" + "content/sources/ai-portfolio/ai-research-and-engineering.md": "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d" } } diff --git a/website/content/sources/ai-portfolio/.gitattributes b/website/content/sources/ai-portfolio/.gitattributes index 1c75029..b182b12 100644 --- a/website/content/sources/ai-portfolio/.gitattributes +++ b/website/content/sources/ai-portfolio/.gitattributes @@ -1,2 +1,2 @@ -# Keep the approved source's byte-level evidence identical across checkouts. +# Use consistent manuscript line endings across platforms. *.md text eol=lf diff --git a/website/content/sources/ai-portfolio/ai-research-and-engineering.md b/website/content/sources/ai-portfolio/ai-research-and-engineering.md new file mode 100644 index 0000000..979fe08 --- /dev/null +++ b/website/content/sources/ai-portfolio/ai-research-and-engineering.md @@ -0,0 +1,806 @@ +# AI Research and Engineering + +**Author** +Theodore Ouyang + +**Updated** +2026-10-27 + +## Confidentiality & Disclosure Note + +The following four research and engineering initiatives were conducted between September 2023 and September 2026 through university–industry collaborations at Duke University. Pursuant to strict non-disclosure agreements (NDAs) with our industry partners—comprising a flagship multimodal foundation model developer, a frontier AI research laboratory, and a leading global alternative asset manager overseeing more than $300 billion in AUM—partner corporate names and proprietary internal system identifiers are anonymized on this public site. Project titles, my role (*Researcher*), geographic locations, dates, architectural paradigms, and self-contained mathematical formulations with reproducible synthetic benchmarks are authorized for public presentation. + +# Project 01: statistical-inference-and-resource-allocation-in-expert-data-production + +**Article Title** +Statistical inference and resource allocation in expert data production + +**Organization** +Duke University × Top-Tier Foundation Model Company + +**Title** +Scalable Expert Quality Control and Adaptive Review + +**URL** +https://www.theodoreoy.com/past-experience/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production/ + +**Position** +Researcher + +**Location** +Durham, NC; Palo Alto, CA + +**Dates** +September 2023 – September 2026 + +**Read More Label** +Read More + +## Summary + +Launched in September 2023 for a flagship multimodal foundation model program, this initiative established the project’s statistical governance and resource-allocation framework for reinforcement learning from human feedback (RLHF) across more than 300 domain specialists in travel, food delivery, consumer goods, and e-commerce advertising *(partner corporate identity anonymized under NDA; role, scope, and mathematical design disclosed)*. Rather than executing routine vendor quality assurance, I architected the pilot’s mathematical and operational framework for governing external expert networks, calibrating evolving acceptance standards, and allocating review capacity at scale. By uniting hierarchical Beta-Binomial reliability modeling, prediction-assisted active statistical inference (inverse-probability-weighted residual correction), and capacity-constrained mixed-integer programming, the framework strictly separates *epistemic measurement value* from *defect remediation value*. In a 10,000-submission synthetic benchmark verified across 200,000 independent sampling repetitions with ideal residual calibration, this allocation architecture reduced population quality estimation variance by **17.79%** versus uniform sampling at equal expected audit cost. A separate 200-item analytical example reduced post-review expected error loss by **8.89%** at equal review volume. + +## Confidentiality & Scope Disclosure + +Conducted through a university–industry research collaboration with a flagship multimodal foundation model developer. While the corporate partner's name is withheld pursuant to non-disclosure agreements, my role, project scope, operational scale, and the underlying mathematical and engineering frameworks are authorized for public disclosure. + +## Key Contributions + +* **Established the Pilot Statistical Governance Framework for Expert Post-Training Data (September 2023 Launch)**: Starting in September 2023, I architected a pilot system governing more than 300 domain contributors across travel, food delivery, consumer goods, and e-commerce advertising. My work defined the project’s mathematical and operational standards for structuring, auditing, and scaling its external expert and outsourcing networks. +* **Bayesian Adaptive Review Routing & Hierarchical Contributor Calibration**: Designed multi-tier adaptive review rules parameterized by Beta-Binomial reliability posteriors, credible intervals, verified sample counts, and task-level risk priors. In a flagship e-commerce advertising workflow, dynamically routed flagged submissions to three additional specialists and initially accepted submissions to one or two; closed the feedback loop by feeding expert adjudications directly back into contributor reliability posteriors, targeted coaching, and dynamic review intensity. +* **Taxonomic Defect Disentanglement & Rubric Evolution**: Combined independent random reference audits with targeted high-risk escalations to convert recurring inter-annotator disagreements into unambiguous acceptance criteria. Formally separated missing context, ambiguous prompt instructions, and genuine contributor defects; systematically revised task instructions, boundary cases, and shared calibration anchors and subsequently compared Dawid-Skene reviewer-error models with cost-sensitive deferral rules (DeCCaF) to inform escalation thresholds. +* **Empirical Cost-Latency Telemetry & Production Data Engineering**: Built end-to-end SQL and Python data pipelines unifying task metadata, review trajectories, acceptance outcomes, turnaround times, and behavioral logs, reconciling platform activity with Hubstaff operational telemetry. Crucially, this cross-system reconciliation served not as clerical time-tracking, but to empirically calibrate unit verification costs ($c_i$), task-stratum latencies ($t_{iaj}$), and hard specialist capacity constraints ($H_j^{\text{review}}$) for our mathematical allocation solvers, while translating client-reported model defects into reproducible engineering investigations and operating SOPs. +* **Econometric Analysis of Exploratory Choice Architecture**: Analyzed user decision behavior across curated (approximately 20 hotels) versus expanded (100+ eligible options) recommendation sets using logistic regression with decision-stage interactions, dwell-time transformations, and distributional modeling. Demonstrated that expanding the choice set increased deliberation browsing time significantly more clearly than click-through conversion, establishing quantitative product guidance for exploratory assistant settings. +* **Unified Mathematical Theory Separating Audit Measurement from Defect Remediation**: Extended our production procedures into a rigorous joint mathematical design connecting prediction-assisted active statistical inference, cost-aware audit allocation, and capacity-constrained review assignment. Proved that optimal operations must separate three distinct quantities that standard QA pipelines conflate: a task's *initial error risk* ($q_i$), the *statistical information value of an independent reference audit* ($m_i / c_i$), and the *expected defect loss that additional human handling can remove* ($\Delta_i = r_{i0} - r_{i1}$). +* **Reproducible Large-Scale Monte Carlo & Remediation Validation**: Validated the estimator and allocation algorithms on a stratified synthetic benchmark of 10,000 submissions checked across 200,000 independent Monte Carlo sampling repetitions. Under ideal residual calibration and an identical expected audit budget, cost-aware residual allocation reduced population quality estimation variance by **17.79%** (cutting standard deviation by **9.33%**) relative to uniform sampling. In a separate 200-item analytical example, marginal-gain review ranking reduced expected post-review defect loss by **8.89%** at equal review volume. + +## Architectural Context: Statistical Governance for Expert Data Production + +As the project scaled multi-domain post-training work from September 2023 onward, it faced a statistical problem: coordinating hundreds of distributed domain specialists and external vendor queues while keeping quality measurements interpretable under selective review. Concentrating human review on heuristic high-risk flags created severe selection bias ($\operatorname{Cov}(\pi(X), q(X)) > 0$) that made raw inspection error rates uninterpretable, while auditing tasks whose errors were already predictable wasted senior specialist capacity without improving statistical visibility. The seven sections below formalize the production design and compare it with subsequent literature. The September 2023 start date describes the collaboration; references published in 2024–2026 provide later methodological context. + +## Manuscript + +## 1 The observation mechanism creates its own failure modes + +In production RLHF pipelines, the routing policy used to inspect submissions alters the statistical properties of the observed sample. Let $`X`$ denote observable task features (encoding domain vertical, prompt complexity, and contributor history) and $`Y \in \{0, 1\}`$ indicate an error in the original submission under fixed acceptance criteria. The true conditional error probability is $`q(x) = P(Y = 1 \mid X = x)`$. Let $`S = 1`$ indicate selection for an independent reference audit. Conditional on $`X`$, selection is randomized with probability $`\pi(x) > 0`$, independently of the reference label that has yet to be observed. + +When operational dashboards use the error fraction among verified submissions to assess the full population, they actually target: + +$$ +E[Y \mid S = 1] = \frac{E[\pi(X)q(X)]}{E[\pi(X)]} +$$ + +$$ +E[Y \mid S = 1] - E[Y] = \frac{\operatorname{Cov}(\pi(X), q(X))}{E[\pi(X)]} +$$ + +When review priorities successfully target high-risk tasks, the covariance between audit probability $`\pi(X)`$ and true defect probability $`q(X)`$ is strictly positive. A higher error detection rate in the reviewed sample cannot then be read directly as a deterioration in overall production quality; it often reflects sharper risk targeting. This identity describes the distribution of a selected item. In a finite batch, the ratio of observed errors to selected items also has a random denominator, so the two quantities should not be conflated. + +Expert rankings are affected by the exact same mechanism. Let $`a_{jh}`$ be the probability that expert $`j`$ is correct on task stratum $`h`$, and let $`p_{jht}`$ describe the mix of tasks assigned to that expert in period $`t`$. Even with unchanged underlying ability, observed accuracy shifts whenever task routing changes: + +$$ +A_{jt} = \sum_h p_{jht} a_{jh}, \qquad A_{j,t+1} - A_{jt} = \sum_h (p_{jh,t+1} - p_{jht}) a_{jh} +$$ + +Valid cross-period comparisons therefore require a fixed target task mix $`w_h`$, reporting standardized accuracy: + +$$ +A_j^{\text{std}} = \sum_h w_h a_{jh} +$$ + +with overlapping assignments or shared anchor cases to support estimation within each stratum. In our production routing engine, we model each contributor's stratum-level reliability via a conjugate Beta-Binomial scheme: given prior $`\operatorname{Beta}(\alpha_{0h}, \beta_{0h})`$ and $`n_{jht}`$ audited tasks with $`k_{jht}`$ verified successes, the posterior reliability distribution $`\operatorname{Beta}(\alpha_{0h} + k_{jht}, \beta_{0h} + n_{jht} - k_{jht})`$ yields shrinkage-regularized point estimates and credible intervals that account for unequal sample sizes across comparable audits. Pooling different difficulty levels, criteria versions, and selection mechanisms into a single unstratified binomial sample leaves the preceding selection bias unresolved. + +These distinctions also guide algorithmic method selection. **[Dawid-Skene](https://doi.org/10.2307/2346806)** models represent differences in reviewer confusion matrices across latent ground truth, whereas **[DeCCaF](https://arxiv.org/html/2403.06906v1)** incorporates task conditions, error costs, and reviewer capacity into human-AI deferral and highlights how assigning difficult cases distorts naive evaluations of expert performance. The architecture developed here addresses the critical interface between selective operational deferral and unbiased population quality inference in ongoing post-training production (*[Dawid-Skene](https://doi.org/10.2307/2346806); [DeCCaF](https://arxiv.org/html/2403.06906v1)*). + +## 2 Correct predictions with audits to recover the population target + +Fix a production batch of $`N`$ original submissions, its acceptance criteria, and its latent reference labels $`Y_1, \dots, Y_N`$. The inferential target is the finite-batch error rate $`\mu = N^{-1} \sum_{i=1}^N Y_i`$. Let $`f_i \in [0, 1]`$ be a fixed, inexpensive risk prediction (synthesized from contributor posterior reliability, task complexity features, and automated heuristic checks) and $`\pi_i \in [\pi_{\min}, 1]`$ be the assigned audit probability, with independent draws $`S_i \sim \operatorname{Bernoulli}(\pi_i)`$. Both predictions and sampling probabilities are locked before the current batch's reference labels are revealed. + +We deploy a prediction-assisted, inverse-probability-weighted (IPW) residual-corrected estimator: + +$$ +\hat{\mu} = \frac{1}{N} \sum_{i=1}^N \left[ f_i + \frac{S_i}{\pi_i} (Y_i - f_i) \right] +$$ + +Its design-based validity follows directly. Write the prediction residual as $`e_i = Y_i - f_i`$ and condition on the fixed batch, predictions, and sampling probabilities: + +$$ +\hat{\mu} - \mu = \frac{1}{N} \sum_{i=1}^N \left( \frac{S_i}{\pi_i} - 1 \right) e_i, \qquad E_S[\hat{\mu} - \mu] = 0 +$$ + +Inverse-probability weighting restores the population representation altered by selective verification, while predictions reduce the residual variation that expensive human expert audits must resolve. Crucially for production robustness, **this design-based unbiasedness does not require $`f_i = q(X_i)`$**: inaccurate predictions can reduce statistical efficiency, but they never introduce selection bias under the specified Bernoulli sampling design. While the estimator builds on established active statistical inference principles (*[Active Statistical Inference: mean estimation and sampling design](https://arxiv.org/html/2403.03208v3)*), our production architecture specifies the exact immutable objects being audited, the rubric version lineage, and the telemetry records required for downstream optimization. + +Specifically, original and revised submissions must be retained as separate database entities. Estimating $`\hat{\mu}_{\text{raw}}`$ on frozen originals and $`\hat{\mu}_{\text{final}}`$ on post-review deliverables cleanly distinguishes *defects generated in initial production* from *defects corrected by review*. Furthermore, whenever post-training criteria evolve, a shared set of anchor cases is adjudicated under both rubric versions; submissions from different periods are then compared under a common reference version, preventing stricter acceptance standards from being mistaken for declining contributor capability. + +## 3 Allocate audit effort according to residuals and cost + +Independent Bernoulli sampling eliminates cross-item covariance terms, yielding the exact design variance: + +$$ +\operatorname{Var}_S(\hat{\mu}) = \frac{1}{N^2} \sum_{i=1}^N \frac{1 - \pi_i}{\pi_i} e_i^2 +$$ + +This expression identifies the true mathematical objective of audit allocation: **audit effort should address what the automated risk predictions have not resolved**. For a binary error label with conditional error probability $`q_i`$, the expected squared residual decomposes into two components: + +$$ +m_i := E[(Y_i - f_i)^2 \mid X_i] = q_i(1 - q_i) + (q_i - f_i)^2 +$$ + +The two terms are intrinsic conditional Bernoulli variance $`q_i(1 - q_i)`$ and squared prediction calibration error $`(q_i - f_i)^2`$. This identity establishes a counter-intuitive operational principle: **a high error probability $`q_i`$ does not necessarily imply high inferential value**. If a task category is almost certain to contain an error ($`q_i \approx 1`$) and the predictor correctly recognizes that fact ($`f_i \approx 1`$), then $`m_i \approx 0`$—spending an independent reference audit on that item yields less statistical information about population quality than auditing a moderately risky task whose outcome is genuinely uncertain ($`q_i \approx 0.5`$). The high-risk task still merits operational correction, which is precisely why the *value of measurement* and the *value of remediation* require separate mathematical models. + +Let $`c_i > 0`$ be the empirical cost of an independent reference audit (calibrated by reconciling platform review logs with Hubstaff active-work telemetry) and $`B`$ be the expected audit-cost budget. First consider the optimal design when the true residual second moments $`m_i > 0`$ are known: + +$$ +\min_{\pi} \sum_{i=1}^N \frac{m_i}{\pi_i} \quad \text{s.t.} \quad \sum_{i=1}^N c_i \pi_i \le B, \quad \pi_{\min} \le \pi_i \le 1 +$$ + +The omitted constant $`-\sum_i m_i`$ in the numerator of the variance does not depend on the allocation $`\pi`$. For probabilities strictly between their lower and upper bounds, the Lagrangian first-order condition yields the clipped square-root allocation law: + +$$ +-\frac{m_i}{\pi_i^2} + \lambda c_i = 0 \implies \pi_i^* = \operatorname{clip}_{[\pi_{\min}, 1]} \left( \sqrt{\frac{m_i}{\lambda c_i}} \right) +$$ + +for budgets below the cost of a full audit, $`\lambda > 0`$ is chosen to satisfy the budget constraint (the multiplier need not be unique when clipping is active), and feasibility requires $`B \ge \pi_{\min} \sum_i c_i`$. If the budget covers a full audit, every item is audited ($`\pi_i^* = 1`$). The square-root law increases sampling intensity for larger expected residuals while penalizing expensive audits, and the positive probability floor $`\pi_{\min}`$ preserves observation coverage and bounded IPW weights across the entire task space. + +Production implementation substitutes regularized stratum estimates $`\hat{m}_i > 0`$ obtained from earlier independent audits or a pilot calibration batch. True $`m_i`$ defines the optimality analysis; estimated values $`\hat{m}_i`$ determine the operational allocation. By estimating residuals at the level of `(task stratum × criteria version × cost stratum)`, the pipeline avoids fitting unstable parameters to individual submissions. (Note that $`B`$ constrains *expected* audit cost per batch. Workflows requiring an exact integer count must use a feasible fixed-size sampling design: its first-order inclusion probabilities sum to that integer, and its variance depends on joint inclusion probabilities. These probabilities need not equal the Bernoulli optimum $`\pi_i^*`$.) + +## 4 Derive the gain and identify when it disappears + +We now derive the exact analytical efficiency gain of cost-aware residual allocation over uniform sampling using the same predictor, and characterize when active allocation fails. Let $`C = \sum_{i=1}^N c_i`$, so that the budget-equivalent uniform sampling probability is $`p = B / C`$. When true residual second moments $`m_i`$ are available and no optimal probability reaches a clipping boundary, averaging design variance over the conditional label distribution gives: + +$$ +V_{\text{uniform}} = \frac{1}{N^2} \left[ \frac{C \sum_i m_i}{B} - \sum_i m_i \right] +$$ + +$$ +V_{\text{optimal}} = \frac{1}{N^2} \left[ \frac{\left(\sum_i \sqrt{m_i c_i}\right)^2}{B} - \sum_i m_i \right] +$$ + +(If a common sampling probability is used within each fixed task stratum, substituting that stratum's finite-batch mean squared residual for $`m_i`$ makes these expressions exact finite-batch identities.) By the Cauchy-Schwarz inequality: + +$$ +V_{\text{uniform}} - V_{\text{optimal}} = \frac{C \sum_i m_i - \left(\sum_i \sqrt{m_i c_i}\right)^2}{N^2 B} \ge 0 +$$ + +Equality holds **if and only if $`m_i / c_i`$ is constant across all items**. Efficiency gains therefore depend strictly on identifiable heterogeneity in residual second moment per unit audit cost; with homogeneous residuals and costs, optimal allocation reduces to uniform sampling. When clipping is active at $`\pi_{\min}`$ or $`1`$, constrained optimization is still no worse than feasible uniform policies, with the exact gain evaluated directly at the clipped primal objective. + +Crucially, the same analysis exposes the exact variance penalty of model misspecification. If the allocation uses inaccurate residual estimates $`\hat{m}_i`$ and neither allocation is clipped, the excess variance relative to the true optimum is: + +$$ +V(\hat{\pi}) - V(\pi^*) = \frac{1}{N^2 B} \left[ \left( \sum_i \sqrt{\hat{m}_i c_i} \right) \left( \sum_i \frac{m_i \sqrt{c_i}}{\sqrt{\hat{m}_i}} \right) - \left( \sum_i \sqrt{m_i c_i} \right)^2 \right] \ge 0 +$$ + +Severely underestimating $`\hat{m}_i`$ in any region of the task space inflates the penalty through the $`\sqrt{\hat{m}_i}`$ denominator. Enforcing a positive sampling floor $`\pi_{\min}`$ caps this worst-case blowup, while mixing active and uniform probabilities via the budget-preserving defensive convex combination: + +$$ +\pi_i(\rho) = (1 - \rho)\hat{\pi}_i + \rho p +$$ + +allows the shrinkage weight $`\rho \in [0, 1]`$ to be tuned across independent calibration batches (*[Robust Sampling for Active Statistical Inference](https://arxiv.org/html/2511.08991v1)*). + +## 5 Allocate review by expected loss reduction + +Whereas statistical auditing seeks to *measure* population quality with minimum variance, operational review seeks to *remove defects* from the current batch before training handoff. For task $`i`$ under handling action $`a`$ (e.g., accept directly, route to peer review, escalate to expert adjudication, or request contributor revision), let $`r_{ia}`$ be the expected error loss remaining *after* the action, $`d_{ia}`$ its operational cost, and $`t_{iaj}`$ the time required from specialist $`j`$ (measured via platform and Hubstaff telemetry). Binary assignment decisions $`z_{ia} \in \{0, 1\}`$ solve: + +$$ +\min_{z} F(z) = \sum_{i,a} z_{ia}(r_{ia} + \eta d_{ia}) \quad \text{s.t.} \quad \sum_a z_{ia} = 1, \quad \sum_{i,a} z_{ia} t_{iaj} \le H_j \quad \forall j +$$ + +where $`H_j`$ is available specialist time and $`\eta \ge 0`$ converts resource cost to the scale of error loss (*[DeCCaF: capacity constraints and global assignment](https://arxiv.org/html/2403.06906v1)*). + +When review costs are equal and each reviewed task consumes one slot, **the optimal priority ranking is by marginal remediation gain $`\Delta_i = r_{i0} - r_{i1}`$, not by initial risk $`r_{i0}`$**. Missing external evidence may make a high-risk task difficult to repair through one additional review ($`\Delta_i`$ small), whereas a self-contained calculation error in a moderate-risk task can be completely eliminated ($`\Delta_k`$ large). Moving a single review slot from task $`i`$ to task $`k`$ changes total expected loss by: + +$$ +\Delta F = (r_{i0} + r_{k1}) - (r_{i1} + r_{k0}) = \Delta_i - \Delta_k +$$ + +Whenever $`\Delta_k > \Delta_i`$, the exchange strictly reduces delivered defect loss, regardless of the tasks' initial risk ordering. + +We can also state the exact calibration precision required of the action-risk estimators. Suppose $`|\hat{r}_{ia} - r_{ia}| \le \delta`$ for every feasible action, costs and capacities are known, and $`\hat{z}`$ minimizes the predicted objective $`\hat{F}(z)`$. Relative to the true optimal feasible assignment $`z^*`$: + +$$ +F(\hat{z}) \le \hat{F}(\hat{z}) + N\delta \le \hat{F}(z^*) + N\delta \le F(z^*) + 2N\delta +$$ + +This bound identifies *post-action residual loss* $`r_{ia}`$ as the critical empirical calibration target, motivating randomized pilot trials across permitted review actions. + +Finally, both budgets—remediation and statistical measurement—combine into a single batch-level **Mixed-Integer Convex Program (MICP)**. Let $`\tau^2`$ be the target variance ceiling for the quality estimate, $`B_{\text{total}}`$ the combined cost budget, and $`V_{\hat{m}}(\pi) = N^{-2}\sum_i (1 - \pi_i)\hat{m}_i / \pi_i`$ the planning variance under the calibrated residual model: + +$$ +\min_{z, \pi} \sum_{i,a} z_{ia}\hat{r}_{ia} \quad \text{s.t.} \quad \sum_{i,a} z_{ia} d_{ia} + \sum_i c_i \pi_i \le B_{\text{total}}, \quad V_{\hat{m}}(\pi) \le \tau^2 +$$ + +$$ +\sum_a z_{ia} = 1, \quad z_{ia} \in \{0, 1\}, \quad \sum_{i,a} z_{ia} t_{iaj} \le H_j^{\text{review}} \ \forall j, \quad \pi_{\min} \le \pi_i \le 1 +$$ + +Here $`H_j^{\text{review}}`$ is hard capacity allocated to remediation actions while reference audits are scheduled independently on frozen original submissions. The precision ceiling is model-based: independent audits must check residual calibration, and expected audit cost does not impose a hard limit on realized audit hours. Infeasibility exposes a conflict between the planning precision target, feasible review assignments, and available engineering budget. + +## 6 Reproducible experiments at equal cost + +The following self-contained synthetic experiments test the mathematical mechanisms derived above (they validate algorithmic behavior on specified public parameters rather than reporting proprietary client telemetry). A fixed batch contains $`N = 10,000`$ submissions partitioned into three strata of $`6,000`$, $`3,000`$, and $`1,000`$ items, containing $`60`$, $`300`$, and $`350`$ errors respectively (true batch defect rate $`\mu = 7.10\%`$). Unit audit costs are $`1`$, $`1.5`$, and $`3`$ units across the three strata. Every policy operates under an expected audit budget of $`B = 2,000`$ units and a minimum sampling floor of $`\pi_{\min} = 3\%`$. + +Risk predictions for the three strata are $`2\%`$, $`8\%`$, and $`30\%`$ (batch average $`6.60\%`$, differing from the true defect rate by $`0.50`$ percentage points). Cost-aware allocation uses the true within-stratum mean squared residuals (ideal calibration), while the misspecified policy reverses the residual ordering across strata. Holding predictions, budget, and submissions fixed, $`200,000`$ independent Monte Carlo repetitions verify the analytical variance formulas: + + + + + + + + +
Audit PolicySampling Probabilities Across the Three StrataStd. Dev. of Corrected Estimate (pct. points)Variance Change Relative to Uniform Sampling
**Uniform sampling with the same residual correction**14.81% / 14.81% / 14.81%0.568Baseline
**Prioritize predicted error probability ($`f_i`$)**3.00% / 11.56% / 43.33%0.657+33.77% increase
**Residual- and cost-aware allocation with ideal calibration**7.89% / 19.37% / 21.84%**0.515****-17.79% reduction**
**Misspecified residual ordering (reversed)**23.00% / 11.77% / 3.00%0.983+199.66% increase
**Misspecified allocation with 70% weight on uniform sampling**17.27% / 13.90% / 11.27%0.615+17.07% increase
+ +Under ideal residual calibration, the standard deviation of the population quality estimate falls by **9.33%** (a **17.79% variance reduction**). Notice that for the naive high-risk-first policy, the uncorrected raw error detection rate averages **19.60%** (wildly distorting true quality), whereas our IPW residual-corrected estimator recovers **7.1015%** while exposing that naive risk-prioritization actually *inflates* estimation variance by **33.77%** due to over-sampling the expensive third stratum. Across all five policies, Monte Carlo variances match analytical formulas to within $`0.5\%`$. + +A separate, exact allocation experiment isolates remediation gains. Two task groups contain $`100`$ items each, with initial error risks of $`0.40`$ and $`0.15`$. Additional review reduces those risks to $`0.30`$ ($`\Delta_1 = 0.10`$) and $`0.01`$ ($`\Delta_2 = 0.14`$), respectively. Review costs are equal and capacity is $`100`$ items. Ranking by initial risk leaves $`100(0.30) + 100(0.15) = 45`$ expected errors; ranking by expected loss reduction ($`\Delta_i`$) leaves $`100(0.40) + 100(0.01) = 41`$ expected errors—cutting expected defect loss by **8.89%** at the exact same review volume. + +## 7 Establish whether the design improves production + +Production deployment tests statistical quality inference and operational defect control separately, recognizing that *detecting more errors in a targeted queue is never, by itself, evidence of better delivered quality*: +1. **Inferential Validation on Frozen Adjudicated Batches**: Compare uniform sampling, legacy high-risk routing, cost-aware residual allocation, and defensive mixtures against complete independent adjudication—evaluating bias, MSE, and interval coverage across individual task strata and rubric versions. +2. **Capacity-Matched Remediation Trials**: Compare legacy tiered rules against marginal-gain ($`\Delta_i`$) action allocation at equal total specialist hours, measuring severe defects surviving into final post-training corpora and total fully-loaded cost per accepted submission. + +The contributor reliability records, tiered review workflows, and random reference audits I established from September 2023 provided the operational foundation for this architecture. The mathematical analysis above states the sampling, calibration, and capacity assumptions under which quality estimation is unbiased and audit allocation is optimal when residual second moments are correctly specified; the synthetic benchmarks test those mechanisms under their stated conditions. + +# Project 02: task-validity-in-financial-synthetic-data + +**Article Title** +Task validity in financial synthetic data + +**Organization** +Duke University × Top-Tier Foundation Model Company + +**Title** +Financial Task Design and Synthetic Data Engineering + +**URL** +https://www.theodoreoy.com/past-experience/artificial-intelligence/task-validity-in-financial-synthetic-data/ + +**Position** +Researcher + +**Location** +Durham, NC; Palo Alto, CA + +**Dates** +September 2023 – September 2026 + +**Read More Label** +Read More + +## Summary + +Initiated in September 2023 for a frontier foundation model program *(partner identity anonymized under NDA)*, this project defined what constitutes high-signal financial post-training data and established an end-to-end paradigm for adversarial task design, constraint-preserving synthetic data engineering, and structured preference supervision. To expose subtle reasoning blind spots where base models and practitioners systematically stumble, I engineered diagnostic, error-inducing financial decision tasks and a five-level pairwise preference protocol that translates doctoral- and CFA-level expert judgments into evidence-grounded training signals for RLHF and direct preference optimization. I paired these task specifications with a four-regime mathematical synthesis architecture—Gaussian copulas for cross-sectional corporate fundamentals, covariance-weighted affine projections enforcing exact accounting identities ($1.21 \times 10^{-14}$ maximum residual across 200,000 12-dimensional draws), CTGAN and TabDDPM for multimodal tables, with Lipschitz margin checks and Gaussian margin-to-noise ($\gamma/s$) analysis to guide label recomputation, and SDEs, neural SDEs, and Brownian bridges with strict temporal filtration isolation ($\mathcal{F}_s$) for continuous-time price and interest-rate trajectories. To bridge research design and massive vendor scale-up, I created **Batch Zero**—an independent execution and calibration gate subsequently adopted across the company for new post-training tracks. + +## Confidentiality & Scope Disclosure + +Conducted through a university–industry research collaboration with a flagship foundation model developer. While the corporate partner's name is withheld pursuant to non-disclosure agreements, my role, project scope, organizational protocols (*Batch Zero*), and mathematical synthesis frameworks are authorized for public disclosure. + +## Key Contributions + +* **Defined the Financial Post-Training Task Environment & Adversarial Diagnostic Paradigm (Sept 2023)**: Led financial and business decision task design for a frontier foundation model program, specifying capability taxonomies, primary evidence standards, input schemas, decisive variables, and acceptance conditions. Rather than authoring routine textbook prompts, I deliberately engineered **error-inducing diagnostic cases** targeting high-risk business decision boundaries where base models and practitioners systematically stumble—ensuring every task required genuine doctoral- or CFA-level domain judgment. +* **Architected the Five-Level Pairwise Preference Protocol for RLHF / DPO Supervision**: To convert tacit human domain expertise into actionable post-training gradients, I designed a structured **five-level pairwise preference protocol** that captures *why* an expert judges one response superior to another. By requiring experts to bind every preference judgment to verifiable evidence citations, accounting rules, and decisive variables, the protocol transformed subjective expert comparisons into high-signal preference data that teaches models which financial judgments are evidence-backed. +* **Principled Four-Regime Financial Synthetic Data Engineering**: Selected and customized mathematical synthesis engines around the exact structural relationships each financial task had to preserve, avoiding naive one-size-fits-all tabular generation: + * *Cross-Sectional Corporate Fundamentals*: Applied **Gaussian and vine copulas alongside conditional sampling** across company scale, operating vintage, and financial ratios to preserve joint cross-column dependencies and prevent unrealistic column-independent random generation. + * *Hard Accounting Identities, Ratio Bounds & Chronological Order*: Deployed **constrained sampling, variable transformations, and covariance-weighted affine manifold projections** to enforce exact balance-sheet/cash-flow identities, ratio ranges, and date ordering—achieving a maximum constraint residual of **$1.21 \times 10^{-14}$** across 200,000 12-dimensional synthetic draws. A separate standardized-residual rejection model yields a $0.0002529\%$ acceptance rate. + * *Multimodal & Mixed Numerical/Categorical Records*: Utilized **CTGAN and TabDDPM** to model multimodal empirical distributions, paired with mandatory post-generation business rule validation and Lipschitz margin checks and Gaussian margin-to-noise ($\gamma/s$) analysis to guide reference recomputation and reduce stale supervision labels. + * *Continuous-Time Price & Interest-Rate Trajectories*: Developed dynamic paths for temporal financial tasks (as distinct from static cross-sectional company comparisons) using **stochastic differential equations (SDEs), neural SDEs, Brownian bridges, and stochastic interpolation** (extending MathWorks-style stochastic interest-rate path constructions), matching each construction to the task's permitted filtration ($\mathcal{F}_s$) to prevent future-information evaluation leakage. +* **Created the "Batch Zero" Pre-Scaling Execution Gate (Adopted Company-Wide)**: Before scaling task families to external vendor production, I created **Batch Zero**—a pilot verification protocol in which doctoral-level specialists independently executed task construction, synthetic expansion, reference solving, and pairwise preference grading. Pilot disagreements and failure modes directly drove instruction revisions, boundary-case refinements, and **Batch One** entry criteria. The foundation model company subsequently adopted **Batch Zero** as a standard organizational gate across other new post-training tracks. +* **Quantified Stale-Label Corruption, Temporal Leakage & Family Clustering**: Proved analytically and via simulation how synthetic perturbations near decision boundaries cause **40.13%** inherited-label error at $\gamma/s = 0.25$ without recomputation, how undisclosed Brownian bridge endpoint conditioning creates an artificial **29.29% RMSE reduction** via future leakage, and how within-family correlation ($\rho = 0.5$) reduces 2,000 nominal synthetic variants across 100 families to **$N_{\text{eff}} \approx 190.48$** effective independent observations. + +## Architectural Context: Why Frontier Financial Post-Training Requires Diagnostic Traps & Manifold-Exact Synthesis + +In September 2023, frontier foundation models could already recite corporate finance definitions, yet they failed unpredictably on real-world financial workflows where small numerical shifts or accounting footnotes reverse an investment recommendation. Our architecture solved two foundational challenges in post-training data engineering: +1. **Why We Engineer "Error-Prone" Prompts**: Easy prompts may yield less discriminative comparisons between candidate responses. By designing prompts specifically around subtle business failure modes—such as covenant breaches, off-balance-sheet adjustments, or near-boundary valuation spreads—we force reasoning errors into the open. Pairing these diagnostic prompts with our five-level pairwise preference protocol translates a specialist's domain intuition (*"Response A is superior because it isolates the working-capital constraint and cites Footnote 4"*) into structured preference supervision that directly steers post-training. +2. **Matching the Mathematical Generator to the Financial Structure**: Generating synthetic financial data is fundamentally different from generating synthetic images or unconstrained text. If a synthetic corporate table violates Total Assets $=$ Total Liabilities $+$ Equity by even a fraction of a percent, or if a synthetic interest-rate trajectory leaks future terminal rates into a historical forecasting prompt, the resulting training data actively corrupts model reasoning. Sections I–VI below establish the complete mathematical framework we built to guarantee constraint validity, label stability, and filtration integrity across scaled synthetic production. + +## Manuscript + +## 1 Why better distributional fit may leave task validity unchanged + +Modern tabular generative models address distinct distributional challenges: **[CTGAN](https://arxiv.org/abs/1907.00503)** handles mixed discrete-continuous types, multimodality, and class imbalance via mode-specific normalization and conditional vectors, while **[TabDDPM](https://proceedings.mlr.press/v202/kotelnikov23a.html)** applies Gaussian and multinomial diffusion to numerical and categorical fields (*[CTGAN](https://arxiv.org/abs/1907.00503); [TabDDPM](https://proceedings.mlr.press/v202/kotelnikov23a.html)*). However, while both learn empirical distributions effectively, **financial task validity requires strict structural invariants that global distributional similarity metrics cannot enforce**: exact accounting identities, chronological ordering of fiscal periods, covenant applicability conditions, and logical consistency between numerical inputs and reference judgments. + +Consider a failure mode that admits an exact mathematical proof. Valid financial inputs $`x \in \mathbb{R}^d`$ must satisfy $`m`$ linear accounting or ratio identities $`Ax = b`$, where $`A \in \mathbb{R}^{m \times d}`$ has full row rank. The valid population distribution $`P`$ is supported entirely on the affine subspace $`M = \{x \in \mathbb{R}^d : Ax = b\}`$ and has finite second moments. Let the columns of $`N \in \mathbb{R}^{d \times m}`$ form an orthonormal basis for the normal space orthogonal to $`M`$. Suppose a generative model captures $`P`$ tangential to the manifold but adds a tiny amount of independent noise in the normal directions: $`X_\sigma = X + \sigma N \varepsilon`$, where $`\varepsilon \sim \mathcal{N}(0, I_m)`$ is independent of $`X \sim P`$. Writing the generated distribution as $`Q_\sigma`$, we have for every $`\sigma > 0`$: + +$$ +W_2(P, Q_\sigma) \le \sqrt{E\|X_\sigma - X\|^2} = \sigma \sqrt{m} \longrightarrow 0 \quad (\text{as } \sigma \to 0^+) +$$ + +$$ +P(M) = 1, \qquad Q_\sigma(M) = 0 +$$ + +$$ +\|P - Q_\sigma\|_{\text{TV}} := \sup_B |P(B) - Q_\sigma(B)| = 1 +$$ + +The first line follows from the coupling $`(X, X_\sigma)`$ constructed on the same probability space. The second follows because $`AN`$ is invertible and a continuous $`m`$-dimensional Gaussian vector equals zero with probability zero. The third follows by taking the Borel set $`B = M`$. Thus, **the 2-Wasserstein distance $`W_2(P, Q_\sigma)`$ can converge to zero while the acceptance rate under exact business constraints remains identically zero ($`Q_\sigma(M) = 0`$) and Total Variation distance remains at its maximum of $`1`$**. + +This counterexample dictates how responsibilities must be partitioned in a production synthetic data architecture (*[SDV constraint-augmented generation](https://docs.sdv.dev/sdv/modeling/constraint-augmented-generation-cag)*): +• **Copula models and conditional sampling** capture cross-sectional rank and tail dependencies across continuous corporate indicators (company scale, operating history, leverage, and margin ratios) so columns are never generated independently. +• **[CTGAN](https://arxiv.org/abs/1907.00503) and [TabDDPM](https://proceedings.mlr.press/v202/kotelnikov23a.html)** model complex mixed-type and multimodal empirical distributions, followed by explicit business-rule validation. +• **Deterministic variable transformations and manifold projections** enforce hard accounting equalities, ratio bounds, and chronological sequences exactly. + +## 2 Enforce business constraints while preserving the correct conditional distribution + +Enforcing hard constraints by generating unconstrained candidates and discarding violations (post-hoc rejection sampling) becomes computationally intractable as multiple financial constraints tighten simultaneously. Suppose a candidate draw has $`m`$ standardized constraint residuals that behave as independent standard normal variables, and each must lie within a tight tolerance $`[-\epsilon, \epsilon]`$. The acceptance probability and expected candidate draws required per valid input are: + +$$ +\alpha_m(\epsilon) = [2\Phi(\epsilon) - 1]^m \sim \left( \sqrt{\frac{2}{\pi}} \epsilon \right)^m \quad (\epsilon \to 0^+), \qquad E[N_{\text{attempts}}] = \frac{1}{\alpha_m(\epsilon)} +$$ + +Each additional accounting or ratio constraint multiplies the expected candidate count. Setting each tolerance to $`\epsilon = 0.05`$ residual standard deviations yields the following exact analytical scaling: + + + + + + +
Simultaneous Constraints ($`m`$)Rejection Sampling Acceptance Rate $`\alpha_m(\epsilon)`$Expected Candidate Draws per Valid Input
**1**3.9878%25.08
**2**0.1590%628.84
**4****0.0002529%****395,442.59**
+ +Generating directly through the degrees of freedom permitted by the constraints eliminates these wasted draws. In the linear Gaussian regime, covariance-weighted affine projection also preserves the exact theoretical conditional distribution. For a Gaussian copula, this result applies only when the constraints are affine in its latent Gaussian coordinates; nonlinear marginal transformations do not generally preserve affine constraints in the observed financial variables. Let the initial unconstrained draw be $`Z \sim \mathcal{N}(\mu, \Sigma)`$ with $`\Sigma \succ 0`$. Rather than overwriting a single balance-sheet line item (which corrupts covariance structure), we apply a **covariance-weighted Mahalanobis projection**: + +$$ +X = \arg\min_{x : Ax = b} \frac{1}{2}(x - Z)^\top \Sigma^{-1}(x - Z) = Z + K(b - AZ), \qquad K = \Sigma A^\top (A \Sigma A^\top)^{-1} +$$ + +The first-order Lagrangian stationarity condition is $`\Sigma^{-1}(X - Z) + A^\top \lambda = 0`$; substituting $`AX = b`$ and eliminating the multiplier $`\lambda`$ yields the closed-form update. Since $`AK = I_m`$, every draw satisfies $`AX = b`$ exactly, with mean and covariance: + +$$ +E[X] = \mu + K(b - A\mu) +$$ + +$$ +\operatorname{Cov}(X) = (I - KA)\Sigma(I - KA)^\top = \Sigma - \Sigma A^\top (A \Sigma A^\top)^{-1} A\Sigma +$$ + +Because an affine transformation preserves Gaussianity, these moments match the exact regular conditional Gaussian law of $`Z \mid AZ = b`$ (*[Majumdar and Majumdar on Gaussian distributions under linear conditioning](https://arxiv.org/abs/1710.09285)*). + +In our synthetic verification benchmark, generating **200,000 synthetic 12-dimensional financial vectors under 4 simultaneous linear constraints** achieved a **maximum absolute constraint residual of $`1.21 \times 10^{-14}`$** and a **relative Frobenius covariance error of 0.662%** against the theoretical conditional covariance—using direct constrained draws without post-hoc rejection. The nearly 400,000 candidate count above belongs to the separate standardized-residual rejection model. Fields fully determined by accounting definitions are computed analytically; range bounds and chronological date orderings that admit bijective transformations (such as log-transforms or positive increments $`t_{k} = t_{k-1} + \exp(\xi_k)`$) are enforced in latent generation space. + +## 3 Use decision margins to determine when reference judgments must be recomputed + +Synthetic expansion of a golden financial case often preserves the qualitative business narrative while perturbing decisive financial figures, covenant thresholds, or macroeconomic inputs. **Retaining the original golden-case reference answer after perturbing the numbers injects systematic label corruption into post-training**, even when every synthetic input satisfies all accounting constraints. Just as *[FinQA](https://aclanthology.org/2021.emnlp-main.300/)* pairs financial questions with executable reasoning programs, synthetic preference engineering requires the link between inputs, reference conclusions, and expert preference explanations to update dynamically (*[FinQA](https://aclanthology.org/2021.emnlp-main.300/)*). + +Consider a formal decision rule for choosing between two corporate actions. Let $`d(x)`$ denote the net valuation or metric advantage of Action A over Action B, so the true reference decision is $`\operatorname{sign} d(x)`$. If $`d`$ is $`L`$-Lipschitz on a feasible neighborhood, a sufficient condition for retaining the original reference decision without recomputation is that the perturbation $`\Delta`$ remains strictly within the decision margin: + +$$ +|d(x + \Delta) - d(x)| \le L\|\Delta\|, \qquad |d(x)| > L\|\Delta\| \implies \operatorname{sign} d(x + \Delta) = \operatorname{sign} d(x) +$$ + +Because our diagnostic tasks deliberately target **contested decision boundaries** where models are most prone to error, $`|d(x)|`$ is intentionally small! We can derive the exact closed-form rate of **stale-label error** for a linear comparison rule $`d(x) = w^\top x - c`$ with initial margin $`\gamma = d(x) > 0`$. Under a feasible manifold perturbation $`\Delta \sim \mathcal{N}(0, \Omega)`$ with projected variance $`s^2 = w^\top \Omega w > 0`$, the probability that retaining the original decision reverses the true ground-truth label is: + +$$ +\Pr\{d(x + \Delta) < 0\} = \Pr\{w^\top \Delta < -\gamma\} = \Phi\left(-\frac{\gamma}{s}\right) +$$ + +Verified across $`1,000,000`$ Monte Carlo draws: +• When the margin-to-noise ratio $`\gamma / s = 2`$, the inherited-label error rate is **2.28%**. +• When $`\gamma / s = 1`$, the inherited-label error rate jumps to **15.87%**. +• Near diagnostic decision boundaries where $`\gamma / s = 0.25`$, **40.13% of retained labels are false!** + +This analytical result explains why our pipeline links case expansion directly to reference recomputation and fresh **five-level pairwise preference grading** (see the later methodological comparison in *[HelpSteer2-Preference](https://arxiv.org/abs/2410.01257)*). Whenever a synthetic perturbation crosses or approaches a decision boundary, the reference solution is recomputed, fresh candidate model trajectories are sampled, and domain experts evaluate the pair on our five-level preference scale—explicitly annotating which trajectory identified the flipped decisive variable and supported its judgment with primary evidence. + +## 4 Define the information available for each temporal task + +While cross-sectional corporate comparisons do not require continuous-time stochastic processes, dynamic financial tasks—such as derivatives valuation, interest-rate term-structure interpolation, and multi-period liquidity stress testing—require path generators. In these temporal regimes, we deploy **stochastic differential equations (SDEs)** and **neural SDEs** to model path distributions beyond single-time marginals (*[Neural SDEs as Infinite-Dimensional GANs](https://proceedings.mlr.press/v139/kidger21b.html)*), alongside **Brownian bridges** and **stochastic interpolation** (aligning with classical MathWorks stochastic interest-rate path constructions) when conditioning on known intermediate observations or endpoints. + +However, path conditioning introduces a severe **temporal evaluation leakage hazard** whose magnitude can be derived in closed form. Consider a scaled Brownian motion $`X_u = \sigma W_u`$ over $`s < t < T`$. Under squared error loss, the optimal squared-error predictor acting at time $`s`$ with access only to the historical filtration $`\mathcal{F}_s`$ has conditional mean squared error $`\sigma^2(t - s)`$. If a synthetic pipeline conditions path generation on the future terminal endpoint $`X_T`$ and inadvertently exposes that conditioned structure to a forecasting task, Gaussian conditioning gives (*[Pitman on the Brownian bridge](https://www.stat.berkeley.edu/~pitman/s205s03/b_bridge.pdf)*): + +$$ +\operatorname{Var}(X_t \mid \mathcal{F}_s) = \sigma^2(t - s) +$$ + +$$ +\operatorname{Var}(X_t \mid \mathcal{F}_s, X_T) = \sigma^2(t - s) - \frac{[\sigma^2(t - s)]^2}{\sigma^2(T - s)} = \sigma^2(t - s)\frac{T - t}{T - s} +$$ + +When $`t = (s + T)/2`$ is the midpoint of the forecasting horizon, conditioning on $`X_T`$ halves the conditional variance, producing an **apparent 29.29% reduction in RMSE** ($`1 - 1/\sqrt{2}`$) that is 100% attributable to future information leakage rather than model capability! (By contrast, if the task explicitly supplies both endpoints $`X_s`$ and $`X_T`$ and asks the model to audit an *interpolation* or back-cast an intervening accounting period, Brownian bridge conditioning is mathematically appropriate.) + +Our task environment specification therefore enforces a strict separation across three information sets: +1. **Construction Information ($`\mathcal{F}_{\text{gen}}`$)**: Complete paths, terminal endpoints, or latent parameters used by the generator to synthesize the scenario. +2. **Permitted Predictor Filtration ($`\mathcal{F}_s`$)**: The exact historical information legally visible to the model at decision time $`s`$. +3. **Evaluation & Oracle Information ($`\mathcal{F}_{\text{eval}}`$)**: Realized post-$`s`$ outcomes reserved strictly for scoring or explicitly disclosed oracle baselines. + +## 5 Test independent execution through Batch Zero and retain the task family as a statistical unit + +Even when constraint projections, decision-margin recomputations, and filtration rules are mathematically sound, a post-training specification must remain reproducible when executed by hundreds of external specialists. To solve this organizational scaling bottleneck, I created **Batch Zero**: +• **The Batch Zero Protocol**: Prior to launching scaled vendor production (*Batch One*), a select cohort of doctoral-level domain specialists with proven delivery records independently executed the entire specification—authoring seed cases, running synthetic expansions, verifying reference solutions, and assigning five-level pairwise preference scores. Every failed case or inter-expert disagreement in Batch Zero triggered surgical revisions to prompt instructions, boundary definitions, and calibration rubrics. Following the pilot’s use in preparing external vendor workflows, **the partner foundation model company adopted Batch Zero as a standard pre-scaling gate across other new post-training tracks**. + +Simultaneously, our statistical governance prevents **synthetic sample-size overstatement**. Suppose $`G`$ independent golden-case families are each expanded into $`k`$ synthetic variants ($`Gk`$ nominal records), where any two variants within a family share correlation $`\rho`$ and common variance $`\sigma_Y^2`$. Expanding the covariance sum for the sample mean yields the exact **effective sample size** $`N_{\text{eff}}`$: + +$$ +\operatorname{Var}(\bar{Y}) = \frac{Gk\sigma_Y^2 + Gk(k - 1)\rho\sigma_Y^2}{(Gk)^2} = \frac{\sigma_Y^2}{Gk}[1 + (k - 1)\rho] \implies N_{\text{eff}} = \frac{Gk}{1 + (k - 1)\rho} +$$ + +With $`G = 100`$ independent families, $`k = 20`$ synthetic variants per family ($`2,000`$ nominal tasks), and within-family correlation $`\rho = 0.5`$, **the nominal 2,000 records provide the exact same statistical precision for the mean as just $`190.48`$ independent records**—and ignoring within-family correlation understates standard errors by a factor of **$`3.24\times`$**! Consequently, our production pipeline enforces strict family-level train/evaluation splits and computes all confidence intervals with family-level and expert-level hierarchical clustering. + +## 6 Establish whether the complete method justifies scaled production + +To evaluate whether our constraint-preserving synthesis and preference architecture justifies scaled production, we hold the task mix, generator training budget, expert review budget, and final acceptance criteria fixed, introducing each methodological layer in staged blind comparisons evaluated across independent task families: + + + + + + + +
Comparison StageMethodological Upgrade IntroducedCore Research & Production Question
**Existing production baseline**Unconstrained generation + manual expert review & reworkWhat is the true fully-loaded cost per finally accepted, valid case?
**Add manifold-constrained generation**Fix generator; enforce affine projections & variable transformsDo invalid candidates and manual expert repairs drop while preserving conditional distributions?
**Add margin-aware reference recomputation**Recompute reference judgments when the sufficient Lipschitz margin check fails; use $`\gamma/s`$ to quantify Gaussian label-flip riskDo stale-label errors decline, producing valid supervision on both sides of decision boundaries?
**Apply complete Batch Zero & filtration criteria**Jointly check $`\mathcal{F}_s`$ timing conditions and independent executionDo post-training gains persist after excluding leakage and clustering by source family?
+ +Because raw acceptance rates reward generating trivial variants of easy templates, and raw generation speed ignores manual expert repair costs, we govern production decisions via **Scenario-Weighted Family Coverage Efficiency**. Let $`j`$ index predefined cells in the financial capability matrix with strategic weights $`w_j \ge 0`$ ($`\sum_j w_j = 1`$) and required independent valid family counts $`n_j^* > 0`$: + +$$ +\operatorname{Coverage} = \sum_j w_j \min\left(1, \frac{n_j^{\text{valid}}}{n_j^*}\right), \qquad \operatorname{Efficiency} = \frac{\operatorname{Coverage}}{C_{\text{generation}} + C_{\text{review}} + C_{\text{rework}} + C_{\text{adjudication}}} +$$ + +Once a cell reaches its target $`n_j^*`$ independent families, repeatedly expanding easy templates yields zero marginal coverage gain (*[Alaa et al. on synthetic data evaluation](https://proceedings.mlr.press/v162/alaa22a.html)*). Together, manifold constraint enforcement, decision-margin label recomputation, temporal filtration isolation, and **Batch Zero** calibration establish a rigorous engineering standard for synthetic preference data in frontier model post-training. + +# Project 03: verification-and-supervision-in-scientific-reasoning-tasks + +**Article Title** +Verification and supervision in scientific reasoning tasks + +**Organization** +Duke University × Top-Tier AI Research Lab + +**Title** +Scientific Reasoning Evaluation and Task Design + +**URL** +https://www.theodoreoy.com/past-experience/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks/ + +**Position** +Researcher + +**Location** +Durham, NC; Palo Alto, CA + +**Dates** +September 2023 – September 2026 + +**Read More Label** +Read More + +## Summary + +Launched in September 2023 through an early academic collaboration with a frontier AI research laboratory *(partner identity anonymized under NDA)*, this work developed a three-pillar evaluation and process-supervision framework for scientific, mathematical, and quantitative risk reasoning. Recognizing in our earliest experiments that simply stumping frontier models on Olympiad- or graduate-level STEM final answers yields minimal actionable feedback for model iteration, I built an acceptance and diagnostic architecture that audits **(1) Chain-of-Thought (CoT) trajectory soundness** via acyclic proof-obligation dependency graphs ($\Gamma_q \vdash C_q$), **(2) strict compliance in sandboxed tool invocation** (requiring execution within auditable enterprise toolkits such as Excel rather than unauthorized Python virtual machines in regulated workflows, and rejecting numerically correct answers obtained via non-compliant tools), and **(3) structural mathematical reduction over brute-force computation**. In an exact-arithmetic stress test of 3,600 constructed submissions across 600 strongly convex quadratic problem families, semantic certificate verification accepted all **1,200 valid submissions (0.0% false rejection, preserving valid alternative Lagrangian and elimination proofs)** and rejected all **2,400 invalid submissions (0.0% false acceptance, compared to 75.0% false acceptance under answer-only checking)**. + +## Confidentiality & Scope Disclosure + +Conducted through an early university–industry research partnership with a frontier AI research laboratory beginning in September 2023. While the partner laboratory's name is withheld pursuant to non-disclosure agreements, my role, evaluation methodology, proof-obligation verification architecture, and exact-arithmetic stress tests are authorized for public disclosure. + +## Key Contributions + +* **Developed Three-Pillar Process and Tool Evaluation from September 2023**: During a university collaboration aimed at stress-testing frontier models with Olympiad- and graduate-level STEM tasks, I identified the limitations of final-answer checks in our workflow and developed an evaluation framework covering reasoning validity, compliance with task-specific tool policies, and structural mathematical insight. + 1. *Chain-of-Thought (CoT) Trajectory & Proof-Obligation Auditing*: Verifying whether intermediate reasoning steps follow a sound logical trajectory rather than reaching a lucky final number through compensating errors or unverified leaps. + 2. *Enterprise-Compliant Tool-Invocation Governance*: Auditing whether the model selects and executes tools strictly within regulatory and institutional compliance boundaries. When a task’s specified data-governance policy permits only the approved Excel environment, an unauthorized Python virtual machine violates that execution policy. Under our acceptance standard, **even if a model computes the exact right numerical answer, invoking a non-compliant tool (e.g., Python instead of mandated Excel) results in immediate failure**. + 3. *Penalizing Brute-Force Compute in Favor of Structural Mathematical Insight*: Designing tasks and scoring rubrics that discourage models from burning inference compute on brute-force algebraic or numerical exhaustion, rewarding instead the identification of domain invariants and structural reductions. +* **Translated Physics, Mathematics & Risk Engineering into Proof-Obligation DAGs**: Formalized task deliverables as directed acyclic dependency graphs ($\Gamma_q \vdash C_q$) that separate physical model formulation, theorem applicability, compliant tool execution, uniform approximation bounds, and conclusion strength—accommodating legitimate alternative proof paths (such as Lagrangian duality versus variable elimination) that brittle reference-matching pipelines falsely reject. +* **Derived Projected-Residual Optimality Certificates for Convex Optimization**: Formulated exact KKT projected-residual bounds ($0 \le f(x) - v^* \le \|P_{\ker A} r\|_2^2 / (2\mu)$) on the optimality gap for equality-constrained strongly convex quadratic programs, replacing heuristic numerical proximity checks with mathematical certificates of solution quality. +* **Engineered Original Kinematics Benchmarks via Path Invariance & Harmonic Asymptotics**: Created original physics and risk-engineering tasks that test deep structural reasoning—using the path invariance of vertical travel time to reduce a 2D kinematics problem to 1D horizontal scheduling, proving an attained minimum completion time and an unattained supremum, and deriving harmonic-number ($H_n$) asymptotics for staircase trajectories approaching the upper bound. +* **Quantified Biased Supervision Gradients & Task-Transformation Error Propagation**: Proved mathematically how false acceptance ($\alpha$) and false rejection ($\beta$) reverse model rankings on leaderboards and bias the binary cross-entropy training gradient of Process Reward Models ($E[\nabla_s \ell(s,V) - \nabla_s \ell(s,Z) \mid h] = p(h) - \tilde{p}(h)$), and derived exact tolerance propagation laws ($a\delta + 2\varepsilon$) under bijective task transformations. +* **Achieved 0.0% False Acceptance and 0.0% False Rejection Across 3,600 Exact-Arithmetic Stress Tests**: Constructed 600 strongly convex quadratic problem families (3,600 exact rational arithmetic submissions) where 75% of invalid proofs report the exact right optimal value $v^*$. Semantic certificate checking accepted all **1,200 valid proofs (0.0% false rejection)** and rejected all **2,400 invalid submissions (0.0% false acceptance)**, outperforming answer-only checking (**75.0% false acceptance**) and strict reference-path matching (**50.0% false rejection**). + +## Architectural Context: From "Stumping Models on Final Answers" to Verifiable Trajectory & Tool Governance + +The collaboration began in autumn 2023 with scientific-reasoning tasks for frontier models. Initial experiments in our workflow showed why final-answer matching was insufficient: a correct number could coexist with an invalid physical assumption, an unsupported computational shortcut, or a tool call that violated the task’s specified execution policy. Sections I–VII formalize the resulting acceptance architecture and relate it to subsequent process-evaluation research; the project start date does not imply that every later citation or mathematical formulation was part of the initial workflow. + +## Manuscript + +## 1 Correct final answers can conceal invalid reasoning and reverse model rankings + +Checking only the final answer is appropriate only when that final value constitutes the entire deliverable. In scientific reasoning, quantitative finance, and risk engineering, the model formulation, theorem assumptions, compliant tool invocation, intermediate derivations, and conclusion must hold together. In the OlympiadBench and Omni-MATH samples later analyzed to construct *ProcessBench*, **32.2% and 51.8% of solutions with correct final answers still contained process errors** (*[ProcessBench, Section 3.3](https://arxiv.org/html/2412.06559v2)*). Answer checking and trajectory validity measure fundamentally different things. + +Suppose problems and model trajectories are drawn from a target capability distribution. Let $`Z = 1`$ indicate that the complete solution (including reasoning steps and compliant tool execution) is valid, and let $`V = 1`$ indicate that the evaluation protocol passes it. For model $`m`$, let $`\theta_m = \Pr_m(Z = 1)`$ be its true validity rate, $`\alpha_m = \Pr_m(V = 1 \mid Z = 0)`$ its **false acceptance rate** on invalid solutions (e.g., passing a flawed proof, a brute-force guess, or a non-compliant Python tool call that yields the right number), and $`\beta_m = \Pr_m(V = 0 \mid Z = 1)`$ its **false rejection rate** on valid solutions (e.g., rejecting a valid alternative derivation). The law of total probability gives: + +$$ +(1) \qquad s_m = E_m[V] = \alpha_m + (1 - \alpha_m - \beta_m)\theta_m, \qquad s_m - \theta_m = \alpha_m(1 - \theta_m) - \beta_m\theta_m +$$ + +The first bias term $`\alpha_m(1 - \theta_m)`$ credits invalid trajectories; the second term $`-\beta_m\theta_m`$ penalizes valid ones. **Adding more benchmark problems reduces sampling variance in $`\hat{s}_m`$, but never removes the structural gap $`s_m - \theta_m`$**. Because different models exhibit different failure modes (e.g., one model adheres strictly to compliant tools and rigorous deduction, while another aggressively invokes unauthorized code interpreters or heuristic shortcuts), their $`\alpha_m`$ and $`\beta_m`$ can differ enough to **strictly reverse their leaderboard ranking**: +• In a concrete arithmetic scenario, suppose Model A and Model B have true validity rates of $`\theta_A = 72\%`$ and $`\theta_B = 64\%`$, respectively. +• If their invalid trajectories pass a final-answer check with probabilities $`\alpha_A = 4\%`$ and $`\alpha_B = 36\%`$ (while $`\beta_A = \beta_B = 0\%`$), their observed benchmark scores become **$`s_A = 73.12\%`$** and **$`s_B = 76.96\%`$**—ranking the less capable, shortcut-prone model higher! + +While process supervision extends evaluation to intermediate steps (*[Let's Verify Step by Step, Sections 2 and 3](https://arxiv.org/html/2305.20050v1)*), the architecture formalized here addresses three practical requirements in scientific post-training: omissions in reference solutions, valid alternative proof methods, and strict enforcement of tool-compliance and structural-insight obligations defined *before* model responses are generated. + +## 2 Define acceptance through proof obligations and allow valid alternative solutions + +A complete task deliverable specifies its domain assumptions and environmental compliance constraints $`\Gamma_q`$, the target conclusion $`C_q`$, and the proof obligations required to establish it. A model trajectory $`\tau`$ may choose its own intermediate lemmas, coordinate representations, or order of deduction; acceptance depends strictly on whether it fulfills the obligations while obeying tool governance. Organizing the response as a directed acyclic dependency graph (DAG) $`G`$ yields the local soundness check for every node $`v_j`$ alongside the global tool-compliance predicate $`\mathcal{T}_{\text{comp}}(\tau) \in \{0, 1\}`$: + +$$ +(2) \qquad \Gamma_q \cup \{v_k : k \in \operatorname{pa}(j)\} \vdash v_j, \qquad \Gamma_q \cup \{v_j : j \in \operatorname{sink}(G)\} \vdash C_q, \qquad \mathcal{T}_{\text{comp}}(\tau) = 1 +$$ + +Here $`\vdash`$ denotes valid derivability within the sound domain inference system, and $`\mathcal{T}_{\text{comp}}(\tau) = 1`$ certifies that every external tool invocation $`t \in \tau`$ belongs to the task's permitted compliance sandbox $`\mathcal{A}_q`$ (for instance, $`\mathcal{A}_q = \{\text{Excel Audited Workbook}\}`$ and $`\text{Python VM} \notin \mathcal{A}_q`$). If the root nodes follow from verified assumptions $`\Gamma_q`$, each dependency edge is sound, and $`\mathcal{T}_{\text{comp}}(\tau) = 1`$, structural induction over a topological ordering of $`G`$ establishes the complete conclusion from local checks. + +In production implementation, I organized the central obligations around five verifiable gates: +1. **Valid Model Formulation**: Checking that physical, financial, or probabilistic state spaces are well-defined. +2. **Compliant Tool Selection ($`\mathcal{T}_{\text{comp}}(\tau) = 1`$)**: When a task’s specified data-governance policy permits only the approved Excel environment, trajectories that invoke an unauthorized Python virtual machine are rejected immediately ($`V = 0`$), even if the final numerical calculation is exact. +3. **Method Applicability & Uniform Error Control**: In physics tasks, pointwise convergence at a fixed parameter value does not establish that an approximation remains valid inside an optimization, an integral, or an exchange of limits. Authors and models must specify the approximation parameter, its domain of validity, and uniform error bounds before claiming an order of accuracy. In risk engineering, the corresponding obligations govern distributional and tail-dependence assumptions behind extreme-value bounds. +4. **Structural Insight vs. Brute-Force Penalization**: Obligations require identifying the governing mathematical invariant rather than exhausting finite grids via raw compute. +5. **First-Error Localization in Feedback**: Algebra that is locally correct under a false premise does not restore proof validity. Recording the first failed obligation node $`v_j`$ and marking its descendant subgraph $`\operatorname{desc}(v_j)`$ as unverified resolves the post-error ambiguity highlighted in *[ProcessBench, Section 3.1](https://arxiv.org/html/2412.06559v2)*. + +## 3 Produce an error certificate as part of acceptance + +An executable optimization subfamily illustrates how formal mathematical certificates replace brittle string matching. Consider the equality-constrained strongly convex quadratic problem: + +$$ +(3) \qquad v^* = \min_{Ax = b} f(x), \qquad f(x) = \frac{1}{2}x^\top H x + c^\top x, \qquad H \succeq \mu I, \quad \mu > 0 +$$ + +where $`H`$ is symmetric, $`Ax = b`$ is feasible, and $`\mu > 0`$ is a verified lower bound on curvature. For any feasible candidate $`x`$ ($`Ax = b`$), choose any dual multiplier vector $`\nu`$ and define the stationarity residual $`r = Hx + c + A^\top \nu`$. Strong convexity gives the first inequality below for every feasible displacement $`d \in \ker A`$; completing the square in $`d`$ yields an exact **optimality error certificate**: + +$$ +(4) \qquad f(x + d) \ge f(x) + r^\top d + \frac{\mu}{2}\|d\|^2 +$$ + +$$ +\inf_{d \in \ker A} \left( r^\top d + \frac{\mu}{2}\|d\|^2 \right) = -\frac{\|P_{\ker A} r\|_2^2}{2\mu} +$$ + +$$ +0 \le f(x) - v^* \le \frac{\|P_{\ker A} r\|_2^2}{2\mu} +$$ + +The operator $`P_{\ker A}`$ is the orthogonal projection onto the null space of feasible directions $`\ker A`$, removing residual components normal to the constraints that do not affect optimality along the feasible manifold (*[Boyd and Vandenberghe, Convex Optimization, Sections 5.5 and 9.1](https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf)*). A zero projected residual ($`P_{\ker A} r = 0`$) certifies exact optimality; otherwise, the certificate bounds the suboptimality gap in objective units: +• Solutions obtained via **Lagrange multipliers** or by **first eliminating the equality constraints** map seamlessly to the exact same semantic obligations and pass verification. +• Conversely, merely reporting the correct optimal value $`v^*`$ does not satisfy the certificate obligations: feasibility and the claimed objective value must be checked independently. If a submission asserts a KKT multiplier witness, the full stationarity equation $`Hx+c+A^\top\nu=0`$ must also be verified. The projected-residual bound alone cannot validate the claimed multiplier, because $`P_{\ker A}A^\top\nu=0`$ for every $`\nu`$; a valid equality-elimination proof need not submit such a witness. + +Optimization certificates are one instance of our broader obligation design. In an original kinematics benchmark I created for the project, I used the **path invariance of vertical travel time** to reduce a 2D trajectory problem to 1D horizontal scheduling—separately proving an **attained minimum completion time**, an **unattained supremum**, and **harmonic-number ($`H_n = \sum_{k=1}^n k^{-1}`$) asymptotics** for a staircase path construction approaching the supremum. This benchmark demonstrates why feasibility, optimal bounds, structural reduction, and attainment require separate proof obligations even when a model guesses the final limiting value correctly. + +## 4 Task expansion must propagate mathematical error + +While symbolic templates (*[GSM-Symbolic, Section 3.1](https://arxiv.org/html/2410.05229v1)*) parameterize variable domains and answer formulas, scientific and engineering benchmarks also require verifiable laws governing how a transformation changes feasible sets, objective values, and acceptance tolerances. + +Let the original problem and its transformed variant have feasible sets $`F`$ and $`F'`$ with objectives $`J`$ and $`J'`$ and finite infima $`v`$ and $`v'`$. Suppose the task designer verifies a bijection $`\Phi : F \to F'`$ satisfying the uniform bound for every feasible $`u \in F`$: + +$$ +(5) \qquad |J'(\Phi(u)) - aJ(u) - b| \le \varepsilon, \quad a > 0 +$$ + +$$ +av + b - \varepsilon \le v' \le av + b + \varepsilon +$$ + +$$ +J(u) - v \le \delta \implies J'(\Phi(u)) - v' \le a\delta + 2\varepsilon +$$ + +Taking infima across the first line yields the second; subtracting the lower bound on $`v'`$ from the upper bound on $`J'(\Phi(u))`$ yields the third. This theorem answers an essential engineering question: after a unit conversion, parameter rescaling, or numerical transformation with bounded error $`\varepsilon`$, what acceptance tolerance must apply to a previously $`\delta`$-optimal solution? Retaining the raw tolerance $`\delta`$ can misclassify valid transformed solutions as model failures, while relaxing tolerances ad-hoc conceals genuine capability gaps. Moreover, when $`\varepsilon = 0`$, the bijection and positive scaling $`a > 0`$ preserve whether an optimum is attained; variants that alter boundary compactness or admissible control classes require fresh attainment proofs. + +## 5 Label quality determines what post-training research optimizes + +Granular step-level process labels improve post-training only if they measure true logical and compliance validity; systematic grading errors are directly absorbed by downstream Process Reward Models (PRMs) and RLHF policies. Let $`h`$ denote the problem and reasoning prefix through the current step, $`Z = 1`$ indicate true step validity under the problem's assumptions, and $`V \in \{0, 1\}`$ be the noisy acceptance label produced by the grader with conditional false acceptance $`\alpha(h)`$ and false rejection $`\beta(h)`$. Writing $`p(h) = \Pr(Z = 1 \mid h)`$, a process evaluator with logit $`s`$ trained via binary cross-entropy optimizes toward the corrupted conditional target: + +$$ +(6) \qquad \tilde{p}(h) = \alpha(h) + [1 - \alpha(h) - \beta(h)]p(h) +$$ + +$$ +\frac{\partial \ell(s, V)}{\partial s} = \sigma(s) - V +$$ + +$$ +E[\nabla_s \ell(s, V) - \nabla_s \ell(s, Z) \mid h] = p(h) - \tilde{p}(h) +$$ + +This expected gradient bias formula proves mathematically why naive evaluation pipelines damage post-training: +• **When false rejection dominates ($`\beta(h)p(h) > \alpha(h)[1-p(h)]`$)**: $`\tilde{p}(h) < p(h)`$, shifting the learned validity target downward and penalizing valid alternative proofs. +• **When false acceptance dominates ($`\alpha(h)[1-p(h)] > \beta(h)p(h)`$)**: $`\tilde{p}(h) > p(h)`$, shifting the learned validity target upward and rewarding flawed proofs or non-compliant tool calls. The two effects cancel in the conditional mean when their weighted contributions are equal. + +Furthermore, while class-conditional noise correction (*[Learning with Noisy Labels, Sections 2 and 3](https://papers.nips.cc/paper_files/paper/2013/file/3871bd64012152bfb53fdf04b401193f-Paper.pdf)*) can unbias the mean estimate when $`\alpha, \beta`$ are known constants ($`\alpha + \beta < 1`$), it inflates estimation variance quadratically: + +$$ +(7) \qquad \hat{\theta}_{\text{corr}} = \frac{\bar{V} - \alpha}{1 - \alpha - \beta}, \qquad \operatorname{Var}(\hat{\theta}_{\text{corr}}) = \frac{\operatorname{Var}(\bar{V})}{(1 - \alpha - \beta)^2} +$$ + +For task families whose validity is exactly decidable by a sound and complete certificate verifier, $`\alpha=\beta=0`$ removes this noise-correction variance penalty. The zero observed error rates in the constructed stress test below do not establish zero population error on unrestricted scientific reasoning. + +## 6 Reproducible stress tests for false acceptance and false rejection + +To validate our certificate verifier reproducibly, I constructed **600 strongly convex quadratic task families** and generated **6 algebraic submissions per family (3,600 total submissions)** using **exact rational arithmetic**: +• **1,200 Valid Certificates (2 per family)**: One derived in Lagrangian multiplier form and one in equality-elimination form. +• **2,400 Controlled-Defect Submissions (4 per family)**: (1) An infeasible construction ($`Ax \neq b`$) yielding the exact optimal value $`v^*`$; (2) failed stationarity along a feasible direction in $`\ker A`$ while reporting $`v^*`$; (3) a claimed multiplier that fails the stationarity equation while reporting $`v^*`$; and (4) an incorrect final objective value. Crucially, the first three defect classes (1,800 submissions, or **75% of all invalid submissions**) report the exact right final answer $`v^*`$! + +We benchmarked three deterministic acceptance protocols across all 3,600 submissions: + + + + + + +
Acceptance ProtocolFalse Acceptance Rate on Invalid Certificates ($`2,400`$)False Rejection Rate on Valid Certificates ($`1,200`$)Valid Share of Accepted Submissions
**Final-answer matching only****75.0%** (admits 1,800 flawed solutions)0.0%40.0%
**Strict reference path (ablation)**0.0%**50.0%** (rejects 600 valid elimination proofs)100.0%
**Semantic certificate verification (Ours)****0.0%****0.0%****100.0%**
+ +Within this exact-arithmetic benchmark, semantic certificate verification admitted **1,800 fewer invalid proofs** than final-answer matching and retained **600 more valid alternative solutions** than strict reference-path matching, while passing all 600 feasible-perturbation checks on the projected-residual bound in Equation (4). + +## 7 Account for dependence within task families and the full cost of expert review + +Generating surface variants of a single problem template does not yield independent observations of scientific capability. Across $`F`$ independent task families with $`K`$ variants per family, common marginal variance $`\sigma^2`$, and common pairwise within-family correlation $`\rho`$, the variance of the sample mean and effective sample size $`n_{\text{eff}}`$ are: + +$$ +(8) \qquad \operatorname{Var}(\bar{Z}) = \frac{\sigma^2}{FK}[1 + (K - 1)\rho], \qquad n_{\text{eff}} = \frac{FK}{1 + (K - 1)\rho} +$$ + +If $`F = 50`$ task families each have $`K = 8`$ variants ($`400`$ total responses) with $`\rho = 0.6`$, the $`400`$ responses provide the same precision as just **$`n_{\text{eff}} \approx 77`$ independent observations**—mandating family-level train/test partitioning and new combinations of physical and mathematical principles rather than superficial numerical templating. + +Finally, full production cost accounting determines when automated certificate verification outperforms manual PhD review. Let $`C_0`$ be the upfront engineering cost of building the proof-obligation certificates and verifier, $`c_e`$ the legacy workflow's expert review cost per submission, $`c_v`$ the automated verifier's unit execution cost, and $`u`$ the fraction of submissions escalated to human adjudication. Once false acceptance and false rejection standards are met, the certificate architecture reduces total cost whenever submission volume $`N`$ exceeds: + +$$ +(9) \qquad C_0 + N(c_v + uc_e) < Nc_e \iff N > \frac{C_0}{(1 - u)c_e - c_v}, \qquad (1 - u)c_e > c_v +$$ + +By connecting capability specifications, CoT proof obligations, tool-compliance sandboxes, and exact optimality certificates, the framework developed from this collaboration provides a rigorous blueprint for verifiable reasoning evaluation and post-training supervision. + +# Project 04: evidence-uncertainty-and-decision-guarantees-in-investment-research + +**Article Title** +Evidence uncertainty and decision guarantees in investment research + +**Organization** +Duke University × Leading Global Alternative Asset Manager + +**Title** +AI-Driven Investment Research and Explainable Risk Analysis + +**URL** +https://www.theodoreoy.com/past-experience/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research/ + +**Position** +Researcher + +**Location** +Durham, NC; Washington, DC + +**Dates** +September 2023 – September 2026 + +**Read More Label** +Read More + +## Summary + +Developed in a multi-year collaboration between Duke University and a leading global alternative asset manager overseeing more than $300 billion in assets under management *(partner identity anonymized under NDA)*, this project built an AI-native investment-research and epistemic-verification workflow designed to counter **AI-era corporate greenwashing and adversarial narrative manipulation**. The workflow addresses the risk that AI-polished corporate disclosures and PR narratives can influence automated ESG and investment-research scores without strengthening the underlying evidence; document-level checks alone may miss shared reporting errors and unsupported claims. Rather than building a black-box system that tells deal teams which company to fund, I architected a decision-guarantee engine (`Multi-Source Ingestion → Multi-Model Rashomon Scoring → Support-Function Ranking Stability Certification → Minimax-Regret Verification Routing → Versioned SHAP Propagation`) that measures **how trustworthy an AI-assisted assessment actually is** and identifies **which exact factual claim analysts should verify next**. In reproducible analytical benchmarks, joint support-function modeling certified a **0.6-point pairwise company advantage** completely obscured by independent confidence intervals, while a single minimax-regret query reduced worst-case decision regret from **0.61 score units to zero**. + +## Confidentiality & Scope Disclosure + +Conducted through a university–industry collaboration with a leading global alternative asset manager overseeing more than $300 billion in assets under management as of June 2026. While the partner institution's name is withheld pursuant to non-disclosure agreements, my role, system architecture, anti-greenwashing methodology, and analytical benchmarks are authorized for public disclosure. + +## Key Contributions + +* **Designed Institutional Controls for AI-Era Greenwashing and Narrative Manipulation**: Addressed the risk that polished corporate disclosures and sustainability narratives can influence automated LLM and ESG scores without corresponding factual support. Designed evidence-tracing and verification controls for private-market and ESG research, where report-level review alone may leave shared-source errors and unresolved factual claims undetected. +* **Built an Epistemic Decision-Support Engine Rather Than a Black-Box Deal Picker**: Designed the internal AI workflow around a strict institutional mandate—**never replace the investment committee by outputting a black-box "invest / pass" verdict; instead, quantify how trustworthy the current score is and prescribe which exact factual claim analysts must verify next to minimize misjudgment**. Implemented the complete closed-loop pipeline: `Multi-Source Evidence Ingestion → Multi-Model Rashomon Scoring → Ranking Stability Certification → Minimax-Regret Verification Routing → Automated Post-Correction Re-scoring & Attribution Propagation`. +* **Converted Heterogeneous Disclosures into Traceable Scoring Rules & Contradiction Detectors**: Transformed operating reports, audited financial statements, sustainability disclosures, management questionnaires, and third-party datasets into structured scoring rubrics with strict evidence lineage. Directed AI-assisted verification specifically toward unsubstantiated corporate claims and cross-document contradictions—signals that warrant targeted factual review. +* **Combined Tree Ensembles, Conditional SHAP & Rashomon Model Multiplicity**: Deployed gradient-boosted tree models and tabular deep learning to capture nonlinear thresholds and interaction effects, paired with SHAP attributions and conditional Shapley analyses across Rashomon sets of near-optimal models ($\mathcal{F}_\alpha$) to distinguish stable score contributors from model-selection artifacts. +* **Formulated Joint Model-and-Evidence Uncertainty Bounds via Convex Support Functions**: Derived failure cases for standard bootstrap resampling and raw SHAP-based verification priorities under shared reporting bias or unverified corporate claims. Represented unresolved company evidence $\mathcal{X}(E)$ and Rashomon model sets $\mathcal{F}_\alpha$ jointly ($\Omega(E) = \mathcal{F}_\alpha \times \mathcal{X}(E)$) and derived convex support-function bounds ($h_U(a_f)$) that cancel shared-source measurement errors when the two companies have identical exposures—certifying, in a synthetic comparison, a **0.6-point pairwise company advantage** that separate company confidence intervals ($[-1.4, 2.6]$) failed to resolve. +* **Derived Minimax-Regret Active Verification Routing ($V(q)/c_q$)**: Formulated a distribution-free verification criterion that prioritizes diligence queries by the **worst-case decision regret they remove** rather than by naive SHAP magnitude. In an eight-scenario benchmark, a single targeted verification query reduced worst-case decision regret from **0.61 score units to zero**, whereas rechecking an already-verified high-SHAP feature yielded zero regret reduction. + +## Architectural Context: Countering Adversarial AI Greenwashing in $300B+ Institutional Diligence + +In private-equity and impact-investment diligence, fluent summaries do not establish the truth of corporate disclosures. The workflow treats AI-polished narratives, unsupported sustainability claims, and shared reporting errors as risks to the credibility of automated ESG and investment scores. Existing disclosure and antifraud obligations do not remove the analyst’s need to verify material evidence. Sections I–VI present the mathematical and systems architecture I developed to audit score credibility, account for shared measurement errors, and route human diligence toward claims capable of changing the comparison. + +## Manuscript + +## 1 Where existing methods leave the decision unresolved + +[SHAP](https://arxiv.org/abs/1705.07874)'s local accuracy property decomposes a given model prediction into additive feature contributions, while conditional Shapley methods address feature dependence and Rashomon-set methods examine explanation stability across near-optimal models (*[SHAP](https://arxiv.org/abs/1705.07874); [Aas et al.](https://arxiv.org/html/1903.10464v3); [Marx et al.](https://proceedings.mlr.press/v206/marx23a.html)*). However, **these explainability methods impose zero requirement that the underlying corporate input facts $`x`$ be true**. In institutional investment diligence—where self-reported disclosures may contain AI-polished greenwashing—a further mathematical layer is required to determine *which factual correction to the evidence could flip the current investment decision*. + +For a linear scoring model under a fixed marginal background $`Q`$, the [SHAP](https://arxiv.org/abs/1705.07874) attribution assigned to feature $`j`$ is $`\phi_j = \beta_j(x_j - E_Q X_j)`$. Suppose feature $`j`$ carries an unresolved factual or greenwashing uncertainty bounded by $`|\delta_j| \le d_j`$. Its true sensitivity to factual verification is $`|\beta_j|d_j`$, and over a box of admissible evidence errors, the exact worst-case swing in the company's score is: + +$$ +\sup_{|\delta_j| \le d_j} |f(x + \delta) - f(x)| = \sum_j |\beta_j| d_j +$$ + +The triangle inequality gives the upper bound, which is attained by choosing the endpoint of each error interval so that all perturbations push the score in the same direction. This identity exposes a fundamental flaw in standard XAI-driven diligence: +• A core financial metric with a massive [SHAP](https://arxiv.org/abs/1705.07874) bar $`|\phi_j|`$ may come from audited statutory filings that have already been thoroughly verified ($`d_j \approx 0`$); re-auditing it wastes diligence budget. +• Conversely, a self-reported ESG or operational metric whose current claimed value sits near the sector average ($`x_j \approx E_Q X_j \implies \phi_j \approx 0`$) appears completely unimportant on a [SHAP](https://arxiv.org/abs/1705.07874) chart, yet may harbor substantial unresolved greenwashing error ($`d_j \gg 0`$) capable of reversing the deal ranking! + +Naive bootstrap resampling of existing records fails for a parallel reason: it cannot detect a shared reporting or AI-marketing bias. Suppose $`n`$ reported observations satisfy $`X_i = \mu + b + \varepsilon_i`$, where $`\varepsilon_i \sim \mathcal{N}(0, \sigma^2)`$ are independent random errors and $`b \neq 0`$ is an unmodeled systemic bias shared across records (such as sector-wide self-reporting inflation or coordinated AI-PR sentiment manipulation). A conventional nominal 95% normal confidence interval centered on the sample mean $`\bar{X}`$ covers the true fundamental parameter $`\mu`$ with probability: + +$$ +\Pr\left\{ \mu \in \left[ \bar{X} - z\frac{\sigma}{\sqrt{n}}, \ \bar{X} + z\frac{\sigma}{\sqrt{n}} \right] \right\} = \Phi\left( z - \frac{b\sqrt{n}}{\sigma} \right) - \Phi\left( -z - \frac{b\sqrt{n}}{\sigma} \right) \longrightarrow 0 \quad (\text{as } n \to \infty) +$$ + +where $`z = \Phi^{-1}(0.975) \approx 1.96`$. **Collecting more biased documents ($`n \to \infty`$) shrinks random variance while driving true confidence coverage to zero!** A bootstrap that resamples the same biased disclosures has no statistical leverage to identify $`b`$. Active evidence verification must therefore bound input uncertainty before stability analysis can provide valid decision guarantees. + +## 2 Joint uncertainty over models and evidence + +To evaluate whether an ESG or investment ranking is genuinely trustworthy, our engine represents **model-selection multiplicity** and **unresolved evidence uncertainty** jointly. + +Fix the prediction target, input definitions, and output scale. Consider $`M`$ candidate scoring models $`\mathcal{F}_0`$ specified before examining a validation set of $`n`$ i.i.d. observations from the target evaluation distribution, independent of model construction, with loss bounded in $`[0, 1]`$. For failure probability $`\alpha \in (0, 1)`$, Hoeffding's inequality and a union bound guarantee: + +$$ +\Pr\left\{ \max_{f \in \mathcal{F}_0} |\hat{L}_n(f) - L(f)| \le t_n \right\} \ge 1 - \alpha, \qquad t_n = \sqrt{\frac{\log(2M / \alpha)}{2n}} +$$ + +Let $`f^*`$ minimize true population loss within $`\mathcal{F}_0`$, and retain the empirical **Rashomon set** of all near-optimal models within $`2t_n`$ of the empirical validation minimum: + +$$ +\mathcal{F}_\alpha = \left\{ f \in \mathcal{F}_0 : \hat{L}_n(f) \le \min_{g \in \mathcal{F}_0} \hat{L}_n(g) + 2t_n \right\} +$$ + +On the simultaneous concentration event, $`\hat{L}_n(f^*) \le L(f^*) + t_n \le \min_g \hat{L}_n(g) + 2t_n`$, guaranteeing $`f^* \in \mathcal{F}_\alpha`$ while every retained model has population excess loss at most $`4t_n`$ (*[Marx et al.](https://proceedings.mlr.press/v206/marx23a.html)*). + +Simultaneously, the **evidence uncertainty set** $`\mathcal{X}(E)`$ encodes all unresolved questions regarding entity boundaries, reporting periods, physical units, measurement definitions, shared third-party data sources, and suspected greenwashing claims under current diligence state $`E`$. Crucially, each element $`X \in \mathcal{X}(E)`$ specifies a *joint* input configuration across all candidate companies—preserving cross-company error dependencies rather than treating each company field as an isolated error bar. Defining the joint uncertainty space $`\Omega(E) = \mathcal{F}_\alpha \times \mathcal{X}(E)`$, we compute the exact worst-case pairwise comparison bounds for Companies $`i`$ and $`k`$: + +$$ +\underline{\Delta}_{ik} = \inf_{(f, X) \in \Omega(E)} [f(x_i) - f(x_k)], \qquad \overline{\Delta}_{ik} = \sup_{(f, X) \in \Omega(E)} [f(x_i) - f(x_k)] +$$ + +Whenever $`\underline{\Delta}_{ik} > 0`$, the strict ordering $`i \succ k`$ is certified across every defensible model in $`\mathcal{F}_\alpha`$ and every admissible factual resolution in $`\mathcal{X}(E)`$. If calibration establishes that $`\mathcal{X}(E)`$ contains the true ground-truth inputs with probability at least $`1 - \beta`$ at the specified certification stage, a union bound guarantees joint inclusion of $`f^*`$ and the true inputs with probability at least $`1 - \alpha - \beta`$. Thus the unconditional probability of issuing a ranking certificate that is false for $`f^*`$ evaluated at the true inputs is at most $`\alpha + \beta`$, without requiring independence between model validation and evidence calibration. This guarantee concerns the best model in the prespecified class; it does not certify model-class adequacy or realized investment outcomes. Repeated adaptive certification requires calibration valid across the stages being reported. + +## 3 Computing comparison bounds with support functions + +Our joint evidence representation also makes worst-case comparison bounds computationally tractable via **convex support functions** (*[Ben-Tal and Nemirovski](https://www2.isye.gatech.edu/~nemirovs/MOR_RCO_1998.pdf), Robust Optimization*). Let $`u \in U`$ collect shared and company-specific unresolved factual quantities (including both shared benchmark errors and company-specific greenwashing tolerances), parameterizing company inputs as $`x_i(u) = \bar{x}_i + B_i u`$. Under affine scoring models, the pairwise score difference between Company $`i`$ and Company $`k`$ takes the form $`\Delta_f(u) = m_f + a_f^\top u`$, where $`a_f = (B_i - B_k)^\top \beta_f`$. Introducing the support function $`h_U(a) = \sup_{u \in U} a^\top u`$ yields exact closed-form comparison bounds: + +$$ +\inf_{u \in U} \Delta_f(u) = m_f - h_U(-a_f), \qquad \sup_{u \in U} \Delta_f(u) = m_f + h_U(a_f) +$$ + +For an ellipsoidal uncertainty set $`U = \{Cz : \|z\|_2 \le r\}`$, the Cauchy-Schwarz inequality yields $`h_U(a_f) = r\|C^\top a_f\|_2`$; when $`U`$ is an $`\ell_\infty`$ box centered at zero, $`h_U(a_f)`$ reduces to a weighted sum of absolute values. + +This formulation delivers a major operational advantage over naive company-by-company confidence intervals: **Exact Cancellation of Shared-Source Errors**. Let $`S`$ index shared-source coordinates on which the two companies have identical loadings, so $`(B_i-B_k)_{:,S}=0`$. Under the same linear scoring model: + +$$ +(a_f)_S=0,\qquad \Delta_f(u)=m_f+(a_f)_{S^c}^{\top}u_{S^c} +$$ + +The shared-source contribution cancels identically in the pairwise comparison; company-specific uncertainty remains in the support-function bound. If all uncertain directions cancel, then $`a_f=0`$ and $`h_U(\pm a_f)=0`$. Constructing separate uncertainty intervals for Company $`i`$ and Company $`k`$ and subtracting them falsely allows the shared source to vary in opposite directions simultaneously, manufacturing a spurious worst case that paralyzes investment decisions. + +Set $`m_f=\Delta_f(0)`$ and $`a_f=\nabla\Delta_f(0)`$. + +For smooth nonlinear models where $`\Delta_f(u)`$ is twice continuously differentiable on $`\|u\|_2 \le r`$ with Hessian operator norm bounded by $`K_f`$, Taylor's remainder theorem gives the certified lower bound: + +$$ +|\Delta_f(u) - m_f - a_f^\top u| \le \kappa_f, \qquad \kappa_f = \frac{1}{2}K_f r^2 \implies \underline{\Delta}_f \ge m_f - r\|a_f\|_2 - \kappa_f +$$ + +while non-smooth tree ensembles are bounded via leaf-region partitioning or mixed-integer linear programming, concentrating compute strictly on company pairs near the zero decision boundary. + +## 4 Choosing verification queries by the regret they can remove + +When $`\underline{\Delta}_{ik} \le 0 < \overline{\Delta}_{ik}`$, the ranking between candidate investments remains unresolved—indicating that suspected greenwashing or data ambiguity could flip the decision. Rather than telling the investor which company to pick, our system recommends **which specific verification query $`q`$ (e.g., auditing a supplier contract, requesting raw utility bills, or cross-checking a regulatory filing) to execute next**. + +Let $`A`$ be the finite set of institutional actions (such as advancing Company $`i`$ to final diligence, advancing Company $`k`$, or deferring rating) and $`U(a, \omega)`$ be the pre-specified decision utility under scenario $`\omega \in \Omega`$, with nonempty $`\Omega`$ and uniformly bounded utilities. The baseline **minimax decision regret** over deterministic actions is: + +$$ +R^*(\Omega) = \min_{a \in A} \sup_{\omega \in \Omega} \left[ \max_{b \in A} U(b, \omega) - U(a, \omega) \right] +$$ + +Executing a verification query $`q`$ produces an observation $`o \in O_q`$ that restricts the compatible scenario set to a nonempty subset $`\Omega_{q,o} \subseteq \Omega`$. The **guaranteed worst-case reduction in decision regret** provided by query $`q`$ is: + +$$ +V(q) = R^*(\Omega) - \sup_{o \in O_q} R^*(\Omega_{q,o}) \ge 0 +$$ + +The non-negativity of $`V(q)`$ follows immediately from set inclusion $`\Omega_{q,o} \subseteq \Omega`$: shrinking the set of plausible greenwashing/evidence scenarios cannot increase worst-case regret for any fixed action, and minimizing over actions $`a \in A`$ preserves the inequality (*[Robust Active Preference Elicitation](https://arxiv.org/html/2003.01899)*). Crucially, **$`V(q)`$ requires no subjective probability prior over whether a corporate disclosure is exaggerated**. At a fixed evidence state, maximizing $`V(q)`$ among affordable queries minimizes worst-case regret after one query. With positive unequal query costs $`c_q`$, $`V(q)/c_q`$ is a one-step efficiency heuristic; complementary or sequential queries require joint or adaptive optimization under the total diligence budget $`B`$. + +## 5 Checking that corrections propagate through attribution + +Once diligence resolves a query and updates a company's verified evidence vector from $`x`$ to $`x'`$, our pipeline guarantees that the factual correction propagates deterministically through all downstream scores, peer rankings, and [SHAP](https://arxiv.org/abs/1705.07874) explanations. Holding the scoring function $`f`$, background reference $`Q`$, and rubric version fixed, **Shapley efficiency** enforces the exact conservation identity (*[SHAP local accuracy property](https://arxiv.org/pdf/1705.07874)*): + +$$ +f(x) = \phi_0 + \sum_j \phi_j(f, Q, x), \qquad \sum_j [\phi_j(f, Q, x') - \phi_j(f, Q, x)] = f(x') - f(x) +$$ + +If approximate Shapley outputs have efficiency residuals $`|\phi_0+\sum_j\phi_j-f(x)|\le\eta`$ and $`|\phi_0+\sum_j\phi'_j-f(x')|\le\eta'`$ under the same baseline $`\phi_0`$, the triangle inequality gives $`|\sum_j \Delta \phi_j - \Delta f| \le \eta + \eta'`$ for the recomputation gate. This checks aggregate conservation, not the accuracy of each attribution or the truth of the input evidence. Every database update logs explicit version identifiers for `(Field Version, Evidence Document ID, Model Version, Background Q, Rubric Version)` so that changes driven by factual diligence corrections are never conflated with model retraining. + +## 6 Reproducible quantitative comparisons + +Three reproducible analytical experiments verify the mathematical mechanisms derived above: + +1. **Collapse of Conventional Confidence Intervals Under Shared Greenwashing / Reporting Bias**: +• Set shared reporting bias $`b = 0.2`$ and independent noise $`\sigma = 1`$. Ignoring the common bias $`b`$, a nominal 95% confidence interval achieves actual coverage of only **48.40% at $`n = 100`$ records**, collapsing to **0.000637% at $`n = 1,000`$ records**. When targeted verification establishes the bias bound $`|b| \le 0.2`$, expanding the interval half-width to $`0.2 + 1.96/\sqrt{n}`$ restores **97.50% actual coverage**. +2. **Shared-Source Error Cancellation in Pairwise Company Ranking**: +• Two target companies have a nominal score difference of $`0.6`$ points and share an additive sector-benchmark error in $`[-1, 1]`$. Subtracting independently constructed company intervals gives **$`[-1.4, 2.6]`$**, failing to certify an ordering. Our joint support-function model cancels the incompatible error combinations identically, yielding **$`[0.6, 0.6]`$** and certifying the ranking. +3. **Minimax-Regret Verification Routing vs. Naive [SHAP](https://arxiv.org/abs/1705.07874) Attribution Ranking**: +• Consider one verification query of equal cost across 8 scenarios formed from 2 Rashomon-set candidate models ($`s \in \{-1, 1\}`$) and 4 unresolved evidence configurations ($`z_1 \in \{-0.4, 0.4\}`$, $`z_2 \in \{-0.2, 0.2\}`$), with pairwise score difference $`\Delta_s = 0.15s + z_1 + 0.3s z_2`$. A further shared feature has a large [SHAP](https://arxiv.org/abs/1705.07874) attribution but has already been verified ($`d_0 = 0`$). The decision action is to select Company $`i`$ or Company $`k`$, with regret measured in model-score units: + + + + + + +
Verification QueryRemaining Regret After Worst Possible ObservationGuaranteed Reduction in Regret $`V(q)`$
**Recheck the already-resolved field with a large [SHAP](https://arxiv.org/abs/1705.07874) attribution**0.61**0**
**Verify the smaller unresolved variable $`z_2`$**0.61**0**
**Verify $`z_1`$, which determines the sign of the score difference****0****0.61**
+ +Enumerating all eight scenarios proves that ranking diligence work by raw [SHAP](https://arxiv.org/abs/1705.07874) magnitude wastes 100% of the query budget ($`V(q) = 0`$), whereas our minimax-regret criterion isolates $`z_1`$ and reduces worst-case decision regret from **0.61 to zero** in a single query. By uniting **Rashomon model multiplicity**, **support-function evidence bounds**, and **minimax-regret verification routing**, this architecture provides institutional investors with a rigorous, auditable defense against AI-era greenwashing. diff --git a/website/content/sources/ai-portfolio/citation-links.json b/website/content/sources/ai-portfolio/citation-links.json deleted file mode 100644 index a49a149..0000000 --- a/website/content/sources/ai-portfolio/citation-links.json +++ /dev/null @@ -1,36 +0,0 @@ -{ - "statistical-inference-and-resource-allocation-in-expert-data-production": [ - { "text": "Active Statistical Inference: mean estimation and sampling design", "href": "https://arxiv.org/html/2403.03208v3" }, - { "text": "Robust Sampling for Active Statistical Inference", "href": "https://arxiv.org/html/2511.08991v1" }, - { "text": "DeCCaF: capacity constraints and global assignment", "href": "https://arxiv.org/html/2403.06906v1" }, - { "text": "Dawid-Skene", "href": "https://doi.org/10.2307/2346806" }, - { "text": "DeCCaF", "href": "https://arxiv.org/html/2403.06906v1" } - ], - "task-validity-in-financial-synthetic-data": [ - { "text": "Majumdar and Majumdar on Gaussian distributions under linear conditioning", "href": "https://arxiv.org/abs/1710.09285" }, - { "text": "Neural SDEs as Infinite-Dimensional GANs", "href": "https://proceedings.mlr.press/v139/kidger21b.html" }, - { "text": "SDV constraint-augmented generation", "href": "https://docs.sdv.dev/sdv/modeling/constraint-augmented-generation-cag" }, - { "text": "Alaa et al. on synthetic data evaluation", "href": "https://proceedings.mlr.press/v162/alaa22a.html" }, - { "text": "Pitman on the Brownian bridge", "href": "https://www.stat.berkeley.edu/~pitman/s205s03/b_bridge.pdf" }, - { "text": "HelpSteer2-Preference", "href": "https://arxiv.org/abs/2410.01257" }, - { "text": "TabDDPM", "href": "https://proceedings.mlr.press/v202/kotelnikov23a.html" }, - { "text": "CTGAN", "href": "https://arxiv.org/abs/1907.00503" }, - { "text": "FinQA", "href": "https://aclanthology.org/2021.emnlp-main.300/" } - ], - "verification-and-supervision-in-scientific-reasoning-tasks": [ - { "text": "Boyd and Vandenberghe, Convex Optimization, Sections 5.5 and 9.1", "href": "https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf" }, - { "text": "Let's Verify Step by Step, Sections 2 and 3", "href": "https://arxiv.org/html/2305.20050v1" }, - { "text": "Learning with Noisy Labels, Sections 2 and 3", "href": "https://papers.nips.cc/paper_files/paper/2013/file/3871bd64012152bfb53fdf04b401193f-Paper.pdf" }, - { "text": "ProcessBench, Section 3.3", "href": "https://arxiv.org/html/2412.06559v2" }, - { "text": "ProcessBench, Section 3.1", "href": "https://arxiv.org/html/2412.06559v2" }, - { "text": "GSM-Symbolic, Section 3.1", "href": "https://arxiv.org/html/2410.05229v1" } - ], - "evidence-uncertainty-and-decision-guarantees-in-investment-research": [ - { "text": "Robust Active Preference Elicitation", "href": "https://arxiv.org/html/2003.01899" }, - { "text": "SHAP local accuracy property", "href": "https://arxiv.org/pdf/1705.07874" }, - { "text": "Ben-Tal and Nemirovski", "href": "https://www2.isye.gatech.edu/~nemirovs/MOR_RCO_1998.pdf" }, - { "text": "Marx et al.", "href": "https://proceedings.mlr.press/v206/marx23a.html" }, - { "text": "Aas et al.", "href": "https://arxiv.org/html/1903.10464v3" }, - { "text": "SHAP", "href": "https://arxiv.org/abs/1705.07874" } - ] -} diff --git a/website/content/sources/ai-portfolio/editorial-revisions.json b/website/content/sources/ai-portfolio/editorial-revisions.json deleted file mode 100644 index 19717bd..0000000 --- a/website/content/sources/ai-portfolio/editorial-revisions.json +++ /dev/null @@ -1,690 +0,0 @@ -{ - "schemaVersion": 1, - "baseCommit": "fc8717e3a1da3861a0e685ea202074e629ec187d", - "sourceFile": "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "sourceSha256": "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283", - "scope": "Reviewed mathematical, historical, and evidence-scope corrections to the four AI entries and article texts. The original manuscript remains frozen.", - "revisions": [ - { - "id": "P1-01.1", - "file": "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json", - "jsonPointer": "/markdown", - "before": "where $`\\lambda > 0`$ is uniquely determined by the budget constraint, and feasibility requires", - "after": "for budgets below the cost of a full audit, $`\\lambda > 0`$ is chosen to satisfy the budget constraint (the multiplier need not be unique when clipping is active), and feasibility requires", - "reason": "Strict convexity gives a unique optimal probability vector, not necessarily a unique dual multiplier after clipping.", - "sourceRefs": [ - "Project 1: printed equations and parameterized examples" - ], - "evidence": [ - "Counterexample: N=2, c=(1,1), m=(1,0.0001), pi_min=0.1, B=1.1. Every lambda in [0.01,1] gives the same optimum pi=(1,0.1)." - ] - }, - { - "id": "P1-02.1", - "file": "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json", - "jsonPointer": "/markdown", - "before": "(Note that $`B`$ constrains *expected* audit cost per batch; workflows requiring an exact integer count of audits in every batch substitute a fixed-size pivotal or stratified sampling design with identical first-order inclusion probabilities $`\\pi_i^*`$.)", - "after": "(Note that $`B`$ constrains *expected* audit cost per batch. Workflows requiring an exact integer count must use a feasible fixed-size sampling design: its first-order inclusion probabilities sum to that integer, and its variance depends on joint inclusion probabilities. These probabilities need not equal the Bernoulli optimum $`\\pi_i^*`$.)", - "reason": "An arbitrary Bernoulli optimum cannot retain its inclusion probabilities under a fixed integer sample size. Even when feasible, dependent inclusion indicators change the variance.", - "sourceRefs": [ - "Project 1: printed equations and parameterized examples" - ], - "evidence": [ - "For any fixed-size design, sum_i pi_i=E[sum_i S_i]=n is an integer. pi=(0.3,0.3) is impossible.", - "With N=2, pi=(0.5,0.5), fixed sample size 1 and residuals (1,1), the residual-corrected estimate has zero design variance; the displayed independent-Bernoulli variance is 0.5." - ] - }, - { - "id": "P1-03.1", - "file": "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json", - "jsonPointer": "/markdown", - "before": "constrained optimization still outperforms feasible uniform policies", - "after": "constrained optimization is still no worse than feasible uniform policies", - "reason": "A constrained optimum is no worse than a feasible comparator, but the improvement need not be strict.", - "sourceRefs": [ - "Project 1: printed equations and parameterized examples" - ], - "evidence": [ - "At B=pi_min sum_i c_i, uniform sampling is the only feasible allocation even with heterogeneous m_i/c_i; the two policies tie." - ] - }, - { - "id": "P1-04.1", - "file": "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json", - "jsonPointer": "/markdown", - "before": "Here $`H_j^{\\text{review}}`$ is hard capacity allocated to remediation actions while reference audits are scheduled independently on frozen original submissions. Whenever this program is infeasible, it surfaces an explicit, mathematically rigorous trade-off between required delivered quality, required measurement precision, and available engineering budget.", - "after": "Here $`H_j^{\\text{review}}`$ is hard capacity allocated to remediation actions while reference audits are scheduled independently on frozen original submissions. The precision ceiling is model-based: independent audits must check residual calibration, and expected audit cost does not impose a hard limit on realized audit hours. Infeasibility exposes a conflict between the planning precision target, feasible review assignments, and available engineering budget.", - "reason": "The constraint is evaluated with estimated residual moments, not true residuals. Audit costs are expectations and hard review capacities do not bound realized audit hours.", - "sourceRefs": [ - "Project 1: printed equations and parameterized examples", - "https://arxiv.org/html/2403.03208v3, Section 2: the label budget is an expectation." - ], - "evidence": [ - "The article defines V_hat_m, not V_m. Underestimated residual moments can satisfy the planning constraint while violating actual target variance." - ] - }, - { - "id": "P1-05.1", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/summary", - "before": "at equal expected audit cost while cutting post-review expected error loss by **8.89%** at equal specialist hours.", - "after": "at equal expected audit cost. A separate 200-item analytical example reduced post-review expected error loss by **8.89%** at equal review volume.", - "reason": "The 17.79% result uses 10,000 submissions and repeated sampling; the 8.89% result is a separate exact 200-item example. The example fixes review volume/cost, not measured specialist hours.", - "sourceRefs": [ - "Project 1: printed equations and parameterized examples" - ], - "evidence": [ - "Project 1 Section 6: two groups of 100 items and capacity 100; (45-41)/45=8.8888889%." - ] - }, - { - "id": "P1-05.2", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/keyContributions/6", - "before": "relative to uniform sampling, while marginal-gain review ranking reduced expected post-review defect loss by **8.89%** at equal review volume.", - "after": "relative to uniform sampling. In a separate 200-item analytical example, marginal-gain review ranking reduced expected post-review defect loss by **8.89%** at equal review volume.", - "reason": "The 17.79% result uses 10,000 submissions and repeated sampling; the 8.89% result is a separate exact 200-item example. The example fixes review volume/cost, not measured specialist hours.", - "sourceRefs": [ - "Project 1: printed equations and parameterized examples" - ], - "evidence": [ - "Project 1 Section 6: two groups of 100 items and capacity 100; (45-41)/45=8.8888889%." - ] - }, - { - "id": "P2-01.1", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "retaining the original reference decision without recomputation is guaranteed valid if and only if the perturbation", - "after": "a sufficient condition for retaining the original reference decision without recomputation is that the perturbation", - "reason": "The displayed inequality proves only the implication shown. A label can remain unchanged even when the perturbation exceeds the sufficient radius.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples" - ], - "evidence": [ - "Counterexample: d(t)=1+t, x=0, L=1, Delta=2. Both labels are positive, while |d(x)|=1 is not greater than L|Delta|=2." - ] - }, - { - "id": "P2-02.1", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/1/summary", - "before": "Lipschitz decision-margin ($\\gamma/s$) label recomputation", - "after": "Lipschitz margin checks and Gaussian margin-to-noise ($\\gamma/s$) analysis for label recomputation", - "reason": "The sufficient deterministic radius is |d(x)|/L in input units. gamma/s is a dimensionless signal-to-noise ratio governing Phi(-gamma/s), not a variable threshold that a perturbed field crosses.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples" - ], - "evidence": [ - "Article Section 3 already gives the two correct and different formulas: |d(x)|>L||Delta|| and Pr(label flips)=Phi(-gamma/s)." - ] - }, - { - "id": "P2-02.2", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "Update reference judgments when decisive variables cross $`\\gamma/s`$", - "after": "Recompute reference judgments when the sufficient Lipschitz margin check fails; use $`\\gamma/s`$ to quantify Gaussian label-flip risk", - "reason": "The sufficient deterministic radius is |d(x)|/L in input units. gamma/s is a dimensionless signal-to-noise ratio governing Phi(-gamma/s), not a variable threshold that a perturbed field crosses.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples" - ], - "evidence": [ - "Article Section 3 already gives the two correct and different formulas: |d(x)|>L||Delta|| and Pr(label flips)=Phi(-gamma/s)." - ] - }, - { - "id": "P2-03.1", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "In the linear Gaussian (or latent Gaussian copula) regime, manifold projection also preserves the exact theoretical conditional distribution.", - "after": "In the linear Gaussian regime, covariance-weighted affine projection also preserves the exact theoretical conditional distribution. For a Gaussian copula, this result applies only when the constraints are affine in its latent Gaussian coordinates; nonlinear marginal transformations do not generally preserve affine constraints in the observed financial variables.", - "reason": "The exact projection theorem requires Gaussian coordinates and affine constraints in those same coordinates. A nonlinear marginal transform generally turns an affine constraint in observed features into a nonlinear latent constraint.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples", - "https://arxiv.org/pdf/1710.09285, introduction and Theorem 4: exact Gaussian conditional law is derived for linear conditioning; nonlinear transformations require separate treatment." - ], - "evidence": [ - "For Y_j=exp(Z_j), the observed constraint Y_1+Y_2=1 is exp(Z_1)+exp(Z_2)=1, not a linear constraint on Z." - ] - }, - { - "id": "P2-04.1", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "In our production verification benchmark,", - "after": "In our synthetic verification benchmark,", - "reason": "The displayed algebra gives exact equality in exact arithmetic. The reported floating-point residual is nonzero. The 395,442.59 rejection count comes from a separate independent-standard-normal-residual toy model, not necessarily the Gaussian covariance benchmark.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples" - ], - "evidence": [ - "Project 2 Section 2 provides the independent standardized-residual assumptions for the rejection example.", - "Reported maximum |Ax-b|=1.21e-14 is not zero. No numerical acceptance tolerance or common distribution for the two experiments is supplied." - ] - }, - { - "id": "P2-04.2", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "—achieving 100% first-draw feasibility rather than rejecting nearly 400,000 candidates per valid record.", - "after": "—using direct constrained draws without post-hoc rejection. The nearly 400,000 candidate count above belongs to the separate standardized-residual rejection model.", - "reason": "The displayed algebra gives exact equality in exact arithmetic. The reported floating-point residual is nonzero. The 395,442.59 rejection count comes from a separate independent-standard-normal-residual toy model, not necessarily the Gaussian covariance benchmark.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples" - ], - "evidence": [ - "Project 2 Section 2 provides the independent standardized-residual assumptions for the rejection example.", - "Reported maximum |Ax-b|=1.21e-14 is not zero. No numerical acceptance tolerance or common distribution for the two experiments is supplied." - ] - }, - { - "id": "P2-05.1", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/1/keyContributions/4", - "before": "cause up to **40.13%** inherited-label error without recomputation", - "after": "cause **40.13%** inherited-label error at $\\gamma/s = 0.25$ without recomputation", - "reason": "For positive gamma/s, the Gaussian label-flip rate approaches 50% as gamma/s tends to zero. The 40.13% value corresponds specifically to gamma/s=0.25.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples" - ], - "evidence": [ - "Phi(-0.25)=0.4012936743; lim_(r downarrow 0) Phi(-r)=0.5." - ] - }, - { - "id": "P2-06.1", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "an honest predictor acting at time", - "after": "the optimal squared-error predictor acting at time", - "reason": "An arbitrary predictor using only the allowed history can have greater MSE. Conditional variance equals the MSE of the conditional-mean predictor.", - "sourceRefs": [ - "Project 2: printed equations and parameterized examples", - "https://www.stat.berkeley.edu/~pitman/s205s03/b_bridge.pdf, page 2: Brownian-bridge covariance and its endpoint-conditioned construction." - ], - "evidence": [ - "For squared loss, E[(X_t-g)^2|F_s]=Var(X_t|F_s)+(E[X_t|F_s]-g)^2." - ] - }, - { - "id": "P3-01", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "• **When valid alternative proofs are rejected ($`\\beta(h) > 0`$)**: $`\\tilde{p}(h) < p(h)`$, shifting the gradient to penalize valid mathematical diversity and force narrow template memorization.\n• **When flawed proofs or non-compliant tool calls pass because the final number is right ($`\\alpha(h) > 0`$)**: $`\\tilde{p}(h) > p(h)`$, training the model to omit boundary checks and violate tool-compliance sandboxes.", - "after": "• **When false rejection dominates ($`\\beta(h)p(h) > \\alpha(h)[1-p(h)]`$)**: $`\\tilde{p}(h) < p(h)`$, shifting the learned validity target downward and penalizing valid alternative proofs.\n• **When false acceptance dominates ($`\\alpha(h)[1-p(h)] > \\beta(h)p(h)`$)**: $`\\tilde{p}(h) > p(h)`$, shifting the learned validity target upward and rewarding flawed proofs or non-compliant tool calls. The two effects cancel in the conditional mean when their weighted contributions are equal.", - "reason": "Nonzero beta or alpha alone does not determine the sign; subtract p from Equation (6). Counterexamples: p=.5, alpha=.2, beta=.1 gives ptilde=.55; p=.9, alpha=.1, beta=.2 gives ptilde=.73.", - "sourceRefs": [ - "Project 3: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P3-02", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "• Conversely, reproducing the correct optimal value $`v^*`$ via brute-force guessing, an infeasible point ($`Ax \\neq b`$), or an unsupported dual multiplier fails the certificate check immediately.", - "after": "• Conversely, merely reporting the correct optimal value $`v^*`$ does not satisfy the certificate obligations: feasibility and the claimed objective value must be checked independently. If a submission asserts a KKT multiplier witness, the full stationarity equation $`Hx+c+A^\\top\\nu=0`$ must also be verified. The projected-residual bound alone cannot validate the claimed multiplier, because $`P_{\\ker A}A^\\top\\nu=0`$ for every $`\\nu`$; a valid equality-elimination proof need not submit such a witness.", - "reason": "Projected residual is independent of nu. With H=I2, A=(1,0), b=0, c=0, x=0, nu=1, the projected residual is zero although the claimed full KKT witness is false.", - "sourceRefs": [ - "Project 3: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P3-03", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "Eliminating $`\\alpha`$ and $`\\beta`$ inside data production via semantic proof-obligation certificates avoids this variance explosion entirely.", - "after": "For task families whose validity is exactly decidable by a sound and complete certificate verifier, $`\\alpha=\\beta=0`$ removes this noise-correction variance penalty. The zero observed error rates in the constructed stress test below do not establish zero population error on unrestricted scientific reasoning.", - "reason": "Soundness alone guarantees no false acceptance, not completeness; finite constructed tests do not certify universal zero errors.", - "sourceRefs": [ - "Project 3: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P3-04", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "Across $`F`$ independent task families with $`K`$ variants per family and within-family correlation $`\\rho`$,", - "after": "Across $`F`$ independent task families with $`K`$ variants per family, common marginal variance $`\\sigma^2`$, and common pairwise within-family correlation $`\\rho`$,", - "reason": "The variance formula uses common variances and a common within-family pair covariance.", - "sourceRefs": [ - "Project 3: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P3-05", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "Retaining the raw tolerance $`\\delta`$ misclassifies valid transformed solutions as model failures,", - "after": "Retaining the raw tolerance $`\\delta`$ can misclassify valid transformed solutions as model failures,", - "reason": "Unconditional claim is false for identity transformations and for exact contractions a<=1.", - "sourceRefs": [ - "Project 3: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-01", - "file": "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "jsonPointer": "/markdown", - "before": "specified before examining an independent validation set of size $`n`$ with loss bounded in $`[0, 1]`$.", - "after": "specified before examining a validation set of $`n`$ i.i.d. observations from the target evaluation distribution, independent of model construction, with loss bounded in $`[0, 1]`$.", - "reason": "An independent holdout set does not necessarily mean observations inside the set are independent; the displayed Hoeffding rate requires this sampling condition.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-02", - "file": "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "jsonPointer": "/markdown", - "before": "If calibration establishes that $`\\mathcal{X}(E)`$ contains the true ground-truth inputs with probability at least $`1 - \\beta`$, a union bound guarantees joint inclusion of $`f^*`$ and the true inputs with probability at least $`1 - \\alpha - \\beta`$—bounding the unconditional false-certificate rate by $`\\alpha + \\beta`$ without assuming independence between model validation and evidence calibration.", - "after": "If calibration establishes that $`\\mathcal{X}(E)`$ contains the true ground-truth inputs with probability at least $`1 - \\beta`$ at the specified certification stage, a union bound guarantees joint inclusion of $`f^*`$ and the true inputs with probability at least $`1 - \\alpha - \\beta`$. Thus the unconditional probability of issuing a ranking certificate that is false for $`f^*`$ evaluated at the true inputs is at most $`\\alpha + \\beta`$, without requiring independence between model validation and evidence calibration. This guarantee concerns the best model in the prespecified class; it does not certify model-class adequacy or realized investment outcomes. Repeated adaptive certification requires calibration valid across the stages being reported.", - "reason": "The probability is a joint-event error bound relative to f*, not conditional error among issued certificates or real-world investment truth. Repeated adaptive stages need simultaneous or sequentially valid coverage.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-03", - "file": "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "jsonPointer": "/markdown", - "before": "This formulation delivers a major operational advantage over naive company-by-company confidence intervals: **Exact Cancellation of Shared-Source Errors**. If two companies evaluated by the same linear model share an identical additive error from a common industry benchmark or third-party sensor feed, then $`B_i = B_k`$ along those coordinates and:\n\n$$\na_f = (B_i - B_k)^\\top \\beta_f = 0 \\implies h_U(\\pm a_f) = 0\n$$\n\nThe common error cancels identically in the pairwise comparison!", - "after": "This formulation delivers a major operational advantage over naive company-by-company confidence intervals: **Exact Cancellation of Shared-Source Errors**. Let $`S`$ index shared-source coordinates on which the two companies have identical loadings, so $`(B_i-B_k)_{:,S}=0`$. Under the same linear scoring model:\n\n$$\n(a_f)_S=0,\\qquad \\Delta_f(u)=m_f+(a_f)_{S^c}^{\\top}u_{S^c}\n$$\n\nThe shared-source contribution cancels identically in the pairwise comparison; company-specific uncertainty remains in the support-function bound. If all uncertain directions cancel, then $`a_f=0`$ and $`h_U(\\pm a_f)=0`$.", - "reason": "Equality only on shared columns kills only the corresponding components of a_f, not all company-specific components. Counterexample loadings Bi=(1,1), Bk=(1,0), beta=1 leaves a=(0,1).", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-04", - "file": "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "jsonPointer": "/markdown", - "before": "and $`U(a, \\omega)`$ be the pre-specified decision utility under scenario $`\\omega \\in \\Omega`$.", - "after": "and $`U(a, \\omega)`$ be the pre-specified decision utility under scenario $`\\omega \\in \\Omega`$, with nonempty $`\\Omega`$ and uniformly bounded utilities.", - "reason": "Ensures finite regret, avoiding undefined infinity-minus-infinity in V(q).", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-05", - "file": "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "jsonPointer": "/markdown", - "before": "Under a finite diligence budget $`B`$, ranking candidate verification queries by cost-adjusted regret reduction $`V(q)/c_q`$ (or solving a small batch discrete optimization over complementary queries) directs human diligence toward shared sources and decisive claims that resolve multiple company comparisons at once.", - "after": "At a fixed evidence state, maximizing $`V(q)`$ among affordable queries minimizes worst-case regret after one query. With positive unequal query costs $`c_q`$, $`V(q)/c_q`$ is a one-step efficiency heuristic; complementary or sequential queries require joint or adaptive optimization under the total diligence budget $`B`$.", - "reason": "Nonnegativity does not prove optimality of cost-ratio ordering for a finite multi-query budget; complementarity can make greedy ratios arbitrarily inadequate.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-06", - "file": "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "jsonPointer": "/markdown", - "before": "If approximate Shapley algorithms carry numerical summation tolerances bounded by $`\\eta`$ and $`\\eta'`$ before and after correction, our automated recomputation gate verifies $`|\\sum_j \\Delta \\phi_j - \\Delta f| \\le \\eta + \\eta'`$.", - "after": "If approximate Shapley outputs have efficiency residuals $`|\\phi_0+\\sum_j\\phi_j-f(x)|\\le\\eta`$ and $`|\\phi_0+\\sum_j\\phi'_j-f(x')|\\le\\eta'`$ under the same baseline $`\\phi_0`$, the triangle inequality gives $`|\\sum_j \\Delta \\phi_j - \\Delta f| \\le \\eta + \\eta'`$ for the recomputation gate. This checks aggregate conservation, not the accuracy of each attribution or the truth of the input evidence.", - "reason": "Floating-point summation error alone does not control approximation error in attribution efficiency; define actual aggregate efficiency residuals.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-07", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/keyContributions/3", - "before": "to distinguish genuine fundamental drivers from model-selection artifacts.", - "after": "to distinguish stable score contributors from model-selection artifacts.", - "reason": "SHAP stability is neither causal identification nor verification of the underlying fundamental facts.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-08", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/keyContributions/4", - "before": "Proved that standard bootstrap resampling and raw SHAP feature rankings break down under shared reporting bias or unverified corporate claims.", - "after": "Derived failure cases for standard bootstrap resampling and raw SHAP-based verification priorities under shared reporting bias or unverified corporate claims.", - "reason": "The supplied mathematics proves failure cases, not universal failure of these method classes.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "P4-09", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/keyContributions/4", - "before": "that cancel shared-source measurement errors—certifying a **0.6-point pairwise company advantage**", - "after": "that cancel shared-source measurement errors when the two companies have identical exposures—certifying, in a synthetic comparison, a **0.6-point pairwise company advantage**", - "reason": "Propagates the identical-loading condition and synthetic benchmark scope into the contribution claim.", - "sourceRefs": [ - "Project 4: mathematical definition and derivation cited in the revision reason" - ], - "evidence": [] - }, - { - "id": "N01", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/summary", - "before": "Launched in September 2023 during the earliest stage of large-scale post-training for a flagship multimodal foundation model, this initiative established the foundational statistical governance and resource-allocation paradigm", - "after": "Launched in September 2023 for a flagship multimodal foundation model program, this initiative established the project’s statistical governance and resource-allocation framework", - "reason": "September 2023 is the supplied project start, not the inception of RLHF or large-model post-training.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2203.02155" - ], - "evidence": [] - }, - { - "id": "N02", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/summary", - "before": "I architected the beta mathematical and operational blueprint that defined how frontier model teams govern external expert networks, calibrate evolving acceptance standards, and allocate review capacity at scale.", - "after": "I architected the pilot’s mathematical and operational framework for governing external expert networks, calibrating evolving acceptance standards, and allocating review capacity at scale.", - "reason": "Keep the personal systems contribution while limiting its reach to the project rather than frontier teams generally.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N03", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/keyContributions/0", - "before": "**Pioneered the 0-to-1 Statistical Governance Paradigm for Early Frontier Post-Training (Sept 2023 Beta Launch)**: At the inception of large-scale reinforcement learning from human feedback (RLHF) in September 2023—when frontier model pipelines lacked formal statistical control over distributed human data production—I architected the foundational pilot system governing more than 300 domain contributors across travel, food delivery, consumer goods, and e-commerce advertising. Rather than executing day-to-day vendor supervision, my work defined the mathematical and operational standards for how frontier model teams structure, audit, and scale external expert and outsourcing networks.", - "after": "**Established the Pilot Statistical Governance Framework for Expert Post-Training Data (September 2023 Launch)**: Starting in September 2023, I architected a pilot system governing more than 300 domain contributors across travel, food delivery, consumer goods, and e-commerce advertising. My work defined the project’s mathematical and operational standards for structuring, auditing, and scaling its external expert and outsourcing networks.", - "reason": "Removes the readily refutable inception claim and unverified field-wide absence of statistical control; retains date, authorship, scale, domains, and 0-to-1 pilot responsibility.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2203.02155" - ], - "evidence": [] - }, - { - "id": "N04", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/architecturalContext/title", - "before": "Architectural Context: Why Frontier RLHF Required a Paradigm Shift in Late 2023", - "after": "Architectural Context: Statistical Governance for Expert Data Production", - "reason": "A problem-specific title carries the engineering contribution without asserting an industry-wide historical turning point.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N05", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/architecturalContext/text", - "before": "When frontier foundation models entered massive multi-domain post-training in late 2023, model builders faced an acute structural paradox: scaling RLHF required transitioning from small internal research cohorts to hundreds of distributed domain specialists and external vendor queues, yet naive quality control actively corrupted both training data and management metrics.", - "after": "As the project scaled multi-domain post-training work from September 2023 onward, it faced a statistical problem: coordinating hundreds of distributed domain specialists and external vendor queues while keeping quality measurements interpretable under selective review.", - "reason": "Scope the operational history to the supplied project, rather than claiming that all frontier models entered the phase in late 2023.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2203.02155" - ], - "evidence": [] - }, - { - "id": "N06", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/architecturalContext/text", - "before": "The seven sections below present the complete mathematical and engineering framework I developed to establish a self-calibrating control plane for expert data production.", - "after": "The seven sections below formalize the production design and compare it with subsequent literature. The September 2023 start date describes the collaboration; references published in 2024–2026 provide later methodological context.", - "reason": "DeCCaF and Active Statistical Inference appeared in 2024; Robust Sampling appeared in 2025; the cited Active Statistical Inference v3 is dated April 2026. Later references are legitimate, but their relationship to the start date should be explicit.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2403.06906", - "https://arxiv.org/abs/2403.03208", - "https://arxiv.org/abs/2511.08991" - ], - "evidence": [] - }, - { - "id": "N07", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/0/keyContributions/2", - "before": "while benchmarking Dawid-Skene reviewer-error models against cost-sensitive deferral rules (DeCCaF) to optimize escalation thresholds.", - "after": "and subsequently compared Dawid-Skene reviewer-error models with cost-sensitive deferral rules (DeCCaF) to inform escalation thresholds.", - "reason": "Makes later comparison explicit without asserting that a March 2024 method was used at the September 2023 launch.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2403.06906" - ], - "evidence": [] - }, - { - "id": "N08", - "file": "content/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production.json", - "jsonPointer": "/markdown", - "before": "The contributor reliability records, tiered review workflows, and random reference audits I established in September 2023 provided the operational foundation for this architecture; the mathematical synthesis above ensured that as our frontier model partner scaled external vendor pipelines, quality evidence remained unbiased, cost-optimal, and invariant to routing policy shifts.", - "after": "The contributor reliability records, tiered review workflows, and random reference audits I established from September 2023 provided the operational foundation for this architecture. The mathematical analysis above states the sampling, calibration, and capacity assumptions under which quality estimation is unbiased and audit allocation is optimal when residual second moments are correctly specified; the synthetic benchmarks test those mechanisms under their stated conditions.", - "reason": "Do not convert conditional theorems and synthetic experiments into a demonstrated production guarantee. The source itself notes residual-calibration misspecification.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N09", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/2/summary", - "before": "this work pioneered a three-pillar evaluation and process-supervision paradigm for scientific, mathematical, and quantitative risk reasoning—anticipating the industry's 2026 agentic evaluation standards by three full years.", - "after": "this work developed a three-pillar evaluation and process-supervision framework for scientific, mathematical, and quantitative risk reasoning.", - "reason": "Process supervision and an LLM-agent benchmark were already published before September 2023. No defined 2026 industry standard or priority comparison is supplied.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2305.20050", - "https://arxiv.org/abs/2308.03688" - ], - "evidence": [] - }, - { - "id": "N10", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/2/keyContributions/0", - "before": "**Pioneered a Three-Pillar Agentic & Process Evaluation Paradigm 3 Years Ahead of Industry Standard (Sept 2023)**: During early university collaborations in September 2023 aimed at stress-testing frontier models with Olympiad- and graduate-level STEM tasks, I demonstrated that merely trying to stump models and checking final answers provides minimal actionable feedback for post-training. Three full years before the industry's 2026 Agent Evaluation wave, I established an evaluation paradigm centered on three pillars:", - "after": "**Developed Three-Pillar Process and Tool Evaluation from September 2023**: During a university collaboration aimed at stress-testing frontier models with Olympiad- and graduate-level STEM tasks, I identified the limitations of final-answer checks in our workflow and developed an evaluation framework covering reasoning validity, compliance with task-specific tool policies, and structural mathematical insight.", - "reason": "Retains the project start and contribution while removing an unsupported three-year industry-priority claim.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2305.20050", - "https://arxiv.org/abs/2308.03688" - ], - "evidence": [] - }, - { - "id": "N11", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/2/architecturalContext/text", - "before": "When frontier AI labs partnered with research universities in autumn 2023 to probe the limits of scientific reasoning, the standard playbook was simple: collect difficult STEM problems and check whether the model's final answer matched the answer key. Our experiments quickly revealed why this approach hit a dead end: models frequently arrived at the right number via broken physical assumptions, brute-force computational exhaustion, or unauthorized tool calls (such as invoking a Python VM in a compliance-restricted workflow that mandated Excel). Sections I–VII below present the formal mathematical and engineering architecture I developed in 2023 to turn scientific benchmarking into a rigorous, certificate-backed supervision system.", - "after": "The collaboration began in autumn 2023 with scientific-reasoning tasks for frontier models. Initial experiments in our workflow showed why final-answer matching was insufficient: a correct number could coexist with an invalid physical assumption, an unsupported computational shortcut, or a tool call that violated the task’s specified execution policy. Sections I–VII formalize the resulting acceptance architecture and relate it to subsequent process-evaluation research; the project start date does not imply that every later citation or mathematical formulation was part of the initial workflow.", - "reason": "Avoids a universal claim about the standard playbook at all AI labs and separates the original task work from later formalization.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2305.20050", - "https://arxiv.org/abs/2412.06559", - "https://arxiv.org/abs/2410.05229" - ], - "evidence": [] - }, - { - "id": "N12", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "our 2023 architecture further addressed three critical gaps in scientific post-training", - "after": "the architecture formalized here addresses three practical requirements in scientific post-training", - "reason": "The surrounding section cites 2024/2025 sources. This retains the conceptual contribution without dating the entire current synthesis to 2023.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2412.06559", - "https://arxiv.org/abs/2410.05229" - ], - "evidence": [] - }, - { - "id": "N13", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "this 2023 framework established a rigorous blueprint", - "after": "the framework developed from this collaboration provides a rigorous blueprint", - "reason": "Avoids projecting the entire revised technical analysis back onto the starting year.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N14", - "file": "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", - "jsonPointer": "/markdown", - "before": "In regulated financial and academic settings where spinning up a Python virtual machine violates data-governance rules, trajectories that invoke Python instead of executing within the mandated Excel environment are rejected immediately", - "after": "When a task’s specified data-governance policy permits only the approved Excel environment, trajectories that invoke an unauthorized Python virtual machine are rejected immediately", - "reason": "Excel-versus-Python compliance is a task or organizational policy, not a generally applicable legal rule or intrinsic property of either tool.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N15", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/summary", - "before": "Just as merchants game AI consumer recommendations through coordinated marketing seeding, corporate issuers increasingly deploy AI-polished disclosures and PR narratives to inflate automated ESG and investment-research scores—exploiting a regulatory and enforcement vacuum where traditional report-level checks fail.", - "after": "The workflow addresses the risk that AI-polished corporate disclosures and PR narratives can influence automated ESG and investment-research scores without strengthening the underlying evidence; document-level checks alone may miss shared reporting errors and unsupported claims.", - "reason": "Frame this as the system’s threat model. Public enforcement exists, and the page does not supply a dataset establishing prevalence, intention, or a cross-industry trend.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://www.sec.gov/newsroom/press-releases/2023-194", - "https://www.sec.gov/newsroom/press-releases/2024-36" - ], - "evidence": [] - }, - { - "id": "N16", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/keyContributions/0", - "before": "**Architected an Institutional Defense Against \"AI-Era Greenwashing\" & Adversarial Narrative Manipulation**: Identified and solved a critical emerging vulnerability in private-market and ESG investment research: corporate executives increasingly use generative AI to craft polished marketing disclosures and sustainability narratives that game automated LLM sentiment and ESG scoring models (analogous to restaurants gaming AI consumer recommendation rankings through coordinated review seeding)—operating in a regulatory and legal vacuum where traditional equipment- or report-level greenwashing checks fail.", - "after": "**Designed Institutional Controls for AI-Era Greenwashing and Narrative Manipulation**: Addressed the risk that polished corporate disclosures and sustainability narratives can influence automated LLM and ESG scores without corresponding factual support. Designed evidence-tracing and verification controls for private-market and ESG research, where report-level review alone may leave shared-source errors and unresolved factual claims undetected.", - "reason": "Preserves the concrete engineering problem and controls while removing unsupported prevalence, intent, solved-problem, and regulatory-vacuum assertions.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://www.sec.gov/newsroom/press-releases/2023-194", - "https://www.sec.gov/newsroom/press-releases/2024-36" - ], - "evidence": [] - }, - { - "id": "N17", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/architecturalContext/text", - "before": "When global alternative asset managers evaluate private equity and impact investments, ESG and operational risk diligence cannot rely on naive LLM summarization. Between 2023 and 2026, we observed a structural shift in corporate self-reporting: just as local merchants learned to manipulate AI recommendation engines via coordinated digital marketing, corporate issuers began deploying AI-optimized sustainability narratives to inflate automated ESG scores. Because regulatory legislation and enforcement mechanisms have not yet caught up with AI-driven narrative manipulation, an investment workflow that takes unverified corporate disclosures at face value—or ranks diligence priorities solely by raw SHAP values—exposes the fund to severe greenwashing risk. Sections I–VI below present the mathematical and systems architecture I developed to audit score credibility, cancel shared measurement errors, and route human diligence toward the exact claims that govern the investment decision.", - "after": "In private-equity and impact-investment diligence, fluent summaries do not establish the truth of corporate disclosures. The workflow treats AI-polished narratives, unsupported sustainability claims, and shared reporting errors as risks to the credibility of automated ESG and investment scores. Existing disclosure and antifraud obligations do not remove the analyst’s need to verify material evidence. Sections I–VI present the mathematical and systems architecture I developed to audit score credibility, account for shared measurement errors, and route human diligence toward claims capable of changing the comparison.", - "reason": "The SEC has taken ESG and AI-marketing enforcement actions. They refute a broad vacuum claim but do not establish that every AI-generated narrative tactic is covered or effectively detected.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://www.sec.gov/newsroom/press-releases/2023-194", - "https://www.sec.gov/newsroom/press-releases/2024-36" - ], - "evidence": [] - }, - { - "id": "N18", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/3/keyContributions/2", - "before": "—the primary signatures of AI-amplified greenwashing.", - "after": "—signals that warrant targeted factual review.", - "reason": "Contradictions and unsupported claims are not specific evidence of AI authorship or deliberate greenwashing.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N20", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "Because Batch Zero prevented costly downstream rework across external vendor queues,", - "after": "Following the pilot’s use in preparing external vendor workflows,", - "reason": "The supplied company-wide adoption claim can remain; the causal cost-reduction claim needs separate measured evidence.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "N21", - "file": "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", - "jsonPointer": "/markdown", - "before": "(*[HelpSteer2-Preference](https://arxiv.org/abs/2410.01257)*)", - "after": "(see the later methodological comparison in *[HelpSteer2-Preference](https://arxiv.org/abs/2410.01257)*)", - "reason": "HelpSteer2-Preference first appeared in October 2024. Its citation need not imply that the user’s protocol originated in that paper or existed because of it at launch.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records", - "https://arxiv.org/abs/2410.01257" - ], - "evidence": [] - }, - { - "id": "N22", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/1/architecturalContext/text", - "before": "Easy prompts produce uniform model outputs that yield zero preference signal.", - "after": "Easy prompts may yield less discriminative comparisons between candidate responses.", - "reason": "An easy task can still elicit differences in correctness, evidence use, and style; zero preference signal is an unnecessary universal claim.", - "sourceRefs": [ - "Original portfolio narrative and cited publication/enforcement records" - ], - "evidence": [] - }, - { - "id": "P2-NEST-01", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/1/keyContributionDetails/2/items/1", - "before": "across 200,000 12-dimensional synthetic draws where naive rejection sampling collapses to a $0.0002529\\%$ acceptance rate.", - "after": "across 200,000 12-dimensional synthetic draws. A separate standardized-residual rejection model yields a $0.0002529\\%$ acceptance rate.", - "reason": "The numerical projection benchmark and the independent-standard-normal-residual rejection example use different experimental assumptions; the text must not imply a paired comparison.", - "sourceRefs": [ - "Project 2, Section II: two separate quantitative examples" - ], - "evidence": [] - }, - { - "id": "P2-NEST-02", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/1/keyContributionDetails/2/items/2", - "before": "Lipschitz decision-margin ($\\gamma/s$) reference recomputation so perturbed inputs never inherit stale supervision labels.", - "after": "Lipschitz margin checks and Gaussian margin-to-noise ($\\gamma/s$) analysis to guide reference recomputation and reduce stale supervision labels.", - "reason": "The deterministic Lipschitz radius and the dimensionless Gaussian margin-to-noise ratio are distinct. The mathematical checks do not establish zero stale-label errors for all production inputs.", - "sourceRefs": [ - "Project 2, Section III: Lipschitz bound and Gaussian label-flip identity" - ], - "evidence": [] - }, - { - "id": "P3-NEST-01", - "file": "content/experience-showcase.json", - "jsonPointer": "/entries/2/keyContributionDetails/0/items/1", - "before": "In high-compliance financial and academic environments, spinning up an arbitrary Python virtual machine introduces severe security and audit risks, requiring tasks to be solved strictly within approved, auditable toolkits (such as Excel).", - "after": "When a task’s specified data-governance policy permits only the approved Excel environment, an unauthorized Python virtual machine violates that execution policy.", - "reason": "Tool compliance depends on the specified organizational task policy; there is no general financial or academic rule that forbids Python or mandates Excel.", - "sourceRefs": [ - "Project 3, Section II: tool-compliance predicate and task-specific policy" - ], - "evidence": [] - } - ] -} diff --git a/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md b/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md deleted file mode 100644 index 9d71fba..0000000 --- a/website/content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md +++ /dev/null @@ -1,668 +0,0 @@ -# 01 AI Research and Engineering — Website Portfolio Copy (`theodoreoy.com`) - -> **Confidentiality & Disclosure Note (For Index Header or Footer)** -> The following four research and engineering initiatives were conducted between September 2023 and September 2026 through university–industry collaborations at Duke University. Pursuant to strict non-disclosure agreements (NDAs) with our industry partners—comprising a flagship multimodal foundation model developer, a frontier AI research laboratory, and a leading global alternative asset manager overseeing more than $300 billion in AUM—partner corporate names and proprietary internal system identifiers are anonymized on this public site. Project titles, my role (*Researcher*), geographic locations, dates, architectural paradigms, and self-contained mathematical formulations with reproducible synthetic benchmarks are authorized for public presentation. - -# PART 1: Index Page Briefs (`/past-experience/` Landing View Before Clicking `Read More >`) - -### Duke University × Top-Tier Foundation Model Company -Launched in September 2023 during the earliest stage of large-scale post-training for a flagship multimodal foundation model, this initiative established the foundational statistical governance and resource-allocation paradigm for reinforcement learning from human feedback (RLHF) across more than 300 domain specialists in travel, food delivery, consumer goods, and e-commerce advertising *(partner corporate identity anonymized under NDA; role, scope, and mathematical design disclosed)*. Rather than executing routine vendor quality assurance, I architected the beta mathematical and operational blueprint that defined how frontier model teams govern external expert networks, calibrate evolving acceptance standards, and allocate review capacity at scale. By uniting hierarchical Beta-Binomial reliability modeling, prediction-assisted active statistical inference (inverse-probability-weighted residual correction), and capacity-constrained mixed-integer programming, the framework strictly separates *epistemic measurement value* from *defect remediation value*. In a 10,000-submission synthetic benchmark verified across 200,000 independent sampling repetitions with ideal residual calibration, this allocation architecture reduced population quality estimation variance by **17.79%** versus uniform sampling at equal expected audit cost while cutting post-review expected error loss by **8.89%** at equal specialist hours. - -[Read More >] - -### Duke University × Top-Tier Foundation Model Company -Initiated in September 2023 for a frontier foundation model program *(partner identity anonymized under NDA)*, this project defined what constitutes high-signal financial post-training data and established an end-to-end paradigm for adversarial task design, constraint-preserving synthetic data engineering, and structured preference supervision. To expose subtle reasoning blind spots where base models and practitioners systematically stumble, I engineered diagnostic, error-inducing financial decision tasks and a five-level pairwise preference protocol that translates doctoral- and CFA-level expert judgments into evidence-grounded training signals for RLHF and direct preference optimization. I paired these task specifications with a four-regime mathematical synthesis architecture—Gaussian copulas for cross-sectional corporate fundamentals, covariance-weighted affine projections enforcing exact accounting identities ($1.21 \times 10^{-14}$ maximum residual across 200,000 12-dimensional draws), CTGAN and TabDDPM with Lipschitz decision-margin ($\gamma/s$) label recomputation for multimodal tables, and SDEs, neural SDEs, and Brownian bridges with strict temporal filtration isolation ($\mathcal{F}_s$) for continuous-time price and interest-rate trajectories. To bridge research design and massive vendor scale-up, I created **Batch Zero**—an independent execution and calibration gate subsequently adopted across the company for new post-training tracks. - -[Read More >] - -### Duke University × Top-Tier AI Research Lab -Launched in September 2023 through an early academic collaboration with a frontier AI research laboratory *(partner identity anonymized under NDA)*, this work pioneered a three-pillar evaluation and process-supervision paradigm for scientific, mathematical, and quantitative risk reasoning—anticipating the industry's 2026 agentic evaluation standards by three full years. Recognizing in our earliest experiments that simply stumping frontier models on Olympiad- or graduate-level STEM final answers yields minimal actionable feedback for model iteration, I built an acceptance and diagnostic architecture that audits **(1) Chain-of-Thought (CoT) trajectory soundness** via acyclic proof-obligation dependency graphs ($\Gamma_q \vdash C_q$), **(2) strict compliance in sandboxed tool invocation** (requiring execution within auditable enterprise toolkits such as Excel rather than unauthorized Python virtual machines in regulated workflows, and rejecting numerically correct answers obtained via non-compliant tools), and **(3) structural mathematical reduction over brute-force computation**. In an exact-arithmetic stress test of 3,600 constructed submissions across 600 strongly convex quadratic problem families, semantic certificate verification accepted all **1,200 valid submissions (0.0% false rejection, preserving valid alternative Lagrangian and elimination proofs)** and rejected all **2,400 invalid submissions (0.0% false acceptance, compared to 75.0% false acceptance under answer-only checking)**. - -[Read More >] - -### Duke University × Leading Global Alternative Asset Manager -Developed in a multi-year collaboration between Duke University and a leading global alternative asset manager overseeing more than $300 billion in assets under management *(partner identity anonymized under NDA)*, this project built an AI-native investment-research and epistemic-verification workflow designed to counter **AI-era corporate greenwashing and adversarial narrative manipulation**. Just as merchants game AI consumer recommendations through coordinated marketing seeding, corporate issuers increasingly deploy AI-polished disclosures and PR narratives to inflate automated ESG and investment-research scores—exploiting a regulatory and enforcement vacuum where traditional report-level checks fail. Rather than building a black-box system that tells deal teams which company to fund, I architected a decision-guarantee engine (`Multi-Source Ingestion → Multi-Model Rashomon Scoring → Support-Function Ranking Stability Certification → Minimax-Regret Verification Routing → Versioned SHAP Propagation`) that measures **how trustworthy an AI-assisted assessment actually is** and identifies **which exact factual claim analysts should verify next**. In reproducible analytical benchmarks, joint support-function modeling certified a **0.6-point pairwise company advantage** completely obscured by independent confidence intervals, while a single minimax-regret query reduced worst-case decision regret from **0.61 score units to zero**. - -[Read More >] - -# PART 2 & PART 3: Full Subpage Content (Header, Key Contributions, Paradigm Context & Unabridged Mathematical/Engineering Derivations) - -# SUBPAGE 01: Statistical Inference and Resource Allocation in Expert Data Production - -**URL**: `https://www.theodoreoy.com/past-experience/artificial-intelligence/statistical-inference-and-resource-allocation-in-expert-data-production/` - -## Duke University × Top-Tier Foundation Model Company: Scalable Expert Quality Control and Adaptive Review - -**Position** -Researcher - -**Location** -Durham, NC; Palo Alto, CA - -**Dates** -September 2023 – September 2026 - -**Confidentiality & Scope Disclosure** -Conducted through a university–industry research collaboration with a flagship multimodal foundation model developer. While the corporate partner's name is withheld pursuant to non-disclosure agreements, my role, project scope, operational scale, and the underlying mathematical and engineering frameworks are authorized for public disclosure. - -## Key Contributions - -* **Pioneered the 0-to-1 Statistical Governance Paradigm for Early Frontier Post-Training (Sept 2023 Beta Launch)**: At the inception of large-scale reinforcement learning from human feedback (RLHF) in September 2023—when frontier model pipelines lacked formal statistical control over distributed human data production—I architected the foundational pilot system governing more than 300 domain contributors across travel, food delivery, consumer goods, and e-commerce advertising. Rather than executing day-to-day vendor supervision, my work defined the mathematical and operational standards for how frontier model teams structure, audit, and scale external expert and outsourcing networks. -* **Bayesian Adaptive Review Routing & Hierarchical Contributor Calibration**: Designed multi-tier adaptive review rules parameterized by Beta-Binomial reliability posteriors, credible intervals, verified sample counts, and task-level risk priors. In a flagship e-commerce advertising workflow, dynamically routed flagged submissions to three additional specialists and initially accepted submissions to one or two; closed the feedback loop by feeding expert adjudications directly back into contributor reliability posteriors, targeted coaching, and dynamic review intensity. -* **Taxonomic Defect Disentanglement & Rubric Evolution**: Combined independent random reference audits with targeted high-risk escalations to convert recurring inter-annotator disagreements into unambiguous acceptance criteria. Formally separated missing context, ambiguous prompt instructions, and genuine contributor defects; systematically revised task instructions, boundary cases, and shared calibration anchors while benchmarking Dawid-Skene reviewer-error models against cost-sensitive deferral rules (DeCCaF) to optimize escalation thresholds. -* **Empirical Cost-Latency Telemetry & Production Data Engineering**: Built end-to-end SQL and Python data pipelines unifying task metadata, review trajectories, acceptance outcomes, turnaround times, and behavioral logs, reconciling platform activity with Hubstaff operational telemetry. Crucially, this cross-system reconciliation served not as clerical time-tracking, but to empirically calibrate unit verification costs ($c_i$), task-stratum latencies ($t_{iaj}$), and hard specialist capacity constraints ($H_j^{\text{review}}$) for our mathematical allocation solvers, while translating client-reported model defects into reproducible engineering investigations and operating SOPs. -* **Econometric Analysis of Exploratory Choice Architecture**: Analyzed user decision behavior across curated (approximately 20 hotels) versus expanded (100+ eligible options) recommendation sets using logistic regression with decision-stage interactions, dwell-time transformations, and distributional modeling. Demonstrated that expanding the choice set increased deliberation browsing time significantly more clearly than click-through conversion, establishing quantitative product guidance for exploratory assistant settings. -* **Unified Mathematical Theory Separating Audit Measurement from Defect Remediation**: Extended our production procedures into a rigorous joint mathematical design connecting prediction-assisted active statistical inference, cost-aware audit allocation, and capacity-constrained review assignment. Proved that optimal operations must separate three distinct quantities that standard QA pipelines conflate: a task's *initial error risk* ($q_i$), the *statistical information value of an independent reference audit* ($m_i / c_i$), and the *expected defect loss that additional human handling can remove* ($\Delta_i = r_{i0} - r_{i1}$). -* **Reproducible Large-Scale Monte Carlo & Remediation Validation**: Validated the estimator and allocation algorithms on a stratified synthetic benchmark of 10,000 submissions checked across 200,000 independent Monte Carlo sampling repetitions. Under ideal residual calibration and an identical expected audit budget, cost-aware residual allocation reduced population quality estimation variance by **17.79%** (cutting standard deviation by **9.33%**) relative to uniform sampling, while marginal-gain review ranking reduced expected post-review defect loss by **8.89%** at equal review volume. - -## Architectural Context: Why Frontier RLHF Required a Paradigm Shift in Late 2023 - -When frontier foundation models entered massive multi-domain post-training in late 2023, model builders faced an acute structural paradox: scaling RLHF required transitioning from small internal research cohorts to hundreds of distributed domain specialists and external vendor queues, yet naive quality control actively corrupted both training data and management metrics. Concentrating human review on heuristic high-risk flags created severe selection bias ($\operatorname{Cov}(\pi(X), q(X)) > 0$) that made raw inspection error rates uninterpretable, while auditing tasks whose errors were already predictable wasted senior specialist capacity without improving statistical visibility. The seven sections below present the complete mathematical and engineering framework I developed to establish a self-calibrating control plane for expert data production. - -## Contents - -* Key Contributions -* I. The observation mechanism creates its own failure modes -* II. Correct predictions with audits to recover the population target -* III. Allocate audit effort according to residuals and cost -* IV. Derive the gain and identify when it disappears -* V. Allocate review by expected loss reduction -* VI. Reproducible experiments at equal cost -* VII. Establish whether the design improves production - -## I. The observation mechanism creates its own failure modes - -In production RLHF pipelines, the routing policy used to inspect submissions alters the statistical properties of the observed sample. Let $X$ denote observable task features (encoding domain vertical, prompt complexity, and contributor history) and $Y \in \{0, 1\}$ indicate an error in the original submission under fixed acceptance criteria. The true conditional error probability is $q(x) = P(Y = 1 \mid X = x)$. Let $S = 1$ indicate selection for an independent reference audit. Conditional on $X$, selection is randomized with probability $\pi(x) > 0$, independently of the reference label that has yet to be observed. - -When operational dashboards use the error fraction among verified submissions to assess the full population, they actually target: - -$$E[Y \mid S = 1] = \frac{E[\pi(X)q(X)]}{E[\pi(X)]}$$ - -$$E[Y \mid S = 1] - E[Y] = \frac{\operatorname{Cov}(\pi(X), q(X))}{E[\pi(X)]}$$ - -When review priorities successfully target high-risk tasks, the covariance between audit probability $\pi(X)$ and true defect probability $q(X)$ is strictly positive. A higher error detection rate in the reviewed sample cannot then be read directly as a deterioration in overall production quality; it often reflects sharper risk targeting. This identity describes the distribution of a selected item. In a finite batch, the ratio of observed errors to selected items also has a random denominator, so the two quantities should not be conflated. - -Expert rankings are affected by the exact same mechanism. Let $a_{jh}$ be the probability that expert $j$ is correct on task stratum $h$, and let $p_{jht}$ describe the mix of tasks assigned to that expert in period $t$. Even with unchanged underlying ability, observed accuracy shifts whenever task routing changes: - -$$A_{jt} = \sum_h p_{jht} a_{jh}, \qquad A_{j,t+1} - A_{jt} = \sum_h (p_{jh,t+1} - p_{jht}) a_{jh}$$ - -Valid cross-period comparisons therefore require a fixed target task mix $w_h$, reporting standardized accuracy: - -$$A_j^{\text{std}} = \sum_h w_h a_{jh}$$ - -with overlapping assignments or shared anchor cases to support estimation within each stratum. In our production routing engine, we model each contributor's stratum-level reliability via a conjugate Beta-Binomial scheme: given prior $\operatorname{Beta}(\alpha_{0h}, \beta_{0h})$ and $n_{jht}$ audited tasks with $k_{jht}$ verified successes, the posterior reliability distribution $\operatorname{Beta}(\alpha_{0h} + k_{jht}, \beta_{0h} + n_{jht} - k_{jht})$ yields shrinkage-regularized point estimates and credible intervals that account for unequal sample sizes across comparable audits. Pooling different difficulty levels, criteria versions, and selection mechanisms into a single unstratified binomial sample leaves the preceding selection bias unresolved. - -These distinctions also guide algorithmic method selection. **Dawid-Skene** models represent differences in reviewer confusion matrices across latent ground truth, whereas **DeCCaF** incorporates task conditions, error costs, and reviewer capacity into human-AI deferral and highlights how assigning difficult cases distorts naive evaluations of expert performance. The architecture developed here addresses the critical interface between selective operational deferral and unbiased population quality inference in ongoing post-training production (*Dawid-Skene; DeCCaF*). - -## II. Correct predictions with audits to recover the population target - -Fix a production batch of $N$ original submissions, its acceptance criteria, and its latent reference labels $Y_1, \dots, Y_N$. The inferential target is the finite-batch error rate $\mu = N^{-1} \sum_{i=1}^N Y_i$. Let $f_i \in [0, 1]$ be a fixed, inexpensive risk prediction (synthesized from contributor posterior reliability, task complexity features, and automated heuristic checks) and $\pi_i \in [\pi_{\min}, 1]$ be the assigned audit probability, with independent draws $S_i \sim \operatorname{Bernoulli}(\pi_i)$. Both predictions and sampling probabilities are locked before the current batch's reference labels are revealed. - -We deploy a prediction-assisted, inverse-probability-weighted (IPW) residual-corrected estimator: - -$$\hat{\mu} = \frac{1}{N} \sum_{i=1}^N \left[ f_i + \frac{S_i}{\pi_i} (Y_i - f_i) \right]$$ - -Its design-based validity follows directly. Write the prediction residual as $e_i = Y_i - f_i$ and condition on the fixed batch, predictions, and sampling probabilities: - -$$\hat{\mu} - \mu = \frac{1}{N} \sum_{i=1}^N \left( \frac{S_i}{\pi_i} - 1 \right) e_i, \qquad E_S[\hat{\mu} - \mu] = 0$$ - -Inverse-probability weighting restores the population representation altered by selective verification, while predictions reduce the residual variation that expensive human expert audits must resolve. Crucially for production robustness, **this design-based unbiasedness does not require $f_i = q(X_i)$**: inaccurate predictions can reduce statistical efficiency, but they never introduce selection bias under the specified Bernoulli sampling design. While the estimator builds on established active statistical inference principles (*Active Statistical Inference: mean estimation and sampling design*), our production architecture specifies the exact immutable objects being audited, the rubric version lineage, and the telemetry records required for downstream optimization. - -Specifically, original and revised submissions must be retained as separate database entities. Estimating $\hat{\mu}_{\text{raw}}$ on frozen originals and $\hat{\mu}_{\text{final}}$ on post-review deliverables cleanly distinguishes *defects generated in initial production* from *defects corrected by review*. Furthermore, whenever post-training criteria evolve, a shared set of anchor cases is adjudicated under both rubric versions; submissions from different periods are then compared under a common reference version, preventing stricter acceptance standards from being mistaken for declining contributor capability. - -## III. Allocate audit effort according to residuals and cost - -Independent Bernoulli sampling eliminates cross-item covariance terms, yielding the exact design variance: - -$$\operatorname{Var}_S(\hat{\mu}) = \frac{1}{N^2} \sum_{i=1}^N \frac{1 - \pi_i}{\pi_i} e_i^2$$ - -This expression identifies the true mathematical objective of audit allocation: **audit effort should address what the automated risk predictions have not resolved**. For a binary error label with conditional error probability $q_i$, the expected squared residual decomposes into two components: - -$$m_i := E[(Y_i - f_i)^2 \mid X_i] = q_i(1 - q_i) + (q_i - f_i)^2$$ - -The two terms are intrinsic conditional Bernoulli variance $q_i(1 - q_i)$ and squared prediction calibration error $(q_i - f_i)^2$. This identity establishes a counter-intuitive operational principle: **a high error probability $q_i$ does not necessarily imply high inferential value**. If a task category is almost certain to contain an error ($q_i \approx 1$) and the predictor correctly recognizes that fact ($f_i \approx 1$), then $m_i \approx 0$—spending an independent reference audit on that item yields less statistical information about population quality than auditing a moderately risky task whose outcome is genuinely uncertain ($q_i \approx 0.5$). The high-risk task still merits operational correction, which is precisely why the *value of measurement* and the *value of remediation* require separate mathematical models. - -Let $c_i > 0$ be the empirical cost of an independent reference audit (calibrated by reconciling platform review logs with Hubstaff active-work telemetry) and $B$ be the expected audit-cost budget. First consider the optimal design when the true residual second moments $m_i > 0$ are known: - -$$\min_{\pi} \sum_{i=1}^N \frac{m_i}{\pi_i} \quad \text{s.t.} \quad \sum_{i=1}^N c_i \pi_i \le B, \quad \pi_{\min} \le \pi_i \le 1$$ - -The omitted constant $-\sum_i m_i$ in the numerator of the variance does not depend on the allocation $\pi$. For probabilities strictly between their lower and upper bounds, the Lagrangian first-order condition yields the clipped square-root allocation law: - -$$-\frac{m_i}{\pi_i^2} + \lambda c_i = 0 \implies \pi_i^* = \operatorname{clip}_{[\pi_{\min}, 1]} \left( \sqrt{\frac{m_i}{\lambda c_i}} \right)$$ - -where $\lambda > 0$ is uniquely determined by the budget constraint, and feasibility requires $B \ge \pi_{\min} \sum_i c_i$. If the budget covers a full audit, every item is audited ($\pi_i^* = 1$). The square-root law increases sampling intensity for larger expected residuals while penalizing expensive audits, and the positive probability floor $\pi_{\min}$ preserves observation coverage and bounded IPW weights across the entire task space. - -Production implementation substitutes regularized stratum estimates $\hat{m}_i > 0$ obtained from earlier independent audits or a pilot calibration batch. True $m_i$ defines the optimality analysis; estimated values $\hat{m}_i$ determine the operational allocation. By estimating residuals at the level of `(task stratum × criteria version × cost stratum)`, the pipeline avoids fitting unstable parameters to individual submissions. (Note that $B$ constrains *expected* audit cost per batch; workflows requiring an exact integer count of audits in every batch substitute a fixed-size pivotal or stratified sampling design with identical first-order inclusion probabilities $\pi_i^*$.) - -## IV. Derive the gain and identify when it disappears - -We now derive the exact analytical efficiency gain of cost-aware residual allocation over uniform sampling using the same predictor, and characterize when active allocation fails. Let $C = \sum_{i=1}^N c_i$, so that the budget-equivalent uniform sampling probability is $p = B / C$. When true residual second moments $m_i$ are available and no optimal probability reaches a clipping boundary, averaging design variance over the conditional label distribution gives: - -$$V_{\text{uniform}} = \frac{1}{N^2} \left[ \frac{C \sum_i m_i}{B} - \sum_i m_i \right]$$ - -$$V_{\text{optimal}} = \frac{1}{N^2} \left[ \frac{\left(\sum_i \sqrt{m_i c_i}\right)^2}{B} - \sum_i m_i \right]$$ - -(If a common sampling probability is used within each fixed task stratum, substituting that stratum's finite-batch mean squared residual for $m_i$ makes these expressions exact finite-batch identities.) By the Cauchy-Schwarz inequality: - -$$V_{\text{uniform}} - V_{\text{optimal}} = \frac{C \sum_i m_i - \left(\sum_i \sqrt{m_i c_i}\right)^2}{N^2 B} \ge 0$$ - -Equality holds **if and only if $m_i / c_i$ is constant across all items**. Efficiency gains therefore depend strictly on identifiable heterogeneity in residual second moment per unit audit cost; with homogeneous residuals and costs, optimal allocation reduces to uniform sampling. When clipping is active at $\pi_{\min}$ or $1$, constrained optimization still outperforms feasible uniform policies, with the exact gain evaluated directly at the clipped primal objective. - -Crucially, the same analysis exposes the exact variance penalty of model misspecification. If the allocation uses inaccurate residual estimates $\hat{m}_i$ and neither allocation is clipped, the excess variance relative to the true optimum is: - -$$V(\hat{\pi}) - V(\pi^*) = \frac{1}{N^2 B} \left[ \left( \sum_i \sqrt{\hat{m}_i c_i} \right) \left( \sum_i \frac{m_i \sqrt{c_i}}{\sqrt{\hat{m}_i}} \right) - \left( \sum_i \sqrt{m_i c_i} \right)^2 \right] \ge 0$$ - -Severely underestimating $\hat{m}_i$ in any region of the task space inflates the penalty through the $\sqrt{\hat{m}_i}$ denominator. Enforcing a positive sampling floor $\pi_{\min}$ caps this worst-case blowup, while mixing active and uniform probabilities via the budget-preserving defensive convex combination: - -$$\pi_i(\rho) = (1 - \rho)\hat{\pi}_i + \rho p$$ - -allows the shrinkage weight $\rho \in [0, 1]$ to be tuned across independent calibration batches (*Robust Sampling for Active Statistical Inference*). - -## V. Allocate review by expected loss reduction - -Whereas statistical auditing seeks to *measure* population quality with minimum variance, operational review seeks to *remove defects* from the current batch before training handoff. For task $i$ under handling action $a$ (e.g., accept directly, route to peer review, escalate to expert adjudication, or request contributor revision), let $r_{ia}$ be the expected error loss remaining *after* the action, $d_{ia}$ its operational cost, and $t_{iaj}$ the time required from specialist $j$ (measured via platform and Hubstaff telemetry). Binary assignment decisions $z_{ia} \in \{0, 1\}$ solve: - -$$\min_{z} F(z) = \sum_{i,a} z_{ia}(r_{ia} + \eta d_{ia}) \quad \text{s.t.} \quad \sum_a z_{ia} = 1, \quad \sum_{i,a} z_{ia} t_{iaj} \le H_j \quad \forall j$$ - -where $H_j$ is available specialist time and $\eta \ge 0$ converts resource cost to the scale of error loss (*DeCCaF: capacity constraints and global assignment*). - -When review costs are equal and each reviewed task consumes one slot, **the optimal priority ranking is by marginal remediation gain $\Delta_i = r_{i0} - r_{i1}$, not by initial risk $r_{i0}$**. Missing external evidence may make a high-risk task difficult to repair through one additional review ($\Delta_i$ small), whereas a self-contained calculation error in a moderate-risk task can be completely eliminated ($\Delta_k$ large). Moving a single review slot from task $i$ to task $k$ changes total expected loss by: - -$$\Delta F = (r_{i0} + r_{k1}) - (r_{i1} + r_{k0}) = \Delta_i - \Delta_k$$ - -Whenever $\Delta_k > \Delta_i$, the exchange strictly reduces delivered defect loss, regardless of the tasks' initial risk ordering. - -We can also state the exact calibration precision required of the action-risk estimators. Suppose $|\hat{r}_{ia} - r_{ia}| \le \delta$ for every feasible action, costs and capacities are known, and $\hat{z}$ minimizes the predicted objective $\hat{F}(z)$. Relative to the true optimal feasible assignment $z^*$: - -$$F(\hat{z}) \le \hat{F}(\hat{z}) + N\delta \le \hat{F}(z^*) + N\delta \le F(z^*) + 2N\delta$$ - -This bound identifies *post-action residual loss* $r_{ia}$ as the critical empirical calibration target, motivating randomized pilot trials across permitted review actions. - -Finally, both budgets—remediation and statistical measurement—combine into a single batch-level **Mixed-Integer Convex Program (MICP)**. Let $\tau^2$ be the target variance ceiling for the quality estimate, $B_{\text{total}}$ the combined cost budget, and $V_{\hat{m}}(\pi) = N^{-2}\sum_i (1 - \pi_i)\hat{m}_i / \pi_i$ the planning variance under the calibrated residual model: - -$$\min_{z, \pi} \sum_{i,a} z_{ia}\hat{r}_{ia} \quad \text{s.t.} \quad \sum_{i,a} z_{ia} d_{ia} + \sum_i c_i \pi_i \le B_{\text{total}}, \quad V_{\hat{m}}(\pi) \le \tau^2$$ - -$$\sum_a z_{ia} = 1, \quad z_{ia} \in \{0, 1\}, \quad \sum_{i,a} z_{ia} t_{iaj} \le H_j^{\text{review}} \ \forall j, \quad \pi_{\min} \le \pi_i \le 1$$ - -Here $H_j^{\text{review}}$ is hard capacity allocated to remediation actions while reference audits are scheduled independently on frozen original submissions. Whenever this program is infeasible, it surfaces an explicit, mathematically rigorous trade-off between required delivered quality, required measurement precision, and available engineering budget. - -## VI. Reproducible experiments at equal cost - -The following self-contained synthetic experiments test the mathematical mechanisms derived above (they validate algorithmic behavior on specified public parameters rather than reporting proprietary client telemetry). A fixed batch contains $N = 10,000$ submissions partitioned into three strata of $6,000$, $3,000$, and $1,000$ items, containing $60$, $300$, and $350$ errors respectively (true batch defect rate $\mu = 7.10\%$). Unit audit costs are $1$, $1.5$, and $3$ units across the three strata. Every policy operates under an expected audit budget of $B = 2,000$ units and a minimum sampling floor of $\pi_{\min} = 3\%$. - -Risk predictions for the three strata are $2\%$, $8\%$, and $30\%$ (batch average $6.60\%$, differing from the true defect rate by $0.50$ percentage points). Cost-aware allocation uses the true within-stratum mean squared residuals (ideal calibration), while the misspecified policy reverses the residual ordering across strata. Holding predictions, budget, and submissions fixed, $200,000$ independent Monte Carlo repetitions verify the analytical variance formulas: - -| Audit Policy | Sampling Probabilities Across the Three Strata | Std. Dev. of Corrected Estimate (pct. points) | Variance Change Relative to Uniform Sampling | -| :--- | :--- | :--- | :--- | -| **Uniform sampling with the same residual correction** | 14.81% / 14.81% / 14.81% | 0.568 | Baseline | -| **Prioritize predicted error probability ($f_i$)** | 3.00% / 11.56% / 43.33% | 0.657 | +33.77% increase | -| **Residual- and cost-aware allocation with ideal calibration** | 7.89% / 19.37% / 21.84% | **0.515** | **-17.79% reduction** | -| **Misspecified residual ordering (reversed)** | 23.00% / 11.77% / 3.00% | 0.983 | +199.66% increase | -| **Misspecified allocation with 70% weight on uniform sampling** | 17.27% / 13.90% / 11.27% | 0.615 | +17.07% increase | - -Under ideal residual calibration, the standard deviation of the population quality estimate falls by **9.33%** (a **17.79% variance reduction**). Notice that for the naive high-risk-first policy, the uncorrected raw error detection rate averages **19.60%** (wildly distorting true quality), whereas our IPW residual-corrected estimator recovers **7.1015%** while exposing that naive risk-prioritization actually *inflates* estimation variance by **33.77%** due to over-sampling the expensive third stratum. Across all five policies, Monte Carlo variances match analytical formulas to within $0.5\%$. - -A separate, exact allocation experiment isolates remediation gains. Two task groups contain $100$ items each, with initial error risks of $0.40$ and $0.15$. Additional review reduces those risks to $0.30$ ($\Delta_1 = 0.10$) and $0.01$ ($\Delta_2 = 0.14$), respectively. Review costs are equal and capacity is $100$ items. Ranking by initial risk leaves $100(0.30) + 100(0.15) = 45$ expected errors; ranking by expected loss reduction ($\Delta_i$) leaves $100(0.40) + 100(0.01) = 41$ expected errors—cutting expected defect loss by **8.89%** at the exact same review volume. - -## VII. Establish whether the design improves production - -Production deployment tests statistical quality inference and operational defect control separately, recognizing that *detecting more errors in a targeted queue is never, by itself, evidence of better delivered quality*: -1. **Inferential Validation on Frozen Adjudicated Batches**: Compare uniform sampling, legacy high-risk routing, cost-aware residual allocation, and defensive mixtures against complete independent adjudication—evaluating bias, MSE, and interval coverage across individual task strata and rubric versions. -2. **Capacity-Matched Remediation Trials**: Compare legacy tiered rules against marginal-gain ($\Delta_i$) action allocation at equal total specialist hours, measuring severe defects surviving into final post-training corpora and total fully-loaded cost per accepted submission. - -The contributor reliability records, tiered review workflows, and random reference audits I established in September 2023 provided the operational foundation for this architecture; the mathematical synthesis above ensured that as our frontier model partner scaled external vendor pipelines, quality evidence remained unbiased, cost-optimal, and invariant to routing policy shifts. - -# SUBPAGE 02: Financial Task Design and Synthetic Data Engineering - -**URL**: `https://www.theodoreoy.com/past-experience/artificial-intelligence/task-validity-in-financial-synthetic-data/` - -## Duke University × Top-Tier Foundation Model Company: Financial Task Design and Synthetic Data Engineering - -**Position** -Researcher - -**Location** -Durham, NC; Palo Alto, CA - -**Dates** -September 2023 – September 2026 - -**Confidentiality & Scope Disclosure** -Conducted through a university–industry research collaboration with a flagship foundation model developer. While the corporate partner's name is withheld pursuant to non-disclosure agreements, my role, project scope, organizational protocols (*Batch Zero*), and mathematical synthesis frameworks are authorized for public disclosure. - -## Key Contributions - -* **Defined the Financial Post-Training Task Environment & Adversarial Diagnostic Paradigm (Sept 2023)**: Led financial and business decision task design for a frontier foundation model program, specifying capability taxonomies, primary evidence standards, input schemas, decisive variables, and acceptance conditions. Rather than authoring routine textbook prompts, I deliberately engineered **error-inducing diagnostic cases** targeting high-risk business decision boundaries where base models and practitioners systematically stumble—ensuring every task required genuine doctoral- or CFA-level domain judgment. -* **Architected the Five-Level Pairwise Preference Protocol for RLHF / DPO Supervision**: To convert tacit human domain expertise into actionable post-training gradients, I designed a structured **five-level pairwise preference protocol** that captures *why* an expert judges one response superior to another. By requiring experts to bind every preference judgment to verifiable evidence citations, accounting rules, and decisive variables, the protocol transformed subjective expert comparisons into high-signal preference data that teaches models which financial judgments are evidence-backed. -* **Principled Four-Regime Financial Synthetic Data Engineering**: Selected and customized mathematical synthesis engines around the exact structural relationships each financial task had to preserve, avoiding naive one-size-fits-all tabular generation: - * *Cross-Sectional Corporate Fundamentals*: Applied **Gaussian and vine copulas alongside conditional sampling** across company scale, operating vintage, and financial ratios to preserve joint cross-column dependencies and prevent unrealistic column-independent random generation. - * *Hard Accounting Identities, Ratio Bounds & Chronological Order*: Deployed **constrained sampling, variable transformations, and covariance-weighted affine manifold projections** to enforce exact balance-sheet/cash-flow identities, ratio ranges, and date ordering—achieving a maximum constraint residual of **$1.21 \times 10^{-14}$** across 200,000 12-dimensional synthetic draws where naive rejection sampling collapses to a $0.0002529\%$ acceptance rate. - * *Multimodal & Mixed Numerical/Categorical Records*: Utilized **CTGAN and TabDDPM** to model multimodal empirical distributions, paired with mandatory post-generation business rule validation and Lipschitz decision-margin ($\gamma/s$) reference recomputation so perturbed inputs never inherit stale supervision labels. - * *Continuous-Time Price & Interest-Rate Trajectories*: Developed dynamic paths for temporal financial tasks (as distinct from static cross-sectional company comparisons) using **stochastic differential equations (SDEs), neural SDEs, Brownian bridges, and stochastic interpolation** (extending MathWorks-style stochastic interest-rate path constructions), matching each construction to the task's permitted filtration ($\mathcal{F}_s$) to prevent future-information evaluation leakage. -* **Created the "Batch Zero" Pre-Scaling Execution Gate (Adopted Company-Wide)**: Before scaling task families to external vendor production, I created **Batch Zero**—a pilot verification protocol in which doctoral-level specialists independently executed task construction, synthetic expansion, reference solving, and pairwise preference grading. Pilot disagreements and failure modes directly drove instruction revisions, boundary-case refinements, and **Batch One** entry criteria. The foundation model company subsequently adopted **Batch Zero** as a standard organizational gate across other new post-training tracks. -* **Quantified Stale-Label Corruption, Temporal Leakage & Family Clustering**: Proved analytically and via simulation how synthetic perturbations near decision boundaries cause up to **40.13%** inherited-label error without recomputation, how undisclosed Brownian bridge endpoint conditioning creates an artificial **29.29% RMSE reduction** via future leakage, and how within-family correlation ($\rho = 0.5$) reduces 2,000 nominal synthetic variants across 100 families to **$N_{\text{eff}} \approx 190.48$** effective independent observations. - -## Architectural Context: Why Frontier Financial Post-Training Requires Diagnostic Traps & Manifold-Exact Synthesis - -In September 2023, frontier foundation models could already recite corporate finance definitions, yet they failed unpredictably on real-world financial workflows where small numerical shifts or accounting footnotes reverse an investment recommendation. Our architecture solved two foundational challenges in post-training data engineering: -1. **Why We Engineer "Error-Prone" Prompts**: Easy prompts produce uniform model outputs that yield zero preference signal. By designing prompts specifically around subtle business failure modes—such as covenant breaches, off-balance-sheet adjustments, or near-boundary valuation spreads—we force reasoning errors into the open. Pairing these diagnostic prompts with our five-level pairwise preference protocol translates a specialist's domain intuition (*"Response A is superior because it isolates the working-capital constraint and cites Footnote 4"*) into structured preference supervision that directly steers post-training. -2. **Matching the Mathematical Generator to the Financial Structure**: Generating synthetic financial data is fundamentally different from generating synthetic images or unconstrained text. If a synthetic corporate table violates Total Assets $=$ Total Liabilities $+$ Equity by even a fraction of a percent, or if a synthetic interest-rate trajectory leaks future terminal rates into a historical forecasting prompt, the resulting training data actively corrupts model reasoning. Sections I–VI below establish the complete mathematical framework we built to guarantee constraint validity, label stability, and filtration integrity across scaled synthetic production. - -## Contents - -* Key Contributions -* I. Why better distributional fit may leave task validity unchanged -* II. Enforce business constraints while preserving the correct conditional distribution -* III. Use decision margins to determine when reference judgments must be recomputed -* IV. Define the information available for each temporal task -* V. Test independent execution through Batch Zero and retain the task family as a statistical unit -* VI. Establish whether the complete method justifies scaled production - -## I. Why better distributional fit may leave task validity unchanged - -Modern tabular generative models address distinct distributional challenges: **CTGAN** handles mixed discrete-continuous types, multimodality, and class imbalance via mode-specific normalization and conditional vectors, while **TabDDPM** applies Gaussian and multinomial diffusion to numerical and categorical fields (*CTGAN; TabDDPM*). However, while both learn empirical distributions effectively, **financial task validity requires strict structural invariants that global distributional similarity metrics cannot enforce**: exact accounting identities, chronological ordering of fiscal periods, covenant applicability conditions, and logical consistency between numerical inputs and reference judgments. - -Consider a failure mode that admits an exact mathematical proof. Valid financial inputs $x \in \mathbb{R}^d$ must satisfy $m$ linear accounting or ratio identities $Ax = b$, where $A \in \mathbb{R}^{m \times d}$ has full row rank. The valid population distribution $P$ is supported entirely on the affine subspace $M = \{x \in \mathbb{R}^d : Ax = b\}$ and has finite second moments. Let the columns of $N \in \mathbb{R}^{d \times m}$ form an orthonormal basis for the normal space orthogonal to $M$. Suppose a generative model captures $P$ tangential to the manifold but adds a tiny amount of independent noise in the normal directions: $X_\sigma = X + \sigma N \varepsilon$, where $\varepsilon \sim \mathcal{N}(0, I_m)$ is independent of $X \sim P$. Writing the generated distribution as $Q_\sigma$, we have for every $\sigma > 0$: - -$$W_2(P, Q_\sigma) \le \sqrt{E\|X_\sigma - X\|^2} = \sigma \sqrt{m} \longrightarrow 0 \quad (\text{as } \sigma \to 0^+)$$ - -$$P(M) = 1, \qquad Q_\sigma(M) = 0$$ - -$$\|P - Q_\sigma\|_{\text{TV}} := \sup_B |P(B) - Q_\sigma(B)| = 1$$ - -The first line follows from the coupling $(X, X_\sigma)$ constructed on the same probability space. The second follows because $AN$ is invertible and a continuous $m$-dimensional Gaussian vector equals zero with probability zero. The third follows by taking the Borel set $B = M$. Thus, **the 2-Wasserstein distance $W_2(P, Q_\sigma)$ can converge to zero while the acceptance rate under exact business constraints remains identically zero ($Q_\sigma(M) = 0$) and Total Variation distance remains at its maximum of $1$**. - -This counterexample dictates how responsibilities must be partitioned in a production synthetic data architecture (*SDV constraint-augmented generation*): -* **Copula models and conditional sampling** capture cross-sectional rank and tail dependencies across continuous corporate indicators (company scale, operating history, leverage, and margin ratios) so columns are never generated independently. -* **CTGAN and TabDDPM** model complex mixed-type and multimodal empirical distributions, followed by explicit business-rule validation. -* **Deterministic variable transformations and manifold projections** enforce hard accounting equalities, ratio bounds, and chronological sequences exactly. - -## II. Enforce business constraints while preserving the correct conditional distribution - -Enforcing hard constraints by generating unconstrained candidates and discarding violations (post-hoc rejection sampling) becomes computationally intractable as multiple financial constraints tighten simultaneously. Suppose a candidate draw has $m$ standardized constraint residuals that behave as independent standard normal variables, and each must lie within a tight tolerance $[-\epsilon, \epsilon]$. The acceptance probability and expected candidate draws required per valid input are: - -$$\alpha_m(\epsilon) = [2\Phi(\epsilon) - 1]^m \sim \left( \sqrt{\frac{2}{\pi}} \epsilon \right)^m \quad (\epsilon \to 0^+), \qquad E[N_{\text{attempts}}] = \frac{1}{\alpha_m(\epsilon)}$$ - -Each additional accounting or ratio constraint multiplies the expected candidate count. Setting each tolerance to $\epsilon = 0.05$ residual standard deviations yields the following exact analytical scaling: - -| Simultaneous Constraints ($m$) | Rejection Sampling Acceptance Rate $\alpha_m(\epsilon)$ | Expected Candidate Draws per Valid Input | -| :--- | :--- | :--- | -| **1** | 3.9878% | 25.08 | -| **2** | 0.1590% | 628.84 | -| **4** | **0.0002529%** | **395,442.59** | - -Generating directly through the degrees of freedom permitted by the constraints eliminates these wasted draws. In the linear Gaussian (or latent Gaussian copula) regime, manifold projection also preserves the exact theoretical conditional distribution. Let the initial unconstrained draw be $Z \sim \mathcal{N}(\mu, \Sigma)$ with $\Sigma \succ 0$. Rather than overwriting a single balance-sheet line item (which corrupts covariance structure), we apply a **covariance-weighted Mahalanobis projection**: - -$$X = \arg\min_{x : Ax = b} \frac{1}{2}(x - Z)^\top \Sigma^{-1}(x - Z) = Z + K(b - AZ), \qquad K = \Sigma A^\top (A \Sigma A^\top)^{-1}$$ - -The first-order Lagrangian stationarity condition is $\Sigma^{-1}(X - Z) + A^\top \lambda = 0$; substituting $AX = b$ and eliminating the multiplier $\lambda$ yields the closed-form update. Since $AK = I_m$, every draw satisfies $AX = b$ exactly, with mean and covariance: - -$$E[X] = \mu + K(b - A\mu)$$ - -$$\operatorname{Cov}(X) = (I - KA)\Sigma(I - KA)^\top = \Sigma - \Sigma A^\top (A \Sigma A^\top)^{-1} A\Sigma$$ - -Because an affine transformation preserves Gaussianity, these moments match the exact regular conditional Gaussian law of $Z \mid AZ = b$ (*Majumdar and Majumdar on Gaussian distributions under linear conditioning*). - -In our production verification benchmark, generating **200,000 synthetic 12-dimensional financial vectors under 4 simultaneous linear constraints** achieved a **maximum absolute constraint residual of $1.21 \times 10^{-14}$** and a **relative Frobenius covariance error of 0.662%** against the theoretical conditional covariance—achieving 100% first-draw feasibility rather than rejecting nearly 400,000 candidates per valid record. Fields fully determined by accounting definitions are computed analytically; range bounds and chronological date orderings that admit bijective transformations (such as log-transforms or positive increments $t_{k} = t_{k-1} + \exp(\xi_k)$) are enforced in latent generation space. - -## III. Use decision margins to determine when reference judgments must be recomputed - -Synthetic expansion of a golden financial case often preserves the qualitative business narrative while perturbing decisive financial figures, covenant thresholds, or macroeconomic inputs. **Retaining the original golden-case reference answer after perturbing the numbers injects systematic label corruption into post-training**, even when every synthetic input satisfies all accounting constraints. Just as *FinQA* pairs financial questions with executable reasoning programs, synthetic preference engineering requires the link between inputs, reference conclusions, and expert preference explanations to update dynamically (*FinQA*). - -Consider a formal decision rule for choosing between two corporate actions. Let $d(x)$ denote the net valuation or metric advantage of Action A over Action B, so the true reference decision is $\operatorname{sign} d(x)$. If $d$ is $L$-Lipschitz on a feasible neighborhood, retaining the original reference decision without recomputation is guaranteed valid if and only if the perturbation $\Delta$ remains strictly within the decision margin: - -$$|d(x + \Delta) - d(x)| \le L\|\Delta\|, \qquad |d(x)| > L\|\Delta\| \implies \operatorname{sign} d(x + \Delta) = \operatorname{sign} d(x)$$ - -Because our diagnostic tasks deliberately target **contested decision boundaries** where models are most prone to error, $|d(x)|$ is intentionally small! We can derive the exact closed-form rate of **stale-label error** for a linear comparison rule $d(x) = w^\top x - c$ with initial margin $\gamma = d(x) > 0$. Under a feasible manifold perturbation $\Delta \sim \mathcal{N}(0, \Omega)$ with projected variance $s^2 = w^\top \Omega w > 0$, the probability that retaining the original decision reverses the true ground-truth label is: - -$$\Pr\{d(x + \Delta) < 0\} = \Pr\{w^\top \Delta < -\gamma\} = \Phi\left(-\frac{\gamma}{s}\right)$$ - -Verified across $1,000,000$ Monte Carlo draws: -* When the margin-to-noise ratio $\gamma / s = 2$, the inherited-label error rate is **2.28%**. -* When $\gamma / s = 1$, the inherited-label error rate jumps to **15.87%**. -* Near diagnostic decision boundaries where $\gamma / s = 0.25$, **40.13% of retained labels are false!** - -This analytical result explains why our pipeline links case expansion directly to reference recomputation and fresh **five-level pairwise preference grading** (*HelpSteer2-Preference*). Whenever a synthetic perturbation crosses or approaches a decision boundary, the reference solution is recomputed, fresh candidate model trajectories are sampled, and domain experts evaluate the pair on our five-level preference scale—explicitly annotating which trajectory identified the flipped decisive variable and supported its judgment with primary evidence. - -## IV. Define the information available for each temporal task - -While cross-sectional corporate comparisons do not require continuous-time stochastic processes, dynamic financial tasks—such as derivatives valuation, interest-rate term-structure interpolation, and multi-period liquidity stress testing—require path generators. In these temporal regimes, we deploy **stochastic differential equations (SDEs)** and **neural SDEs** to model path distributions beyond single-time marginals (*Neural SDEs as Infinite-Dimensional GANs*), alongside **Brownian bridges** and **stochastic interpolation** (aligning with classical MathWorks stochastic interest-rate path constructions) when conditioning on known intermediate observations or endpoints. - -However, path conditioning introduces a severe **temporal evaluation leakage hazard** whose magnitude can be derived in closed form. Consider a scaled Brownian motion $X_u = \sigma W_u$ over $s < t < T$. Under squared error loss, an honest predictor acting at time $s$ with access only to the historical filtration $\mathcal{F}_s$ has conditional mean squared error $\sigma^2(t - s)$. If a synthetic pipeline conditions path generation on the future terminal endpoint $X_T$ and inadvertently exposes that conditioned structure to a forecasting task, Gaussian conditioning gives (*Pitman on the Brownian bridge*): - -$$\operatorname{Var}(X_t \mid \mathcal{F}_s) = \sigma^2(t - s)$$ - -$$\operatorname{Var}(X_t \mid \mathcal{F}_s, X_T) = \sigma^2(t - s) - \frac{[\sigma^2(t - s)]^2}{\sigma^2(T - s)} = \sigma^2(t - s)\frac{T - t}{T - s}$$ - -When $t = (s + T)/2$ is the midpoint of the forecasting horizon, conditioning on $X_T$ halves the conditional variance, producing an **apparent 29.29% reduction in RMSE** ($1 - 1/\sqrt{2}$) that is 100% attributable to future information leakage rather than model capability! (By contrast, if the task explicitly supplies both endpoints $X_s$ and $X_T$ and asks the model to audit an *interpolation* or back-cast an intervening accounting period, Brownian bridge conditioning is mathematically appropriate.) - -Our task environment specification therefore enforces a strict separation across three information sets: -1. **Construction Information ($\mathcal{F}_{\text{gen}}$)**: Complete paths, terminal endpoints, or latent parameters used by the generator to synthesize the scenario. -2. **Permitted Predictor Filtration ($\mathcal{F}_s$)**: The exact historical information legally visible to the model at decision time $s$. -3. **Evaluation & Oracle Information ($\mathcal{F}_{\text{eval}}$)**: Realized post-$s$ outcomes reserved strictly for scoring or explicitly disclosed oracle baselines. - -## V. Test independent execution through Batch Zero and retain the task family as a statistical unit - -Even when constraint projections, decision-margin recomputations, and filtration rules are mathematically sound, a post-training specification must remain reproducible when executed by hundreds of external specialists. To solve this organizational scaling bottleneck, I created **Batch Zero**: -* **The Batch Zero Protocol**: Prior to launching scaled vendor production (*Batch One*), a select cohort of doctoral-level domain specialists with proven delivery records independently executed the entire specification—authoring seed cases, running synthetic expansions, verifying reference solutions, and assigning five-level pairwise preference scores. Every failed case or inter-expert disagreement in Batch Zero triggered surgical revisions to prompt instructions, boundary definitions, and calibration rubrics. Because Batch Zero prevented costly downstream rework across external vendor queues, **the partner foundation model company adopted Batch Zero as a standard pre-scaling gate across other new post-training tracks**. - -Simultaneously, our statistical governance prevents **synthetic sample-size overstatement**. Suppose $G$ independent golden-case families are each expanded into $k$ synthetic variants ($Gk$ nominal records), where any two variants within a family share correlation $\rho$ and common variance $\sigma_Y^2$. Expanding the covariance sum for the sample mean yields the exact **effective sample size** $N_{\text{eff}}$: - -$$\operatorname{Var}(\bar{Y}) = \frac{Gk\sigma_Y^2 + Gk(k - 1)\rho\sigma_Y^2}{(Gk)^2} = \frac{\sigma_Y^2}{Gk}[1 + (k - 1)\rho] \implies N_{\text{eff}} = \frac{Gk}{1 + (k - 1)\rho}$$ - -With $G = 100$ independent families, $k = 20$ synthetic variants per family ($2,000$ nominal tasks), and within-family correlation $\rho = 0.5$, **the nominal 2,000 records provide the exact same statistical precision for the mean as just $190.48$ independent records**—and ignoring within-family correlation understates standard errors by a factor of **$3.24\times$**! Consequently, our production pipeline enforces strict family-level train/evaluation splits and computes all confidence intervals with family-level and expert-level hierarchical clustering. - -## VI. Establish whether the complete method justifies scaled production - -To evaluate whether our constraint-preserving synthesis and preference architecture justifies scaled production, we hold the task mix, generator training budget, expert review budget, and final acceptance criteria fixed, introducing each methodological layer in staged blind comparisons evaluated across independent task families: - -| Comparison Stage | Methodological Upgrade Introduced | Core Research & Production Question | -| :--- | :--- | :--- | -| **Existing production baseline** | Unconstrained generation + manual expert review & rework | What is the true fully-loaded cost per finally accepted, valid case? | -| **Add manifold-constrained generation** | Fix generator; enforce affine projections & variable transforms | Do invalid candidates and manual expert repairs drop while preserving conditional distributions? | -| **Add margin-aware reference recomputation** | Update reference judgments when decisive variables cross $\gamma/s$ | Do stale-label errors decline, producing valid supervision on both sides of decision boundaries? | -| **Apply complete Batch Zero & filtration criteria** | Jointly check $\mathcal{F}_s$ timing conditions and independent execution | Do post-training gains persist after excluding leakage and clustering by source family? | - -Because raw acceptance rates reward generating trivial variants of easy templates, and raw generation speed ignores manual expert repair costs, we govern production decisions via **Scenario-Weighted Family Coverage Efficiency**. Let $j$ index predefined cells in the financial capability matrix with strategic weights $w_j \ge 0$ ($\sum_j w_j = 1$) and required independent valid family counts $n_j^* > 0$: - -$$\operatorname{Coverage} = \sum_j w_j \min\left(1, \frac{n_j^{\text{valid}}}{n_j^*}\right), \qquad \operatorname{Efficiency} = \frac{\operatorname{Coverage}}{C_{\text{generation}} + C_{\text{review}} + C_{\text{rework}} + C_{\text{adjudication}}}$$ - -Once a cell reaches its target $n_j^*$ independent families, repeatedly expanding easy templates yields zero marginal coverage gain (*Alaa et al. on synthetic data evaluation*). Together, manifold constraint enforcement, decision-margin label recomputation, temporal filtration isolation, and **Batch Zero** calibration establish a rigorous engineering standard for synthetic preference data in frontier model post-training. - -# SUBPAGE 03: Scientific Reasoning Evaluation and Task Design - -**URL**: `https://www.theodoreoy.com/past-experience/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks/` - -## Duke University × Top-Tier AI Research Lab: Scientific Reasoning Evaluation and Task Design - -**Position** -Researcher - -**Location** -Durham, NC; Palo Alto, CA - -**Dates** -September 2023 – September 2026 - -**Confidentiality & Scope Disclosure** -Conducted through an early university–industry research partnership with a frontier AI research laboratory beginning in September 2023. While the partner laboratory's name is withheld pursuant to non-disclosure agreements, my role, evaluation methodology, proof-obligation verification architecture, and exact-arithmetic stress tests are authorized for public disclosure. - -## Key Contributions - -* **Pioneered a Three-Pillar Agentic & Process Evaluation Paradigm 3 Years Ahead of Industry Standard (Sept 2023)**: During early university collaborations in September 2023 aimed at stress-testing frontier models with Olympiad- and graduate-level STEM tasks, I demonstrated that merely trying to stump models and checking final answers provides minimal actionable feedback for post-training. Three full years before the industry's 2026 Agent Evaluation wave, I established an evaluation paradigm centered on three pillars: - 1. *Chain-of-Thought (CoT) Trajectory & Proof-Obligation Auditing*: Verifying whether intermediate reasoning steps follow a sound logical trajectory rather than reaching a lucky final number through compensating errors or unverified leaps. - 2. *Enterprise-Compliant Tool-Invocation Governance*: Auditing whether the model selects and executes tools strictly within regulatory and institutional compliance boundaries. In high-compliance financial and academic environments, spinning up an arbitrary Python virtual machine introduces severe security and audit risks, requiring tasks to be solved strictly within approved, auditable toolkits (such as Excel). Under our acceptance standard, **even if a model computes the exact right numerical answer, invoking a non-compliant tool (e.g., Python instead of mandated Excel) results in immediate failure**. - 3. *Penalizing Brute-Force Compute in Favor of Structural Mathematical Insight*: Designing tasks and scoring rubrics that discourage models from burning inference compute on brute-force algebraic or numerical exhaustion, rewarding instead the identification of domain invariants and structural reductions. -* **Translated Physics, Mathematics & Risk Engineering into Proof-Obligation DAGs**: Formalized task deliverables as directed acyclic dependency graphs ($\Gamma_q \vdash C_q$) that separate physical model formulation, theorem applicability, compliant tool execution, uniform approximation bounds, and conclusion strength—accommodating legitimate alternative proof paths (such as Lagrangian duality versus variable elimination) that brittle reference-matching pipelines falsely reject. -* **Derived Projected-Residual Optimality Certificates for Convex Optimization**: Formulated exact KKT projected-residual bounds ($0 \le f(x) - v^* \le \|P_{\ker A} r\|_2^2 / (2\mu)$) on the optimality gap for equality-constrained strongly convex quadratic programs, replacing heuristic numerical proximity checks with mathematical certificates of solution quality. -* **Engineered Original Kinematics Benchmarks via Path Invariance & Harmonic Asymptotics**: Created original physics and risk-engineering tasks that test deep structural reasoning—using the path invariance of vertical travel time to reduce a 2D kinematics problem to 1D horizontal scheduling, proving an attained minimum completion time and an unattained supremum, and deriving harmonic-number ($H_n$) asymptotics for staircase trajectories approaching the upper bound. -* **Quantified Biased Supervision Gradients & Task-Transformation Error Propagation**: Proved mathematically how false acceptance ($\alpha$) and false rejection ($\beta$) reverse model rankings on leaderboards and bias the binary cross-entropy training gradient of Process Reward Models ($E[\nabla_s \ell(s,V) - \nabla_s \ell(s,Z) \mid h] = p(h) - \tilde{p}(h)$), and derived exact tolerance propagation laws ($a\delta + 2\varepsilon$) under bijective task transformations. -* **Achieved 0.0% False Acceptance and 0.0% False Rejection Across 3,600 Exact-Arithmetic Stress Tests**: Constructed 600 strongly convex quadratic problem families (3,600 exact rational arithmetic submissions) where 75% of invalid proofs report the exact right optimal value $v^*$. Semantic certificate checking accepted all **1,200 valid proofs (0.0% false rejection)** and rejected all **2,400 invalid submissions (0.0% false acceptance)**, outperforming answer-only checking (**75.0% false acceptance**) and strict reference-path matching (**50.0% false rejection**). - -## Architectural Context: From "Stumping Models on Final Answers" to Verifiable Trajectory & Tool Governance - -When frontier AI labs partnered with research universities in autumn 2023 to probe the limits of scientific reasoning, the standard playbook was simple: collect difficult STEM problems and check whether the model's final answer matched the answer key. Our experiments quickly revealed why this approach hit a dead end: models frequently arrived at the right number via broken physical assumptions, brute-force computational exhaustion, or unauthorized tool calls (such as invoking a Python VM in a compliance-restricted workflow that mandated Excel). Sections I–VII below present the formal mathematical and engineering architecture I developed in 2023 to turn scientific benchmarking into a rigorous, certificate-backed supervision system. - -## Contents - -* Key Contributions -* I. Correct final answers can conceal invalid reasoning and reverse model rankings -* II. Define acceptance through proof obligations and allow valid alternative solutions -* III. Produce an error certificate as part of acceptance -* IV. Task expansion must propagate mathematical error -* V. Label quality determines what post-training research optimizes -* VI. Reproducible stress tests for false acceptance and false rejection -* VII. Account for dependence within task families and the full cost of expert review - -## I. Correct final answers can conceal invalid reasoning and reverse model rankings - -Checking only the final answer is appropriate only when that final value constitutes the entire deliverable. In scientific reasoning, quantitative finance, and risk engineering, the model formulation, theorem assumptions, compliant tool invocation, intermediate derivations, and conclusion must hold together. In the OlympiadBench and Omni-MATH samples later analyzed to construct *ProcessBench*, **32.2% and 51.8% of solutions with correct final answers still contained process errors** (*ProcessBench, Section 3.3*). Answer checking and trajectory validity measure fundamentally different things. - -Suppose problems and model trajectories are drawn from a target capability distribution. Let $Z = 1$ indicate that the complete solution (including reasoning steps and compliant tool execution) is valid, and let $V = 1$ indicate that the evaluation protocol passes it. For model $m$, let $\theta_m = \Pr_m(Z = 1)$ be its true validity rate, $\alpha_m = \Pr_m(V = 1 \mid Z = 0)$ its **false acceptance rate** on invalid solutions (e.g., passing a flawed proof, a brute-force guess, or a non-compliant Python tool call that yields the right number), and $\beta_m = \Pr_m(V = 0 \mid Z = 1)$ its **false rejection rate** on valid solutions (e.g., rejecting a valid alternative derivation). The law of total probability gives: - -$$(1) \qquad s_m = E_m[V] = \alpha_m + (1 - \alpha_m - \beta_m)\theta_m, \qquad s_m - \theta_m = \alpha_m(1 - \theta_m) - \beta_m\theta_m$$ - -The first bias term $\alpha_m(1 - \theta_m)$ credits invalid trajectories; the second term $-\beta_m\theta_m$ penalizes valid ones. **Adding more benchmark problems reduces sampling variance in $\hat{s}_m$, but never removes the structural gap $s_m - \theta_m$**. Because different models exhibit different failure modes (e.g., one model adheres strictly to compliant tools and rigorous deduction, while another aggressively invokes unauthorized code interpreters or heuristic shortcuts), their $\alpha_m$ and $\beta_m$ can differ enough to **strictly reverse their leaderboard ranking**: -* In a concrete arithmetic scenario, suppose Model A and Model B have true validity rates of $\theta_A = 72\%$ and $\theta_B = 64\%$, respectively. -* If their invalid trajectories pass a final-answer check with probabilities $\alpha_A = 4\%$ and $\alpha_B = 36\%$ (while $\beta_A = \beta_B = 0\%$), their observed benchmark scores become **$s_A = 73.12\%$** and **$s_B = 76.96\%$**—ranking the less capable, shortcut-prone model higher! - -While process supervision extends evaluation to intermediate steps (*Let's Verify Step by Step, Sections 2 and 3*), our 2023 architecture further addressed three critical gaps in scientific post-training: omissions in reference solutions, valid alternative proof methods, and strict enforcement of tool-compliance and structural-insight obligations defined *before* model responses are generated. - -## II. Define acceptance through proof obligations and allow valid alternative solutions - -A complete task deliverable specifies its domain assumptions and environmental compliance constraints $\Gamma_q$, the target conclusion $C_q$, and the proof obligations required to establish it. A model trajectory $\tau$ may choose its own intermediate lemmas, coordinate representations, or order of deduction; acceptance depends strictly on whether it fulfills the obligations while obeying tool governance. Organizing the response as a directed acyclic dependency graph (DAG) $G$ yields the local soundness check for every node $v_j$ alongside the global tool-compliance predicate $\mathcal{T}_{\text{comp}}(\tau) \in \{0, 1\}$: - -$$(2) \qquad \Gamma_q \cup \{v_k : k \in \operatorname{pa}(j)\} \vdash v_j, \qquad \Gamma_q \cup \{v_j : j \in \operatorname{sink}(G)\} \vdash C_q, \qquad \mathcal{T}_{\text{comp}}(\tau) = 1$$ - -Here $\vdash$ denotes valid derivability within the sound domain inference system, and $\mathcal{T}_{\text{comp}}(\tau) = 1$ certifies that every external tool invocation $t \in \tau$ belongs to the task's permitted compliance sandbox $\mathcal{A}_q$ (for instance, $\mathcal{A}_q = \{\text{Excel Audited Workbook}\}$ and $\text{Python VM} \notin \mathcal{A}_q$). If the root nodes follow from verified assumptions $\Gamma_q$, each dependency edge is sound, and $\mathcal{T}_{\text{comp}}(\tau) = 1$, structural induction over a topological ordering of $G$ establishes the complete conclusion from local checks. - -In production implementation, I organized the central obligations around five verifiable gates: -1. **Valid Model Formulation**: Checking that physical, financial, or probabilistic state spaces are well-defined. -2. **Compliant Tool Selection ($\mathcal{T}_{\text{comp}}(\tau) = 1$)**: In regulated financial and academic settings where spinning up a Python virtual machine violates data-governance rules, trajectories that invoke Python instead of executing within the mandated Excel environment are rejected immediately ($V = 0$), even if the final numerical calculation is exact. -3. **Method Applicability & Uniform Error Control**: In physics tasks, pointwise convergence at a fixed parameter value does not establish that an approximation remains valid inside an optimization, an integral, or an exchange of limits. Authors and models must specify the approximation parameter, its domain of validity, and uniform error bounds before claiming an order of accuracy. In risk engineering, the corresponding obligations govern distributional and tail-dependence assumptions behind extreme-value bounds. -4. **Structural Insight vs. Brute-Force Penalization**: Obligations require identifying the governing mathematical invariant rather than exhausting finite grids via raw compute. -5. **First-Error Localization in Feedback**: Algebra that is locally correct under a false premise does not restore proof validity. Recording the first failed obligation node $v_j$ and marking its descendant subgraph $\operatorname{desc}(v_j)$ as unverified resolves the post-error ambiguity highlighted in *ProcessBench, Section 3.1*. - -## III. Produce an error certificate as part of acceptance - -An executable optimization subfamily illustrates how formal mathematical certificates replace brittle string matching. Consider the equality-constrained strongly convex quadratic problem: - -$$(3) \qquad v^* = \min_{Ax = b} f(x), \qquad f(x) = \frac{1}{2}x^\top H x + c^\top x, \qquad H \succeq \mu I, \quad \mu > 0$$ - -where $H$ is symmetric, $Ax = b$ is feasible, and $\mu > 0$ is a verified lower bound on curvature. For any feasible candidate $x$ ($Ax = b$), choose any dual multiplier vector $\nu$ and define the stationarity residual $r = Hx + c + A^\top \nu$. Strong convexity gives the first inequality below for every feasible displacement $d \in \ker A$; completing the square in $d$ yields an exact **optimality error certificate**: - -$$(4) \qquad f(x + d) \ge f(x) + r^\top d + \frac{\mu}{2}\|d\|^2$$ - -$$\inf_{d \in \ker A} \left( r^\top d + \frac{\mu}{2}\|d\|^2 \right) = -\frac{\|P_{\ker A} r\|_2^2}{2\mu}$$ - -$$0 \le f(x) - v^* \le \frac{\|P_{\ker A} r\|_2^2}{2\mu}$$ - -The operator $P_{\ker A}$ is the orthogonal projection onto the null space of feasible directions $\ker A$, removing residual components normal to the constraints that do not affect optimality along the feasible manifold (*Boyd and Vandenberghe, Convex Optimization, Sections 5.5 and 9.1*). A zero projected residual ($P_{\ker A} r = 0$) certifies exact optimality; otherwise, the certificate bounds the suboptimality gap in objective units: -* Solutions obtained via **Lagrange multipliers** or by **first eliminating the equality constraints** map seamlessly to the exact same semantic obligations and pass verification. -* Conversely, reproducing the correct optimal value $v^*$ via brute-force guessing, an infeasible point ($Ax \neq b$), or an unsupported dual multiplier fails the certificate check immediately. - -Optimization certificates are one instance of our broader obligation design. In an original kinematics benchmark I created for the project, I used the **path invariance of vertical travel time** to reduce a 2D trajectory problem to 1D horizontal scheduling—separately proving an **attained minimum completion time**, an **unattained supremum**, and **harmonic-number ($H_n = \sum_{k=1}^n k^{-1}$) asymptotics** for a staircase path construction approaching the supremum. This benchmark demonstrates why feasibility, optimal bounds, structural reduction, and attainment require separate proof obligations even when a model guesses the final limiting value correctly. - -## IV. Task expansion must propagate mathematical error - -While symbolic templates (*GSM-Symbolic, Section 3.1*) parameterize variable domains and answer formulas, scientific and engineering benchmarks also require verifiable laws governing how a transformation changes feasible sets, objective values, and acceptance tolerances. - -Let the original problem and its transformed variant have feasible sets $F$ and $F'$ with objectives $J$ and $J'$ and finite infima $v$ and $v'$. Suppose the task designer verifies a bijection $\Phi : F \to F'$ satisfying the uniform bound for every feasible $u \in F$: - -$$(5) \qquad |J'(\Phi(u)) - aJ(u) - b| \le \varepsilon, \quad a > 0$$ - -$$av + b - \varepsilon \le v' \le av + b + \varepsilon$$ - -$$J(u) - v \le \delta \implies J'(\Phi(u)) - v' \le a\delta + 2\varepsilon$$ - -Taking infima across the first line yields the second; subtracting the lower bound on $v'$ from the upper bound on $J'(\Phi(u))$ yields the third. This theorem answers an essential engineering question: after a unit conversion, parameter rescaling, or numerical transformation with bounded error $\varepsilon$, what acceptance tolerance must apply to a previously $\delta$-optimal solution? Retaining the raw tolerance $\delta$ misclassifies valid transformed solutions as model failures, while relaxing tolerances ad-hoc conceals genuine capability gaps. Moreover, when $\varepsilon = 0$, the bijection and positive scaling $a > 0$ preserve whether an optimum is attained; variants that alter boundary compactness or admissible control classes require fresh attainment proofs. - -## V. Label quality determines what post-training research optimizes - -Granular step-level process labels improve post-training only if they measure true logical and compliance validity; systematic grading errors are directly absorbed by downstream Process Reward Models (PRMs) and RLHF policies. Let $h$ denote the problem and reasoning prefix through the current step, $Z = 1$ indicate true step validity under the problem's assumptions, and $V \in \{0, 1\}$ be the noisy acceptance label produced by the grader with conditional false acceptance $\alpha(h)$ and false rejection $\beta(h)$. Writing $p(h) = \Pr(Z = 1 \mid h)$, a process evaluator with logit $s$ trained via binary cross-entropy optimizes toward the corrupted conditional target: - -$$(6) \qquad \tilde{p}(h) = \alpha(h) + [1 - \alpha(h) - \beta(h)]p(h)$$ - -$$\frac{\partial \ell(s, V)}{\partial s} = \sigma(s) - V$$ - -$$E[\nabla_s \ell(s, V) - \nabla_s \ell(s, Z) \mid h] = p(h) - \tilde{p}(h)$$ - -This expected gradient bias formula proves mathematically why naive evaluation pipelines damage post-training: -* **When valid alternative proofs are rejected ($\beta(h) > 0$)**: $\tilde{p}(h) < p(h)$, shifting the gradient to penalize valid mathematical diversity and force narrow template memorization. -* **When flawed proofs or non-compliant tool calls pass because the final number is right ($\alpha(h) > 0$)**: $\tilde{p}(h) > p(h)$, training the model to omit boundary checks and violate tool-compliance sandboxes. - -Furthermore, while class-conditional noise correction (*Learning with Noisy Labels, Sections 2 and 3*) can unbias the mean estimate when $\alpha, \beta$ are known constants ($\alpha + \beta < 1$), it inflates estimation variance quadratically: - -$$(7) \qquad \hat{\theta}_{\text{corr}} = \frac{\bar{V} - \alpha}{1 - \alpha - \beta}, \qquad \operatorname{Var}(\hat{\theta}_{\text{corr}}) = \frac{\operatorname{Var}(\bar{V})}{(1 - \alpha - \beta)^2}$$ - -Eliminating $\alpha$ and $\beta$ inside data production via semantic proof-obligation certificates avoids this variance explosion entirely. - -## VI. Reproducible stress tests for false acceptance and false rejection - -To validate our certificate verifier reproducibly, I constructed **600 strongly convex quadratic task families** and generated **6 algebraic submissions per family (3,600 total submissions)** using **exact rational arithmetic**: -* **1,200 Valid Certificates (2 per family)**: One derived in Lagrangian multiplier form and one in equality-elimination form. -* **2,400 Controlled-Defect Submissions (4 per family)**: (1) An infeasible construction ($Ax \neq b$) yielding the exact optimal value $v^*$; (2) failed stationarity along a feasible direction in $\ker A$ while reporting $v^*$; (3) a claimed multiplier that fails the stationarity equation while reporting $v^*$; and (4) an incorrect final objective value. Crucially, the first three defect classes (1,800 submissions, or **75% of all invalid submissions**) report the exact right final answer $v^*$! - -We benchmarked three deterministic acceptance protocols across all 3,600 submissions: - -| Acceptance Protocol | False Acceptance Rate on Invalid Certificates ($2,400$) | False Rejection Rate on Valid Certificates ($1,200$) | Valid Share of Accepted Submissions | -| :--- | :--- | :--- | :--- | -| **Final-answer matching only** | **75.0%** (admits 1,800 flawed solutions) | 0.0% | 40.0% | -| **Strict reference path (ablation)** | 0.0% | **50.0%** (rejects 600 valid elimination proofs) | 100.0% | -| **Semantic certificate verification (Ours)** | **0.0%** | **0.0%** | **100.0%** | - -Within this exact-arithmetic benchmark, semantic certificate verification admitted **1,800 fewer invalid proofs** than final-answer matching and retained **600 more valid alternative solutions** than strict reference-path matching, while passing all 600 feasible-perturbation checks on the projected-residual bound in Equation (4). - -## VII. Account for dependence within task families and the full cost of expert review - -Generating surface variants of a single problem template does not yield independent observations of scientific capability. Across $F$ independent task families with $K$ variants per family and within-family correlation $\rho$, the variance of the sample mean and effective sample size $n_{\text{eff}}$ are: - -$$(8) \qquad \operatorname{Var}(\bar{Z}) = \frac{\sigma^2}{FK}[1 + (K - 1)\rho], \qquad n_{\text{eff}} = \frac{FK}{1 + (K - 1)\rho}$$ - -If $F = 50$ task families each have $K = 8$ variants ($400$ total responses) with $\rho = 0.6$, the $400$ responses provide the same precision as just **$n_{\text{eff}} \approx 77$ independent observations**—mandating family-level train/test partitioning and new combinations of physical and mathematical principles rather than superficial numerical templating. - -Finally, full production cost accounting determines when automated certificate verification outperforms manual PhD review. Let $C_0$ be the upfront engineering cost of building the proof-obligation certificates and verifier, $c_e$ the legacy workflow's expert review cost per submission, $c_v$ the automated verifier's unit execution cost, and $u$ the fraction of submissions escalated to human adjudication. Once false acceptance and false rejection standards are met, the certificate architecture reduces total cost whenever submission volume $N$ exceeds: - -$$(9) \qquad C_0 + N(c_v + uc_e) < Nc_e \iff N > \frac{C_0}{(1 - u)c_e - c_v}, \qquad (1 - u)c_e > c_v$$ - -By connecting capability specifications, CoT proof obligations, tool-compliance sandboxes, and exact optimality certificates, this 2023 framework established a rigorous blueprint for verifiable reasoning evaluation and post-training supervision. - -# SUBPAGE 04: AI-Driven Investment Research and Explainable Risk Analysis - -**URL**: `https://www.theodoreoy.com/past-experience/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research/` - -## Duke University × Leading Global Alternative Asset Manager: AI-Driven Investment Research and Explainable Risk Analysis - -**Position** -Researcher - -**Location** -Durham, NC; Washington, DC - -**Dates** -September 2023 – September 2026 - -**Confidentiality & Scope Disclosure** -Conducted through a university–industry collaboration with a leading global alternative asset manager overseeing more than $300 billion in assets under management as of June 2026. While the partner institution's name is withheld pursuant to non-disclosure agreements, my role, system architecture, anti-greenwashing methodology, and analytical benchmarks are authorized for public disclosure. - -## Key Contributions - -* **Architected an Institutional Defense Against "AI-Era Greenwashing" & Adversarial Narrative Manipulation**: Identified and solved a critical emerging vulnerability in private-market and ESG investment research: corporate executives increasingly use generative AI to craft polished marketing disclosures and sustainability narratives that game automated LLM sentiment and ESG scoring models (analogous to restaurants gaming AI consumer recommendation rankings through coordinated review seeding)—operating in a regulatory and legal vacuum where traditional equipment- or report-level greenwashing checks fail. -* **Built an Epistemic Decision-Support Engine Rather Than a Black-Box Deal Picker**: Designed the internal AI workflow around a strict institutional mandate—**never replace the investment committee by outputting a black-box "invest / pass" verdict; instead, quantify how trustworthy the current score is and prescribe which exact factual claim analysts must verify next to minimize misjudgment**. Implemented the complete closed-loop pipeline: `Multi-Source Evidence Ingestion → Multi-Model Rashomon Scoring → Ranking Stability Certification → Minimax-Regret Verification Routing → Automated Post-Correction Re-scoring & Attribution Propagation`. -* **Converted Heterogeneous Disclosures into Traceable Scoring Rules & Contradiction Detectors**: Transformed operating reports, audited financial statements, sustainability disclosures, management questionnaires, and third-party datasets into structured scoring rubrics with strict evidence lineage. Directed AI-assisted verification specifically toward unsubstantiated corporate claims and cross-document contradictions—the primary signatures of AI-amplified greenwashing. -* **Combined Tree Ensembles, Conditional SHAP & Rashomon Model Multiplicity**: Deployed gradient-boosted tree models and tabular deep learning to capture nonlinear thresholds and interaction effects, paired with SHAP attributions and conditional Shapley analyses across Rashomon sets of near-optimal models ($\mathcal{F}_\alpha$) to distinguish genuine fundamental drivers from model-selection artifacts. -* **Formulated Joint Model-and-Evidence Uncertainty Bounds via Convex Support Functions**: Proved that standard bootstrap resampling and raw SHAP feature rankings break down under shared reporting bias or unverified corporate claims. Represented unresolved company evidence $\mathcal{X}(E)$ and Rashomon model sets $\mathcal{F}_\alpha$ jointly ($\Omega(E) = \mathcal{F}_\alpha \times \mathcal{X}(E)$) and derived convex support-function bounds ($h_U(a_f)$) that cancel shared-source measurement errors—certifying a **0.6-point pairwise company advantage** that separate company confidence intervals ($[-1.4, 2.6]$) failed to resolve. -* **Derived Minimax-Regret Active Verification Routing ($V(q)/c_q$)**: Formulated a distribution-free verification criterion that prioritizes diligence queries by the **worst-case decision regret they remove** rather than by naive SHAP magnitude. In an eight-scenario benchmark, a single targeted verification query reduced worst-case decision regret from **0.61 score units to zero**, whereas rechecking an already-verified high-SHAP feature yielded zero regret reduction. - -## Architectural Context: Countering Adversarial AI Greenwashing in $300B+ Institutional Diligence - -When global alternative asset managers evaluate private equity and impact investments, ESG and operational risk diligence cannot rely on naive LLM summarization. Between 2023 and 2026, we observed a structural shift in corporate self-reporting: just as local merchants learned to manipulate AI recommendation engines via coordinated digital marketing, corporate issuers began deploying AI-optimized sustainability narratives to inflate automated ESG scores. Because regulatory legislation and enforcement mechanisms have not yet caught up with AI-driven narrative manipulation, an investment workflow that takes unverified corporate disclosures at face value—or ranks diligence priorities solely by raw SHAP values—exposes the fund to severe greenwashing risk. Sections I–VI below present the mathematical and systems architecture I developed to audit score credibility, cancel shared measurement errors, and route human diligence toward the exact claims that govern the investment decision. - -## Contents - -* Key Contributions -* I. Where existing methods leave the decision unresolved -* II. Joint uncertainty over models and evidence -* III. Computing comparison bounds with support functions -* IV. Choosing verification queries by the regret they can remove -* V. Checking that corrections propagate through attribution -* VI. Reproducible quantitative comparisons - -## I. Where existing methods leave the decision unresolved - -SHAP's local accuracy property decomposes a given model prediction into additive feature contributions, while conditional Shapley methods address feature dependence and Rashomon-set methods examine explanation stability across near-optimal models (*SHAP; Aas et al.; Marx et al.*). However, **these explainability methods impose zero requirement that the underlying corporate input facts $x$ be true**. In institutional investment diligence—where self-reported disclosures may contain AI-polished greenwashing—a further mathematical layer is required to determine *which factual correction to the evidence could flip the current investment decision*. - -For a linear scoring model under a fixed marginal background $Q$, the SHAP attribution assigned to feature $j$ is $\phi_j = \beta_j(x_j - E_Q X_j)$. Suppose feature $j$ carries an unresolved factual or greenwashing uncertainty bounded by $|\delta_j| \le d_j$. Its true sensitivity to factual verification is $|\beta_j|d_j$, and over a box of admissible evidence errors, the exact worst-case swing in the company's score is: - -$$\sup_{|\delta_j| \le d_j} |f(x + \delta) - f(x)| = \sum_j |\beta_j| d_j$$ - -The triangle inequality gives the upper bound, which is attained by choosing the endpoint of each error interval so that all perturbations push the score in the same direction. This identity exposes a fundamental flaw in standard XAI-driven diligence: -* A core financial metric with a massive SHAP bar $|\phi_j|$ may come from audited statutory filings that have already been thoroughly verified ($d_j \approx 0$); re-auditing it wastes diligence budget. -* Conversely, a self-reported ESG or operational metric whose current claimed value sits near the sector average ($x_j \approx E_Q X_j \implies \phi_j \approx 0$) appears completely unimportant on a SHAP chart, yet may harbor substantial unresolved greenwashing error ($d_j \gg 0$) capable of reversing the deal ranking! - -Naive bootstrap resampling of existing records fails for a parallel reason: it cannot detect a shared reporting or AI-marketing bias. Suppose $n$ reported observations satisfy $X_i = \mu + b + \varepsilon_i$, where $\varepsilon_i \sim \mathcal{N}(0, \sigma^2)$ are independent random errors and $b \neq 0$ is an unmodeled systemic bias shared across records (such as sector-wide self-reporting inflation or coordinated AI-PR sentiment manipulation). A conventional nominal 95% normal confidence interval centered on the sample mean $\bar{X}$ covers the true fundamental parameter $\mu$ with probability: - -$$\Pr\left\{ \mu \in \left[ \bar{X} - z\frac{\sigma}{\sqrt{n}}, \ \bar{X} + z\frac{\sigma}{\sqrt{n}} \right] \right\} = \Phi\left( z - \frac{b\sqrt{n}}{\sigma} \right) - \Phi\left( -z - \frac{b\sqrt{n}}{\sigma} \right) \longrightarrow 0 \quad (\text{as } n \to \infty)$$ - -where $z = \Phi^{-1}(0.975) \approx 1.96$. **Collecting more biased documents ($n \to \infty$) shrinks random variance while driving true confidence coverage to zero!** A bootstrap that resamples the same biased disclosures has no statistical leverage to identify $b$. Active evidence verification must therefore bound input uncertainty before stability analysis can provide valid decision guarantees. - -## II. Joint uncertainty over models and evidence - -To evaluate whether an ESG or investment ranking is genuinely trustworthy, our engine represents **model-selection multiplicity** and **unresolved evidence uncertainty** jointly. - -Fix the prediction target, input definitions, and output scale. Consider $M$ candidate scoring models $\mathcal{F}_0$ specified before examining an independent validation set of size $n$ with loss bounded in $[0, 1]$. For failure probability $\alpha \in (0, 1)$, Hoeffding's inequality and a union bound guarantee: - -$$\Pr\left\{ \max_{f \in \mathcal{F}_0} |\hat{L}_n(f) - L(f)| \le t_n \right\} \ge 1 - \alpha, \qquad t_n = \sqrt{\frac{\log(2M / \alpha)}{2n}}$$ - -Let $f^*$ minimize true population loss within $\mathcal{F}_0$, and retain the empirical **Rashomon set** of all near-optimal models within $2t_n$ of the empirical validation minimum: - -$$\mathcal{F}_\alpha = \left\{ f \in \mathcal{F}_0 : \hat{L}_n(f) \le \min_{g \in \mathcal{F}_0} \hat{L}_n(g) + 2t_n \right\}$$ - -On the simultaneous concentration event, $\hat{L}_n(f^*) \le L(f^*) + t_n \le \min_g \hat{L}_n(g) + 2t_n$, guaranteeing $f^* \in \mathcal{F}_\alpha$ while every retained model has population excess loss at most $4t_n$ (*Marx et al.*). - -Simultaneously, the **evidence uncertainty set** $\mathcal{X}(E)$ encodes all unresolved questions regarding entity boundaries, reporting periods, physical units, measurement definitions, shared third-party data sources, and suspected greenwashing claims under current diligence state $E$. Crucially, each element $X \in \mathcal{X}(E)$ specifies a *joint* input configuration across all candidate companies—preserving cross-company error dependencies rather than treating each company field as an isolated error bar. Defining the joint uncertainty space $\Omega(E) = \mathcal{F}_\alpha \times \mathcal{X}(E)$, we compute the exact worst-case pairwise comparison bounds for Companies $i$ and $k$: - -$$\underline{\Delta}_{ik} = \inf_{(f, X) \in \Omega(E)} [f(x_i) - f(x_k)], \qquad \overline{\Delta}_{ik} = \sup_{(f, X) \in \Omega(E)} [f(x_i) - f(x_k)]$$ - -Whenever $\underline{\Delta}_{ik} > 0$, the strict ordering $i \succ k$ is certified across every defensible model in $\mathcal{F}_\alpha$ and every admissible factual resolution in $\mathcal{X}(E)$. If calibration establishes that $\mathcal{X}(E)$ contains the true ground-truth inputs with probability at least $1 - \beta$, a union bound guarantees joint inclusion of $f^*$ and the true inputs with probability at least $1 - \alpha - \beta$—bounding the unconditional false-certificate rate by $\alpha + \beta$ without assuming independence between model validation and evidence calibration. - -## III. Computing comparison bounds with support functions - -Our joint evidence representation also makes worst-case comparison bounds computationally tractable via **convex support functions** (*Ben-Tal and Nemirovski, Robust Optimization*). Let $u \in U$ collect shared and company-specific unresolved factual quantities (including both shared benchmark errors and company-specific greenwashing tolerances), parameterizing company inputs as $x_i(u) = \bar{x}_i + B_i u$. Under affine scoring models, the pairwise score difference between Company $i$ and Company $k$ takes the form $\Delta_f(u) = m_f + a_f^\top u$, where $a_f = (B_i - B_k)^\top \beta_f$. Introducing the support function $h_U(a) = \sup_{u \in U} a^\top u$ yields exact closed-form comparison bounds: - -$$\inf_{u \in U} \Delta_f(u) = m_f - h_U(-a_f), \qquad \sup_{u \in U} \Delta_f(u) = m_f + h_U(a_f)$$ - -For an ellipsoidal uncertainty set $U = \{Cz : \|z\|_2 \le r\}$, the Cauchy-Schwarz inequality yields $h_U(a_f) = r\|C^\top a_f\|_2$; when $U$ is an $\ell_\infty$ box centered at zero, $h_U(a_f)$ reduces to a weighted sum of absolute values. - -This formulation delivers a major operational advantage over naive company-by-company confidence intervals: **Exact Cancellation of Shared-Source Errors**. If two companies evaluated by the same linear model share an identical additive error from a common industry benchmark or third-party sensor feed, then $B_i = B_k$ along those coordinates and: - -$$a_f = (B_i - B_k)^\top \beta_f = 0 \implies h_U(\pm a_f) = 0$$ - -The common error cancels identically in the pairwise comparison! Constructing separate uncertainty intervals for Company $i$ and Company $k$ and subtracting them falsely allows the shared source to vary in opposite directions simultaneously, manufacturing a spurious worst case that paralyzes investment decisions. - -For smooth nonlinear models where $\Delta_f(u)$ is twice continuously differentiable on $\|u\|_2 \le r$ with Hessian operator norm bounded by $K_f$, Taylor's remainder theorem gives the certified lower bound: - -$$|\Delta_f(u) - m_f - a_f^\top u| \le \kappa_f, \qquad \kappa_f = \frac{1}{2}K_f r^2 \implies \underline{\Delta}_f \ge m_f - r\|a_f\|_2 - \kappa_f$$ - -while non-smooth tree ensembles are bounded via leaf-region partitioning or mixed-integer linear programming, concentrating compute strictly on company pairs near the zero decision boundary. - -## IV. Choosing verification queries by the regret they can remove - -When $\underline{\Delta}_{ik} \le 0 < \overline{\Delta}_{ik}$, the ranking between candidate investments remains unresolved—indicating that suspected greenwashing or data ambiguity could flip the decision. Rather than telling the investor which company to pick, our system recommends **which specific verification query $q$ (e.g., auditing a supplier contract, requesting raw utility bills, or cross-checking a regulatory filing) to execute next**. - -Let $A$ be the finite set of institutional actions (such as advancing Company $i$ to final diligence, advancing Company $k$, or deferring rating) and $U(a, \omega)$ be the pre-specified decision utility under scenario $\omega \in \Omega$. The baseline **minimax decision regret** over deterministic actions is: - -$$R^*(\Omega) = \min_{a \in A} \sup_{\omega \in \Omega} \left[ \max_{b \in A} U(b, \omega) - U(a, \omega) \right]$$ - -Executing a verification query $q$ produces an observation $o \in O_q$ that restricts the compatible scenario set to a nonempty subset $\Omega_{q,o} \subseteq \Omega$. The **guaranteed worst-case reduction in decision regret** provided by query $q$ is: - -$$V(q) = R^*(\Omega) - \sup_{o \in O_q} R^*(\Omega_{q,o}) \ge 0$$ - -The non-negativity of $V(q)$ follows immediately from set inclusion $\Omega_{q,o} \subseteq \Omega$: shrinking the set of plausible greenwashing/evidence scenarios cannot increase worst-case regret for any fixed action, and minimizing over actions $a \in A$ preserves the inequality (*Robust Active Preference Elicitation*). Crucially, **$V(q)$ requires no subjective probability prior over whether a corporate disclosure is exaggerated**. Under a finite diligence budget $B$, ranking candidate verification queries by cost-adjusted regret reduction $V(q)/c_q$ (or solving a small batch discrete optimization over complementary queries) directs human diligence toward shared sources and decisive claims that resolve multiple company comparisons at once. - -## V. Checking that corrections propagate through attribution - -Once diligence resolves a query and updates a company's verified evidence vector from $x$ to $x'$, our pipeline guarantees that the factual correction propagates deterministically through all downstream scores, peer rankings, and SHAP explanations. Holding the scoring function $f$, background reference $Q$, and rubric version fixed, **Shapley efficiency** enforces the exact conservation identity (*SHAP local accuracy property*): - -$$f(x) = \phi_0 + \sum_j \phi_j(f, Q, x), \qquad \sum_j [\phi_j(f, Q, x') - \phi_j(f, Q, x)] = f(x') - f(x)$$ - -If approximate Shapley algorithms carry numerical summation tolerances bounded by $\eta$ and $\eta'$ before and after correction, our automated recomputation gate verifies $|\sum_j \Delta \phi_j - \Delta f| \le \eta + \eta'$. Every database update logs explicit version identifiers for `(Field Version, Evidence Document ID, Model Version, Background Q, Rubric Version)` so that changes driven by factual diligence corrections are never conflated with model retraining. - -## VI. Reproducible quantitative comparisons - -Three reproducible analytical experiments verify the mathematical mechanisms derived above: - -1. **Collapse of Conventional Confidence Intervals Under Shared Greenwashing / Reporting Bias**: - * Set shared reporting bias $b = 0.2$ and independent noise $\sigma = 1$. Ignoring the common bias $b$, a nominal 95% confidence interval achieves actual coverage of only **48.40% at $n = 100$ records**, collapsing to **0.000637% at $n = 1,000$ records**. When targeted verification establishes the bias bound $|b| \le 0.2$, expanding the interval half-width to $0.2 + 1.96/\sqrt{n}$ restores **97.50% actual coverage**. -2. **Shared-Source Error Cancellation in Pairwise Company Ranking**: - * Two target companies have a nominal score difference of $0.6$ points and share an additive sector-benchmark error in $[-1, 1]$. Subtracting independently constructed company intervals gives **$[-1.4, 2.6]$**, failing to certify an ordering. Our joint support-function model cancels the incompatible error combinations identically, yielding **$[0.6, 0.6]$** and certifying the ranking. -3. **Minimax-Regret Verification Routing vs. Naive SHAP Attribution Ranking**: - * Consider one verification query of equal cost across 8 scenarios formed from 2 Rashomon-set candidate models ($s \in \{-1, 1\}$) and 4 unresolved evidence configurations ($z_1 \in \{-0.4, 0.4\}$, $z_2 \in \{-0.2, 0.2\}$), with pairwise score difference $\Delta_s = 0.15s + z_1 + 0.3s z_2$. A further shared feature has a large SHAP attribution but has already been verified ($d_0 = 0$). The decision action is to select Company $i$ or Company $k$, with regret measured in model-score units: - -| Verification Query | Remaining Regret After Worst Possible Observation | Guaranteed Reduction in Regret $V(q)$ | -| :--- | :--- | :--- | -| **Recheck the already-resolved field with a large SHAP attribution** | 0.61 | **0** | -| **Verify the smaller unresolved variable $z_2$** | 0.61 | **0** | -| **Verify $z_1$, which determines the sign of the score difference** | **0** | **0.61** | - -Enumerating all eight scenarios proves that ranking diligence work by raw SHAP magnitude wastes 100% of the query budget ($V(q) = 0$), whereas our minimax-regret criterion isolates $z_1$ and reduces worst-case decision regret from **0.61 to zero** in a single query. By uniting **Rashomon model multiplicity**, **support-function evidence bounds**, and **minimax-regret verification routing**, this architecture provides institutional investors with a rigorous, auditable defense against AI-era greenwashing. diff --git a/website/content/visible-copy-manifest.json b/website/content/visible-copy-manifest.json index 8a82a8d..4e64900 100644 --- a/website/content/visible-copy-manifest.json +++ b/website/content/visible-copy-manifest.json @@ -744,9 +744,9 @@ "formulaCount": 965 }, "/experience/": { - "textSha256": "a898405ebfde962833e13dab533e281084fb777b9a61b3f3f20327418d34a23b", - "characterCount": 12695, - "wordCount": 1619, + "textSha256": "b189dc1cb7b7e5ca6370402a1fe1ed772cd0d2bd95e423c5f907d66b1c1903f5", + "characterCount": 12701, + "wordCount": 1620, "preview": "Skip to main content Theodore Ouyang About Experience Experience 01 AI Research and Engineering 02 Data Science 03 Legal Research and Policy Analysis 04 Finance", "altTexts": [], "ariaLabels": [ @@ -994,9 +994,9 @@ } }, "/past-experience/": { - "textSha256": "a898405ebfde962833e13dab533e281084fb777b9a61b3f3f20327418d34a23b", - "characterCount": 12695, - "wordCount": 1619, + "textSha256": "b189dc1cb7b7e5ca6370402a1fe1ed772cd0d2bd95e423c5f907d66b1c1903f5", + "characterCount": 12701, + "wordCount": 1620, "preview": "Skip to main content Theodore Ouyang About Experience Experience 01 AI Research and Engineering 02 Data Science 03 Legal Research and Policy Analysis 04 Finance", "altTexts": [], "ariaLabels": [ @@ -1034,9 +1034,9 @@ } }, "/past-experience/artificial-intelligence/": { - "textSha256": "2e80618aebe1f4ae456b4921d2a0db0b6d817d6978ee1f5668896724ae0d948e", - "characterCount": 6835, - "wordCount": 872, + "textSha256": "76f26ad93d7038ddb3ee5a13033114d704a0c01a98c22a143eebeed868aff4b0", + "characterCount": 6841, + "wordCount": 873, "preview": "Skip to main content Theodore Ouyang About Experience Experience AI Research and Engineering Confidentiality & Disclosure Note The following four research and e", "altTexts": [], "ariaLabels": [ @@ -1396,9 +1396,9 @@ } }, "/past-experience/legal-research-and-policy-analysis/autonomous-authority-in-space/": { - "textSha256": "660e36b3f4f681c522eb1135466647b4eb5a908fae76a1848b8758e0a653800c", - "characterCount": 65617, - "wordCount": 9201, + "textSha256": "1496b1f3c5aa218da9b981c7bbe73cc3fd12e8338405400a306b51574a7c8d27", + "characterCount": 65609, + "wordCount": 9193, "preview": "Skip to main content Theodore Ouyang About Experience Experience Duke Law School: Autonomous Authority in Space — Risk Tradeoffs and the Law of Delegation Posit", "altTexts": [], "ariaLabels": [ @@ -1480,9 +1480,9 @@ } }, "/past-experience/legal-research-and-policy-analysis/small-states-and-strategic-space-dependence/": { - "textSha256": "3500e10e2c6b787402119c552e0783a8d797582468b99e871faf718fb982c0c2", - "characterCount": 49684, - "wordCount": 6803, + "textSha256": "3950c945d4b5fdcfaa990d250408bdc1fe519b7b8b0113d2b7d2a9e0bde08a02", + "characterCount": 49677, + "wordCount": 6796, "preview": "Skip to main content Theodore Ouyang About Experience Experience Duke Law School: Small States and the Governance of Strategic Space Dependence Position Author ", "altTexts": [], "ariaLabels": [ @@ -1568,9 +1568,9 @@ } }, "/past-experience/legal-research-and-policy-analysis/solar-geoengineering-comparison-and-continuity/": { - "textSha256": "f7a66943b3ca3e46e798f04604f62833919d440cbe65c704e98644eec58e7399", - "characterCount": 47448, - "wordCount": 6608, + "textSha256": "7c4d0b1347f8ff2e88cd6eaf6add992a528654a4152cbaf5cf4b0c32a05ce112", + "characterCount": 47439, + "wordCount": 6599, "preview": "Skip to main content Theodore Ouyang About Experience Experience Duke Law School: Who May Choose the Lesser Risk — Solar Geoengineering and the Legal Duties of ", "altTexts": [], "ariaLabels": [ @@ -1652,9 +1652,9 @@ } }, "/past-experience/legal-research-and-policy-analysis/mangrove-restoration-and-compensatory-mitigation/": { - "textSha256": "0e6125d2b54688fbab993b6d9b823cf91d2e3a6e8be37ea8ca1fb90dc8816283", - "characterCount": 61743, - "wordCount": 8626, + "textSha256": "3065bf4ab6d152eef2837b599c5bc10679954f4d9837d843905828ae470c553b", + "characterCount": 61732, + "wordCount": 8615, "preview": "Skip to main content Theodore Ouyang About Experience Experience Duke Law School: Mangrove Restoration and the Limits of Compensatory Mitigation — Lessons from ", "altTexts": [], "ariaLabels": [ @@ -1758,9 +1758,9 @@ } }, "/now/": { - "textSha256": "a898405ebfde962833e13dab533e281084fb777b9a61b3f3f20327418d34a23b", - "characterCount": 12695, - "wordCount": 1619, + "textSha256": "b189dc1cb7b7e5ca6370402a1fe1ed772cd0d2bd95e423c5f907d66b1c1903f5", + "characterCount": 12701, + "wordCount": 1620, "preview": "Skip to main content Theodore Ouyang About Experience Experience 01 AI Research and Engineering 02 Data Science 03 Legal Research and Policy Analysis 04 Finance", "altTexts": [], "ariaLabels": [ diff --git a/website/scripts/check-protected-sources.mjs b/website/scripts/check-protected-sources.mjs index 8e77ca3..fc0ddf8 100644 --- a/website/scripts/check-protected-sources.mjs +++ b/website/scripts/check-protected-sources.mjs @@ -46,9 +46,7 @@ const PROTECTED_PATHS = [ "content/artificial-intelligence/task-validity-in-financial-synthetic-data.json", "content/artificial-intelligence/verification-and-supervision-in-scientific-reasoning-tasks.json", "content/artificial-intelligence/evidence-uncertainty-and-decision-guarantees-in-investment-research.json", - "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md", - "content/sources/ai-portfolio/citation-links.json", - "content/sources/ai-portfolio/editorial-revisions.json", + "content/sources/ai-portfolio/ai-research-and-engineering.md", "scripts/import-ai-portfolio.mjs", ]; diff --git a/website/scripts/import-ai-portfolio.mjs b/website/scripts/import-ai-portfolio.mjs index 792bf07..aaad23f 100644 --- a/website/scripts/import-ai-portfolio.mjs +++ b/website/scripts/import-ai-portfolio.mjs @@ -1,157 +1,121 @@ import assert from "node:assert/strict"; import { createHash } from "node:crypto"; import { readFileSync, writeFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; import path from "node:path"; +import { fileURLToPath } from "node:url"; const root = fileURLToPath(new URL("../", import.meta.url)); -const sourceFile = "content/sources/ai-portfolio/theodore_ouyang_ai_portfolio_optimized.md"; -const sourceBytes = readFileSync(path.join(root, sourceFile)); -const source = sourceBytes.toString("utf8").replace(/\r\n?/gu, "\n"); -const links = JSON.parse(readFileSync(path.join(root, "content/sources/ai-portfolio/citation-links.json"), "utf8")); -const revisions = JSON.parse(readFileSync(path.join(root, "content/sources/ai-portfolio/editorial-revisions.json"), "utf8")); -assert.equal(revisions.schemaVersion, 1, "Unsupported AI editorial revision schema"); -assert.equal(revisions.sourceFile, sourceFile, "Revisions must target the frozen AI manuscript"); -assert.equal(createHash("sha256").update(source).digest("hex"), revisions.sourceSha256, "The frozen manuscript changed"); -const showcasePath = path.join(root, "content/experience-showcase.json"); -const showcase = JSON.parse(readFileSync(showcasePath, "utf8")); -const beforeOtherEntries = JSON.stringify(showcase.entries.slice(4)); -const generated = new Map([["content/experience-showcase.json", showcase]]); +const sourceFile = "content/sources/ai-portfolio/ai-research-and-engineering.md"; +const source = readFileSync(path.join(root, sourceFile), "utf8").replace(/\r\n?/gu, "\n"); +const projects = [ + ["statistical-inference-and-resource-allocation-in-expert-data-production", 7], + ["task-validity-in-financial-synthetic-data", 6], + ["verification-and-supervision-in-scientific-reasoning-tasks", 7], + ["evidence-uncertainty-and-decision-guarantees-in-investment-research", 6], +]; const sourceDocument = { fileName: path.basename(sourceFile), path: sourceFile, - sha256: createHash("sha256").update(sourceBytes).digest("hex"), + sha256: createHash("sha256").update(source).digest("hex"), }; +const showcasePath = path.join(root, "content/experience-showcase.json"); +const showcase = JSON.parse(readFileSync(showcasePath, "utf8")); +const otherEntries = JSON.stringify(showcase.entries.slice(projects.length)); +const generated = new Map([["content/experience-showcase.json", showcase]]); -function capture(text, expression, label) { - const match = text.match(expression); - assert.ok(match, `Missing ${label}`); - return match[1].trim(); -} - -function restoreLinks(markdown, definitions) { - const escaped = definitions.map(({ text }) => text.replace(/[.*+?^${}()|[\]\\]/gu, "\\$&")); - const byText = new Map(definitions.map(({ text, href }) => [text, href])); - // One replacement pass prevents a short label from matching inside a longer link. - return markdown.replace(new RegExp(escaped.join("|"), "gu"), (text) => `[${text}](${byText.get(text)})`); -} - -function manuscript(markdown, definitions) { - const numerals = ["I", "II", "III", "IV", "V", "VI", "VII"]; - let result = restoreLinks(markdown, definitions) - .replace(/^## ([IVX]+)\. /gmu, (_, numeral) => `## ${numerals.indexOf(numeral) + 1} `) - .replace(/^\s*\* /gmu, "• "); - // Use the site's existing equation syntax while retaining every TeX expression. - result = result.replace(/\$\$([\s\S]*?)\$\$|(? display === undefined ? `$\`${inline}\`$` : `$$\n${display.trim()}\n$$`); - result = result.replace(/^\|[^\n]+\|\n\|[ :|-]+\|\n(?:\|[^\n]+\|(?:\n|$))+/gmu, (table) => { - const rows = table.trim().split("\n").map((row) => row.slice(1, -1).split("|").map((cell) => cell.trim())); - assert.ok(rows[1].every((cell) => /^:?-+:?$/u.test(cell)), "Expected table header separator"); - rows.splice(1, 1); - assert.ok(rows.every((row) => row.length === rows[0].length), "Expected rectangular table"); - return `\n${rows.map((row) => `${row.map((cell) => ``).join("")}`).join("\n")}\n
${cell}
\n`; - }); - return result.trim(); +function one(text, expression, label) { + const matches = [...text.matchAll(expression)]; + assert.equal(matches.length, 1, `Expected exactly one ${label}`); + return matches[0]; } -function contributionDetails(contributions) { +function contributions(text) { + const items = []; const details = {}; - let parent = -1; - for (const line of contributions.split("\n")) { - if (line.startsWith("* ")) parent += 1; + for (const line of text.split("\n")) { + if (!line.trim()) continue; + if (line.startsWith("* ")) { + assert.ok(line.slice(2).trim(), "Contributions cannot be empty"); + items.push(line.slice(2)); + continue; + } const child = line.match(/^ {2}(\*|\d+\.) (.+)$/u); - if (!child) continue; - assert.ok(parent >= 0, "Nested contributions require a parent contribution"); + assert.ok(child && items.length, `Invalid contribution list line: ${line}`); + const parent = items.length - 1; const ordered = child[1] !== "*"; details[parent] ??= { ordered, items: [] }; - assert.equal(details[parent].ordered, ordered, "A contribution detail list cannot mix list types"); - if (ordered) assert.equal(Number.parseInt(child[1], 10), details[parent].items.length + 1, "Ordered details must retain their source sequence"); + assert.equal(details[parent].ordered, ordered, "Contribution details cannot mix list types"); + if (ordered) assert.equal(Number.parseInt(child[1], 10), details[parent].items.length + 1, "Contribution details must be sequential"); details[parent].items.push(child[2]); } - return details; -} - -function applyEditorialRevisions() { - const ids = new Set(); - for (const revision of revisions.revisions) { - assert.ok(!ids.has(revision.id), `Duplicate revision ${revision.id}`); - ids.add(revision.id); - assert.ok(revision.reason && revision.sourceRefs.length, `Revision ${revision.id} requires a reason and source references`); - assert.ok(revision.before && revision.after && revision.before !== revision.after, `Empty revision ${revision.id}`); - const target = generated.get(revision.file); - assert.ok(target, `Revision ${revision.id} is outside the four AI articles and showcase`); - if (revision.file === "content/experience-showcase.json") { - assert.match(revision.jsonPointer, /^\/entries\/[0-3]\/(?:summary|keyContributions\/\d+|architecturalContext\/(?:title|text)|keyContributionDetails\/\d+\/items\/\d+)$/u, `Revision ${revision.id} targets unapproved entry fields`); - } else { - assert.equal(revision.jsonPointer, "/markdown", "Article revisions may change manuscript text only"); - } - const keys = revision.jsonPointer.slice(1).split("/"); - let parent = target; - for (const key of keys.slice(0, -1)) { - assert.ok(Object.hasOwn(parent, key), `Revision ${revision.id} targets a missing field`); - parent = parent[key]; - } - const key = keys.at(-1); - assert.ok(Object.hasOwn(parent, key) && typeof parent[key] === "string", `Revision ${revision.id} must target text`); - assert.equal(parent[key].split(revision.before).length - 1, 1, `Revision ${revision.id} must match exactly once`); - parent[key] = parent[key].replace(revision.before, () => revision.after); - assert.equal(parent[key].split(revision.after).length - 1, 1, `Revision ${revision.id} must produce one exact reviewed fragment`); - } + assert.ok(items.length >= 5, "A project must include its complete key contributions"); + return { items, details }; } +const boundaries = [...source.matchAll(/^# Project (\d{2}): ([a-z0-9-]+)$/gmu)]; +assert.equal(boundaries.length, projects.length, "Expected four complete AI projects"); +const introduction = source.slice(0, boundaries[0].index); +const intro = one(introduction, /^# AI Research and Engineering\n\n\*\*Author\*\*\n([^\n]+)\n\n\*\*Updated\*\*\n(\d{4}-\d{2}-\d{2})\n\n## (Confidentiality & Disclosure Note)\n\n([^\n]+)\n\n$/gu, "portfolio introduction"); showcase.aiSourceDocument = sourceDocument; -showcase.categories.find(({ id }) => id === "01").disclosure = { - title: "Confidentiality & Disclosure Note", - text: capture(source, /^> (The following[^\n]+)$/mu, "index disclosure"), -}; -const briefs = [...source.slice(source.indexOf("# PART 1:"), source.indexOf("# PART 2 & PART 3:")).matchAll(/^### ([^\n]+)\n([^\n]+)\n\n\[Read More >\]/gmu)]; -const pages = source.split(/^# SUBPAGE \d{2}: [^\n]+\n/gmu).slice(1); -assert.equal(briefs.length, 4); -assert.equal(pages.length, 4); -for (let index = 0; index < 4; index += 1) { - const page = pages[index]; - const url = capture(page, /\*\*URL\*\*: `([^`]+)`/u, "page URL"); - const slug = new URL(url).pathname.split("/").filter(Boolean).at(-1); - const articlePath = path.join(root, "content/artificial-intelligence", `${slug}.json`); - const article = JSON.parse(readFileSync(articlePath, "utf8")); +const category = showcase.categories.find(({ id }) => id === "01"); +assert.ok(category, "Missing AI showcase category"); +category.disclosure = { title: intro[3], text: intro[4] }; + +for (const [index, [slug, chapterCount]] of projects.entries()) { + const boundary = boundaries[index]; + assert.equal(Number(boundary[1]), index + 1, "Projects must retain their directory order"); + assert.equal(boundary[2], slug, "Unexpected AI project route"); + const page = source.slice(boundary.index + boundary[0].length, boundaries[index + 1]?.index).trim(); + const parts = page.split("\n## Manuscript\n"); + assert.equal(parts.length, 2, `Expected one manuscript for ${slug}`); + const body = parts[1].trim(); + const chapters = [...body.matchAll(/^## (\d+) [^\n]+$/gmu)]; + assert.deepEqual(chapters.map((chapter) => Number(chapter[1])), Array.from({ length: chapterCount }, (_, i) => i + 1), `Incomplete chapter sequence for ${slug}`); + assert.ok(body.startsWith("## 1 "), "The manuscript must start at its first chapter"); + + const sectionHeadings = [...parts[0].matchAll(/^## ([^\n]+)$/gmu)]; + assert.equal(sectionHeadings.length, 4, `Incomplete project sections for ${slug}`); + assert.deepEqual(sectionHeadings.slice(0, 3).map((heading) => heading[1]), ["Summary", "Confidentiality & Scope Disclosure", "Key Contributions"]); + assert.match(sectionHeadings[3][1], /^Architectural Context: .+/u); + const sections = sectionHeadings.map((heading, i) => parts[0].slice(heading.index + heading[0].length, sectionHeadings[i + 1]?.index).trim()); + assert.ok(sections.every(Boolean), `Empty project section for ${slug}`); + + const metadataText = parts[0].slice(0, sectionHeadings[0].index); + const metadataPattern = /^\*\*([^*\n]+)\*\*\n([^\n]+)$/gmu; + const metadataFields = [...metadataText.matchAll(metadataPattern)]; + assert.deepEqual(metadataFields.map((field) => field[1]), ["Article Title", "Organization", "Title", "URL", "Position", "Location", "Dates", "Read More Label"], `Incomplete metadata for ${slug}`); + assert.equal(metadataText.replace(metadataPattern, "").trim(), "", `Unexpected metadata content for ${slug}`); + const metadata = Object.fromEntries(metadataFields.map((field) => [field[1], field[2]])); + assert.equal(metadata.URL, `https://www.theodoreoy.com/past-experience/artificial-intelligence/${slug}/`, "The source URL must match the fixed project route"); const entry = showcase.entries[index]; - entry.organization = briefs[index][1]; - entry.summary = briefs[index][2]; - entry.detailTitle = capture(page, /^## Duke University[^:\n]+: ([^\n]+)$/mu, "page title"); - for (const field of ["Position", "Location", "Dates"]) { - entry.metadata[field.toLowerCase()] = capture(page, new RegExp(`\\*\\*${field}\\*\\*\\n([^\\n]+)`, "u"), field); - } - entry.disclosure = { - title: "Confidentiality & Scope Disclosure", - text: capture(page, /\*\*Confidentiality & Scope Disclosure\*\*\n([^\n]+)/u, "page disclosure"), - }; - const contributions = capture(page, /## Key Contributions\n([\s\S]+?)\n## Architectural Context:/u, "contributions"); - entry.keyContributions = contributions.split("\n").filter((line) => line.startsWith("* ")).map((line) => line.slice(2)); - assert.ok(entry.keyContributions.length >= 5); - const details = contributionDetails(contributions); - if (Object.keys(details).length) entry.keyContributionDetails = details; + assert.equal(entry.id, index + 1); + assert.equal(entry.categoryId, "01"); + entry.organization = metadata.Organization; + entry.summary = sections[0]; + entry.detailTitle = metadata.Title; + entry.readMoreLabel = metadata["Read More Label"]; + entry.metadata = { position: metadata.Position, location: metadata.Location, dates: metadata.Dates }; + entry.disclosure = { title: sectionHeadings[1][1], text: sections[1] }; + entry.contributionsLabel = sectionHeadings[2][1]; + const parsedContributions = contributions(sections[2]); + entry.keyContributions = parsedContributions.items; + if (Object.keys(parsedContributions.details).length) entry.keyContributionDetails = parsedContributions.details; else delete entry.keyContributionDetails; - entry.architecturalContext = { - title: capture(page, /^## (Architectural Context:[^\n]+)$/mu, "architectural title"), - text: capture(page, /^## Architectural Context:[^\n]+\n([\s\S]+?)\n## Contents/mu, "architectural context"), - }; + entry.architecturalContext = { title: sectionHeadings[3][1], text: sections[3] }; entry.notes = []; - let body = page.slice(page.indexOf("## I. ")); - assert.ok(body.startsWith("## I. ")); - if (index === 3) { - // Retain these two definitions from the preceding published manuscript. - body = body.replace("For smooth nonlinear models where", String.raw`Set $m_f=\Delta_f(0)$ and $a_f=\nabla\Delta_f(0)$. -For smooth nonlinear models where`); - } - article.markdown = manuscript(body, links[slug]); + const articleFile = `content/artificial-intelligence/${slug}.json`; + const article = JSON.parse(readFileSync(path.join(root, articleFile), "utf8")); + assert.equal(article.slug, slug); + article.title = metadata["Article Title"]; + article.markdown = body; article.sourceUrl = `https://github.com/130U/130U.github.io/blob/main/website/${sourceFile}`; - article.sourceEditedAt = "2026-10-27"; + article.sourceEditedAt = intro[2]; article.sourceDocument = sourceDocument; - generated.set(`content/artificial-intelligence/${slug}.json`, article); + generated.set(articleFile, article); } -applyEditorialRevisions(); -assert.equal(JSON.stringify(showcase.entries.slice(4)), beforeOtherEntries, "Only the four AI entries may change"); -// Validate every revision before writing any generated content. + +assert.equal(JSON.stringify(showcase.entries.slice(projects.length)), otherEntries, "Only AI entries may change"); +// Validate the complete source before writing any generated content. for (const [file, document] of generated) writeFileSync(path.join(root, file), `${JSON.stringify(document, null, 2)}\n`); -console.log(`Imported four AI manuscripts and all nested contributions; applied ${revisions.revisions.length} reviewed editorial revisions.`); +console.log("Imported the four AI projects from the canonical manuscript."); diff --git a/website/scripts/integrate-architecture.mjs b/website/scripts/integrate-architecture.mjs index 2355bd2..a4c52b6 100644 --- a/website/scripts/integrate-architecture.mjs +++ b/website/scripts/integrate-architecture.mjs @@ -106,6 +106,11 @@ function applyReaderLayout(viewer) { return viewer.replace("", `${readerRules}\n`); } +function applyExportTypography(viewer) { + const rule = "svg { font-family: 'Inter', sans-serif; font-optical-sizing: none; font-variation-settings: 'opsz' 14; font-variant-numeric: lining-nums proportional-nums; font-synthesis: none; }"; + return viewer.replace(/svg \{ font-family: '(?:Inter|JetBrains Mono)'[^}]+\}/gu, rule); +} + function applyViewerLabels(viewer) { const labels = { "viewer.owner.route": "route probe", @@ -162,6 +167,7 @@ source = source.replace(/(\.semantic-passport-reach-actions strong\s*\{[^}]*font source = source.replace(/(\.semantic-passport-reach-actions strong\s*\{[^}]*line-height:)\s*[^;]+;/gu, "$1 var(--interface-leading);"); source = applyViewerLabels(source); +source = applyExportTypography(source); if (source.includes("data-site-typography")) { assert.doesNotMatch(source, /fonts\.googleapis|fonts\.gstatic|JetBrains|Georgia|Times New Roman/u); @@ -258,8 +264,6 @@ if (source.includes("data-site-typography")) { viewer = replaceOnce(viewer, / \/\/ Prepend a local\(\)-only[\s\S]*? var style =/u, " var fontFaces = exportFontCss;\n\n var style ="); viewer = viewer.replaceAll("fontFallback +", "fontFaces +"); - viewer = viewer.replace(/svg \{ font-family: 'JetBrains Mono'[^}]+\}/gu, - "svg { font-family: 'Inter', sans-serif; font-optical-sizing: auto; font-synthesis: none; }"); viewer = viewer.replace(/'JetBrains Mono', ui-monospace, SFMono-Regular, Menlo, Consolas, monospace/gu, "'Inter', sans-serif"); for (const [name, argument] of [["rasterize", "format"], ["renderShareCard", "options"], ["recordWebm", "options"]]) { diff --git a/website/tests/ai-research-content.test.mjs b/website/tests/ai-research-content.test.mjs index 2c0cdd2..531c1a5 100644 --- a/website/tests/ai-research-content.test.mjs +++ b/website/tests/ai-research-content.test.mjs @@ -7,9 +7,9 @@ import { fileURLToPath } from "node:url"; const ROOT = fileURLToPath(new URL("../", import.meta.url)); const AI_ROUTE = "/past-experience/artificial-intelligence/"; -const APPROVED_DOCUMENT = path.join(ROOT, "content", "sources", "ai-portfolio", "theodore_ouyang_ai_portfolio_optimized.md"); -const APPROVED_REVISIONS = path.join(ROOT, "content", "sources", "ai-portfolio", "editorial-revisions.json"); -const REVISIONS_SHA256 = "6171190332fee0deae11d95e60f3278f885bfac7cb75093ab3608a299e9cf3c5"; +const SOURCE_FILE = "content/sources/ai-portfolio/ai-research-and-engineering.md"; +const CANONICAL_DOCUMENT = path.join(ROOT, SOURCE_FILE); +const SOURCE_SHA256 = "bc688374f1ca56e7c5eafbd06a7f33e6d7d752f4c8d0dd98b05f30164921405d"; // Structure and benchmark-table shapes are checked independently of the renderer. const PROJECTS = [ { @@ -68,57 +68,58 @@ function readableMarkdown(markdown) { .replace(/<[^>]+>/gu, " ")); } -function comparableManuscript(markdown) { - return readableMarkdown(markdown - .replace(/^## (?:[IVXLCDM]+\.|\d+\.?) /gmu, "## ") - .replace(/^\|\s*:?-+[:\s|\-]*\|\s*$/gmu, "") - .replace(/^\s*(?:\*|•) /gmu, "") - .replace(/^\|(.+)\|$/gmu, (_, row) => row.replaceAll("|", " "))); -} - -async function approvedPortfolio() { - const raw = await readFile(APPROVED_DOCUMENT); - const document = raw.toString("utf8").replace(/\r\n?/gu, "\n"); - assert.equal(createHash("sha256").update(document).digest("hex"), "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283", "The frozen copy must be the exact user-approved manuscript, allowing Git line-ending normalization"); - const projects = document.split(/^# SUBPAGE \d{2}: .+$/gmu).slice(1); - assert.equal(projects.length, 4); - return { document, projects }; -} - -async function approvedRevisions() { - const text = (await readFile(APPROVED_REVISIONS, "utf8")).replace(/\r\n?/gu, "\n"); - assert.equal(createHash("sha256").update(text).digest("hex"), REVISIONS_SHA256, "The exact independently reviewed revision manifest must remain frozen"); - const manifest = JSON.parse(text); - assert.equal(manifest.sourceSha256, "211f94a29754ca89b196f5de9fec47a990b95d587c2b5c5d89acfeabb775d283"); - assert.equal(manifest.revisions.length, 52); - assert.equal(new Set(manifest.revisions.map(({ id }) => id)).size, 52); - return manifest.revisions; -} - -function restoreFrozenCopy(file, document, revisions) { - const restored = structuredClone(document); - for (const revision of revisions.filter((item) => item.file === file).toReversed()) { - const keys = revision.jsonPointer.slice(1).split("/"); - let parent = restored; - for (const key of keys.slice(0, -1)) { - assert.ok(Object.hasOwn(parent, key), `Missing reviewed field for ${revision.id}`); - parent = parent[key]; - } - const key = keys.at(-1); - assert.equal(typeof parent[key], "string"); - assert.equal(parent[key].split(revision.after).length - 1, 1, `The exact approved text for ${revision.id} must appear once`); - parent[key] = parent[key].replace(revision.after, () => revision.before); - } - return restored; +async function canonicalPortfolio() { + const document = (await readFile(CANONICAL_DOCUMENT, "utf8")).replace(/\r\n?/gu, "\n"); + assert.equal(createHash("sha256").update(document).digest("hex"), SOURCE_SHA256, "The canonical manuscript must match its approved content hash after LF normalization"); + const parts = document.split(/^# Project (\d{2}): ([a-z0-9-]+)$/gmu); + assert.equal(parts.length, 13, "The canonical portfolio must contain exactly four projects"); + const projects = PROJECTS.map(({ slug }, index) => { + assert.equal(Number(parts[index * 3 + 1]), index + 1); + assert.equal(parts[index * 3 + 2], slug); + return parts[index * 3 + 3]; + }); + return { + projects, + sourceDocument: { fileName: path.basename(SOURCE_FILE), path: SOURCE_FILE, sha256: SOURCE_SHA256 }, + updated: parts[0].match(/\*\*Updated\*\*\n(\d{4}-\d{2}-\d{2})/u)?.[1], + disclosure: { + title: "Confidentiality & Disclosure Note", + text: parts[0].split("## Confidentiality & Disclosure Note\n")[1].trim(), + }, + }; } -function frozenContributionDetails(project) { - const section = project.split("## Key Contributions\n")[1].split("\n## Architectural Context:")[0]; - const parents = section.split(/^\* /gmu).slice(1); - return Object.fromEntries(parents.flatMap((parent, index) => { +function canonicalEntry(project, index) { + const field = (label) => { + const value = project.match(new RegExp(`\\*\\*${label}\\*\\*\\n([^\\n]+)`, "u"))?.[1]; + assert.ok(value, `Missing canonical field ${label}`); + return value; + }; + const contributions = project.split("## Key Contributions\n")[1].split("\n## Architectural Context:")[0]; + const parents = contributions.split(/^\* /gmu).slice(1); + const details = Object.fromEntries(parents.flatMap((parent, i) => { const lines = [...parent.matchAll(/^ {2}(\*|\d+\.) (.+)$/gmu)]; - return lines.length ? [[index, { ordered: lines[0][1] !== "*", items: lines.map((line) => line[2]) }]] : []; + return lines.length ? [[i, { ordered: lines[0][1] !== "*", items: lines.map((line) => line[2]) }]] : []; })); + const entry = { + id: index + 1, + categoryId: "01", + organization: field("Organization"), + summary: project.split("## Summary\n")[1].split("\n## Confidentiality & Scope Disclosure")[0].trim(), + readMoreLabel: field("Read More Label"), + detailTitle: field("Title"), + metadata: { position: field("Position"), location: field("Location"), dates: field("Dates") }, + contributionsLabel: "Key Contributions", + keyContributions: parents.map((parent) => parent.split("\n")[0]), + notes: [], + disclosure: { title: "Confidentiality & Scope Disclosure", text: project.split("## Confidentiality & Scope Disclosure\n")[1].split("\n## Key Contributions")[0].trim() }, + architecturalContext: { + title: project.match(/^## (Architectural Context:[^\n]+)$/mu)?.[1], + text: project.match(/^## Architectural Context:[^\n]+\n([\s\S]+?)\n## Manuscript/mu)?.[1].trim(), + }, + }; + if (Object.keys(details).length) entry.keyContributionDetails = details; + return { entry, articleTitle: field("Article Title"), url: field("URL") }; } function readableHtml(html) { @@ -237,44 +238,34 @@ function tableCells(html, source) { } for (const expected of PROJECTS) { - test(`AI project preserves the frozen manuscript, reviewed corrections, and complete mathematical rendering: ${expected.slug}`, async () => { - const { document, projects } = await approvedPortfolio(); - const revisions = await approvedRevisions(); - const approvedProject = projects[PROJECTS.indexOf(expected)]; - let approvedBody = approvedProject.slice(approvedProject.indexOf("## I. ")).trim(); - if (PROJECTS.indexOf(expected) === 3) { - // The preceding published article defines these two symbols; the revised - // document uses them without definitions. Preserve that original math. - approvedBody = approvedBody.replace("For smooth nonlinear models where", String.raw`Set $m_f=\Delta_f(0)$ and $a_f=\nabla\Delta_f(0)$. - -For smooth nonlinear models where`); - } + test(`AI project preserves the canonical manuscript and complete mathematical rendering: ${expected.slug}`, async () => { + const { projects, sourceDocument, updated, disclosure } = await canonicalPortfolio(); + const index = PROJECTS.indexOf(expected); + const project = projects[index]; + const bodyParts = project.split("\n## Manuscript\n"); + assert.equal(bodyParts.length, 2); + const canonicalBody = bodyParts[1].trim(); + const canonical = canonicalEntry(project, index); const source = JSON.parse(await readFile(path.join(ROOT, "content", "artificial-intelligence", `${expected.slug}.json`), "utf8")); const showcaseFile = JSON.parse(await readFile(path.join(ROOT, "content", "experience-showcase.json"), "utf8")); - const showcase = showcaseFile.entries.find(({ id }) => id === PROJECTS.indexOf(expected) + 1); - const originalShowcase = restoreFrozenCopy("content/experience-showcase.json", showcaseFile, revisions).entries[PROJECTS.indexOf(expected)]; - const originalArticle = restoreFrozenCopy(`content/artificial-intelligence/${expected.slug}.json`, source, revisions); + const showcase = showcaseFile.entries.find(({ id }) => id === index + 1); const html = await readFile(path.join(ROOT, "out", ...AI_ROUTE.split("/").filter(Boolean), expected.slug, "index.html"), "utf8"); - const briefs = [...document.slice(document.indexOf("# PART 1:"), document.indexOf("# PART 2 & PART 3:")).matchAll(/^### ([^\n]+)\n([^\n]+)\n\n\[Read More >\]/gmu)]; - const index = PROJECTS.indexOf(expected); - assert.equal(showcase.organization, briefs[index][1]); - assert.equal(originalShowcase.summary, briefs[index][2], "Reversing only the reviewed edits must recover the complete frozen brief"); - for (const field of ["Position", "Location", "Dates"]) { - assert.equal(showcase.metadata[field.toLowerCase()], approvedProject.match(new RegExp(`\\*\\*${field}\\*\\*\\n([^\\n]+)`, "u"))?.[1]); - } - const approvedContributions = approvedProject.split("## Key Contributions\n")[1].split("\n## Architectural Context:")[0].split("\n").filter((line) => line.startsWith("* ")).map((line) => line.slice(2)); - assert.deepEqual(originalShowcase.keyContributions, approvedContributions, "Every frozen contribution must survive except the exact reviewed revisions"); - const approvedDetails = frozenContributionDetails(approvedProject); - assert.deepEqual(originalShowcase.keyContributionDetails ?? {}, approvedDetails, "All original nested contributions must survive in order, except the exact reviewed revisions"); - assert.equal(Object.values(approvedDetails).reduce((count, detail) => count + detail.items.length, 0), [0, 4, 3, 0][index]); - assert.equal(showcase.disclosure.title, "Confidentiality & Scope Disclosure"); - assert.equal(showcase.disclosure.text, approvedProject.match(/\*\*Confidentiality & Scope Disclosure\*\*\n([^\n]+)/u)?.[1]); - assert.equal(originalShowcase.architecturalContext.title, approvedProject.match(/^## (Architectural Context:[^\n]+)$/mu)?.[1]); - assert.equal(originalShowcase.architecturalContext.text, approvedProject.match(/^## Architectural Context:[^\n]+\n([\s\S]+?)\n## Contents/mu)?.[1].trim()); - assert.equal(source.slug, expected.slug); + assert.deepEqual(showcaseFile.aiSourceDocument, sourceDocument, "The generated directory must record the current canonical source"); + assert.deepEqual(showcaseFile.categories.find(({ id }) => id === "01").disclosure, disclosure); + assert.deepEqual(showcase, canonical.entry, "Every generated entry field, including nested details, must match the canonical manuscript"); + assert.deepEqual(source, { + slug: expected.slug, + title: canonical.articleTitle, + sourceUrl: `https://github.com/130U/130U.github.io/blob/main/website/${SOURCE_FILE}`, + sourceEditedAt: updated, + markdown: canonicalBody, + sourceDocument, + }, "Every generated article field and its exact manuscript must match the canonical source"); assert.equal(source.title, expected.title); - assert.equal(comparableManuscript(originalArticle.markdown), comparableManuscript(approvedBody), "Reversing the reviewed edits must recover every original paragraph and table"); - assert.deepEqual(formulas(originalArticle.markdown).map(({ kind, tex }) => ({ kind, tex: tex.trim() })), formulas(approvedBody).map(({ kind, tex }) => ({ kind, tex: tex.trim() })), "Reversing the reviewed edits must recover every original TeX expression in order"); + assert.equal(canonical.url, `https://www.theodoreoy.com${AI_ROUTE}${expected.slug}/`); + assert.deepEqual(formulas(source.markdown), formulas(canonicalBody), "Every mathematical expression must retain its exact TeX and order"); + const canonicalDetails = canonical.entry.keyContributionDetails ?? {}; + assert.equal(Object.values(canonicalDetails).reduce((count, detail) => count + detail.items.length, 0), [0, 4, 3, 0][index]); assert.equal(readableHtml(html.match(/]*>([\s\S]*?)<\/h1>/iu)?.[1] ?? ""), `${showcase.organization}: ${showcase.detailTitle}`); assert.match(html, new RegExp(`href="/experience/#${expected.slug}"`, "u")); assert.match(html, /href="#key-contributions"/u); @@ -282,14 +273,14 @@ For smooth nonlinear models where`); assert.ok(html.indexOf('id="key-contributions"') < html.indexOf(`data-research-block="${expected.bodyStart}"`)); const detailLists = [...html.matchAll(/<([ou]l)\b[^>]*\bdata-contribution-details="(\d+)"[^>]*>([\s\S]*?)<\/\1>/giu)]; - assert.equal(detailLists.length, Object.keys(approvedDetails).length, "Every nested source list must be rendered once"); + assert.equal(detailLists.length, Object.keys(canonicalDetails).length, "Every nested source list must be rendered once"); for (const list of detailLists) { const details = showcase.keyContributionDetails[list[2]]; - assert.ok(details, "Rendered details must have a reviewed source parent"); + assert.ok(details, "Rendered details must have a canonical source parent"); assert.equal(list[1], details.ordered ? "ol" : "ul"); const children = [...list[3].matchAll(/]*\bdata-contribution-detail="(\d+)"[^>]*>([\s\S]*?)<\/li>/giu)]; assert.deepEqual(children.map((child) => Number(child[1])), details.items.map((_, child) => child)); - assert.deepEqual(children.map((child) => readableHtml(child[2])), details.items.map(readableMarkdown), "All seven nested details must retain their reviewed text and order"); + assert.deepEqual(children.map((child) => readableHtml(child[2])), details.items.map(readableMarkdown), "All seven nested details must retain their canonical text and order"); for (const [childIndex, child] of children.entries()) { const annotations = [...child[2].matchAll(/]*encoding="application\/x-tex"[^>]*>([\s\S]*?)<\/annotation>/giu)].map((item) => decodeHtml(item[1])); assert.deepEqual(annotations, formulas(details.items[childIndex]).map(({ tex }) => tex), "Nested contribution TeX must remain exact"); @@ -334,10 +325,6 @@ For smooth nonlinear models where`); const originalLinks = [...source.markdown.matchAll(/\[([^\]]+)\]\((https?:\/\/[^\s)]+)\)/gu)].map((match) => [readableMarkdown(match[1]), match[2]]); const actualLinks = [...content.matchAll(/]*)>([\s\S]*?)<\/a>/giu)].map((match) => [readableHtml(match[2]), attr(match[1], "href")]); assert.deepEqual(actualLinks, originalLinks, "Citation labels, targets, and order must remain exact"); - const retainedCitations = JSON.parse(await readFile(path.join(ROOT, "content", "sources", "ai-portfolio", "citation-links.json"), "utf8"))[expected.slug]; - for (const citation of retainedCitations) { - assert.ok(actualLinks.some(([, href]) => href === citation.href), `The original reference must remain clickable: ${citation.text}`); - } const sourceFormulas = formulas(source.markdown); const numericVisuals = sourceFormulas.map(expectedNumberVisual); @@ -379,7 +366,7 @@ For smooth nonlinear models where`); } } assert.equal(math.filter((match) => attr(match[1], "data-research-math") === "display").length, expected.display); - assert.equal(sourceFormulas.filter(({ kind }) => kind === "inline").length, expected.inline, "The independently reviewed inline formula count must remain exact"); + assert.equal(sourceFormulas.filter(({ kind }) => kind === "inline").length, expected.inline, "The canonical inline formula count must remain exact"); assert.equal(math.filter((match) => attr(match[1], "data-research-math") === "inline").length, sourceFormulas.filter(({ kind }) => kind === "inline").length); assert.deepEqual(math.map((match) => ({ kind: attr(match[1], "data-research-math"), diff --git a/website/tests/reader-red-team.test.mjs b/website/tests/article-reader.test.mjs similarity index 100% rename from website/tests/reader-red-team.test.mjs rename to website/tests/article-reader.test.mjs diff --git a/website/tests/rendered-html.test.mjs b/website/tests/rendered-html.test.mjs index 86f56ee..e86ede0 100644 --- a/website/tests/rendered-html.test.mjs +++ b/website/tests/rendered-html.test.mjs @@ -575,7 +575,7 @@ test("the ten nonresearch detail routes keep metadata in separate label/value ro for (const [, declarations] of metadataRules) { assert.match(declarations, /grid-template-columns:\s*1fr\s*;/u, "Metadata must remain a single column at every defined breakpoint"); } - assert.match(metadataStyles, /\.metadata\s*>\s*div\s*\{[^}]*display:\s*grid[^}]*grid-template-columns:\s*90px minmax\(0,\s*1fr\)[^}]*align-items:\s*baseline/u, "Desktop and tablet metadata labels and values share a baseline within each row"); + assert.match(metadataStyles, /\.metadata\s*>\s*div\s*\{[^}]*display:\s*grid[^}]*grid-template-columns:\s*6\.5em minmax\(0,\s*1fr\)[^}]*align-items:\s*baseline/u, "Metadata labels and values share a baseline, with label width scaling with the text size"); assert.match(metadataStyles, /@media\s*\(width\s*<\s*420px\)\s*\{\s*\.metadata\s*>\s*div\s*\{[^}]*grid-template-columns:\s*1fr/u, "Narrow screens stack each label above its value without rearranging metadata rows"); for (const route of EXPERIENCE_DETAIL_ROUTES) { const html = await routeHtml(route); @@ -879,7 +879,7 @@ test("shared typography source roles preserve Inter numerals, article names, met assert.match(rule(math, ".number"), /font-variant-numeric:\s*lining-nums\s+proportional-nums\s*;/u); assert.match(rule(math, ".number sup"), /font-size:\s*0\.75em\s*;/u); assert.match(rule(math, ".number sup"), /vertical-align:\s*super\s*;/u); - assert.match(rule(experience, ".metadata > div"), /grid-template-columns:\s*90px minmax\(0,\s*1fr\)\s*;/u); + assert.match(rule(experience, ".metadata > div"), /grid-template-columns:\s*6\.5em minmax\(0,\s*1fr\)\s*;/u); assert.match(rule(experience, ".contributions > ul > li + li"), /border-top:\s*1px solid var\(--rule\)\s*;/u); assert.match(rule(experience, ".contributions > ul > li + li"), /padding-top:\s*16px\s*;/u); assert.doesNotMatch(experience, /\.contributions\s+li(?:\s*\+\s*li)?\s*\{/u, "Contribution spacing and dividers must not cascade into nested items"); diff --git a/website/tests/static-security.test.mjs b/website/tests/static-security.test.mjs index 2c8b90f..2eb8af5 100644 --- a/website/tests/static-security.test.mjs +++ b/website/tests/static-security.test.mjs @@ -39,9 +39,9 @@ test("every exported HTML page permits only its authored inline scripts", async }); test("static policies are deterministic and normalize browser line endings", () => { - const document = ''; + const document = ''; const secured = secureStaticHtml(document); - assert.ok(secured.includes(hash('const label = "你好";\n'))); + assert.ok(secured.includes(hash('const label = "café";\n'))); assert.equal(secureStaticHtml(secured), secured); assert.equal((secured.match(/http-equiv="Content-Security-Policy"/gu) ?? []).length, 1); assert.ok(secured.includes("script-src-attr 'none'"));