Use rocm-docs-core's llms.txt generator

[j-stephan]

Motivation

These docs generated llms.txt / llms-full.txt with a hand-rolled build-finished function in conf.py and shipped a manually-curated llms.txt via html_extra_path. rocm-docs-core 1.38.0 now provides this natively (behind the optional llms extra), producing both files from Sphinx's resolved doctree. Switching to it removes ~230 lines of bespoke, untested prose-filtering code per repo, keeps the output consistent with every other ROCm project, and can't drift from the docs since it's generated from the same doctree Sphinx builds.

Technical Details

• docs/sphinx/requirements.in: add the llms extra and pin rocm-docs-core 1.38.0 (rocm-docs-core[llms]==1.38.0). requirements.txt regenerated with pip-compile in an Ubuntu / Python 3.12 environment so no Windows-only packages (e.g. pywin32) leak into the RTD build;
  this adds sphinx-markdown-builder==0.6.10.
• docs/conf.py:
  • Enable generation with rocm_docs_generate_llms = True.
  • Remove html_extra_path = ["llms.txt"] and delete the curated docs/llms.txt; the index is now generated.
  • Remove the custom generate_combined_markdown build-finished handler and all its helpers (is_prose_line, should_skip, the EXCLUDED_DIRS / MARKUP_PREFIXES / regex constants, MIN_PROSE_LINES) and the imports they orphaned (re, pathlib.Path).
  • Per-page Markdown download button (use_download_button) is enabled (already present in these repos).
• Unrelated conf.py code (e.g. the primary-sidebar collapse handler) is left untouched.

The feature is documented in the rocm-docs-core guide: https://rocm.docs.amd.com/projects/rocm-docs-core/en/latest/user_guide/llms.html

Test Plan

• Regenerated requirements.txt in an Ubuntu / Python 3.12 container and confirmed it is free of Windows-only dependencies.
• Ran a full sphinx -b html build with rocm-docs-core 1.38.0 + the llms extra installed.
• Verified conf.py byte-compiles and has no references to the removed functions/constants.

Test Result

• Build succeeded (exit 0).
• llms.txt and llms-full.txt are generated in the HTML output directory (replacing the removed custom output); llms-full.txt is more complete than before because it inlines the resolved doctree rather than a prose-line heuristic.
• Per-page Markdown download button renders (dropdown-download-buttons present in built pages).
• requirements.txt pins rocm-docs-core[llms]==1.38.0 and sphinx-markdown-builder==0.6.10, with no pywin32.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.