Correct link resolution and add a step-by-step link fixer

.claude/skills/wso2-doc-frontmatter/SKILL.md

@@ -89,19 +89,19 @@ can tell which parts are theirs.
 The report classifies every finding by **cause**, because the causes have
 completely different fixes and completely different risk:
 
-| Tier | Cause | Fix |
-|---|---|---|
-| 0 | `{{base_path}}` and the resource exists | Exact rewrite to a relative path |
-| — | `{{base_path}}` and it does not | **Leave alone.** May be served by a redirect |
-| 0 | Malformed link syntax | Exact rewrite. No judgement. Safe in bulk. |
-| 1 | Wrong relative depth | Exact rewrite. No judgement. Safe in bulk. |
-| 2 | Renamed or moved target | A file of that name exists elsewhere; proposed, with confidence |
-| 3 | Pre-migration domain | Needs the new equivalent page — human |
-| 4 | Missing anchor | Heading was reworded — human |
-| 5 | No target anywhere | Was it dropped, missed, or merged? — human |
-
-It ends with a **ready-to-paste prompt for an AI coding agent**, deliberately
-scoped to tiers 0 and 1 and the high-confidence half of tier 2. Do not widen that scope.
+| Tier name | Cause | Fix | Applied by `fix_links.py`? |
+|---|---|---|---|
+| `templated_fixable` | `{{base_path}}` and the resource exists | Exact rewrite to a relative path | Yes |
+| `malformed` | Malformed link syntax | Exact rewrite. No judgement. | Yes |
+| `depth` | Wrong relative depth | Exact rewrite. No judgement. | Yes |
+| `renamed` | Renamed or moved target | A file of that name exists elsewhere; proposed, with confidence | Yes, `high` confidence only by default |
+| `templated` | `{{base_path}}` and it does not exist | **Leave alone.** May be served by a redirect | No — refused |
+| `stale` | Pre-migration domain | Needs the new equivalent page | No — refused |
+| `anchor` | Missing anchor | Heading was reworded | No — refused |
+| `gone` | No target anywhere | Was it dropped, missed, or merged? | No — refused |
+
+The report also ends with a **ready-to-paste prompt for an AI coding agent**, for
+when someone wants to hand the work off rather than run step 5 here.
 
 Two rules the reporter enforces, and you must not work around:
 
@@ -111,15 +111,69 @@ Two rules the reporter enforces, and you must not work around:
 - **Never propose a target in a different version.** If a page under one version
   links to something missing, the replacement must live under that same version. A
   cross-version link silently sends a reader to a different release.
-Tiers 3 to 5 need information that is not in the repo, and an agent asked to fix
-them produces confident links to the wrong pages — worse than a visibly broken
-link, because a plausible wrong link never gets re-checked.
+`stale`, `anchor` and `gone` need information that is not in the repo, and an agent
+asked to fix them produces confident links to the wrong pages — worse than a
+visibly broken link, because a plausible wrong link never gets re-checked.
 
 When you report back, give the tier counts and say plainly how many need a human. A
 raw total is alarming and useless on its own; "N have an exact fix, M need a
 decision" is what someone can act on.
 
-### 5. Re-audit, and verify against a real build
+### 5. Fix the links one tier at a time, and ask before each tier
+
+The report is a plan, not a change. `scripts/fix_links.py` is the only thing that
+applies it, and it takes **one tier per run**:
+
+```bash
+# show what this tier would do — nothing is written
+python3 scripts/fix_links.py en/docs --plan BROKEN-LINKS-<scope>.json --tier malformed
+# apply it, and record what changed
+python3 scripts/fix_links.py en/docs --plan BROKEN-LINKS-<scope>.json --tier malformed \
+    --apply --journal /tmp/fixed-malformed.json
+```
+
+Work the tiers in this order, easiest and safest first:
+
+1. `malformed` — link syntax
+2. `depth` — wrong number of `../`
+3. `renamed` — target moved (`high` confidence only; `--min-confidence` widens it)
+4. `templated_fixable` — if the scope has any
+
+**The rule for each tier: dry-run it, show the person a sample, tell them how many
+would change and how many the verifier refused, and wait for an explicit yes before
+`--apply`.** Never chain tiers in one go, and never apply a tier the person has not
+seen. Their answer for one tier is not their answer for the next — `depth` is
+arithmetic, but `renamed` is a proposal, and someone may want every one of those
+eyeballed.
+
+After each applied tier, report what actually changed and stop:
+
+```
+tier `depth`: 667 verified, 3 refused, 132 files changed, 667 links rewritten.
+Refused: 3 where the anchor no longer exists. Next tier is `renamed` (448, 406
+verified). Apply it?
+```
+
+**Regenerate the plan between tiers.** Fixing one tier changes what the others
+resolve to, so a plan written before the last tier was applied is stale — and
+`fix_links.py` will skip entries whose link text it can no longer find rather than
+guess.
+
+Two things the script does that you should not work around:
+
+- **It verifies every rewrite against the disk before writing anything.** A
+  proposal whose target does not resolve, or whose anchor does not exist on the new
+  page, is refused rather than applied. Those refusals are the useful output — they
+  are the cases where the report was optimistic.
+- **It refuses `templated`, `stale`, `anchor` and `gone` outright.** Those need
+  information that is not in the repo. Do not hand-apply them in bulk to save
+  time; a plausible wrong link is worse than a visibly broken one, because nobody
+  re-checks it.
+
+Finish by re-running the link checker and quoting the before/after, since that —
+not the number of files touched — is what says the fix worked.
+
+### 6. Re-audit, and verify against a real build
 
 ```bash
 python3 scripts/fm_audit.py en/docs --gate
@@ -132,7 +186,11 @@ Re-auditing is not optional: it's the only thing that proves the fix worked rath
 
 Where the repo can be built, `mkdocs build` is the authoritative check on links — the link checker is calibrated against it and finds a superset of what it reports. If you have the dependencies, run it and reconcile any difference rather than assuming the script is right.
 
-## The other checkers
+## The other scripts
+
+`scripts/fix_links.py` applies one tier of a `report_links.py` plan. Dry run by
+default; `--apply` writes; `--journal` records every rewrite so a tier can be
+reviewed or undone. It is the only script here that edits link text.
 
 `scripts/check_redirects.py` validates `redirect_maps` in `mkdocs.yml` — targets exist, no source shadowed by a real file, no chains (the plugin doesn't follow them), no map left pointing at a superseded version after a version bump.
 

.claude/skills/wso2-doc-frontmatter/references/conventions.md

@@ -118,6 +118,15 @@ fixable, and `report_links.py` splits them accordingly:
 Redirects themselves belong either in a `redirects.yml` file or in a `redirects`
 block inside `mkdocs.yml`.
 
+## Applying a link plan
+
+`report_links.py` proposes; `fix_links.py` is the only script that rewrites link
+text. It applies one tier per run, verifies each rewrite against the disk first,
+and refuses the tiers that need a person (`templated`, `stale`, `anchor`, `gone`).
+
+Regenerate the plan between tiers: fixing one tier changes what the others resolve
+to. Entries whose link text can no longer be found are skipped rather than guessed.
+
 ## Adding another source of documentation
 
 Nothing in the scripts is tied to a particular product or version. Versions are

.claude/skills/wso2-doc-frontmatter/scripts/check_links.py

@@ -11,6 +11,8 @@
 _ap.add_argument("--json", dest="json_out", default=None,
                  help="Write full findings to this path. Omitted = summary only.")
 _ap.add_argument("--gate", action="store_true", help="Exit 1 if any blocking finding.")
+_ap.add_argument("--mkdocs-yml", dest="mkdocs_yml", default=None,
+                 help="Where to read `toc_depth` from. Defaults to <docs_root>/../mkdocs.yml.")
 _args = _ap.parse_args()
 DOCS = _args.docs_root.rstrip("/")
 SITE = "https://wso2.com/api-platform/docs"
@@ -33,24 +35,82 @@ def slug(h):
     h = re.sub(r"[^\w\s-]", "", h).strip().lower()
     return re.sub(r"[-\s]+", "-", h)
 
+def toc_depth(mkdocs_yml):
+    """The `toc_depth` configured for the python-markdown toc extension.
+
+    This is load-bearing, not cosmetic. With `toc_depth: 3`, python-markdown
+    assigns NO `id` to h4 and deeper — so `#some-h4-heading` links resolve to
+    nothing in the built site even though the heading is right there in the
+    Markdown. Neither this checker nor `mkdocs build` on 1.4.x flags that on its
+    own, so a page can look clean and still have dead in-page links.
+    """
+    if not os.path.isfile(mkdocs_yml):
+        return 6
+    text = open(mkdocs_yml, encoding="utf-8", errors="replace").read()
+    m = re.search(r"^\s*toc_depth:\s*['\"]?(\d)", text, re.M)
+    return int(m.group(1)) if m else 6
+
+
+MKDOCS_YML = _args.mkdocs_yml or os.path.join(os.path.dirname(DOCS) or ".", "mkdocs.yml")
+TOC_DEPTH = toc_depth(MKDOCS_YML)
+
 # harvest anchors per file (headings + explicit <a name>/{#id})
-anchors = {}
+#
+# `anchors` is what the build actually produces. `deep_anchors` holds the ids a
+# heading WOULD have if toc_depth allowed it — kept separately so a link to one
+# can be reported as its own cause rather than as a generic missing anchor.
+#
+# Do NOT suggest `{#id}` as the fix for a deep heading here, even though
+# `attr_list` would honour it: the `markdownextradata` plugin runs every page
+# through Jinja BEFORE Markdown, and `{#` opens a Jinja comment. An unterminated
+# one fails the whole build with "Missing end of comment tag". The safe additive
+# fix is `<a name="...">` immediately above the heading — inert HTML, already used
+# in ~300 pages here, and it leaves the heading level and the TOC untouched.
+anchors, deep_anchors = {}, {}
 for p in md_files:
     txt = open(os.path.join(DOCS, p), encoding="utf-8", errors="replace").read()
     txt = re.sub(r"<!--.*?-->", "", txt, flags=re.S)
     txt = re.sub(r"```.*?```", "", txt, flags=re.S)
-    a = set()
-    for m in re.finditer(r"^#{1,6}\s+(.+?)\s*$", txt, re.M):
-        h = m.group(1)
+    a, deep = set(), set()
+    for m in re.finditer(r"^(#{1,6})\s+(.+?)\s*$", txt, re.M):
+        level, h = len(m.group(1)), m.group(2)
         exp = re.search(r"\{#([\w-]+)\}", h)
-        if exp: a.add(exp.group(1)); h = h[:exp.start()]
-        a.add(slug(h))
-    for m in re.finditer(r'<a[^>]+(?:name|id)="([^"]+)"', txt): a.add(m.group(1))
+        if exp:
+            a.add(exp.group(1))            # explicit id survives any toc_depth
+            h = h[:exp.start()]
+        (a if level <= TOC_DEPTH else deep).add(slug(h))
+    for m in re.finditer(r"""<a[^>]+(?:name|id)=(["'])(.*?)\1""", txt): a.add(m.group(2))
     for m in re.finditer(r'\{#([\w-]+)\}', txt): a.add(m.group(1))
     anchors[p] = a
+    deep_anchors[p] = deep - a
 
 LINK = re.compile(r'(!?)\[([^\]]*)\]\(\s*<?([^)\s>]+)>?(?:\s+"[^"]*")?\s*\)')
-HTML_SRC = re.compile(r'<img[^>]+src="([^"]+)"')
+# Tag name is captured so an `<a href>` is reported as a broken LINK and an
+# `<img src>` as a missing IMAGE. Lumping them together mislabels every raw-HTML
+# link as an image, which sends whoever reads the report looking for the wrong thing.
+#
+# The quote character is captured and back-referenced, so single-quoted attributes
+# are matched too. HTML allows either, the migrated pages use both, and a
+# double-quote-only pattern skips the single-quoted ones silently — they look
+# checked when they were never read.
+HTML_SRC = re.compile(r"""<(img|a|source|iframe)[^>]+(?:src|href)=(["'])(.*?)\2""")
+
+
+def url_base(rel):
+    """Directory the RENDERED page sits in, under `use_directory_urls: true`.
+
+    `a/b/page.md` is served at `/a/b/page/` — one level deeper than the source —
+    while `a/b/index.md` is served at `/a/b/`, the same level.
+
+    mkdocs rewrites relative targets written in Markdown syntax, resolving them
+    against the source file, but passes raw HTML through untouched, so the browser
+    resolves an `<img src>` against the rendered URL instead. The identical string
+    is therefore correct in one syntax and broken in the other. Resolving both the
+    same way is how a working image gets "fixed" into a broken one.
+    """
+    d = os.path.dirname(rel)
+    stem = os.path.basename(rel)[:-3] if rel.endswith(".md") else os.path.basename(rel)
+    return d if stem in ("index", "README") else (f"{d}/{stem}" if d else stem)
 
 findings = []
 def add(f, sev, code, msg):
@@ -64,10 +124,10 @@ def add(f, sev, code, msg):
     body = re.sub(r"`[^`\n]*`", "", body)
     d = os.path.dirname(p)
 
-    targets = [(m.group(1) == "!", m.group(3)) for m in LINK.finditer(body)]
-    targets += [(True, m.group(1)) for m in HTML_SRC.finditer(body)]
+    targets = [(m.group(1) == "!", m.group(3), False) for m in LINK.finditer(body)]
+    targets += [(m.group(1).lower() != "a", m.group(3), True) for m in HTML_SRC.finditer(body)]
 
-    for is_img, t in targets:
+    for is_img, t, is_html in targets:
         if t.startswith(("mailto:", "tel:", "#!")):
             continue
         # Build-time template variables (e.g. `{{base_path}}`) are not paths. They
@@ -92,15 +152,44 @@ def add(f, sev, code, msg):
         if t.startswith("#"):
             frag = urllib.parse.unquote(t[1:])
             if frag and frag not in anchors.get(p, set()):
-                add(p, "should-fix", "ANCHOR_MISSING", f"In-page anchor `{t}` has no matching heading.")
+                if frag in deep_anchors.get(p, set()):
+                    add(p, "blocking", "ANCHOR_TOO_DEEP",
+                        f"`{t}` names a heading deeper than h{TOC_DEPTH}, and `toc_depth: {TOC_DEPTH}` "
+                        f"means the build gives it no id — so the link goes nowhere. Add "
+                        f"`<a name=\"{frag}\"></a>` just above the heading, or promote the heading "
+                        f"to h{TOC_DEPTH}.")
+                else:
+                    add(p, "should-fix", "ANCHOR_MISSING", f"In-page anchor `{t}` has no matching heading.")
             continue
 
         path, _, frag = t.partition("#")
         path = urllib.parse.unquote(path)
         frag = urllib.parse.unquote(frag)
         if not path:
             continue
-        cand = os.path.normpath(os.path.join(d, path)) if not path.startswith("/") else path.lstrip("/")
+        # WHICH BASE APPLIES — verified against a real mkdocs build, not inferred.
+        #
+        # mkdocs rewrites a Markdown target only when the literal path names a file
+        # that exists in docs_dir (`../c/target.md`, `../img.png`). Then, and only
+        # then, is the target resolved against the SOURCE directory.
+        #
+        # Everything else is passed through verbatim and resolved by the browser
+        # against the RENDERED URL, which sits one level deeper for a non-index page:
+        #   * raw HTML (`<img src>`, `<a href>`)
+        #   * directory-style Markdown links (`../c/target/`)
+        #   * extensionless Markdown links (`../c/target`) — passed through even
+        #     when `target.md` exists right there
+        #
+        # Judging a passed-through link source-relative is how a link that is broken
+        # in the browser gets reported as clean.
+        if path.startswith("/"):
+            cand = path.lstrip("/")
+            rewritten = False
+        else:
+            literal = os.path.normpath(os.path.join(d, path)).replace("\\", "/")
+            rewritten = (not is_html) and literal in all_files
+            rel_base = d if rewritten else url_base(p)
+            cand = os.path.normpath(os.path.join(rel_base, path))
         if cand.startswith(".."):
             add(p, "blocking", "LINK_ESCAPES_ROOT", f"Link `{t}` resolves outside the docs root.")
             continue
@@ -118,8 +207,13 @@ def add(f, sev, code, msg):
             continue
         if frag and resolved.endswith(".md"):
             if frag not in anchors.get(resolved, set()):
-                add(p, "should-fix", "ANCHOR_MISSING",
-                    f"Anchor `#{frag}` not found in `{resolved}` (link was `{t}`).")
+                if frag in deep_anchors.get(resolved, set()):
+                    add(p, "blocking", "ANCHOR_TOO_DEEP",
+                        f"`{t}` names a heading in `{resolved}` deeper than h{TOC_DEPTH}, which "
+                        f"`toc_depth: {TOC_DEPTH}` leaves without an id, so the link goes nowhere.")
+                else:
+                    add(p, "should-fix", "ANCHOR_MISSING",
+                        f"Anchor `#{frag}` not found in `{resolved}` (link was `{t}`).")
 
     # Alt text. The style guide is specific here and it is easy to get wrong:
     #   - alt="" is CORRECT for purely decorative images or screenshots that