feat(search): score the curated description and keywords front matter

The corpus is built from the markdown mirrors, and the mirrors carry `# Title` plus a
Source/Published/Author block and nothing else — front matter is not in them by design. So
the two most deliberately curated relevance signals on this site were absent from its own
search: `description` was ignored in favour of each page's first paragraph, and `keywords`
was unused entirely. 186 curated phrases across 29 posts, 140 of which appear nowhere in
the indexed text.

The consequence, measured as the rank of the page each phrase was WRITTEN for:

  handle traffic spikes microservices   absent of 32  ->  #1
  nodejs backpressure microservices     absent of 32  ->  #5
  overload resilience                   absent        ->  #1
  message queue throughput              #32 of 134    ->  #3
  nodejs job queue                      #36 of 93     ->  #3
  imqueue benchmark                     #2 of 140     ->  #1

Weighted at 300 — BELOW emphasis — and scored on coverage alone: no density, no position,
no phrase bonus. A keyword list is a handful of comma-separated phrases, so
occurrences-per-token sits near 1.0 for anything that matches at all, and density would
rank by brevity while rewarding padding; order in a comma list means nothing, so position
would be noise.

That placement is the point rather than caution for its own sake. Google has ignored
<meta name="keywords"> since 2009 and Bing treats a stuffed one as a spam signal — because
neither can trust the author. A first-party index can: the author is the site. What does
not change is that a self-declared list is cheap to pad, so it sits under the signals that
cost something to fake, and check:search-ranking asserts "safeDelivery" still ranks the
symbol first — the failure mode meta keywords earned its reputation for is an identifier
query returning articles that merely list the identifier.

Plumbed through a build intermediate: src/search-frontmatter.11ty.js emits url ->
{description, keywords} and the generator reads it and DELETES it, so it never ships. A
template rather than a front-matter parse because mapping a source file back to its URL
means re-deriving permalink resolution, which Eleventy owns and which this repo has already
been bitten by duplicating; `item.url` cannot drift from what Eleventy published. Keyword
text also feeds the vocabulary, so inflections in a keyword list get lemmas like any other
word.

Tier 1 grows 65.2 -> 67.5 KB gz on org, 0.9 -> 1.1 on com.

One defect worth recording: the new record was first named `entry`, inside a loop whose
variable is the directory entry. Shadowing it in the same block is a temporal-dead-zone
error reported at the line ABOVE, which reads as readdir having failed.

Author: Mikhus

scripts/check-search-ranking.js

@@ -322,6 +322,45 @@ if (!sendTop || sendName !== 'send') {
   pass(`"send" ranks ${sendTop.record.t} first among symbols — identifiers stay literal`);
 }
 
+// ---- curated keywords ---------------------------------------------------------
+// `keywords:` front matter states which queries a page exists to answer, and it was absent
+// from the index entirely until this element existed — 140 of 186 curated phrases appeared
+// nowhere in the indexed text, so the page written for "nodejs backpressure microservices"
+// was not among that query's 32 results at all. Each case below is a measured before/after.
+const KEYWORD_CASES = [
+  ['handle traffic spikes microservices', '/blog/backpressure-nodejs-services/', 'was absent entirely'],
+  ['nodejs backpressure microservices', '/blog/backpressure-nodejs-services/', 'was absent entirely'],
+  ['overload resilience', '/blog/backpressure-nodejs-services/', 'was absent entirely'],
+  ['message queue throughput', '/blog/benchmarking-imqueue-throughput/', 'was #32'],
+  ['nodejs job queue', '/blog/imqueue-vs-bullmq/', 'was #36'],
+  ['imqueue benchmark', '/blog/benchmarking-imqueue-throughput/', 'was #2'],
+];
+
+for (const [query, target, was] of KEYWORD_CASES) {
+  const hits = run(query);
+  const at = hits.findIndex((hit) => hit.record.u.split('#')[0] === target);
+
+  if (at === -1) {
+    fail(`"${query}" does not reach ${target} at all (${was}) — curated keywords are not being scored`);
+  } else if (at > 5) {
+    fail(`"${query}" ranks ${target} at #${at + 1} (${was}); the page written for this phrase should be in the top few`);
+  } else {
+    pass(`"${query}" ranks its target page #${at + 1} (${was})`);
+  }
+}
+
+// The counterweight. Keywords are self-declared and cheap to pad, so the element sits
+// below emphasis and is scored on coverage alone. If it ever outgrew that, an identifier
+// query would start returning articles that merely LIST the identifier — which is the
+// failure mode <meta name="keywords"> earned its reputation for.
+const identifier = run('safeDelivery')[0];
+
+if (!identifier || identifier.record.g !== 1) {
+  fail(`"safeDelivery" no longer ranks a symbol first — keyword weight may have overtaken the reference`);
+} else {
+  pass('"safeDelivery" still ranks the symbol first — keywords cannot outrank reference');
+}
+
 // ---- cross-site search -------------------------------------------------------
 // imqueue.org and imqueue.com search each other, reading the peer's index from their own
 // origin (scripts/copy-peer-index.js). Two properties matter, and they pull against each

scripts/lib/search-corpus.js

@@ -389,6 +389,16 @@ function verifyAnchor(part, outputDir, url) {
  * knows which terms are stopwords), so the exclusion lives in one place rather than
  * being duplicated here as a second list that could drift.
  */
+function vocabularyOf(text, into) {
+  for (const word of String(text).toLowerCase().split(/[^a-z0-9]+/)) {
+    if (word) {
+      into.add(word);
+    }
+  }
+
+  return into;
+}
+
 function lemmaMap(words) {
   const map = {};
 
@@ -403,13 +413,38 @@ function lemmaMap(words) {
   return map;
 }
 
+/**
+ * Curated `description`/`summary` and `keywords` per URL, from the build intermediate
+ * src/search-frontmatter.11ty.js writes.
+ *
+ * Deleted after reading: it duplicates facts already public in llms.txt and in each page's
+ * meta description, and publishing a third copy would be a maintenance surface with no
+ * consumer. Absent is legal — the corpus then falls back to first paragraphs, exactly as
+ * it did before front matter was wired in.
+ */
+function takeFrontmatter(outputDir) {
+  const file = path.join(outputDir, "search-frontmatter.json");
+
+  if (!fs.existsSync(file)) {
+    return {};
+  }
+
+  const data = JSON.parse(fs.readFileSync(file, "utf8"));
+
+  fs.unlinkSync(file);
+
+  return data;
+}
+
 function buildCorpus(outputDir) {
   const docs = [];
   const faq = [];
   const pages = [];
   const sections = [];
   const vocabulary = new Set();
 
+  const frontmatter = takeFrontmatter(outputDir);
+
   idCache.clear();
   unanchored = 0;
 
@@ -442,25 +477,38 @@ function buildCorpus(outputDir) {
 
       const parts = splitSections(mirror.body).map((part) => verifyAnchor(part, outputDir, mirror.url));
 
-      docs.push({
+      const meta = frontmatter[mirror.url] || {};
+      // Named `record`, not `entry`: the enclosing loop's variable is the directory entry,
+      // and shadowing it in the same block is a temporal-dead-zone error at the line ABOVE
+      // this one, which reads as the readdir having failed.
+      const record = {
         g: 0,
         t: mirror.title,
         u: mirror.url,
-        s: summarize(mirror.body),
+        // The CURATED description wins over the first paragraph. It was written to say
+        // what the page answers, in one sentence; a lead paragraph was written to be read
+        // next. Falling back matters for the pages that have neither.
+        s: meta.d ? summarize(meta.d) : summarize(mirror.body),
         k: groupFor(mirror.url),
-      });
+      };
+
+      // Curated keywords, as their own scoring element — see E.keywords in search.js. This
+      // is the one signal no amount of body analysis can reconstruct: it is the author
+      // stating which queries the page exists to answer.
+      if (meta.k) {
+        vocabularyOf(meta.k, vocabulary);
+        record.w = meta.k;
+      }
+
+      docs.push(record);
 
       faq.push(...faqRecords(parts, mirror));
 
       const pageIdx = pages.length;
 
       pages.push([mirror.url, mirror.title, groupFor(mirror.url)]);
 
-      for (const word of `${mirror.title} ${mirror.body}`.toLowerCase().split(/[^a-z0-9]+/)) {
-        if (word) {
-          vocabulary.add(word);
-        }
-      }
+      vocabularyOf(`${mirror.title} ${mirror.body}`, vocabulary);
 
       for (const part of parts) {
         if (part.text) {
@@ -506,6 +554,7 @@ function buildCorpus(outputDir) {
       // headings and its markdown mirror have drifted apart.
       unanchored,
       vocabulary: vocabulary.size,
+      keyworded: docs.filter((d) => d.w).length,
       lemmas: Object.keys(lemmas).length,
       // Stems a detachment produced, that the dictionary rejected, and that the corpus
       // uses as words: candidates for scripts/data/project-words.txt. Reported rather

src/_shared/js/search.js

@@ -94,6 +94,13 @@
     title: 430,
     header: 360,
     emphasis: 200,
+    // Curated `keywords` front matter: the author stating which queries this page exists to
+    // answer. BELOW emphasis deliberately. Google has ignored <meta name="keywords"> since
+    // 2009 and Bing treats a stuffed one as a spam signal — because neither can trust the
+    // author. This index can: the author is the site. What does not change is that a
+    // self-declared list is cheap to pad, so it sits under the signals that cost something
+    // to fake and is scored on COVERAGE ONLY (see keywordScore).
+    keywords: 300,
     body: 120,
   };
 
@@ -641,6 +648,36 @@
     ) + positionBonus(E.title, lower, q);
   }
 
+  /**
+   * The keywords element. Coverage only — no density, no position, no phrase bonus.
+   *
+   * A keyword list is a handful of comma-separated phrases, so occurrences-per-token sits
+   * near 1.0 for every page that matches at all: density would rank by list brevity, and
+   * reward padding. Order in a comma list means nothing, so the position bonus would be
+   * noise. What the list can honestly say is "these words describe this page", and coverage
+   * is exactly that statement.
+   */
+  function keywordScore(record, q) {
+    if (!record._w) {
+      return 0;
+    }
+
+    var matched = 0;
+
+    for (var i = 0; i < q.terms.length; i++) {
+      var hit = scanFor(record._w, q.terms[i], q.whole[i]);
+
+      if (!hit.n && q.lemmas[i]) {
+        hit = scanFor(record._w, q.lemmas[i], true);
+      }
+      if (hit.n) {
+        matched += q.weights[i];
+      }
+    }
+
+    return matched ? E.keywords * Math.min(1, q.weightSum ? matched / q.weightSum : 0) : 0;
+  }
+
   function scoreRecord(record, q) {
     if (q.filters.pkg && fold(record.p || "").indexOf(q.filters.pkg) === -1) {
       return 0;
@@ -658,7 +695,8 @@
 
       for (var t = 0; t < q.terms.length; t++) {
         if (q.weights[t] === 1 &&
-          (record._l.indexOf(q.terms[t]) !== -1 || record._s.indexOf(q.terms[t]) !== -1)) {
+          (record._l.indexOf(q.terms[t]) !== -1 || record._s.indexOf(q.terms[t]) !== -1 ||
+            (record._w && record._w.indexOf(q.terms[t]) !== -1))) {
           found++;
         }
       }
@@ -671,7 +709,9 @@
     // scored as a body — same weight a section's prose gets — so a symbol whose
     // description happens to use the query words cannot outrank the symbol named
     // after them.
-    var score = titleScore(record, q) + elementScore(E.body, record._s, record._sn, q, "");
+    var score = titleScore(record, q) +
+      keywordScore(record, q) +
+      elementScore(E.body, record._s, record._sn, q, "");
 
     if (score < MIN_SCORE) {
       return 0;
@@ -770,6 +810,7 @@
       // splitting on whitespace is within a token or two of idTokens() here and this
       // runs over 1,325 records at load.
       r._sn = r._s ? r._s.split(" ").length : 0;
+      r._w = r.w ? fold(r.w) : "";
     }
 
     return index;

src/search-frontmatter.11ty.js

@@ -0,0 +1,68 @@
+// Emits /search-frontmatter.json — every page's curated `description` and `keywords`,
+// keyed by URL, for scripts/lib/search-corpus.js to fold into the search index.
+//
+// A BUILD INTERMEDIATE, not a published feed: the generator reads it and deletes it, so
+// it never reaches the deployed site. That is deliberate — it duplicates data already
+// public in llms.txt and in each page's meta description, and one more machine-readable
+// index of the same facts is a maintenance surface with no consumer.
+//
+// WHY IT HAS TO EXIST AT ALL
+//
+// The search corpus is built from the markdown MIRRORS (see search-corpus.js for why),
+// and the mirrors carry `# Title` plus a Source/Published/Author block and nothing else.
+// Front matter is not in them by design. So the two most deliberately curated relevance
+// signals on the site were absent from its own search:
+//
+//   * `description` — the corpus used each page's first paragraph as its summary instead
+//   * `keywords`    — unused entirely. 186 curated phrases across 29 posts, 140 of which
+//                     appear nowhere in the indexed text, so the page written for
+//                     "nodejs backpressure microservices" did not appear in that query's
+//                     32 results at all
+//
+// WHY A TEMPLATE AND NOT A FRONT-MATTER PARSE
+//
+// The generator runs post-build over `_site-*`, and mapping a source file back to its URL
+// means re-deriving permalink resolution — which Eleventy owns, and which this repo has
+// already been bitten by duplicating (see the two `search-index.11ty.js` headers). Asking
+// Eleventy for `item.url` cannot drift from what Eleventy actually published.
+//
+// `/api/` is excluded: those descriptions are generated from the source's own doc comments
+// by scripts/lib/api-summary.js, and /api/search-index.json already carries them.
+
+module.exports = class SearchFrontmatter {
+  data() {
+    return {
+      permalink: "/search-frontmatter.json",
+      eleventyExcludeFromCollections: true,
+    };
+  }
+
+  render(data) {
+    const pages = {};
+
+    for (const item of data.collections.all || []) {
+      const url = item.url || "";
+
+      if (!url || url.startsWith("/api/")) {
+        continue;
+      }
+
+      const description = item.data.description || "";
+      // `summary` is the blog's longer form; a post has both, and the longer one names
+      // more of what the post actually answers.
+      const summary = item.data.summary || "";
+      const keywords = item.data.keywords || "";
+
+      if (!description && !keywords && !summary) {
+        continue;
+      }
+
+      pages[url] = {
+        d: String(summary || description).replace(/\s+/g, " ").trim(),
+        k: String(keywords).replace(/\s+/g, " ").trim(),
+      };
+    }
+
+    return JSON.stringify(pages);
+  }
+};