Switch to an aggregated Changes system with much lower collision rates []

Author: wchilders-nvidia

.agents/skills/edg-changes-example/SKILL.md

@@ -1,14 +1,16 @@
 ---
 name: edg-changes-example
 description: >-
-  Encodes and decodes non-UTF-8 example bytes in Changes changelogs with
-  edg-changes-example («HHHH...» markers) and writes Changes files as UTF-8.
-  Use when editing any Changes file (src/Changes, util/Changes, …), adding
+  Writes Changes entries as pending_changes/*.txt files (collected
+  nightly by edg-collect-changes, checked with edg-collect-changes --check),
+  and encodes/decodes non-UTF-8 example bytes with edg-changes-example
+  («HHHH...» markers). Use when adding a Changes entry, editing any Changes
+  file (src/Changes, util/Changes, …) or pending_changes/ entry, adding
   examples with Latin-1/EUC-JP/Shift-JIS/other non-UTF-8 bytes, or verifying
   changelog encoding.
 ---
 
-# `edg-changes-example` and Changes files
+# Changes entries, `edg-collect-changes`, and `edg-changes-example`
 
 ## Policy (all `Changes` files)
 
@@ -74,7 +76,50 @@ bytes. No other escapes are interpreted.
 
 ## Workflow for a new changelog entry
 
-1. Draft the entry as UTF-8 prose.
+New entries are **not** written into a `Changes` file. Each `Changes` file
+has a `pending_changes/` directory beside it (`src/pending_changes/`,
+`util/pending_changes/`, …); add one entry file there per change, named
+`*.txt` (suggested: `gh-<N>-<short-slug>.txt`). A nightly job
+(`.github/workflows/collect-changes.yml`, midnight US Eastern) runs
+`edg-collect-changes`, which moves every entry into its `Changes` file under
+the newest `Version x.y` line, adds the date, orders that night's entries by
+summary line, and deletes the entry files.
+
+Entry file format:
+
+```text
+//allowed_words: reflectable memoized
+[GH #335]
+Type transformations of qualified function types
+
+Previously, the __add_lvalue_reference, __add_rvalue_reference, ...
+For example, with --c++11:
+
+  using F = void() const;
+  using F = __add_pointer(F);  // Previously a spurious error, now okay.
+```
+
+- Optional leading `//key: value` directives (EDG C++ line directive
+  syntax). `//allowed_words:` lists space-separated words the spell check
+  should accept (identifiers, jargon); repeat the line as needed. Words the
+  target `Changes` file already uses (`constexpr`, `lvalue`, …) are accepted
+  without it. Directives are dropped when the entry is collected.
+- The tag block: one or more comma-separated `GH #N` tags in brackets
+  (`[GH #12,GH #34]`; only `GH #N`, never `[]`). It may wrap onto several
+  lines; the tool re-wraps it the way existing entries are wrapped.
+- **No date.** The tool adds it (`m/d/yy`, padded to 8 columns) in front of
+  the tags.
+- The summary line, one blank line, then the description, in the same style
+  as existing entries (unindented prose wrapped at 79 columns, two spaces
+  after periods, examples indented 2 spaces).
+- UTF-8, no tabs, at most **79** columns per line. Trailing whitespace is
+  tolerated (it's stripped on collection).
+
+Steps:
+
+1. Draft the entry as UTF-8 prose in
+   `<dir>/pending_changes/<name>.txt` (the `pending_changes/` beside
+   the `Changes` file the entry belongs in, usually `src/`).
 2. If an example needs non-UTF-8 bytes, author the example in a small raw
    file (or produce the markers by hand only when the bytes are already
    known), then run `edg-changes-example encode` and paste the UTF-8 result
@@ -83,20 +128,35 @@ bytes. No other escapes are interpreted.
 
    `Note: The above example must be decoded using the edg-changes-example tool.`
 
-4. Insert the entry into the appropriate `Changes` file with a **UTF-8**
-   write (keep the rest of the file valid UTF-8).
-5. Verify the file still decodes as UTF-8 (raises on failure):
+4. Check the entry (format, spelling, typos, line length). This is the
+   check `edg-check-file-changes` does for `pending_changes/*.txt` files,
+   which pull request CI runs and which makes the pre-push hook refuse the
+   push (no override) if anything is wrong:
 
    ```bash
-   python3 -c "open('path/to/Changes', encoding='utf-8').read()"
+   edg-exec -- edg-collect-changes --check src/Changes src/pending_changes
    ```
 
-   Use the real path (for example `src/Changes`). A clean exit means the
-   whole file is valid UTF-8; a `UnicodeDecodeError` means the write
-   introduced non-UTF-8 bytes and must be fixed before continuing.
+   (Name the `Changes` file and `pending_changes/` directory the entry is
+   in. `edg-exec` runs it in the dev-env container, which has `aspell`.) For
+   misspellings that are really identifiers or terms, add them to an
+   `//allowed_words:` line at the very top of the entry, as the failure
+   message shows; fix real misspellings instead.
+5. Optionally preview the result without changing anything:
+   `edg-collect-changes --dry-run src/Changes src/pending_changes`.
+
+Edit a `Changes` file directly only to correct an entry that has already
+been collected; then write it as **UTF-8** and verify the file still decodes
+(raises on failure):
+
+```bash
+python3 -c "open('path/to/Changes', encoding='utf-8').read()"
+```
 
 ## Do not
 
+- Add new entries directly to a `Changes` file, or put a date in a
+  `pending_changes/` entry.
 - Commit raw Latin-1 / Shift-JIS / etc. bytes inside a `Changes` file.
 - Invent alternate note wording; match the existing sentence above.
 - Run `encode` on the entire historical `src/Changes` and rewrite it unless

.agents/skills/edg-cpfe-frontend/SKILL.md

@@ -45,15 +45,19 @@ Language: `--c++11` … `--c++26`. Emulation: `--gnu=XXYYZZ`,
 
 ## Editing `Changes` files
 
-Write every `Changes` file as **UTF-8** (`src/Changes`, `util/Changes`, and
-any other `Changes` in the tree). For non-UTF-8 example bytes, `«HHHH…»`
-markers, the required note line, encode/decode commands, and verification,
-follow skill **edg-changes-example** — do not invent a parallel procedure here.
+Add new entries as `pending_changes/*.txt` files beside the `Changes`
+file (usually `src/pending_changes/`), not in the `Changes` file itself; a
+nightly job collects them with `edg-collect-changes`. Write everything as
+**UTF-8**. For the entry format, `//allowed_words:`, `edg-collect-changes
+--check`, non-UTF-8 example bytes (`«HHHH…»` markers and the note line), and
+verification, follow skill **edg-changes-example** — do not invent a parallel
+procedure here.
 
 ## Related skills
 
 - **edg-cpfe-style** — coding style; [code_style.rst](../../../doc/source/code_style.rst).
-- **edg-changes-example** — UTF-8 Changes writes; `edg-changes-example` CLI.
+- **edg-changes-example** — Changes entries (`pending_changes/`), UTF-8;
+  `edg-collect-changes` and `edg-changes-example` CLIs.
 - **edg-cpfe-build-test** — rebuild, `edg-docker-test`, recording.
 - **edg-cpfe-bisect** — `edg-docker-test-bisect` between good/bad commits.
 - Do not invent a separate defect-tracker workflow; use the project's current

.git-hooks/pre-push.py

@@ -42,6 +42,7 @@
 import sys
 import tempfile
 
+import edgchanges
 import edgutil
 
 from pathlib import Path
@@ -170,6 +171,11 @@ def confirm_okay(self):
         print('Too many attempts.')
         sys.exit(1)
 
+  def reject(self):
+    '''Record a problem that can't be accepted, without asking.'''
+    print('    Not allowed: CI would reject this; fix it and push again.')
+    self.errors = 1
+
   def check_commits(self):
     '''Walk commits until one already on a remote branch is found.
 
@@ -287,6 +293,23 @@ def check_changed_files(self, first_new_commit):
       for file_path_str, file_results in results.items():
         print(f"  Checking file: {file_path_str}")
 
+        # A pending Changes entry with any of these problems fails the
+        # changes-fragments CI check, so there's no accepting them here.
+        is_changes_entry = edgchanges.is_fragment_path(file_path_str)
+        confirm_okay = (
+          self.reject if is_changes_entry else self.confirm_okay
+        )
+
+        format_errors = [
+          edgchanges.FormatError(*error)
+          for error in file_results['format_errors']
+        ]
+        if len(format_errors) != 0:
+          print('    Badly formatted Changes entry:')
+          for error in format_errors:
+            print(f"    - {edgchanges.describe_error(file_path_str, error)}")
+          confirm_okay()
+
         misspellings = file_results['misspellings']
         if misspellings is None:
           print('    Spell check failed:')
@@ -297,14 +320,20 @@ def check_changed_files(self, first_new_commit):
           print('    New misspelled words:')
           for misspelling in misspellings:
             print(f"    - {misspelling}")
-          self.confirm_okay()
+          if is_changes_entry:
+            hint_lines = edgchanges.allowed_words_hint(
+              file_path_str,
+              misspellings
+            )
+            print_indented('\n'.join(hint_lines), '    ')
+          confirm_okay()
 
         typos = file_results['typos']
         if len(typos) != 0:
           print('    New typos:')
           for typo in typos:
             print(f"    - {typo}")
-          self.confirm_okay()
+          confirm_okay()
 
         coding_errors = file_results['coding_errors']
         if len(coding_errors) != 0:
@@ -327,14 +356,9 @@ def check_changed_files(self, first_new_commit):
 
         overlong_lines = file_results['overlong_lines']
         if len(overlong_lines) != 0:
-          for line_no, (line_start, lines) in overlong_lines.items():
-            print(f"    Line {line_no} too long:")
-            line_end = line_start + (len(lines) - 1)
-            print(f" {line_start} -> {line_end} ".center(79, '='))
-            for line in lines:
-              print(line, end = '')
-            print('=' * 79)
-          self.confirm_okay()
+          for line in edgutil.format_overlong_lines(overlong_lines, '    '):
+            print(line)
+          confirm_okay()
 
   def run(self) -> bool:
     '''Execute the full pre-push check sequence.'''

.github/workflows/build-and-test-ci.yml

@@ -30,6 +30,10 @@ on:
   push:
     branches:
       - main
+    # The nightly collect-changes job's commits only move Changes entries.
+    paths-ignore:
+      - '**/Changes'
+      - '**/pending_changes/**'
   pull_request:
 
 concurrency:

.github/workflows/changes-fragments.yml

@@ -0,0 +1,69 @@
+# Part of the EDG Compiler Project, under the Apache License v2.0 with LLVM
+# Exceptions.
+# See https://edgcpp.org/LICENSE.txt for license information.
+# SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+#
+# Check the pending Changes entries (*/pending_changes/*.txt) with
+# `edg-collect-changes --check` (format, spelling, typos, line length), so a
+# bad entry is caught before the nightly collect-changes job meets it, and run
+# the unit tests of the tools involved.
+
+name: Pending Changes Entries
+
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - '**/pending_changes/**'
+      - .github/workflows/changes-fragments.yml
+      - dev_tools/bin/edg-check-file-changes
+      - dev_tools/bin/edg-collect-changes
+      - dev_tools/pylibs/edgchanges/**
+      - dev_tools/pylibs/edgutil/**
+      - dev_tools/tests/**
+  pull_request:
+    paths:
+      - '**/pending_changes/**'
+      - .github/workflows/changes-fragments.yml
+      - dev_tools/bin/edg-check-file-changes
+      - dev_tools/bin/edg-collect-changes
+      - dev_tools/pylibs/edgchanges/**
+      - dev_tools/pylibs/edgutil/**
+      - dev_tools/tests/**
+
+permissions:
+  contents: read
+
+jobs:
+  check:
+    name: check-pending-changes
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    env:
+      PYTHONPATH: ${{ github.workspace }}/dev_tools/pylibs
+      # Each holds a Changes file and its pending_changes/ directory (keep in
+      # step with the list in collect-changes.yml).
+      CHANGES_DIRS: src util lib_src include_c++ nt_util
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v7
+
+      # The same dictionary as the dev-env image
+      # (dev_tools/docker/dev-env/setup-scripts/install-software.sh).
+      - name: Install aspell
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y aspell aspell-en
+
+      - name: Run unit tests
+        run: python3 -m unittest discover -s dev_tools/tests -v
+
+      - name: Check pending Changes entries
+        run: |
+          status=0
+          for dir in ${CHANGES_DIRS}; do
+            dev_tools/bin/edg-collect-changes --check \
+              "${dir}/Changes" "${dir}/pending_changes" || status=1
+          done
+          exit "${status}"