Skip to content

ops: rationalize scheduled workflows and recurring organization maintenance #485

Open
1 of 1 issue completed
Open
ops: rationalize scheduled workflows and recurring organization maintenance#485
1 of 1 issue completed
Assignees
ss-o
Labels
area:ciContinuous integration or GitHub Actions work.area:metaOrganization-wide policy, templates, or meta-repo work.meta:org-trackedIndicates this issue has cross-repository tracking implications (synced to Linear).type:maintenanceNon-feature maintenance, cleanup, or org work.

Description

@ss-o
ss-o
opened on Jul 26, 2026

Purpose

Create one coordination point for auditing, classifying, and improving recurring organization maintenance without replacing the owning issues for individual workstreams.

The target operating model is hybrid:

  • GitHub Actions owns deterministic repository checks and genuinely time-driven jobs.
  • Read-only agent reviews produce drafts for maintainer evaluation.
  • Mutating or high-judgment operations remain explicitly approved and are not placed behind broad unattended credentials.
  • Existing runbooks, issues, and Linear records remain the sources of truth for their respective workstreams.

Verified public baseline - 2026-07-27

A default-branch Trees and Contents API inventory covered all 86 active, public, non-fork repositories.

  • 93 scheduled workflow files exist across 51 repositories.
  • 33 are obsolete sync-labels.yml callers, already tracked by ci: 33 repositories schedule a sync-labels workflow that no longer exists #472. All 33 are currently disabled_inactivity; activity can reactivate the broken schedule.
  • 24 are scheduled Trunk callers. Eighteen use the same Friday 05:00 UTC boundary.
  • 11 are stale or lock workflows.
  • 5 are CodeQL schedules.
  • 20 are repository-specific monitoring, documentation, metrics, container-test, translation, or site workflows.

GitHub documents that scheduled workflows can be delayed or dropped during high load, particularly at the start of an hour, and that public-repository schedules are disabled after 60 days without repository activity. The current inventory therefore needs classification based on purpose and live behavior, not file presence alone.

Existing owning tracks

Do not duplicate these records:

This issue coordinates the cross-cutting recurring-operations design and links outcomes back to those owners.

Priority and complexity

P0 - high impact

  1. Complete Pilot weekly org review cadence and prompt tuning #412 cycles 2 and 3 (Medium, time-gated), then record whether the organization review stays manual, becomes a read-only scheduled draft, or is retired.
  2. Privately audit organization-level Actions credential and execution defaults (High). Inventory workflows that rely on implicit write access before proposing narrower defaults. Do not publish sensitive admin details in this issue.

P1 - standardization

  1. Remove the 33 obsolete label-sync callers under ci: 33 repositories schedule a sync-labels workflow that no longer exists #472 (Low per repository, High coordination).
  2. Classify all retained schedules (Medium) as retain, replace with an event trigger, consolidate through a reusable workflow, document as a manual runbook, or remove.
  3. Review the 24 scheduled Trunk callers (Medium). Remove unchanged-code schedules that add no regression signal, and stagger retained schedules away from shared hour boundaries.
  4. Decide stale and lock policy (Medium, maintainer judgment). Do not let unattended automation close useful work without an explicit organization rule.
  5. Review CodeQL schedule ownership (High). Prefer organization security configuration or default setup where it meets the repository's language and policy needs; retain advanced workflows only for justified exceptions.

P2 - ongoing governance

  1. Define a monthly workflow-health review (Low) using Actions usage, queue-time, duration, and failure-rate evidence.
  2. Evaluate repository classification metadata (Medium) for maintenance tier, release class, and review cadence without making unavailable paid-plan features a dependency.
  3. Define a quarterly governance review (Low) covering runbooks, ADRs, release models, documentation drift, and recurring investigations.

Classification record

Every scheduled workflow must receive these fields in the audit artifact:

Repository:
Workflow path:
Purpose:
Current state:
Cron and timezone:
Owning runbook or issue:
Required permissions:
Failure and retry behavior:
Observed value:
Classification: retain | event-driven | reusable | manual | remove
Decision evidence:
Follow-up owner:

Acceptance criteria

  • The inventory is reproducible from a committed read-only audit command rather than a one-time code search.
  • All public scheduled workflows are classified with an owner and evidence.
  • ci: 33 repositories schedule a sync-labels workflow that no longer exists #472 is completed without adding another label-sync scheduler.
  • Pilot weekly org review cadence and prompt tuning #412 records its three-cycle decision before any agent review is scheduled.
  • Retained schedules avoid unnecessary top-of-hour concentration and declare UTC or an intentional IANA timezone.
  • Repeated deterministic logic uses tested reusable workflows while callers retain repository-specific triggers and least-privilege permissions.
  • Stale/lock behavior has an explicit maintainer-approved policy.
  • Organization-level Actions and security-default recommendations are reviewed privately and rolled out through canaries before broader application.
  • The recurring-operations runbook distinguishes scheduled drafts, event-driven checks, and manual approval gates.
  • GitHub and Linear state are reconciled before closure.

Safety boundary

This issue authorizes planning and read-only inventory. It does not authorize bulk workflow deletion, organization-setting changes, unattended issue closure, repository writes, or broad credential provisioning. Each implementation batch remains separately reviewable and uses the owning repository's issue and pull request.

Activity

  1. ss-o
    self-assigned this
    on Jul 26, 2026
  2. ss-o
    added
    type:maintenanceNon-feature maintenance, cleanup, or org work.
    area:ciContinuous integration or GitHub Actions work.
    area:metaOrganization-wide policy, templates, or meta-repo work.
    meta:org-trackedIndicates this issue has cross-repository tracking implications (synced to Linear).
    on Jul 26, 2026

  3. ss-o commented on Jul 27, 2026

    @ss-o
    ss-o
    on Jul 27, 2026 – with ChatGPT Codex Connector
    MemberAuthor

    Task 1 checkpoint - 2026-07-27

    The reproducible scheduled-workflow inventory is complete in a reviewed local implementation:

    • branch: codex/recurring-operations-audit
    • commit: 32b7a44f (feat(ci): add scheduled workflow audit)
    • verification: 19 fixture tests, both Ruby syntax checks, 68 organization-policy tests, policy validation, and the existing label smoke suite passed
    • live public result: 93 scheduled workflow records, one versioned schema, and zero structured errors
    • safety: GitHub requests are GET-only; disabled workflows are included by default; private-capable JSON requires an explicit mode-0600 destination; public Markdown redaction passed
    • independent task review: specification compliant and approved, with no remaining findings

    Publication boundary: the commit is local only. It has not been pushed, opened as a pull request, merged, or reflected in the private workspace gitlink.

    This completes the inventory-command implementation unit, not issue #485. The next active gate is Cycle 2 of #412, followed by the remaining schedule classifications and policy decisions.

  4. ss-o commented on Jul 27, 2026

    @ss-o
    ss-o
    on Jul 27, 2026 – with ChatGPT Codex Connector
    MemberAuthor

    GitHub Actions authority audit ready for canary approval

    The read-only audit covered 88 active, non-fork repositories and 222
    default-branch workflow files. Of those workflows, 104 declare top-level
    permissions, 5 use job-only declarations, and 112 still depend on an implicit
    default for at least one job.

    The dependency matrix identifies 43 workflows with evidence-backed likely
    permission failures under a read default, plus 3 credential-fallback cases.
    Separately, 32 of the 49 unresolved workflows call a missing reusable target;
    they currently stop at target resolution and are not counted as permission
    failures or as safe cases. No pull-request approval operation was observed in
    default-branch workflow source.

    Decision: narrow repository canary.

    z-shell/.github is a viable proposed canary: all 16 of its workflows declare
    top-level permissions and the audit found no missing permission for recognized
    operations. No canary ran, no settings changed, and no organization default
    change is recommended from this audit.

    Next gate: explicit maintainer approval for the repository-only canary, followed
    by the documented success and rollback checks. Organization-wide evaluation
    remains deferred until that canary succeeds and rollback is tested.

  5. ss-o commented on Jul 27, 2026

    @ss-o
    ss-o
    on Jul 27, 2026
    MemberAuthor

    Task 5 read-only Trunk schedule decision audit

    The current live set still contains exactly 24 public default-branch
    .github/workflows/trunk-check.yml callers. Nine are active and 15 are
    disabled by inactivity. No open pull request modifies a scoped caller.

    Provisional classifications after current-source, reusable-target, action, and
    latest-run review:

    • 21 retain: 20 mutable main reusable targets can change and be exercised
      without a caller-repository event; the wiki schedule targets external URL
      drift.
    • 2 event-driven: zsh-eza and zsh-lint use immutable targets and have no
      accepted health probe or historical missed defect. Their version-tag pushes
      and manual runs provide all-files validation, so remove only schedule.
    • 1 reusable: zsh-editing-workbench currently executes no job on its exact
      schedule and points to a mutable legacy target, so it needs consolidation
      before a final trigger decision.
    • 0 manual or remove.

    The master-plan rule is authoritative: materially broader input is supporting
    evidence, not sufficient by itself for retention. At both executed Trunk action
    revisions, tag and first-commit pushes take the zero-before all-files path. All
    22 callers previously described by broader input include version-tag triggers.

    Run evidence covers 36 latest schedule runs across 12 workflows, plus
    structured no-run results for 12 workflows. There were 33 successes and three
    wiki failures. All three wiki failures are the same schedule-specific argument
    collision; push runs on the same heads succeeded, so the samples do not prove a
    defect missed by event-driven CI. The wiki external-link intent supports
    retention, but the schedule path needs an owning issue and repair.

    All current schedules use minute zero. Observed creation was 01:18:11 to
    03:29:10 after the cron boundary while runner queue time was zero. A
    deterministic unique non-zero minute is proposed for each retained caller,
    keeping its current weekday and hour.

    Next gates are maintainer review, owning issues for the two event-driven
    changes and wiki failure, consolidation of the zsh-editing-workbench
    exception, and a joint ref-and-trigger decision for each mutable main caller.
    Enablement, caller modernization, and minute staggering remain separate
    implementation decisions.

    This was a GET-only audit. No workflow, trigger, issue, pull request, run, or
    setting was changed.

  6. ss-o
    added this to Z-shell Deliveryon Aug 17, 2026
  7. ss-o
    moved this to Triage in Z-shell Deliveryon Aug 17, 2026

  8. ss-o commented on Aug 29, 2026

    @ss-o
    ss-o
    on Aug 29, 2026
    MemberAuthor

    F-Sy-H schedule classification update, 2026-08-29

    F-Sy-H's Trunk caller was previously in the mutable-target population. The prepared local repository change pins the reusable Trunk workflow to immutable .github commit 2a6edcbf14017df18cfac6309e434e24cc345891. With that premise changed, the Friday schedule no longer exercises newer reusable-workflow code and has no separately pinned time-driven input. Push, pull-request, tag, and manual triggers cover repository changes.

    Classification after the pin: event-driven. The local patch removes only the schedule trigger.

    The same review classified .github/workflows/assign.yml as remove: it targets retired Project 4, legacy labels, and a mutable action reference, while the Project 28 reconciler now owns organization issue inclusion.

    The matching strategic mirror note was added to Linear ZSH-30. These changes are local only; no workflow commit, push, pull request, or settings mutation has occurred.

  9. ss-o commented on Aug 30, 2026

    @ss-o
    ss-o
    on Aug 30, 2026
    MemberAuthor

    Scoped update on the metrics and Pages portion of this issue. PR #578 covers it.

    Root cause of the empty organization summary

    Not a permissions failure. metrics.yml ran green on every schedule while
    publishing placeholder output.

    lowlighter/metrics rejects GITHUB_TOKEN because it fetches data outside the
    current repository, detecting it via the absence of an x-oauth-scopes header.
    Four of seven steps passed ${{ github.token }}. In run 33300567276 the first
    step reported Token validity | (could not verify) and consumed
    REST 0 / GraphQL 0, then exited Success because plugins_errors_fatal was
    unset. base/index.mjs writes {totalCount: NaN} when the org field query
    fails, which produced the NaN members string on the wiki.

    Confirmed fixed by dispatching the corrected workflow (run 33313188835). All six
    remaining steps now report Token validity | seems ok:

    image before after
    metrics.svg NaN members 4 members
    repositories_metrics.svg 0 Stargazers, 0 Forkers, 0 Releases 1574 Stargazers, 166 Forkers, 48 Releases
    followup.svg all zeros 62 open / 767 closed issues, 2025 merged PRs

    METRICS_TOKEN does carry organization read scope; that was the open question
    and it is now settled by observed output rather than inspection.

    Redundant workflows

    gh-deploy-assets.yml and gh-push.yml existed only on the metrics branch.
    Both were unreachable: absent from main, unregistered in gh workflow list,
    and publishing to a gh-pages branch that does not exist in this repository.
    No consolidation was needed, only deletion.

    They were stale because the metrics branch had been created by branching from
    main rather than as an empty branch, so it carried a full repository copy
    frozen at 2026-05-18. The branch is now reset to generated assets only, 119
    files down to 15, with every retained blob verified byte-identical
    (029e83a1 -> 4fec54d3, recovery SHA recorded).

    Pages

    Pages was serving the entire repository from main via path: '.', exposing
    runbooks/, decisions/, lib/ and .trunk/ while omitting the generated
    SVGs. Verified against the live site: /AGENTS.md and /decisions/README.md
    returned 200, /metrics/plugin/metrics.svg and / returned 404.

    static.yml now stages a curated set (profile imagery from main, metrics
    assets from the metrics branch, filtered to images, with a generated index)
    and triggers on completion of Z Metrics, Z Reader and Z PageSpeed Insights.

    Also removed: the stargazers worldmap step. GOOGLE_MAP_TOKEN is defined in
    neither repository nor organization secrets, so it only ever rendered
    Google Maps API token is not set.

    Remaining

  10. ss-o commented on Aug 30, 2026

    @ss-o
    ss-o
    on Aug 30, 2026
    MemberAuthor

    Follow-up: the remaining items from the previous comment are now complete.

    PR #578 merged (0063360c). The Pages rework is live and verified:

    200  /                                          (generated index)
200  /metrics/plugin/metrics.svg
200  /metrics/plugin/repositories_metrics.svg
200  /profile/img/logo.svg
404  /AGENTS.md
404  /decisions/README.md
404  /runbooks/triage.md
404  /lib/labels.yml

    Pages now serves 30 curated image assets instead of the entire repository. The
    staging filter also correctly excluded profile/img/*.tar.gz.

    Orphaned renders removed. The four SVGs with no generator
    (people/*-repository.svg, projects/projects.svg) are gone; the metrics
    branch now holds exactly the eleven assets current CI regenerates. Their
    generator was activity_metrics.yml, which used ORG_TOKEN with
    plugin_people and plugin_projects and was removed from main. Restoring
    those renders means restoring that workflow, which is a separate decision and
    not currently proposed.

    Wiki repointed. z-shell/wiki#902 merged; https://wiki.zshell.dev/docs now
    loads both summary images from the Pages origin, with no remaining
    raw.githubusercontent.com references. Live output:

    • Z-Shell ... 4 members ... Verified
    • 93 Repositories, 48 Releases, 1574 Stargazers, 166 Forkers, 30 Watchers

    The originally reported defect is resolved end to end: generation, publication,
    and consumption.

  11. ss-o commented on Sep 30, 2026

    @ss-o
    ss-o
    on Sep 30, 2026
    MemberAuthor

    Project 28 triage pass. Blank values filled on this issue (existing values left unchanged):

    • Impact: Medium. Covers 93 scheduled workflows across 51 repositories and an implicit-write-permission audit: organization-wide operational risk with no current user breakage.
    • Effort: L. A coordination issue with ten workstreams and ten acceptance criteria.
  12. ss-o
    marked this as blocked on Oct 1, 2026

  13. ss-o commented on Oct 1, 2026

    @ss-o
    ss-o
    on Oct 1, 2026
    MemberAuthor

    Project 28 triage pass (2026-10-02): stays in Triage.

  14. ss-o
    marked this as blocked on Oct 1, 2026
  15. ss-o
    added a sub-issue on Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area:ciContinuous integration or GitHub Actions work.area:metaOrganization-wide policy, templates, or meta-repo work.meta:org-trackedIndicates this issue has cross-repository tracking implications (synced to Linear).type:maintenanceNon-feature maintenance, cleanup, or org work.

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions