INTEROP-8976, INTEROP-8979: Token expiry alerts and per-rule Slack notifications

[amp-rh]

Summary

Two related features unified under a shared Slack notification infrastructure.

Token expiry alerts (INTEROP-8976)

Jira Cloud does not support programmatic API token rotation. This PR adds tooling for scheduled expiry monitoring with manual rotation:

• check-token-expiry.py: Reads expires_at from the Vault KV secret and posts Slack alerts at 30/14/7/3/1 days before expiry. Exit code 2 if expired. Uses SlackClient.post_webhook() from the shared Slack infrastructure.
• prow-periodic.yaml: Reference Prow periodic job config (daily at 08:00 UTC). To be merged into openshift/release once approved.
• docs/vault-schema.md: Full Vault KV secret schema documenting all 10 fields (5 managed by rotation, 5 that must not be modified). Warns against vault kv put in favor of vault kv patch.
• docs/rotation-runbook.md: Step-by-step manual rotation procedure with staging verification, rollback, and troubleshooting. Tested against stage-redhat.atlassian.net.

Per-rule Slack notifications (INTEROP-8979)

Adds optional slack_channel field to firewatch config rules. When set, firewatch sends a Slack notification after creating or updating a Jira issue. Empty or absent value skips notification. Supports !default to read from $FIREWATCH_DEFAULT_SLACK_CHANNEL.

Changed files:

• src/objects/rule.py: slack_channel field with !default / env var pattern
• src/objects/slack_base.py: post_webhook() static method for Slack incoming webhooks
• src/commands/report.py: --slack-bot-token and --slack-webhook-url CLI options
• src/objects/configuration.py: Stores Slack credentials
• src/report/report.py: _notify_slack(), _slack_new_issue(), _slack_duplicate(), _slack_success() hooks

Config example:

{
  "failure_rules": [
    {
      "step": "install",
      "failure_type": "pod_failure",
      "classification": "Infrastructure",
      "jira_project": "LPINTEROP",
      "slack_channel": "#ocp-ci-firewatch-tool"
    }
  ]
}

Shared infrastructure

Both features post to Slack via SlackClient.post_webhook(). The token-expiry script previously had its own slack_post() function; it now uses the shared method. Errors propagate to callers: the expiry script surfaces them as exit codes while Report notifications log and continue.

Vault schema change

The Vault secret at kv/selfservice/firewatch-tool/jira-credentials contains 10 fields. Five are managed during token rotation:

Field	Purpose
email	Jira account email (firewatch@redhat.com)
access_token	Production Jira API token
access_token_msi	Copy of access_token (kept in sync)
access_token_stage	Staging-only token (rotated independently)
expires_at	ISO 8601 date for alert script (firewatch itself does not read this)

The remaining 5 fields (account credentials, secretsync config) must not be modified during rotation. Always use vault kv patch, never vault kv put, to avoid destroying these fields.

Doc updates from staging walkthrough

The rotation runbook and vault schema were validated against the Jira staging environment (stage-redhat.atlassian.net). Key fixes applied:

• Changed vault kv put to vault kv patch (a put would destroy 7 undocumented fields)
• Documented all 10 actual Vault fields, not just 3
• Added staging verification step before CI
• Noted that API tokens are account-scoped (work across all Atlassian Cloud instances)
• Specified vault login -method=oidc for Vault authentication
• Added access_token_msi to the rotation patch command

Test plan

• Review check-token-expiry.py for correctness (Vault read, date parsing, shared Slack webhook)
• Verify prow-periodic.yaml is valid Prow config
• Review docs/vault-schema.md (all 10 fields documented, patch not put)
• Review docs/rotation-runbook.md (staging verification, OIDC auth, patch command)
• Review Slack notification hooks in report.py
• Review slack_channel field parsing in rule.py
• After merge: deploy Prow periodic job, configure slack_channel in team configs

Relates: https://issues.redhat.com/browse/INTEROP-8976
Relates: https://issues.redhat.com/browse/INTEROP-8979

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[amp-rh]

/test all