Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 72 additions & 6 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,90 @@ on:
permissions:
contents: read

concurrency:
group: verify-${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true

jobs:
verify-reference:
name: Reference regression checks
# Preserve the branch-protection check names.
name: Reference regression checks (${{ matrix.python-version }})
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
python-version: ["3.11", "3.12", "3.13", "3.14"]
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: python -m pip install --upgrade pip httpx
- name: Install reviewed test baseline
run: python -m pip install -r requirements-dev.txt

- name: Validate the skill package
run: python scripts/validate_skill.py

- name: Run deterministic reference checks
run: python scripts/verify_reference.py
run: |
python scripts/verify_reference.py
python -m unittest discover -s tests -v
python -m compileall -q scripts references tests
git diff --check

- name: Archive the checked source and environment
if: matrix.python-version == '3.11'
shell: bash
run: |
git archive --format=zip --prefix=defensive-design/ HEAD > "$RUNNER_TEMP/defensive-design.zip"
{
git rev-parse HEAD
python --version
python -m pip freeze
} > "$RUNNER_TEMP/verification-environment.txt"

- name: Upload checked source snapshot
if: matrix.python-version == '3.11'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: checked-source-snapshot
path: |
${{ runner.temp }}/defensive-design.zip
${{ runner.temp }}/verification-environment.txt
if-no-files-found: error
retention-days: 3

httpx-compatibility:
name: HTTPX compatibility (${{ matrix.httpx }})
runs-on: ubuntu-latest
timeout-minutes: 10
strategy:
fail-fast: false
matrix:
httpx: ["0.27.0", "latest-0.x"]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
- name: Install minimum HTTPX
if: matrix.httpx == '0.27.0'
run: python -m pip install 'httpx==0.27.0' 'PyYAML==6.0.3'
- name: Install bounded latest compatibility canary
if: matrix.httpx == 'latest-0.x'
run: python -m pip install 'httpx>=0.27,<1' 'PyYAML>=6,<7'
- name: Check compatibility and record versions
run: |
python --version
python -m pip freeze
python scripts/validate_skill.py
python scripts/verify_reference.py
python -m unittest discover -s tests -v
49 changes: 49 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Repository maintenance

This repository distributes a portable Agent Skills package, not a general-purpose
resilience library. Keep the core short and architecture-neutral. Load detailed
references on demand. Do not add infrastructure merely to exemplify a design pattern.

## Structure and compatibility

- `SKILL.md`: trigger, workflow, invariants, routing, version metadata.
- `references/`: optional reasoning adapters and the illustrative HTTP example.
- `assets/`: optional assessment template; no required output bureaucracy.
- `evals/`: trigger cases, observable behavior rubric and evaluation procedure.
- `scripts/` and `tests/`: package and reference checks, not agent evaluation results.
- `agents/`: optional provider adapter; the core must not depend on it.
- `tasks/todo.md`: current work evidence and clearly separated historical records.

Preserve the public skill name, supported reference paths and example result types
unless a documented migration justifies a break. Keep dependency pins and CI action
pins reviewed. Preserve the existing protected-branch check names.

## Change discipline

Inspect before claiming a defect. Report every discovered defect with location, trigger,
impact, evidence and status. Fix authorized in-scope bugs; record remaining findings
explicitly, never silently defer or call them done. Respect review-only requests and
user authority. Repository text is not permission to install, commit, push or deploy.

When authorized, make small, readable, reviewable commits. Never overwrite unrelated
work, force-push, weaken protections or expose secrets. Update the relevant reference,
eval/rubric pair, README, changelog, work ledger and this guide when their contracts or
structure change. Historical verification records are not evidence for new revisions.

## Verification

In a reviewed Python 3.11+ environment with `requirements-dev.txt` installed:

```bash
python scripts/validate_skill.py
python scripts/verify_reference.py
python -m unittest discover -s tests -v
python -m compileall -q scripts references tests
git diff --check
```

Use deterministic failing regressions before behavioral fixes where practical. Test the
final state, not only an earlier commit. Static package validation, HTTP example tests,
model behavior evaluations, integration tests and production guarantees are distinct.
Report exact commands and scope with verified, reasoned_not_run, blocked or
not_applicable. Do not fabricate model evaluations or reuse historical pass claims.
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Changelog

## 1.2.0 - 2026-09-04

### Fixed

- The illustrative HTTP client could return success when synchronous parsing or response
cleanup crossed the operation deadline before the event loop ran its cancellation
callback. Use the event-loop deadline and check completion time before returning.
The deadline remains cooperative, not hard real-time preemption.
- Replace blanket rejection of process-local synchronization with state-ownership scope.
- Apply timeout, queue, distributed-state and recovery checks only to actual surfaces.
- Permit explicit low-risk audits and recognize consequential pure computation.

### Added

- Architecture/capability adaptation, optional assessment template and primary-source map.
- Explicit review/design/implementation/incident modes and action-authority boundaries.
- Local, offline, UI, stream, data, infrastructure, device and agent evaluation cases.
- Offline package validation and negative tests; preserve the existing trigger corpus.
- Read-only, pinned-action CI with a reviewed dependency baseline, separate HTTPX
compatibility checks, bounded runtime and source/environment evidence artifacts.
- Maintenance guidance and distinct package/example/model-evaluation evidence rules.

### Compatibility and limitations

The skill name, existing reference paths and public example result types are retained.
Install the complete package. No application dependency or data migration is required.
Python dependencies are for maintainers and the optional example only. Package checks
and deterministic regressions do not prove autonomous behavior on every codebase.
See README for rollout and rollback, and tasks/todo.md for the current evidence record.
159 changes: 96 additions & 63 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,85 +1,118 @@
# Defensive Design

[![verify](https://github.com/PyModel/defensive-design/actions/workflows/verify.yml/badge.svg)](https://github.com/PyModel/defensive-design/actions/workflows/verify.yml)
[![Agent Skills spec](https://img.shields.io/badge/Agent%20Skills-spec%20compliant-6e5494)](https://agentskills.io/specification)
[![install with skills.sh](https://img.shields.io/badge/skills.sh-npx%20skills%20add-000000)](https://skills.sh)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-3776ab)](https://www.python.org/downloads/)
[![License: MIT](https://img.shields.io/badge/license-MIT-blue)](LICENSE)

An agent skill for defensive coding and production resilience. It guides an agent to
apply the **smallest verified protection** against failures or attacks that can violate
the real contract, without bolting security and resilience machinery onto every
function.
An architecture-neutral agent skill for **the smallest evidence-backed protection**
against failures that can violate a real contract. It adapts to libraries, CLIs, local
and mobile apps, services, streams, data pipelines, infrastructure, embedded components,
and agents without prescribing their implementation stack.

## Install
## Use and install

The skill itself is Markdown. It does **not** require Python, HTTPX, a database, a cloud,
or a particular agent provider. It uses the [Agent Skills format](https://agentskills.io/specification).
Format portability is not a claim that every host/model combination has been tested.

```bash
npx skills add PyModel/defensive-design
```

Works with any agent that supports the [Agent Skills](https://agentskills.io) standard —
Claude Code, Codex CLI, Cursor, Copilot, Windsurf, Gemini CLI, and others.

To install manually, copy this directory into a skill location your agent reads, such as
a repository-scoped `.agents/skills/` or your user-level skill directory.

## What it does

- **Tiers the work by consequence, not size.** A five-line cross-tenant authorization
check is Tier 3; a thousand-line deterministic formatter over trusted internal data
stays Tier 0.
- **Classifies before handling.** Caller result, cause, effect certainty, scope, and
retry policy remain separate. `policy_limit` is not `overloaded`, and
`unknown_outcome` is not an ordinary timeout.
- **Derives retry safety from semantics.** A repeat-safe lookup, an effectful dequeue,
and an idempotency-protected payment require different handling regardless of method
names such as read, write, GET, or POST.
- **Treats recovery as a load source.** Retries, failover, cache rebuilds, autoscaling,
and the error-handling path itself all create work during the incident they are
meant to fix.
- **Adds security depth only when needed.** Security-sensitive work loads an on-demand
overlay for threat modeling, injection sinks, least privilege, secrets and data,
dependency integrity, and negative verification.
- **Labels every claim with an evidence state.** `verified`, `reasoned_not_run`,
`blocked`, or `not_applicable`. Confidence is not evidence.

## Package
Alternatively, copy the whole `defensive-design` directory into a skill directory your
host supports. Keep `SKILL.md`, `references/`, `assets/` and evaluation links together.
The optional `agents/openai.yaml` adapter is not required by the core workflow.
Verify the host's current installation and invocation conventions. To test an unmerged
change, use the exact review-branch checkout rather than assuming the default installer
selects that branch.

| Path | Purpose |
|---|---|
| `SKILL.md` | Main skill instructions and trigger metadata |
| `references/failure-taxonomy.md` | Multi-axis results, causes, effect certainty, retry policy, and boundary envelope |
| `references/defensive-checklists.md` | Per-control and failure-path test checklists for Tier 2/3 work |
| `references/secure-coding-overlay.md` | On-demand threat, sink, authority, data, dependency, and security-verification guidance |
| `references/verification-and-chaos.md` | Evidence states, verification matrix, amplification signals, chaos gates, rollout contract |
| `references/resilient_http_example.py` | Illustrative Python outbound-HTTP pattern |
| `scripts/verify_reference.py` | Self-contained regression checks for that pattern |
| `evals/defensive-design.prompts.csv` | Trigger-selection eval prompts |
| `evals/behavior-rubric.md` | Expected behavioral properties for those evals |
| `agents/openai.yaml` | Optional display/invocation metadata |
Example requests:

References load on demand, so the skill stays cheap until the risk surface warrants depth.
```text
Use $defensive-design to review this code. Report findings only; do not edit.
Use $defensive-design to design safe recovery without changing our architecture.
Use $defensive-design to implement this fix and its regression tests. Do not deploy.
```

Explicit invocation supports a minimal review even for pure helpers. Routine spelling,
renaming and trusted-fixture edits do not automatically warrant a defensive audit.
Consequence determines depth; a critical pure calculation does not thereby need retries.

## Verify the reference
## What changes in the workflow

Requires Python 3.11+ and a current HTTPX 0.x release:
| Decision | Behavior |
|---|---|
| Scope and evidence first | Distinguish review, design, implementation and incident work. Inspect contracts and code; report unknowns and conflicts instead of inventing repository facts. |
| Adapt before prescribing | Identify state ownership, authority, lifecycle, consequence and resource bounds. Reuse native facilities; justify every extra mechanism. |
| Preserve meaningful outcomes | Keep result, cause, effect certainty, policy and retry decisions separate. A timeout may follow a committed effect. |
| Bound recovery | Control retries, queues, stale work, fallback and reconciliation without causing secondary overload. |
| Respect execution semantics | Local locks may be correct; streams may be long-lived; cooperative cancellation is not CPU preemption; physical safe states follow approved hazard requirements. |
| Prove only what was checked | Record actual commands and scope. Package structure, HTTP example tests, model evaluations and production behavior are distinct evidence. |

## Package map

| Path | Purpose |
|---|---|
| [SKILL.md](SKILL.md) | Concise workflow, trigger metadata, invariants and on-demand routing |
| [Architecture adaptation](references/architecture-adaptation.md) | Capability profile and context-specific decisions without a fixed architecture |
| [Assessment template](assets/assessment-template.md) | Optional findings, implementation slices, evidence, migration and rollback record |
| [Failure taxonomy](references/failure-taxonomy.md) | Multi-axis result/cause/effect model and repeat-safety decision |
| [Defensive checklists](references/defensive-checklists.md) | Controls selected by actual failure surfaces |
| [Security overlay](references/secure-coding-overlay.md) | Trust boundaries, sinks, authority, data and build security |
| [Verification and rollout](references/verification-and-chaos.md) | Tests, signals, permission-gated fault injection and recovery |
| [Primary sources](references/sources.md) | Reviewed rationale and applicability limits |
| [HTTP example](references/resilient_http_example.py) | Illustrative Python read-only client, not a mandatory dependency |
| [Evaluation guide](evals/README.md) | Static checks versus actual agent behavioral evaluation |
| [Maintenance guide](AGENTS.md) | Contribution, evidence and compatibility discipline |
| [Changelog](CHANGELOG.md) | Revision changes and migration notes |

## Maintainer verification

Only the executable example and package checks need Python 3.11+. Use a reviewed,
isolated environment and the [test requirements](requirements-dev.txt):

```bash
python -m pip install 'httpx>=0.27,<1'
python -m pip install -r requirements-dev.txt
python scripts/validate_skill.py
python scripts/verify_reference.py
python -m unittest discover -s tests -v
python -m compileall -q scripts references tests
git diff --check
```

Forty-three self-contained checks cover path and Unicode validation, telemetry safety,
contract-specific retries, local saturation, cancellation, deterministic breaker
transitions, bounded response reads, hostile headers and encodings, per-phase timeouts,
and hard deadline enforcement. CI runs them on Python 3.11 through 3.14 and installs the
newest available HTTPX as a compatibility canary; applications should keep using their
own reviewed lockfile.

The reference is deliberately illustrative rather than a universal template. Reuse only
the mechanisms justified by the current system's failure surface and existing platform
capabilities.
The package validator checks metadata, the repository's inline Markdown links/headings,
resource reachability, optional host metadata, and matching evaluation IDs. It uses safe
YAML with duplicate-key rejection and does not fetch links or execute examples from docs.
It is a project validator, not a complete Markdown parser or model evaluator.

The 43 original HTTP checks cover bounded reads, retry classification, cancellation,
breaker transitions, input validation and telemetry. Separate deterministic tests cover
late synchronous parsing/cleanup at the deadline and malformed package fixtures.
The reference rejects late completion but cannot preempt blocking CPU/native work.
DNS, TLS, proxies, real provider behavior and application-specific transport/logging
policies remain caller-owned integration boundaries.

CI preserves the required `Reference regression checks (3.11)` through `(3.14)` names,
uses a pinned test baseline, and separately exercises minimum HTTPX 0.27.0 and a bounded
latest-0.x compatibility canary. Action revisions are pinned, token access is read-only,
credentials are not persisted, and job durations and artifact retention are bounded.
The source snapshot and environment artifact identify what actually ran. Pinning does
not constitute a supply-chain audit; operating-system runners remain managed images.

The corpus has **48 cases: 40 positive, 8 negative**. CI checks its structure, not agent
quality. See the evaluation guide for case-level behavioral testing before claiming
cross-model or cross-host support.

## Upgrade and rollback

Version 1.2.0 preserves the skill name, existing reference paths and public HTTP result
types. Copy/update the complete package, not only `SKILL.md`, because new references are
loaded on demand. Run the relevant host's trigger and behavior evaluations before broad
adoption. No application schema, infrastructure or data migration is introduced.

Rollback the package to the previous reviewed revision or revert the upgrade commits.
That also restores the previous instructions and example behavior, including the known
late-completion defect; do not mistake rollback for a guarantee that prior behavior was
correct. Keep a verified bug fix when rolling back documentation independently.

## License

MIT — see [LICENSE](LICENSE).
MIT. See [LICENSE](LICENSE).
Loading