Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
{
"name": "phpstan",
"owner": {
"name": "PHPStan"
},
"plugins": [
{
"name": "tally",
"source": "./",
"description": "Agent Skills for benchmarking PHP code with Tally."
}
]
}
7 changes: 7 additions & 0 deletions .claude-plugin/plugin.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"name": "tally",
"description": "Agent Skills for benchmarking PHP code with Tally.",
"homepage": "https://github.com/phpstan/tally",
"repository": "https://github.com/phpstan/tally",
"license": "MIT"
}
1 change: 1 addition & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
*.php text eol=lf

.agents export-ignore
.claude-plugin export-ignore
.bashunitrc export-ignore
.github export-ignore
bench export-ignore
Expand Down
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Tally benchmarks PHP code by counting instructions, cycles, and memory under Cod
- `src/Complexity` – `ComplexityClass`, the interface of `#[Complexity]` and its presets (`#[Linear]` and the others, which compose a `Complexity`), and `GrowthPoint`, where a case is on its growth curve. `Discovery\GrowthPoints` computes them at discovery, `Comparator::compareGrowth()` compares a case with its base. These cases have no numbers to accept: `PresetFit` (pure) finds the lowest preset the measured instructions of a method grow within, which `tally accept` writes instead of the method's preset through the `ComplexityAttribute` of its `GrowthPoint`.
- `src/Quantity` – `QuantityFormat` formats and parses numbers with units (`3.119B`, `44.49MiB`).
- `src/Expectation` – where expected metrics are stored. `ExpectationSource` is the interface: `read()` the expectation, `file()` the file it lives in, the pure `edits(code, metrics)` that returns the `CodeEdit`s that write new expectations into the file's code, and `location()`, which throws `ExpectationException` when the source cannot be rewritten. `AttributeExpectations` implements it for `#[Expected]` and `#[ExpectedFor]` through `ExpectedAttributeRewriter`, which reads a file once into an `AttributeIndex` of every method's attributes, with `Tokens`, the token list it walks, and locates and edits each case in the index; `ComplexityAttribute` replaces the complexity class of a method through `ExpectedAttributeRewriter::complexityEdits()`, in the same index; `DirectorySuiteLoader` indexes each file once for all its cases, because tokenizing it for every case made locating and rewriting quadratic in the cases of a file. `ExpectationChanges` plans the rewrites of many cases and complexity classes (pure): it applies the edits of all of them in a file in one pass, one `FileChange` per file with every change in it, in a `ChangePlan` together with the cases that failed. `ExpectationFiles` reads and writes the files.
- `src/Discovery` – `DirectorySuiteLoader` finds and loads `*Bench.php` files (I/O), `CaseExpander` turns one method into its cases (pure, works on reflection); `hooks()` finds a class's `#[Before]` and `#[After]` methods once per class, because scanning them for every benchmark method made loading a class quadratic in its size. `AttributeFactory` creates every attribute Tally reads: it checks the arguments against the attribute's constructor first (names, count, types through `ValueTypeCheck`, as under strict_types), so a mistake is an `InvalidBenchmarkException` naming the argument, not a PHP error. Everything a benchmark author can get wrong, including a file that does not load and a data provider that throws, ends as an `InvalidBenchmarkException` that names the benchmark or file; PHP errors with stack traces are bugs. `ArgumentResolver` checks that a method takes the arguments of a data set (a hook the leading ones) and `ConstructorArguments` that a constructor takes only `Meter` and `ScratchDirectory`; discovery checks both, so a mismatch fails `tally list` instead of the worker.
- `src/Discovery` – `DirectorySuiteLoader` finds and loads `*Bench.php` files and takes the class named like each file as its benchmark class, so helpers can live next to it (I/O), `CaseExpander` turns one method into its cases (pure, works on reflection); `hooks()` finds a class's `#[Before]` and `#[After]` methods once per class, because scanning them for every benchmark method made loading a class quadratic in its size. `AttributeFactory` creates every attribute Tally reads: it checks the arguments against the attribute's constructor first (names, count, types through `ValueTypeCheck`, as under strict_types), so a mistake is an `InvalidBenchmarkException` naming the argument, not a PHP error. Everything a benchmark author can get wrong, including a file that does not load and a data provider that throws, ends as an `InvalidBenchmarkException` that names the benchmark or file; PHP errors with stack traces are bugs. `ArgumentResolver` checks that a method takes the arguments of a data set (a hook the leading ones) and `ConstructorArguments` that a constructor takes only `Meter` and `ScratchDirectory`; discovery checks both, so a mismatch fails `tally list` instead of the worker.
- `src/Selection` – `CaseSelector` implementations: `FilterSelector`, `ShardSelector` (longest-processing-time sharding by expected instructions). Both keep a case compared by its growth with its base: the filter adds the base, the shards keep a method's cases together.
- `src/Runner` – running one case in this process. `CaseRunner` calls the hooks uncounted, prepares the counter, calls the warm-up uncounted, then counts a `#warmup` span, an empty `#baseline` call, and one call of the benchmark, and subtracts the baseline's memory. The `#warmup` span exists because the first counted span costs more than later ones. `CodSpeedCounter` starts Callgrind's instrumentation once, when it is prepared, stops it when it is finished after the benchmark's span, so the after methods and the end of the process run at Valgrind's quickest, and only toggles collecting for each span: starting the instrumentation empties the simulated caches, and doing it for every span made the cycles of small benchmarks depend on the cache lines the hooks touched. `MeterSession` is the `Meter` the benchmark object gets in its constructor, one per case: every call `CaseRunner` makes is a span of it with its own `Counter` (`CodSpeedCounter` over `InstrumentHooks` for the counted ones, `IdleCounter` for hooks and warm-up), and it measures memory above the start of the span through `Memory`. `ExtensionInstrumentHooks` calls the `tally_hooks` extension, the only C that Tally calls.
- `src/Comparison` – `Comparator` turns a case's expectations and measurement into a `CaseComparison` of `MetricComparison`s with a `Status`. `TolerancePolicy` is implemented by the tolerance attributes; `CaseExpander` picks the method's, else the class's, else that of the `Configuration`, which has one for expected numbers and one for growth. `Percent` formats shares.
Expand All @@ -26,6 +26,7 @@ Tally benchmarks PHP code by counting instructions, cycles, and memory under Cod
- `examples/` – short, artificial benchmarks for README.md, one feature each. `examples/Scripts` shows a data provider with its own `ExpectationSource`, which keeps the numbers in a comment at the top of each script. Keep them readable over realistic; real measurements of Tally belong in `bench/`. CI runs them under CodSpeed next to `bench/`, so they stay valid.
- `build/` – scripts for the build, such as the coverage check. `build-codspeed.sh` builds CodSpeed's Valgrind (Callgrind only, static launcher) at a pinned tag into `codspeed/linux-<machine>`, which `1.x` ignores, and `build-tally-hooks.sh` builds `ext/tally_hooks` with `phpize` next to it; `make codspeed` runs both for this machine and PHP, CI builds `tally_hooks` for PHP 8.2 to 8.6, NTS and ZTS, in the official `php` Docker images. `build-release.sh` turns `1.x` into the `release/1.x` tree by committing the builds of both platforms; CI pushes it on every push to `1.x`, and releases are tagged there with `tag-release.sh <version>`, which checks that the generated commit sits on the `1.x` commit to release. `check-release.sh` installs a release tree into a new project with Composer and counts a benchmark, which CI runs on every pull request on x86_64 and aarch64, with PHP 8.2 and 8.5, and 8.5 ZTS. `build/PHPStan` holds Tally's own PHPStan rules and `BenchmarkUsageProvider`, which tells the dead code detector about the methods Tally calls through reflection. `build/stubs` holds the stubs for what PHPStan cannot see, such as the functions of `tally_hooks`.
- `e2e/` – end-to-end tests with [bashunit](https://bashunit.com), `<name>/<name>_test.sh`; `e2e/bootstrap.sh`, set in `.bashunitrc`, gives each test a directory of its own, which is also its `TMPDIR`. `stripped-extensions` counts a benchmark with CodSpeed's Valgrind while PHP loads the extensions in `E2E_EXTENSIONS` in several ways (its scan directory, another one, the php.ini in `PHPRC`, `-c`): the case must not see them and must count what it counts on a PHP without them. CI installs bashunit into `e2e/` and runs it once per PHP setup (Xdebug, PCOV, both); locally, with the extensions loaded and bashunit installed the same way: `E2E_EXTENSIONS='xdebug pcov' e2e/bashunit test e2e`. Like `tests/Fixtures`, their PHP files are only linted.
- `skills/` – Agent Skills for projects that use Tally, shipped in the Composer package, where `extra.skills` declares them for `llm/skills`, and as the plugin `tally` of the marketplace in `.claude-plugin/`, which Claude Code and Codex both read. `tally-rewind` turns past performance fixes of a project into benchmarks, replays them over its history, and improves on the best numbers; its `scripts/measure.sh` measures bench files at commits of the history, each in a worktree, with the Tally version the project has installed.
- `tests/Unit` – pure logic, no file system, processes, or network. `tests/Integration` – adapters and the CLI. `tests/Fixtures` – benchmark classes and configuration files the tests load; excluded from PHPStan and the coding standard because some are invalid on purpose.

## Design rules
Expand Down
41 changes: 40 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ Tally runs each benchmark once under [CodSpeed](https://codspeed.io)'s CPU simul
- [Contributing a benchmark](#contributing-a-benchmark)
- [Storing expectations elsewhere](#storing-expectations-elsewhere)
- [Lifecycle of a case](#lifecycle-of-a-case)
- [Agent skills](#agent-skills)

## Installation

Expand All @@ -60,7 +61,7 @@ The counts depend on the PHP build and the system libraries. Numbers measured on

## Writing benchmarks

A benchmark is a public, non-static method whose name starts with `bench`, in a class in a file whose name ends with `Bench.php`. Tally creates the class with the constructor arguments described [below](#meter-and-scratch-directory). A `bench*` method that is static or not public is an error, so is a path without any benchmarks.
A benchmark is a public, non-static method whose name starts with `bench`, in a file whose name ends with `Bench.php`, in the class named like the file, such as `StrBench` in `StrBench.php`. Other classes in the file, such as helpers, are not benchmarks. Tally creates the class with the constructor arguments described [below](#meter-and-scratch-directory). A `bench*` method that is static or not public is an error, so is a path without any benchmarks.

```php
use Tally\Attribute\DataProvider;
Expand Down Expand Up @@ -358,6 +359,8 @@ Accept the new numbers
vendor/bin/tally accept tmp/tally
```

The commands under "Reproduce locally" and "Accept the new numbers" run Tally the way you ran it, such as `tools/tally/vendor/bin/tally` when Tally has a Composer project of its own.

The progress bar shows one character per case: `.` unchanged, `S` slower, `F` faster, `N` new, `E` not measured.

A case is **slower** when a metric exceeds its expected value by more than the tolerance, and **faster** when it is lower by more. A case that throws, exits, or times out is **not measured**. A case without expected numbers is **new**. Slower and not measured cases fail the run. Faster and new cases pass, but their numbers should be accepted, so a later slowdown back to the old numbers does not go unnoticed.
Expand Down Expand Up @@ -607,6 +610,42 @@ Instructions and cycles come from Callgrind's profile of the benchmark's span. M

Cycles are Valgrind's estimate: instructions, plus 10 for each first-level cache miss and 100 for each last-level cache miss of the simulated cache. The cache sizes are the same on every machine.

## Agent skills

Tally comes with [Agent Skills](https://agentskills.io) for coding agents such as Claude Code and Codex, in [skills/](skills):

- `tally-rewind` finds the commits of a project that fixed performance problems, turns each into a Tally benchmark locked at the fix, and replays it on every release since. It then makes today's code beat the best numbers the code ever had, keeping only changes that make a real workload faster in wall-clock time. It adds the benchmarks to the project, and sets up Tally and its CI job when the project has none.

Install them as a plugin, for all your projects:

```bash
claude plugin marketplace add phpstan/tally
claude plugin install tally@phpstan

codex plugin marketplace add phpstan/tally
codex plugin add tally@phpstan
```

Or into one project, in `.agents/skills/`, with [skills](https://github.com/vercel-labs/skills), which links them into `.claude/skills/` for Claude Code:

```bash
npx skills add phpstan/tally -a claude-code -a codex
```

Or with Composer and [llm/skills](https://github.com/roxblnfk/skills), which copies the skills of the packages a project requires into `.agents/skills/` after every `composer install` and `update`, so they stay in the version of Tally the project has:

```bash
composer require --dev llm/skills
```

Claude Code reads `.claude/skills/`, so for it add a `skills.json` next to `composer.json`, then run `composer skills:update`:

```json
{
"aliases": [".claude/skills"]
}
```

## License

MIT
5 changes: 4 additions & 1 deletion bin/tally
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,8 @@ use Tally\Cli\ApplicationFactory;
$iniFiles[] = $iniFile;
}

exit((new ApplicationFactory())->create(STDOUT, STDERR, $workingDirectory, $environment, $colors, $errorColors, PHP_BINARY, $iniFiles, __FILE__, $codspeed)->run($arguments));
// the commands in the report run Tally the way the user did, such as tools/tally/vendor/bin/tally
$program = $arguments[0] ?? 'vendor/bin/tally';

exit((new ApplicationFactory())->create(STDOUT, STDERR, $workingDirectory, $environment, $colors, $errorColors, PHP_BINARY, $iniFiles, __FILE__, $program, $codspeed)->run($arguments));
})();
7 changes: 6 additions & 1 deletion composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,10 @@
},
"bin": [
"bin/tally"
]
],
"extra": {
"skills": {
"source": "skills"
}
}
}
Loading
Loading