Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,13 +1,22 @@
*.php text eol=lf

.agents export-ignore
.github export-ignore
bench export-ignore
examples export-ignore
tests export-ignore
tmp export-ignore
build/PHPStan export-ignore
build/stubs export-ignore
build/build-release.sh export-ignore
build/check-coverage.php export-ignore
build/check-release.sh export-ignore
build/tag-release.sh export-ignore
.editorconfig export-ignore
.gitattributes export-ignore
.gitignore export-ignore
CLAUDE.md export-ignore
infection.json5 export-ignore
Makefile export-ignore
phpcs.xml export-ignore
phpstan.neon export-ignore
Expand Down
4 changes: 2 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Tally benchmarks PHP code by counting instructions, cycles, and memory under Cod
- `src/Expression` – the expression language of complexity classes, all pure: `Lexer` (tokens with offsets), `Parser` (recursive descent into the `Node` tree, sealed), `Printer` (the canonical form, parsing it gives the same tree), `TypeChecker` (finds what the parameter types reveal, through `ValueType`), `Evaluator`, and the `MathFunction`s of the `FunctionLibrary`. Every mistake is an `ExpressionException` that quotes the expression and underlines the part that is wrong; `ExpressionErrors` words them the same for the checker and the evaluator.
- `src/Complexity` – `ComplexityClass`, the interface of `#[Complexity]` and its presets (`#[Linear]` and the others, which compose a `Complexity`), and `GrowthPoint`, where a case is on its growth curve. `Discovery\GrowthPoints` computes them at discovery, `Comparator::compareGrowth()` compares a case with its base. These cases have no numbers to accept.
- `src/Quantity` – `QuantityFormat` formats and parses numbers with units (`3.119B`, `44.49MiB`).
- `src/Expectation` – where expected metrics are stored. `ExpectationSource` is the interface: `read()` the expectation, `file()` the file it lives in, the pure `rewrite(code, metrics)` that returns the file's code with new expectations, and `location()`, which throws `ExpectationException` when the source cannot be rewritten. `AttributeExpectations` implements it for `#[Expected]` and `#[ExpectedFor]` through `ExpectedAttributeRewriter`, the pure code-to-code rewrite, and `Tokens`, the token list it walks. `ExpectationChanges` plans the rewrites of many cases (pure): one `FileChange` per file with every case in it, in a `ChangePlan` together with the cases that failed. `ExpectationFiles` reads and writes the files.
- `src/Expectation` – where expected metrics are stored. `ExpectationSource` is the interface: `read()` the expectation, `file()` the file it lives in, the pure `edits(code, metrics)` that returns the `CodeEdit`s that write new expectations into the file's code, and `location()`, which throws `ExpectationException` when the source cannot be rewritten. `AttributeExpectations` implements it for `#[Expected]` and `#[ExpectedFor]` through `ExpectedAttributeRewriter`, which reads a file once into an `AttributeIndex` of every method's attributes, with `Tokens`, the token list it walks, and locates and edits each case in the index; `DirectorySuiteLoader` indexes each file once for all its cases, because tokenizing it for every case made locating and rewriting quadratic in the cases of a file. `ExpectationChanges` plans the rewrites of many cases (pure): it applies the edits of all cases of a file in one pass, one `FileChange` per file with every case in it, in a `ChangePlan` together with the cases that failed. `ExpectationFiles` reads and writes the files.
- `src/Discovery` – `DirectorySuiteLoader` finds and loads `*Bench.php` files (I/O), `CaseExpander` turns one method into its cases (pure, works on reflection); `hooks()` finds a class's `#[Before]` and `#[After]` methods once per class, because scanning them for every benchmark method made loading a class quadratic in its size. `AttributeFactory` creates every attribute Tally reads: it checks the arguments against the attribute's constructor first (names, count, types through `ValueTypeCheck`, as under strict_types), so a mistake is an `InvalidBenchmarkException` naming the argument, not a PHP error. Everything a benchmark author can get wrong, including a file that does not load and a data provider that throws, ends as an `InvalidBenchmarkException` that names the benchmark or file; PHP errors with stack traces are bugs. `ArgumentResolver` checks that a method takes the arguments of a data set (a hook the leading ones) and `ConstructorArguments` that a constructor takes only `Meter` and `ScratchDirectory`; discovery checks both, so a mismatch fails `tally list` instead of the worker.
- `src/Selection` – `CaseSelector` implementations: `FilterSelector`, `ShardSelector` (longest-processing-time sharding by expected instructions). Both keep a case compared by its growth with its base: the filter adds the base, the shards keep a method's cases together.
- `src/Runner` – running one case in this process. `CaseRunner` calls the hooks uncounted, prepares the counter, calls the warm-up uncounted, then counts a `#warmup` span, an empty `#baseline` call, and one call of the benchmark, and subtracts the baseline's memory. The `#warmup` span exists because the first counted span costs more than later ones. `CodSpeedCounter` starts Callgrind's instrumentation once, when it is prepared, stops it when it is finished after the benchmark's span, so the after methods and the end of the process run at Valgrind's quickest, and only toggles collecting for each span: starting the instrumentation empties the simulated caches, and doing it for every span made the cycles of small benchmarks depend on the cache lines the hooks touched. `MeterSession` is the `Meter` the benchmark object gets in its constructor, one per case: every call `CaseRunner` makes is a span of it with its own `Counter` (`CodSpeedCounter` over `InstrumentHooks` for the counted ones, `IdleCounter` for hooks and warm-up), and it measures memory above the start of the span through `Memory`. `ExtensionInstrumentHooks` calls the `tally_hooks` extension, the only C that Tally calls.
Expand All @@ -24,7 +24,7 @@ Tally benchmarks PHP code by counting instructions, cycles, and memory under Cod
- `ext/tally_hooks` – the PHP extension the counted worker calls: functions in `Tally\Hooks` that make Callgrind's client requests through Valgrind's `valgrind.h` and `callgrind.h`. Built with `-DTALLY_HOOKS_RECORDING`, it records the calls instead, for the tests. Keep it to these few functions; the logic belongs in PHP.
- `bench/` – Tally's benchmarks of itself, listed in `tally.php`. They cover the code whose cost grows with the suite, a benchmark file, or a profile, and that runs once per case or once per worker: locating and rewriting expectations, planning and diffing new expectations, expanding cases, sharding, reading profiles, and rendering reports. `SuiteSource` generates the code and cases of large benchmark classes as their input. Build inputs in data providers and before methods, which are not counted, and keep the counted call to the one method being measured. `#[Warmup(1)]` keeps autoloading out of the count.
- `examples/` – short, artificial benchmarks for README.md, one feature each. `examples/Scripts` shows a data provider with its own `ExpectationSource`, which keeps the numbers in a comment at the top of each script. Keep them readable over realistic; real measurements of Tally belong in `bench/`. CI runs them under CodSpeed next to `bench/`, so they stay valid.
- `build/` – scripts for the build, such as the coverage check. `build-codspeed.sh` builds CodSpeed's Valgrind (Callgrind only, static launcher) at a pinned tag into `codspeed/linux-<machine>`, which `1.x` ignores, and `build-tally-hooks.sh` builds `ext/tally_hooks` with `phpize` next to it; `make codspeed` runs both for this machine and PHP, CI builds `tally_hooks` for PHP 8.2 to 8.6, NTS and ZTS, in the official `php` Docker images. `build-release.sh` turns `1.x` into the `release/1.x` tree by committing the builds of both platforms; CI pushes it on every push to `1.x`, and releases are tagged there. `check-release.sh` installs a release tree into a new project with Composer and counts a benchmark, which CI runs on every pull request on x86_64 and aarch64, with PHP 8.2 and 8.5, and 8.5 ZTS. `build/PHPStan` holds Tally's own PHPStan rules and `BenchmarkUsageProvider`, which tells the dead code detector about the methods Tally calls through reflection. `build/stubs` holds the stubs for what PHPStan cannot see, such as the functions of `tally_hooks`.
- `build/` – scripts for the build, such as the coverage check. `build-codspeed.sh` builds CodSpeed's Valgrind (Callgrind only, static launcher) at a pinned tag into `codspeed/linux-<machine>`, which `1.x` ignores, and `build-tally-hooks.sh` builds `ext/tally_hooks` with `phpize` next to it; `make codspeed` runs both for this machine and PHP, CI builds `tally_hooks` for PHP 8.2 to 8.6, NTS and ZTS, in the official `php` Docker images. `build-release.sh` turns `1.x` into the `release/1.x` tree by committing the builds of both platforms; CI pushes it on every push to `1.x`, and releases are tagged there with `tag-release.sh <version>`, which checks that the generated commit sits on the `1.x` commit to release. `check-release.sh` installs a release tree into a new project with Composer and counts a benchmark, which CI runs on every pull request on x86_64 and aarch64, with PHP 8.2 and 8.5, and 8.5 ZTS. `build/PHPStan` holds Tally's own PHPStan rules and `BenchmarkUsageProvider`, which tells the dead code detector about the methods Tally calls through reflection. `build/stubs` holds the stubs for what PHPStan cannot see, such as the functions of `tally_hooks`.
- `tests/Unit` – pure logic, no file system, processes, or network. `tests/Integration` – adapters and the CLI. `tests/Fixtures` – benchmark classes and configuration files the tests load; excluded from PHPStan and the coding standard because some are invalid on purpose.

## Design rules
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -224,7 +224,7 @@ Without `#[Warmup]`, the counted call is the first one. The difference can be la
```php
public function benchCold(): int|false
{
// 73.16K instructions, mostly compiling the regular expression
// 74.63K instructions, mostly compiling the regular expression
return preg_match('~^(?<user>[a-z]+)@(?<domain>[a-z]+)\.cold$~', 'tally@example.cold');
}

Expand Down Expand Up @@ -562,7 +562,7 @@ An `ExpectationSource` has four methods:
| --- | --- |
| `read(): ?Metrics` | The expected numbers, `null` for a new case. |
| `file(): string` | The file that keeps them. |
| `rewrite(string $code, Metrics $measured): string` | The code of the file with the measured numbers instead. Takes and returns code without reading or writing the file, so `tally accept` writes it and `tally run` shows it as a patch. |
| `edits(string $code, Metrics $measured): list<CodeEdit>` | The edits of the file's code that write the measured numbers instead, each a `Tally\Expectation\CodeEdit` of an offset, a length, and the text that replaces those bytes. Takes code and returns edits without reading or writing the file, so `tally accept` writes the edited code and `tally run` shows it as a patch. Tally applies the edits of all cases of a file in one pass: they must not overlap, equal edits, such as the same import, are made once, and insertions at one offset go in the natural order of their text. |
| `location(): SourceLocation` | Where reports point to. Throws `ExpectationException` when the case's numbers cannot be rewritten, so `tally run` fails before counting. |

[examples/Scripts](examples/Scripts) is the whole example.
Expand All @@ -571,11 +571,11 @@ An `ExpectationSource` has four methods:

`tally run` loads the benchmarks, then runs each case in two processes of its own: first natively, which fails quickly when the case is broken, then under CodSpeed's Valgrind, which counts it. Several cases run at once, see `--jobs`.

Every process loads the whole suite, so **data providers run in every process**: in `tally run` itself, and twice for each case. Every `tally` command runs them too. Keep them cheap and do expensive preparation in a `#[Before]` method, which runs only for the case that needs it.
`tally run` loads the whole suite, and each process of a case loads only the file of the case's class, so **the data providers of a file run twice for each of its cases**, and once in `tally run`. Every `tally` command runs them too. Keep them cheap and do expensive preparation in a `#[Before]` method, which runs only for the case that needs it. A benchmark class finds the other classes it uses through the autoloader, not because another `*Bench.php` file declared them.

In each of the two processes of a case, Tally:

1. loads the benchmarks and calls every data provider, not counted,
1. loads the file of the case's class and calls the data providers of its benchmarks, not counted,
2. creates the benchmark class, once, with its `Meter` and `ScratchDirectory`,
3. calls the `#[Before]` methods, not counted,
4. starts CodSpeed's CPU simulation, with empty caches,
Expand Down
19 changes: 12 additions & 7 deletions bench/Discovery/CaseExpanderBench.php
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
use Tally\Discovery\GrowthPoints;
use Tally\Discovery\InvalidBenchmarkException;
use Tally\Discovery\ValueTypeCheck;
use Tally\Expectation\AttributeIndex;
use Tally\Expectation\ExpectedAttributeRewriter;
use Tally\Expression\Evaluator;
use Tally\Expression\ExpressionErrors;
Expand All @@ -35,20 +36,23 @@
use function sprintf;

/**
* Every tally command, and every worker tally run starts for a case, loads the whole suite, so
* the cases of each benchmark method are expanded once per case of the run.
* Every tally command loads the whole suite, and every worker tally run starts for a case loads
* the file of its class, so the cases of each benchmark method are expanded once per case of
* their file.
*/
#[Warmup(1)]
final class CaseExpanderBench
{

/**
* @return iterable<string, array{int, int}> the methods and data sets of the class
* @return iterable<string, array{int, int, AttributeIndex}> the methods and data sets of the class,
* and the attributes of its file, which the loader indexes once before expanding its methods
*/
public static function classes(): iterable
{
$source = new SuiteSource();
foreach ([10, 100] as $size) {
yield sprintf('size-%d', $size) => [$size, $size];
yield sprintf('size-%d', $size) => [$size, $size, (new ExpectedAttributeRewriter())->index($source->code($size, $size, true))];
}
}

Expand All @@ -73,12 +77,13 @@ public function __construct(private ScratchDirectory $scratch)
public function declareClass(int $methods, int $dataSets): void
{
$source = new SuiteSource();
$code = $source->code($methods, $dataSets, true);
if (class_exists($source->className($methods, $dataSets), false)) {
return;
}

$file = sprintf('%s/Suite%dx%dBench.php', $this->scratch->path(), $methods, $dataSets);
file_put_contents($file, $source->code($methods, $dataSets, true));
file_put_contents($file, $code);
require $file;
}

Expand All @@ -88,7 +93,7 @@ public function declareClass(int $methods, int $dataSets): void
#[DataProvider('classes')]
#[ExpectedFor('size-10', instructions: '1.835M', cycles: '2.451M', memory: '19.59KiB')]
#[ExpectedFor('size-100', instructions: '19.02M', cycles: '25.07M', memory: '174.3KiB')]
public function benchExpand(int $methods, int $dataSets): void
public function benchExpand(int $methods, int $dataSets, AttributeIndex $attributes): void
{
$name = (new SuiteSource())->className($methods, $dataSets);
if (!class_exists($name, false)) {
Expand All @@ -102,7 +107,7 @@ public function benchExpand(int $methods, int $dataSets): void
continue;
}

$this->expander->expand($class, $method, $hooks, $this->configuration);
$this->expander->expand($class, $method, $hooks, $attributes, $this->configuration);
}
}

Expand Down
4 changes: 2 additions & 2 deletions bench/Expectation/ExpectationChangesBench.php
Original file line number Diff line number Diff line change
Expand Up @@ -39,8 +39,8 @@ public static function newCases(): iterable
* @param array<string, string> $code
*/
#[DataProvider('newCases')]
#[ExpectedFor('cases-2', instructions: '641.4K', cycles: '749.8K', memory: '35.08KiB')]
#[ExpectedFor('cases-200', instructions: '1.895B', cycles: '2.071B', memory: '1.043MiB')]
#[ExpectedFor('cases-2', instructions: '143.8K', cycles: '194.3K', memory: '5.148KiB')]
#[ExpectedFor('cases-200', instructions: '21.63M', cycles: '26.19M', memory: '291.0KiB')]
public function benchPlan(array $accepted, array $code): void
{
(new ExpectationChanges())->plan($accepted, $code);
Expand Down
49 changes: 24 additions & 25 deletions bench/Expectation/ExpectedAttributeRewriterBench.php
Original file line number Diff line number Diff line change
Expand Up @@ -6,67 +6,66 @@
use Tally\Attribute\ExpectedFor;
use Tally\Attribute\Warmup;
use Tally\Bench\SuiteSource;
use Tally\Expectation\AttributeIndex;
use Tally\Expectation\ExpectationException;
use Tally\Expectation\ExpectedAttributeRewriter;
use Tally\Expectation\MethodNotFoundException;
use Tally\Metrics;
use function sprintf;

/**
* tally run and tally compare locate the expectation of every case in its file, and tally accept
* rewrites it, so a file with many cases is tokenized once per case.
* Every tally command indexes the attributes of each benchmark file once, and tally run, compare,
* and accept then locate and rewrite the expectation of each case in the index.
*/
#[Warmup(1)]
final class ExpectedAttributeRewriterBench
{

/**
* @return iterable<string, array{string, string, string}> the code, the class, and the data set whose attribute to find
* @return iterable<string, array{string}>
*/
public static function files(): iterable
{
$source = new SuiteSource();
foreach ([10, 100] as $size) {
yield sprintf('size-%d', $size) => [
$source->code($size, $size, true),
$source->className($size, $size),
$source->dataSet($size),
];
yield sprintf('size-%d', $size) => [$source->code($size, $size, true)];
}
}

/**
* @throws ExpectationException
* @throws MethodNotFoundException
*/
#[DataProvider('files')]
#[ExpectedFor('size-10', instructions: '1.906M', cycles: '2.077M', memory: '128.6KiB')]
#[ExpectedFor('size-100', instructions: '16.84M', cycles: '18.60M', memory: '1.002MiB')]
public function benchLocate(string $code, string $class, string $dataSet): void
#[ExpectedFor('size-10', instructions: '2.007M', cycles: '2.209M', memory: '135.9KiB')]
#[ExpectedFor('size-100', instructions: '17.36M', cycles: '19.11M', memory: '1.093MiB')]
public function benchIndex(string $code): void
{
(new ExpectedAttributeRewriter())->locate($code, $class, SuiteSource::SIZED_METHOD, $dataSet);
(new ExpectedAttributeRewriter())->index($code);
}

/**
* @return iterable<string, array{string, string, string, Metrics}> files() with the measured metrics
* @return iterable<string, array{AttributeIndex, string, string, Metrics}> the index, the class, the data set whose attribute to rewrite, and its metrics
*/
public static function rewrites(): iterable
public static function indexes(): iterable
{
foreach (self::files() as $name => $arguments) {
yield $name => [...$arguments, (new SuiteSource())->measured(1)];
$source = new SuiteSource();
foreach ([10, 100] as $size) {
yield sprintf('size-%d', $size) => [
(new ExpectedAttributeRewriter())->index($source->code($size, $size, true)),
$source->className($size, $size),
$source->dataSet($size),
$source->measured(1),
];
}
}

/**
* @throws ExpectationException
* @throws MethodNotFoundException
*/
#[DataProvider('rewrites')]
#[ExpectedFor('size-10', instructions: '1.947M', cycles: '2.133M', memory: '128.7KiB')]
#[ExpectedFor('size-100', instructions: '16.91M', cycles: '18.68M', memory: '1.007MiB')]
public function benchRewrite(string $code, string $class, string $dataSet, Metrics $measured): void
#[DataProvider('indexes')]
#[ExpectedFor('size-10', instructions: '47.67K', cycles: '65.38K', memory: '1.445KiB')]
#[ExpectedFor('size-100', instructions: '80.65K', cycles: '103.6K', memory: '1.445KiB')]
public function benchEdits(AttributeIndex $index, string $class, string $dataSet, Metrics $measured): void
{
(new ExpectedAttributeRewriter())->rewrite($code, $class, SuiteSource::SIZED_METHOD, $dataSet, $measured, false);
(new ExpectedAttributeRewriter())->edits($index, $class, SuiteSource::SIZED_METHOD, $dataSet, $measured, false);
}

}
Loading
Loading