Startup takes long

[maziac]

Hi, thanks for this very good tool.
I only run into one problem: for me it takes about 10 secs until the benchmarking starts.
(I.e. stepping over "run()" takes 10 seconds, even with "time" set to 100)
So even a single test takes that long.
Is this normal?
Or are there any options to get the time down?

[jerome-benoit]

Hi, the ~10s delay before results is the warmup phase, which is enabled by default. Before the actual measurement, tinybench runs each task through a warmup pass to let the JavaScript engine optimize the code (JIT compilation, inline caches, hidden class stabilization). Without warmup, the first iterations include compilation overhead, which inflates the measured times and reduces result accuracy.

The defaults are:

warmup: true              // enabled by default
warmupTime: 250           // 250ms minimum per task
warmupIterations: 16      // 16 iterations minimum per task

If you have several tasks or your task is slow, warmup alone can account for multiple seconds.

You can reduce it or disable it depending on your needs:

// Faster warmup (still lets the engine optimize, just less)
const bench = new Bench({ time: 100, warmupTime: 50, warmupIterations: 2 })

// No warmup at all (fastest, but first samples may include JIT overhead)
const bench = new Bench({ time: 100, warmup: false })

Disabling warmup is fine for quick iteration during development, but for publishable benchmarks the warmup is recommended to get stable, representative numbers.

[maziac]

With warmup: false I get about 3 secs instead of 11. That's a good improvement. But still quite big.
'time' is set to 100ms. Is it possible to get down to that number?

[Uzlopak]

If you provide a test case as reproduction, we could investigate.

[maziac]

Here is a sample project:

tinybench-537-b.zip

The property 'warmup" does not exist, I therefore used warmupTime: 0.
I measured the time to run bench.run().
The time now goes down to 600ms for running 100ms tests (time: 100).
Still strange that it consumes so much time.

But more drastic it becomes when it is run in debug mode.
Then it takes 7000 ms on my system (mac m2).
OK, probably some setup is done and there is an overhead in debug mode, but 7 seconds seems big.

To run the program use:

npm i
npm run build
npm run dev

For debugging I started the program in vscode, there is a launch.json provided.

[jerome-benoit]

The property 'warmup" does not exist, I therefore used warmupTime: 0.

Of course it exists:

tinybench/src/types.ts

Line 219 in 634ec44

warmup?: boolean

and

tinybench/src/bench.ts

Line 189 in 634ec44

this.warmup = restOptions.warmup ?? true

And warmupTime: 0 ≠ warmup: false

Please check the version you are using to actually use the current version.

[maziac]

Sorry, when creating the testcase I somehow installed a quite old version. I now updated to 6.0.2
and used warmup: false but the problems remain the same.

With

{
		time: 100,
		warmup: false,
}

the run function takes about 800 ms and in debug mode 7 seconds (7080ms).

Here is the code:

tinybench-537-c.zip

[jerome-benoit]

Each sample is t1 - t0 around fn(), so it carries the cost of one now() call:

sample = X + C + ε

where X is the true cost of fn, C the timestamp provider call cost (~30–100 ns for performance.now() / hrtime.bigint() on Node), ε jitter. C is irreducible in a per-call measurement model — both timestamps run on the same thread as fn, you can't avoid the second now() landing in the sample.

For X of the same order as C, the reported number is dominated by the timer, not by your function. That's the floor.

[maziac]

?
My question is about the 7 seconds. time is set to 100 ms. What is the rest of the 6.9 seconds used for?

[jerome-benoit]

The incompressible floor is additive per sample.