Skip to content

Update CerbiStream benchmarks to current governance stack #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Zeroshi merged 2 commits into from
Jul 16, 2026
Merged

Update CerbiStream benchmarks to current governance stack #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1c71c13
Update benchmarks governance stack
Zeroshi Jul 14, 2026
aa8e649
Preserve full benchmark run entry point
Zeroshi Jul 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions .github/workflows/benchmark-smoke.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
name: Benchmark smoke

on:
pull_request:
branches: [ master, main ]
workflow_dispatch:

jobs:
benchmark-smoke:
name: Restore, build, and smoke benchmarks
runs-on: ubuntu-latest
timeout-minutes: 20

steps:
- name: Checkout
uses: actions/checkout@v4

- name: Install .NET SDK
uses: actions/setup-dotnet@v4
with:
dotnet-version: '9.0.x'

- name: Restore
run: dotnet restore Cerbi-Benchmark-Tests.sln

- name: Build Release
run: dotnet build Cerbi-Benchmark-Tests.sln --configuration Release --no-restore

- name: BenchmarkDotNet smoke run
run: >-
dotnet run --project Cerbi-Benchmark-Tests/Cerbi-Benchmark-Tests.csproj
--configuration Release --no-restore --no-build --
--filter '*MS_Log_Plain*'
--job Dry
--artifacts ./BenchmarkDotNet.Artifacts
Comment thread
Zeroshi marked this conversation as resolved.

- name: Upload smoke benchmark artifacts
uses: actions/upload-artifact@v4
if: always()
with:
name: benchmark-smoke-results
path: BenchmarkDotNet.Artifacts/**
if-no-files-found: warn

- name: Full benchmark note
run: echo "Smoke results prove benchmark discovery/execution only; full performance numbers require the dedicated full benchmark run."
11 changes: 7 additions & 4 deletions Cerbi-Benchmark-Tests/Cerbi-Benchmark-Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<OutputType>Exe</OutputType>
Expand All @@ -15,9 +15,9 @@
<PackageReference Include="BenchmarkDotNet" Version="0.15.8" />

<!-- CerbiStream ecosystem (latest) -->
<PackageReference Include="CerbiStream" Version="1.1.88" />
<PackageReference Include="Cerbi.Governance.Runtime" Version="1.1.10" />
<PackageReference Include="Cerbi.Governance.Core" Version="1.0.15" />
<PackageReference Include="CerbiStream" Version="2.0.6" />
<PackageReference Include="Cerbi.Governance.Runtime" Version="2.0.43" />
<PackageReference Include="Cerbi.Governance.Core" Version="2.2.37" />

<!-- Competing loggers (latest stable) -->
<PackageReference Include="log4net" Version="3.2.0" />
Expand All @@ -37,6 +37,9 @@
<None Update="log4net.config">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="governance.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="nlog.config">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
Expand Down
20 changes: 18 additions & 2 deletions Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using BenchmarkDotNet.Attributes;
using CerbiStream.Logging;
using CerbiStream.Configuration;
using log4net;
Expand Down Expand Up @@ -31,10 +30,10 @@
[MinColumn, MaxColumn, MeanColumn, MedianColumn]
public class PopularLoggerBenchmarks
{
private ILogger _msLoggerPlain, _msLoggerEncrypted;

Check warning on line 33 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_msLoggerEncrypted' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.

Check warning on line 33 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_msLoggerPlain' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.
private ILogger _serilogPlain, _serilogEncrypted, _serilogAsync;

Check warning on line 34 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_serilogAsync' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.

Check warning on line 34 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_serilogEncrypted' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.

Check warning on line 34 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_serilogPlain' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.
private ILogger _nlogPlain, _nlogEncrypted;

Check warning on line 35 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_nlogEncrypted' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.

Check warning on line 35 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_nlogPlain' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.
private ILogger _log4netPlain, _log4netEncrypted;

Check warning on line 36 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Non-nullable field '_log4netPlain' must contain a non-null value when exiting constructor. Consider adding the 'required' modifier or declaring the field as nullable.
private ILogger _cerbiPlain, _cerbiEncrypted, _cerbiAsync, _cerbiFile;
private ILogger _cerbiEncryptedRsa; // new RSA variant
// RSA state
Expand Down Expand Up @@ -99,7 +98,7 @@

_log4netPlain = BuildLogger(s =>
{
ILoggerRepository repo = Log4NetManager.CreateRepository(Assembly.GetEntryAssembly(), typeof(log4net.Repository.Hierarchy.Hierarchy));

Check warning on line 101 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Possible null reference argument for parameter 'repositoryAssembly' in 'ILoggerRepository LogManager.CreateRepository(Assembly repositoryAssembly, Type repositoryType)'.
XmlConfigurator.Configure(repo, new FileInfo("log4net.config"));
s.AddLogging(b => b.AddLog4Net("log4net.config"));
});
Expand Down Expand Up @@ -264,7 +263,7 @@
// No-op logger provider to act as a null sink for all loggers
private class NoopLogger : Microsoft.Extensions.Logging.ILogger
{
public IDisposable BeginScope<TState>(TState state) => NullScope.Instance;

Check warning on line 266 in Cerbi-Benchmark-Tests/PopularLoggerBenchmarks.cs

View workflow job for this annotation

GitHub Actions / Restore, build, and smoke benchmarks 

Nullability in constraints for type parameter 'TState' of method 'PopularLoggerBenchmarks.NoopLogger.BeginScope<TState>(TState)' doesn't match the constraints for type parameter 'TState' of interface method 'ILogger.BeginScope<TState>(TState)'. Consider using an explicit interface implementation instead.
public bool IsEnabled(Microsoft.Extensions.Logging.LogLevel logLevel) => false;
public void Log<TState>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, TState state, Exception? exception, Func<TState, Exception?, string> formatter) { }
}
Expand Down Expand Up @@ -391,6 +390,7 @@
}

[Benchmark(Baseline = true)]
[BenchmarkCategory("Baseline", "NoCerbi")]
public void MS_Log_Plain() => _msLoggerPlain.LogInformation("MS Logger: Logging at {time}", DateTime.UtcNow);

[Benchmark]
Expand Down Expand Up @@ -447,6 +447,7 @@
// --- New Cerbi toggle benchmarks ---

[Benchmark]
[BenchmarkCategory("CerbiStream", "GovernanceDisabled")]
public void Cerbi_Minimal() => _cerbi_minimal.LogInformation("Cerbi Minimal: Logging at {time}", DateTime.UtcNow);

[Benchmark]
Expand All @@ -459,9 +460,21 @@
public void Cerbi_Minimal_Async() => _cerbi_minimal_async.LogInformation("Cerbi Minimal+Async: Logging at {time}", DateTime.UtcNow);

// --- Governance simulation scenarios ---
[Benchmark]
[BenchmarkCategory("Governance", "NoViolation")]
public void Cerbi_Governance_NoViolation()
{
const string msg = "User completed operation without sensitive data";
if (!ContainsPII(msg))
{
_cerbiPlain.LogInformation("{msg}", msg);
}
}


// Redact simple PII in a single string then log
[Benchmark]
[BenchmarkCategory("Governance", "Violation")]
public void Cerbi_Governance_Redact_Simple()
{
var msg = $"User email={_piiEmail} cc={_piiCc} ssn={_piiSsn} note={_piiNoteBig}";
Expand All @@ -471,6 +484,7 @@

// Validate presence of PII without redacting; drop or log accordingly
[Benchmark]
[BenchmarkCategory("Governance", "Violation")]
public void Cerbi_Governance_ValidateOnly()
{
var msg = $"email={_piiEmail} cc={_piiCc} note={_piiNoteBig}";
Expand Down Expand Up @@ -600,6 +614,7 @@

// Benchmarks using governance JSON
[Benchmark]
[BenchmarkCategory("GovernanceJson", "Violation")]
public void Cerbi_GovernanceJson_Redact_Simple()
{
var msg = $"User email={_piiEmail} cc={_piiCc} ssn={_piiSsn} note={_piiNoteBig}";
Expand Down Expand Up @@ -739,6 +754,7 @@
// This benchmark shows runtime cost is ZERO when using analyzers
// Competitors: No Roslyn analyzer for logging governance
[Benchmark]
[BenchmarkCategory("Governance", "NoViolation", "Relaxed")]
public void Cerbi_Feature_DesignTimeGovernance_RuntimeCost()
{
// With Roslyn analyzers, unsafe logging is blocked at compile time
Expand Down
12 changes: 9 additions & 3 deletions Cerbi-Benchmark-Tests/Program.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,18 @@
using BenchmarkDotNet.Running;
using System.Reflection;

public class Program
{
public static void Main(string[] args)
{
Console.WriteLine(">>> Running updated PopularLoggerBenchmarks (subset for quick validation)...");
if (args is null || args.Length == 0)
{
Console.WriteLine("Running the complete PopularLoggerBenchmarks suite.");
BenchmarkRunner.Run<CerbiBenchmark.PopularLoggerBenchmarks>();
return;
}

// For quicker iteration you can filter benchmarks via BenchmarkSwitcher or regex; left simple here
BenchmarkRunner.Run<CerbiBenchmark.PopularLoggerBenchmarks>();
Console.WriteLine("Running the requested filtered benchmark selection.");
BenchmarkSwitcher.FromAssembly(Assembly.GetExecutingAssembly()).Run(args);

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 14, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve non-interactive full benchmark runs

When users follow the repo's documented reproduction command with no BenchmarkDotNet arguments (dotnet run -c Release --project ... in cerbi-benchmarks-page-v0.md), this BenchmarkSwitcher path no longer behaves like the previous BenchmarkRunner.Run<PopularLoggerBenchmarks>(): BenchmarkDotNet documents the switcher as choosing benchmarks via command-line arguments or console input, so no-arg script/CI invocations can block waiting for interactive selection instead of producing artifacts. Consider preserving the full-suite runner when args is empty, or update the documented full-run command to pass an explicit filter.

Useful? React with 👍 / 👎.

}
}
12 changes: 12 additions & 0 deletions cerbi-benchmarks-page-v0.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,18 @@ A comprehensive benchmarks page for cerbi.io showcasing CerbiStream performance

---

## Benchmark Execution Policy

These benchmark commands intentionally separate full performance runs from pull request smoke validation:

- Running `dotnet run --project Cerbi-Benchmark-Tests/Cerbi-Benchmark-Tests.csproj --configuration Release` with no BenchmarkDotNet arguments executes the complete `PopularLoggerBenchmarks` suite. This preserves existing non-interactive full-run scripts and reproduction commands.
- Passing BenchmarkDotNet CLI arguments, such as `--filter`, runs only the selected benchmark set requested by the caller. Use this for targeted diagnostics and benchmark discovery checks.
- Pull request CI uses an explicit filtered Dry job for `MS_Log_Plain` only. It proves restore, Release build, benchmark discovery, execution, and artifact generation without running the full long suite on every PR.
- PR smoke results must not be published as performance claims or copied into the website metrics. Dry smoke output is validation evidence only.
- Official published numbers require a dedicated full benchmark run with the complete suite, stable hardware/runtime conditions, and archived BenchmarkDotNet results.

---

## Hero Section

### Headline
Expand Down
Loading