Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -170,11 +170,11 @@ jobs:
'| --- | ---: | ---: | ---: | ---: | ---: | --- |'
) | Add-Content -LiteralPath $env:GITHUB_STEP_SUMMARY -Encoding utf8
$modules = @(
@{ Name = 'Agentstration.Api.Tests'; Tests = 191; Warning = 1700; Failure = 2048 },
@{ Name = 'Agentstration.Management.Api.Tests'; Tests = 43; Warning = 900; Failure = 1024 },
@{ Name = 'Agentstration.Management.Bootstrap.Tests'; Tests = 23; Warning = 500; Failure = 700 },
@{ Name = 'Agentstration.Management.Security.Tests'; Tests = 38; Warning = 800; Failure = 1024 },
@{ Name = 'Agentstration.Management.Aep.Tests'; Tests = 11; Warning = 450; Failure = 700 },
@{ Name = 'Agentstration.Management.Sources.Tests'; Tests = 93; Warning = 550; Failure = 900 }
@{ Name = 'Agentstration.Management.Aep.Tests'; Tests = 11; Warning = 450; Failure = 700 }
)
foreach ($module in $modules) {
./scripts/ci/run-test-module-with-diagnostics.ps1 `
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ jobs:
- name: Build
run: dotnet build Agentstration.slnx --configuration Release --no-restore
- name: Test fast lane
run: dotnet test --solution Agentstration.Tests.Fast.slnx --configuration Release --no-build --minimum-expected-tests 139 --max-parallel-test-modules 4
run: dotnet test --solution Agentstration.Tests.Fast.slnx --configuration Release --no-build --minimum-expected-tests 336 --max-parallel-test-modules 4
- name: Test integration lane
run: dotnet test --solution Agentstration.Tests.Integration.slnx --configuration Release --no-build --minimum-expected-tests 692 --max-parallel-test-modules 2
run: dotnet test --solution Agentstration.Tests.Integration.slnx --configuration Release --no-build --minimum-expected-tests 515 --max-parallel-test-modules 2
- name: Restore autonomous AEP SDK
run: dotnet restore aep/Aep.slnx -p:NuGetAudit=true -p:NuGetAuditMode=all
- name: Build autonomous AEP SDK
Expand Down
4 changes: 2 additions & 2 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,8 @@ Run from the repository root:
```powershell
dotnet restore Agentstration.slnx
dotnet build Agentstration.slnx --configuration Release --no-restore
dotnet test --solution Agentstration.Tests.Fast.slnx --configuration Release --no-build --minimum-expected-tests 139 --max-parallel-test-modules 4
dotnet test --solution Agentstration.Tests.Integration.slnx --configuration Release --no-build --minimum-expected-tests 692 --max-parallel-test-modules 2
dotnet test --solution Agentstration.Tests.Fast.slnx --configuration Release --no-build --minimum-expected-tests 336 --max-parallel-test-modules 4
dotnet test --solution Agentstration.Tests.Integration.slnx --configuration Release --no-build --minimum-expected-tests 515 --max-parallel-test-modules 2
```

The two test solutions together form the required deterministic, offline functional suite. Performance and live-provider workloads are explicit opt-ins documented in `docs/contributing/testing.md`. For a focused iteration, run the affected test project first, then run both functional lanes before handoff. Do not suppress warnings or disable analyzers to make a change pass.
Expand Down
2 changes: 2 additions & 0 deletions Agentstration.Tests.Fast.slnx
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
<Solution>
<Folder Name="/tests/fast/">
<Project Path="tests/Agentstration.ArchitectureTests/Agentstration.ArchitectureTests.csproj" />
<Project Path="tests/Agentstration.Console.Client.Tests/Agentstration.Console.Client.Tests.csproj" />
<Project Path="tests/Agentstration.Console.Components.Tests/Agentstration.Console.Components.Tests.csproj" />
<Project Path="tests/Agentstration.Management.Core.Tests/Agentstration.Management.Core.Tests.csproj" />
<Project Path="tests/Agentstration.Tools.SourceRegistry.Tests/Agentstration.Tools.SourceRegistry.Tests.csproj" />
<Project Path="tests/Agentstration.Web.Components.Tests/Agentstration.Web.Components.Tests.csproj" />
Expand Down
2 changes: 1 addition & 1 deletion Agentstration.Tests.Integration.slnx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
<Solution>
<Folder Name="/tests/integration/">
<Project Path="tests/Agentstration.Api.Tests/Agentstration.Api.Tests.csproj" />
<Project Path="tests/Agentstration.Application.Tests/Agentstration.Application.Tests.csproj" />
<Project Path="tests/Agentstration.Management.Storage.Tests/Agentstration.Management.Storage.Tests.csproj" />
<Project Path="tests/Agentstration.Management.Sources.Tests/Agentstration.Management.Sources.Tests.csproj" />
Expand All @@ -11,7 +12,6 @@
<Project Path="tests/Agentstration.Runtime.Tests/Agentstration.Runtime.Tests.csproj" />
<Project Path="tests/Agentstration.SourceProviders.Git.Tests/Agentstration.SourceProviders.Git.Tests.csproj" />
<Project Path="tests/Agentstration.Web.Tests/Agentstration.Web.Tests.csproj" />
<Project Path="tests/Agentstration.Work.Api.Tests/Agentstration.Work.Api.Tests.csproj" />
<Project Path="tests/Agentstration.Workplace.Web.Tests/Agentstration.Workplace.Web.Tests.csproj" />
</Folder>
</Solution>
4 changes: 3 additions & 1 deletion Agentstration.slnx
Original file line number Diff line number Diff line change
Expand Up @@ -56,8 +56,11 @@
<Project Path="src/Agentstration.Workplace.Web/Agentstration.Workplace.Web.csproj" />
</Folder>
<Folder Name="/tests/">
<Project Path="tests/Agentstration.Api.Tests/Agentstration.Api.Tests.csproj" />
<Project Path="tests/Agentstration.Application.Tests/Agentstration.Application.Tests.csproj" />
<Project Path="tests/Agentstration.ArchitectureTests/Agentstration.ArchitectureTests.csproj" />
<Project Path="tests/Agentstration.Console.Client.Tests/Agentstration.Console.Client.Tests.csproj" />
<Project Path="tests/Agentstration.Console.Components.Tests/Agentstration.Console.Components.Tests.csproj" />
<Project Path="tests/Agentstration.Management.Core.Tests/Agentstration.Management.Core.Tests.csproj" />
<Project Path="tests/Agentstration.Management.Storage.Tests/Agentstration.Management.Storage.Tests.csproj" />
<Project Path="tests/Agentstration.Management.Sources.Tests/Agentstration.Management.Sources.Tests.csproj" />
Expand All @@ -73,7 +76,6 @@
<Project Path="tests/Agentstration.Web.Tests/Agentstration.Web.Tests.csproj" />
<Project Path="tests/Agentstration.Web.Components.Tests/Agentstration.Web.Components.Tests.csproj" />
<Project Path="tests/Agentstration.Web.FlowDesigner.Tests/Agentstration.Web.FlowDesigner.Tests.csproj" />
<Project Path="tests/Agentstration.Work.Api.Tests/Agentstration.Work.Api.Tests.csproj" />
<Project Path="tests/Agentstration.Workplace.Components.Tests/Agentstration.Workplace.Components.Tests.csproj" />
<Project Path="tests/Agentstration.Workplace.Web.Tests/Agentstration.Workplace.Web.Tests.csproj" />
</Folder>
Expand Down
3 changes: 3 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,11 @@ src/
Agentstration.Work.Storage.Abstractions/
Agentstration.Work.Storage.Sqlite/
tests/
Agentstration.Api.Tests/
Agentstration.Application.Tests/
Agentstration.ArchitectureTests/
Agentstration.Console.Client.Tests/
Agentstration.Console.Components.Tests/
Agentstration.Management.Core.Tests/
Agentstration.Management.Storage.Tests/
Agentstration.Management.Sources.Tests/
Expand Down
22 changes: 12 additions & 10 deletions docs/contributing/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,15 +7,15 @@ Agentstration separates test execution by runtime cost and dependency type. The
Fast tests do not start an ASP.NET Core host, open a relational database, spawn a child process, or contact a remote provider. They cover architecture rules, component behavior, mapping, validation, and in-process command behavior.

```powershell
dotnet test --solution Agentstration.Tests.Fast.slnx --configuration Release --no-build --minimum-expected-tests 139 --max-parallel-test-modules 4
dotnet test --solution Agentstration.Tests.Fast.slnx --configuration Release --no-build --minimum-expected-tests 336 --max-parallel-test-modules 4
```

## Integration lane

Integration tests exercise a real boundary such as `WebApplicationFactory`, SQLite, Git, persistent Identity, or runtime reconstruction. This lane remains deterministic and offline by default. Tests marked `Integration` for a live model provider are opt-in and report inconclusive unless their documented environment variables are supplied.

```powershell
dotnet test --solution Agentstration.Tests.Integration.slnx --configuration Release --no-build --minimum-expected-tests 692 --max-parallel-test-modules 2
dotnet test --solution Agentstration.Tests.Integration.slnx --configuration Release --no-build --minimum-expected-tests 515 --max-parallel-test-modules 2
```

Run both fast and integration solutions for complete required functional validation. Their union is the functional test inventory represented by the root solution.
Expand Down Expand Up @@ -47,41 +47,43 @@ The report records workload parameters, provider, elapsed time, runtime and OS m

## CI concurrency and memory diagnostics

The fast, integration, and performance lanes cap concurrent test modules at 4, 2, and 1 respectively. Host-heavy Management modules also use one class worker per assembly. CI reruns the designated hosted modules sequentially through `scripts/ci/run-test-module-with-diagnostics.ps1`; each JSON artifact contains the discovered count, duration, process peak working set, process peak private memory, aggregate peak working set for active `dotnet` processes, runtime, and OS. The diagnostic artifact deliberately excludes test output and payloads.
The fast, integration, and performance lanes cap concurrent test modules at 4, 2, and 1 respectively. Host-heavy API and Management modules also use one class worker per assembly. CI reruns the designated hosted modules sequentially through `scripts/ci/run-test-module-with-diagnostics.ps1`; each JSON artifact contains the discovered count, duration, process peak working set, process peak private memory, aggregate peak working set for active `dotnet` processes, runtime, and OS. The diagnostic artifact deliberately excludes test output and payloads.

The initial budgets below use Release runs on Windows 11 10.0.26200 with .NET 10.0.10/10.0.11, collected during #298. A warning is evidence to review the Linux and Windows trend; a failure protects constrained runners from returning to the original greater-than-1-GiB process. Adjust these values only after retaining representative artifacts from both runner families.
The Management budgets below use Release runs on Windows 11 10.0.26200 with .NET 10.0.10/10.0.11, collected during #298. The consolidated `Agentstration.Api.Tests` baseline was collected on the same Windows build with .NET 10.0.11 during #142: 191 tests in 200 seconds, 1503.7 MiB peak working set, 1176.6 MiB peak private memory, and 1621.2 MiB aggregate `dotnet` working set. A warning is evidence to review the Linux and Windows trend; a failure caps regression relative to the retained baseline. Adjust these values only after retaining representative artifacts from both runner families.

| Module | Baseline peak (MiB) | Warning (MiB) | Failure (MiB) | Minimum tests |
| --- | ---: | ---: | ---: | ---: |
| `Agentstration.Api.Tests` | 1503.7 | 1700 | 2048 | 191 |
| `Agentstration.Management.Api.Tests` | 749.3 | 900 | 1024 | 43 |
| `Agentstration.Management.Bootstrap.Tests` | 355.3 | 500 | 700 | 23 |
| `Agentstration.Management.Security.Tests` | 730.1 | 800 | 1024 | 38 |
| `Agentstration.Management.Aep.Tests` | 322.1 | 450 | 700 | 11 |
| `Agentstration.Management.Sources.Tests` | 431.5 | 550 | 900 | 93 |

## Project classification

| Test project | Lane | Boundary or rationale |
| --- | --- | --- |
| `Agentstration.ArchitectureTests` | Fast | Assembly dependency rules |
| `Agentstration.Console.Client.Tests` | Fast | HTTP and SignalR mappings, pagination, errors, retries, and credential forwarding without an authoritative server |
| `Agentstration.Console.Components.Tests` | Fast | Console Razor behavior, presentation state, permissions, localization, and static assets with mocked clients |
| `Agentstration.Management.Core.Tests` | Fast | Pure Management validation and in-memory use cases |
| `Agentstration.Tools.SourceRegistry.Tests` | Fast | In-process CLI and manifest validation |
| `Agentstration.Web.Components.Tests` | Fast | bUnit component behavior |
| `Agentstration.Web.FlowDesigner.Tests` | Fast | bUnit and graph projection behavior |
| `Agentstration.Workplace.Components.Tests` | Fast | bUnit component behavior |
| `Agentstration.Application.Tests` | Integration | Includes SQLite Flow and Work storage contracts |
| `Agentstration.Application.Tests` | Integration | Application use cases and their SQLite-backed contracts without the executable host |
| `Agentstration.Management.Storage.Tests` | Integration | SQLite control-plane, Identity persistence, secrets, audit, and trigger storage |
| `Agentstration.Management.Sources.Tests` | Integration | Source, registry, provider, and Pack distribution boundaries |
| `Agentstration.Management.Sources.Tests` | Integration | Source provider, registry transport, and Pack composition behavior without the executable host |
| `Agentstration.Api.Tests` | Integration | Hosted HTTP, authentication, OpenAPI, MCP, SignalR, Flow, Runtime, Work, and cross-cutting transport behavior |
| `Agentstration.Management.Api.Tests` | Integration | Hosted Management API tests using the API-only test profile |
| `Agentstration.Management.Bootstrap.Tests` | Integration | Declarative bootstrap catalog, application, and hosted startup scenarios using the API-only test profile |
| `Agentstration.Management.Security.Tests` | Integration | Identity, authorization, local-account, and interactive Security boundaries |
| `Agentstration.Management.Aep.Tests` | Integration | AEP enrollment lifecycle and extension inventory boundaries using the API-only test profile |
| `Agentstration.ModelProviders.Tests` | Integration, provider-optional | AEP test hosts plus opt-in live-provider checks |
| `Agentstration.Runtime.Tests` | Integration | SQLite reconstruction and hosted runtime endpoints |
| `Agentstration.Runtime.Tests` | Integration | Runtime business behavior and SQLite reconstruction without the executable host |
| `Agentstration.SourceProviders.Git.Tests` | Integration | Real Git processes and file-system repositories |
| `Agentstration.Web.Tests` | Integration | Full Web host plus a bounded deterministic SQLite contention correctness smoke |
| `Agentstration.Web.Tests` | Integration | Combined standalone composition, startup, storage profiles, workers, and lifecycle |
| `Agentstration.Performance.Tests` | Performance, opt-in | SQLite/PostgreSQL relational storage concurrency workloads and machine-readable reports |
| `Agentstration.Work.Api.Tests` | Integration | Full Work API host |
| `Agentstration.Workplace.Web.Tests` | Integration | Workplace HTTP host |

The autonomous AEP SDK keeps its own `aep/Aep.slnx` validation because it can be built and released independently from the product solution.
4 changes: 2 additions & 2 deletions scripts/ci/run-functional-coverage.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -24,14 +24,14 @@ $lanes = @(
@{
Name = 'Fast'
Solution = 'Agentstration.Tests.Fast.slnx'
MinimumTests = 139
MinimumTests = 336
ParallelModules = 4
ResultsDirectory = Join-Path $resolvedCoverageRoot 'raw/fast'
},
@{
Name = 'Integration'
Solution = 'Agentstration.Tests.Integration.slnx'
MinimumTests = 692
MinimumTests = 515
ParallelModules = 2
ResultsDirectory = Join-Path $resolvedCoverageRoot 'raw/integration'
}
Expand Down
1 change: 1 addition & 0 deletions src/Agentstration.Api/Agentstration.Api.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="Agentstration.Web.Tests" />
<InternalsVisibleTo Include="Agentstration.Api.Tests" />
<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" />
<PackageReference Include="Microsoft.AspNetCore.Authentication.OpenIdConnect" />
<PackageReference Include="Microsoft.AspNetCore.OpenApi" />
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<RootNamespace>Agentstration.Web</RootNamespace>
</PropertyGroup>
<ItemGroup>
<InternalsVisibleTo Include="Agentstration.Web.Tests" />
<InternalsVisibleTo Include="Agentstration.Console.Components.Tests" />
<FrameworkReference Include="Microsoft.AspNetCore.App" />
<PackageReference Include="YamlDotNet" />
<ProjectReference Include="../Agentstration.Console.Client/Agentstration.Console.Client.csproj" />
Expand Down
2 changes: 1 addition & 1 deletion src/Agentstration.Web/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -108,4 +108,4 @@ Interactive Server components invoke canonical APIs through server-side typed cl

## Tests

`Agentstration.Web.Tests` covers API client mapping, model discovery, profile filtering and selection rules, conditional request headers, Problem Details, editor and runner payload mapping, simulated CRUD, SSE processing, retry, and dashboard aggregation. `Agentstration.Web.Components.Tests` covers focused UI state services. Both use MSTest and remain offline.
`Agentstration.Api.Tests` owns hosted transport behavior. `Agentstration.Console.Client.Tests` covers client mappings, conditional requests, Problem Details, streaming, and retries without starting the authoritative server. `Agentstration.Console.Components.Tests` covers Console presentation behavior with mocked clients. `Agentstration.Web.Tests` is reserved for combined standalone composition, startup, storage, workers, and lifecycle. All use MSTest and remain offline by default.
Loading