DotPython

A managed implementation of Python for .NET 10 / C# 14. DotPython owns the full execution pipeline — tokenizer, PEG parser, AST, symbol binder, bytecode compiler, and a managed stack VM. The default runtime runs Python without loading or hosting CPython; an optional, explicitly selected CPython worker provider is planned for unchanged native packages.

DotPython is usable as a command-line application, as an embedded scripting/runtime service inside a .NET solution, and as an early SDK-style .dpyproj project language whose compiled libraries can be referenced from C# and other managed languages.

Status: early, active development. Today the CLI executes a growing managed subset of the language (literals, names, assignment, arithmetic, calls, control flow, functions, explicit exceptions, and managed package imports). The compiler also emits deterministic .dpyc module artifacts, the interop layer statically compiles an initial .pyi subset into typed CLR export contracts, and the prototype DotPython.Sdk generates typed C# facades for single-module projects. A versioned worker boundary now loads and executes one pinned Stable-ABI conformance fixture through an experimental, hash-pinned symbol allowlist. This is not general native-extension support.

Compatibility contract

• Targets the Python 3.14 language surface through an explicit compatibility profile.
• CPython is a differential reference for managed execution and is never an implicit fallback.
• CPython bytecode and arbitrary C-extension binaries are unsupported by the managed runtime today. One internal Stable-ABI fixture is executable only through its explicit worker option.
• A provider-neutral worker protocol and managed worker host are implemented. The optional CPython provider itself is not implemented or qualified.
• The managed interpreter is the semantic reference; any future JIT tier must fall back to it.
• Host/.NET access is capability based; arbitrary assembly loading and reflection are off by default.

The native-extension direction is layered: managed abi3 and HPy work begins in workers, while unchanged version-specific packages use an explicitly selected CPython worker provider. The first managed abi3 experiment is intentionally fixture-specific; no package or universal ABI compatibility is qualified yet.

Requirements

• .NET SDK 10.0.301 or later (pinned in global.json).
• Rust 1.85 or later with Cargo, rustfmt, and Clippy for the experimental Stable-ABI native build.
• Optionally just for the developer task shortcuts below.

Getting started

# Restore, format-check, and build with analyzers and warnings-as-errors
just lint

# Run the CLI (arguments after `--` are passed through unchanged)
just run -- -c "print(1 + 2)"

Without just, invoke the CLI directly:

dotnet run --project src/DotPython.Cli/DotPython.Cli.csproj -- -c "print(1 + 2)"

CLI usage

dotpython -c command      # execute a Python snippet
dotpython -                # read and execute a program from stdin
dotpython script.py        # execute a Python source file
dotpython wheel inspect x.whl # classify a wheel without loading native code
dotpython -V | --version   # print implementation and target language version
dotpython -h | --help      # print usage

Exit codes follow familiar conventions: 0 success, 1 execution/diagnostic error, 2 usage error, 130 cancelled. Interactive (REPL) mode is not yet implemented.

wheel inspect reports canonical filename and embedded tags, platform/libc/architecture, free-threaded ABI selection, SHA-256, native archive entries, imported symbols when their ELF, Mach-O, or PE tables are readable, and actionable incompatibility diagnostics. Classification never loads or executes a library and does not change the managed runtime's CPython ABI support.

Managed modules and packages

Embedded callers can provide an immutable catalog of source modules and packages. A package is a catalog entry that has registered dot-separated child modules. Each engine owns its module objects and initialization cache; imports execute in the calling VM and therefore share its cancellation token, output, and instruction limit.

var modules = new Dictionary<string, SourceText>
{
    ["helpers"] = new("from . import values", "helpers/__init__.py"),
    ["helpers.values"] = new("answer = 42", "helpers/values.py"),
};
var engine = new ManagedPythonEngine(modules);
engine.Execute("import helpers.values; print(helpers.values.answer)", "main.py", output);

The current slice supports dotted absolute imports, explicit packages, relative from imports, aliases, parenthesized import lists, submodule fallback, and module attribute reads. Catalog size, source size, module-name length, and active import depth are bounded. Every dotted child requires an explicit parent-package source entry.

An engine can instead take an ordered set of package roots. Startup discovers regular packages from __init__.py, source modules, validated DotPython .dpyc artifacts, and top-level *.dist-info/METADATA records. The snapshot is immutable after construction; the first configured root wins across roots, while ambiguous source/artifact identities within one root are rejected.

var engine = new ManagedPythonEngine(
    new ManagedModuleDiscoveryOptions
    {
        SearchPaths = ["/opt/my-application/python"],
    }
);
engine.Execute("import helpers.values; print(helpers.values.answer)", "main.py", output);

The CLI applies the same discovery contract to the script directory, or to the current directory for -c and standard-input execution. A minimal runtime-owned importlib.metadata.version() reads the startup metadata snapshot, including normalized distribution names. Discovery has fixed entry, depth, per-file, and aggregate payload limits, uses strict UTF-8, and does not traverse symbolic links or reparse points.

Namespace packages, wheel/zip imports, wildcard imports, reload, and import hooks are not implemented. Native .so and .pyd files are recognized but never loaded; importing one produces the actionable DPY4027 unsupported-native-extension diagnostic. The managed runtime currently reports an empty executable native-extension capability list.

Managed exceptions

The managed compiler and VM support explicit raise, including bare re-raise and raise ... from ..., plus try / except / else / finally. Handlers can match the implemented built-in exception hierarchy, tuples of exception types, or a final bare clause, and can bind an exception with as. Exception propagation crosses managed function and module frames; uncaught explicit exceptions produce DPY4031 at the original raise span.

finally executes for normal completion, returns, explicit exceptions, existing runtime faults, cancellation, and the normal instruction limit. Deferred cleanup has its own fixed instruction budget so cancellation or a resource-limit failure cannot open an unbounded cleanup path.

Language-level VM operation failures are converted into catchable built-in exceptions such as TypeError, NameError, ZeroDivisionError, LookupError, AttributeError, and ImportError. The VM keeps a private raised-error indicator, separate from the exception currently handled by an except block, so callback bridges can save and restore propagating error state. If a converted operation remains uncaught or is re-raised, its original DPY4xxx code, message, and source span remain the host-facing diagnostic.

Cancellation, instruction limits, exception-block limits, deferred-cleanup limits, and VM invariant failures are host/runtime control signals and cannot be swallowed by Python handlers. User-defined exception classes, exception groups and except*, sys.exception(), public traceback objects, and exact deletion of an except ... as target remain follow-up work. The internal error indicator does not implement or enable CPython's native error ABI.

Runtime ownership and shutdown

ManagedPythonModuleRuntime serializes module loading and invocation onto one dedicated owning thread. Normal work admission is limited to 1,024 pending calls. Runtime-owned logical resource leases are limited to 4,096 registrations; explicit disposal schedules an exactly-once release on the owner thread, while a managed finalizer can only enqueue the same release and never executes it directly.

Runtime disposal rejects new work, drains work that was already admitted, releases remaining resources in reverse registration order, clears runtime module state, and then stops the owning thread. Cancellation remains cooperative after a call begins. A synchronous host callback can re-enter the same runtime inline on its owning thread, with active and explicit cancellation linked and nesting limited to 64 calls. Cross-runtime entry from an owning thread, execution during resource/finalization cleanup, and asynchronous callback suspension are rejected to avoid hidden queue waits and deadlocks.

An internal managed-only native-boundary simulator now exercises this lifecycle without loading a library. Logical handles carry runtime identity, generation, and handle identity; borrowed, new, stolen, immortal, and nullable reference transitions are explicit. New references are backed by the existing scheduler-owned leases, so explicit disposal and forced-GC cleanup release on the owning thread and shutdown preserves reverse registration order. The simulator also keeps a separate native-style raised-error indicator and saves/restores it around bounded re-entrant callbacks.

This lifecycle foundation does not load native code or change the compatibility contract. No CPython ABI, HPy, Anyver, or NumPy execution support is enabled.

Worker protocol foundation

DotPython.Protocol, DotPython.Worker, and DotPython.Worker.Host establish the worker-first boundary required by the native-extension roadmap. The host and child negotiate protocol, provider/runtime/environment identity, feature flags, message/output/session/concurrency limits, and an explicit worker generation before accepting work. Four-byte little-endian length prefixes bound source-generated JSON envelopes carrying correlation IDs, deadlines, cancellation, and structured faults.

Worker protocol v3.0 reserves the retired experimental Stable-ABI request/response message IDs. Qualified native modules execute only through ordinary managed source imports and the generic native-object path; there is no direct module load/invoke/release RPC or worker module handle API.

The parent starts an explicitly configured executable without a shell, clears inherited environment variables, and passes only allowlisted values and package roots. Sessions and logical object handles are scoped to provider, worker identity, generation, and session. Clean disposal closes sessions and drains the child; recycling, malformed protocol, crashes, and hard timeout invalidate the generation. Hard timeout sends cooperative cancellation, waits a bounded grace period, and terminates the process tree before a replacement is admitted.

The first transport uses redirected standard streams as a private framed channel. Protocol output never shares the Python standard-output payload, which is returned inside a bounded response. This contains crashes and enables hard termination, but it is not by itself a complete native-code sandbox; platform OS resource and network enforcement remain separate work. No provider is selected implicitly.

Managed Stable-ABI experiments

DotPython.Runtime.Native loads checked-in conformance fixtures or exact pinned native entries through an explicitly configured worker. The parent supplies a bounded catalog of absolute module and manifest paths plus their SHA-256 values; every entry shares one pinned bridge identity. Before execution, the worker freezes the catalog, rejects duplicate module names and artifact paths, and validates bounded regular files, platform and architecture, the versioned manifest, PyInit_*, and every required bridge and module export. Imports select entries by qualified module name and exact path. Native addresses remain inside the dedicated native-owner lane.

The Anyver manifest records the immutable wheel and native-entry hashes and exactly 90 imported Stable-ABI symbols. inspect-anyver-wheel.sh rejects wheel, entry-point, or import-surface drift. Bridge ABI v6 exposes only the generic opaque-object surface for qualified modules: attribute discovery, positional calls, primitive and sequence conversion, attribute/item access, display and representation conversion, length, rich comparison, and explicit owner-thread release. The legacy scalar module query/call helpers have been removed. The worker can therefore import the unchanged Anyver Python wrapper from the staged wheel and use its native exports without Anyver-specific native exports, worker messages, or managed call methods. Package identity and process-pinned lifetime are manifest data; the runtime path is package-neutral. Two independent conformance modules verify catalog routing and failure isolation in one worker. The Anyver acceptance path covers comparisons, sorting, Version construction/display/representation/properties/indexing, length, rich comparison, and dictionary round trips. PyO3 heap-type and intern caches are retained for the worker lifetime; transient module objects are released on logical-module disposal and all remaining native state is reclaimed by worker termination.

The macOS ARM64 native CI lane downloads the exact PyPI wheel, verifies its pinned SHA-256 and native symbol surface, and treats every skipped worker test as a failure. For local acceptance, set DOTPYTHON_ANYVER_WHEEL to the pinned wheel when building DotPython.WorkerTests; the native harness additionally accepts its prepared module through ANYVER_MODULE. The checked-in anyver-compatibility.json records 1 successful unchanged-package import, 16 passing public checks, and all 325 upstream pytest cases as skipped because the complete required Python/pytest surface is not yet claimed. This remains a package-, artifact-, and platform-specific partial result; SupportsCpythonAbi remains false, and this does not claim NumPy, HPy, or general abi3 compatibility.

Typed module contracts

DotPython can parse typed module stubs without importing or executing Python:

from decimal import Decimal
from contracts import OrderDto

def calculate(order: OrderDto, discount: Decimal | None = ...) -> Decimal: ...
async def validate(order: OrderDto) -> list[str]: ...

The initial contract mapper supports None, bool, arbitrary-size int, float, str, bytes, selected decimal/uuid/datetime types, nullable T | None/Optional[T], and read-only list/dictionary shapes. Referenced DTO types require an explicit Python-to-CLR mapping; DotPython does not load assemblies or evaluate annotations while parsing contracts. Contracts are persisted as deterministic, versioned JSON for build tooling and generated facades.

DotPython project references

The prototype DotPython.Sdk compiles one .py source and matching .pyi contract before the normal C# CoreCompile boundary. It embeds the deterministic artifact and contract JSON, then compiles an abstraction-only typed facade into the resulting managed assembly.

<Project Sdk="Microsoft.NET.Sdk">
  <Sdk Name="DotPython.Sdk" />

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <RootNamespace>PricingRules</RootNamespace>
    <DotPythonModuleName>pricing</DotPythonModuleName>
    <DotPythonClrTypeName>PricingModule</DotPythonClrTypeName>
  </PropertyGroup>

  <ItemGroup>
    <PythonCompile Include="pricing.py" />
    <PythonContract Include="pricing.pyi" />
  </ItemGroup>
</Project>

A C# project can use an ordinary project reference and bind the generated facade to an explicitly owned runtime:

<ProjectReference Include="../PricingRules/PricingRules.dpyproj" />

await using var rules = await PricingModule.LoadAsync(runtime, cancellationToken);
BigInteger total = await rules.AddAsync(left, right, cancellationToken);

The SDK is not published yet. The build-integration suite packs it into an isolated local feed and proves restore, C# ProjectReference, embedded-resource execution, incremental reuse, and clean rebuild equivalence. The initial SDK accepts one synchronous, positional, scalar-only module.

Project layout

Project	Purpose
src/DotPython.Language	Tokenizer, AST, syntax, source text, and diagnostics.
src/DotPython.ParserGenerator	PEG grammar and generated parser.
src/DotPython.Compiler	Symbol/scope binding and DotPython bytecode compilation.
src/DotPython.Runtime.Managed	Managed stack VM, object model, and execution engine.
src/DotPython.Abstractions	Backend-independent public API surface.
src/DotPython.Hosting	Embedded hosting, runtime/session lifecycle, and DI.
src/DotPython.Interop	Static .pyi contracts, value conversion, and the capability-limited .NET bridge.
src/DotPython.StdLib	Managed / pure-Python standard-library surface.
src/DotPython.Cli	dotpython command-line front end.
src/DotPython.Build.Tasks	Deterministic out-of-process module compiler and C# facade generator.
src/DotPython.Sdk	Additive MSBuild SDK props, targets, and package layout.
src/DotPython.Protocol	Versioned, bounded worker envelopes, framing, handshake, and faults.
src/DotPython.Runtime.Native	Experimental manifest-bound Stable-ABI fixture loader and logical module boundary.
src/DotPython.Worker	Worker policy, process lifecycle, sessions, recycling, and logical handles.
src/DotPython.Worker.Host	Executable managed worker host and test-only failure injection.
native/dotpython-abi3	Minimal Stable-ABI bridge, pinned fixture, manifest generator, and native harness.
benchmarks/DotPython.Benchmarks	Managed front-end, compiler, and runtime performance baselines.

Development

just            # list available tasks
just format     # format C#, project files, and native Rust sources
just native-format # format only the native Stable-ABI workspace with rustfmt
just native-lint   # check native formatting and run Clippy
just native-test   # build, verify, test, and stage native artifacts
just parser-generate # regenerate the checked-in parser from the pinned PEG subset
just parser-check    # verify deterministic parser regeneration has no drift
just lint       # check managed/native formatting and analyzers, then build Release
just run -- ... # run the CLI

Cargo is the authoritative native build graph:

cargo fmt --manifest-path native/dotpython-abi3/Cargo.toml --all -- --check
cargo clippy --manifest-path native/dotpython-abi3/Cargo.toml --workspace --all-targets -- -D warnings
cargo test --manifest-path native/dotpython-abi3/Cargo.toml --workspace
native/dotpython-abi3/build.sh build/native-abi3

The checked-in build.sh is the authoritative verification and explicit-staging wrapper used by MSBuild. AddressSanitizer and UndefinedBehaviorSanitizer require a separate nightly Rust workflow; the stable workflow relies on Rust checks, Clippy, symbol-boundary checks, and harness leak accounting.

Build settings (Directory.Build.props) enforce C# 14, nullable reference types, TreatWarningsAsErrors, all analyzers enabled, and deterministic builds.

Testing

dotnet test DotPython.sln

Test project	Focus
tests/DotPython.ParserTests	Tokenizer and parser behavior.
tests/DotPython.CompilerTests	Binding and bytecode compilation.
tests/DotPython.RuntimeTests	Managed VM execution.
tests/DotPython.InteropTests	Static export contracts and Python-to-CLR type mapping.
tests/DotPython.DifferentialTests	Behavior compared against the CPython reference.
tests/DotPython.PackageCompatibilityTests	Package/language compatibility matrix.
tests/DotPython.BuildIntegrationTests	.dpyproj SDK packaging, ProjectReference, incremental, and runtime execution.
tests/DotPython.WorkerTests	Framing, handshake, process lifecycle, limits, cancellation, crash, timeout, and recycling.

Benchmarking

Run a short managed-runtime sample in Release mode:

dotnet run -c Release --project benchmarks/DotPython.Benchmarks -- \
  --filter '*RuntimeBenchmarks*' --job short

The benchmark project reports time and managed allocations separately for tokenization, parsing, compilation, precompiled artifact execution, and end-to-end source execution. Treat results as machine-specific observations; do not compare results collected on different hardware or runtime configurations.

License

Released under the MIT License.