Skip to content

About

A Rust-like language with a Python backend and a native PyO3 extension backend.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

190 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

polyglot-lang

A Rust-like language with a shared lexer, parser, and AST, plus a Python backend and a native PyO3 extension backend.

Why polyglot-lang?

You want Rust's syntax, type safety, and explicit error handling. But you don't want to rewrite your whole project in Rust, or set up Cargo.toml + PyO3 + maturin just to speed up one function.

polyglot-lang lets you write one .poly file:

fn add(a: i64, b: i64) -> i64 { a + b }

and choose how to run it:

  • Python backend — polyglot run add.poly (no build step, no Rust)
  • Native backend — polyglot build add.poly (compiles to a PyO3 extension)

Same source, same tests, same error messages. No Cargo.toml, no pyproject.toml scaffolding.

Quick Start

New to Rust-like syntax? See GUIDE.md for a step-by-step tutorial.

pip install polyglot-lang

Create hello.poly:

fn add(a: i64, b: i64) -> i64 { a + b }
fn main() -> i64 { add(2, 3) }

Run it:

polyglot run hello.poly

Output:

5

Build a native extension (requires a Rust toolchain and Cargo):

polyglot build hello.poly
python -c "import hello; print(hello.add(2, 3))"

If you don't have a Rust toolchain, you can skip this step. The Python backend (polyglot run) works without it.

First native build takes about 2 minutes. Subsequent builds use a cache and take about 3 seconds. The Python backend (polyglot run) has no build step.

Status

Milestone Status Scope
N0 Done Empty program → Cargo → load module
N1 Done Free functions, i64/f64/bool/String, Vec, structs with impl exposed as PyO3 classes, and if
N2a Done Native while, loop, and match
N2b Done Vec methods: push, pop, len, and indexing
N2c Done Multiple structs/enums as pyclass
N2d Done Improved caching and Rust-error mapping to source lines
N3 Done Vec iteration, contains, sort/sort_by, map/filter/fold, slicing, and mutation methods
N4 Done String length/iteration, split/join, replace/trim, case conversion, parse, format!, write!, slicing, and comparison
N5 Done Option/Result, ?, if let, async free functions, match exhaustiveness warnings, and Python sandboxing
N6 Done Additional Vec and String methods, including search, slicing, rotation, and sort helpers
N7 Done Distinct usize/isize types, explicit casts, and target-width-aware boundary checks
N8 Done Collections-layer HashMap operations and Python/native dictionary boundaries
N9 Done Platform-defined Option/Result combinators and native Result continuations
N10 Done SPEC v0.2.7 and final conformance status
M10 Done Opt-in termination checking for #[terminates] functions
F1 Done Installable package, CLI, examples, and delivery documentation
M3 Done HashMap entry completion, BTreeMap, HashSet, and nested HashMap depth 1
M4 Done Additional fixed-width integer types and the cross-backend cast matrix
M5 Done Send-aware PyFuture, async &self impl methods, drop-based cancellation, and a polling timeout helper

Current test status: 686 passed, 5 skipped, 19 deselected (pytest -q); 19 passed, 691 deselected (pytest -q -m slow).

Support matrix

Feature Python backend Native backend
Vec, String, Option, Result ✅ ✅
HashMap (core ops) ✅ ✅
HashMap mutable iterators (iter_mut, values_mut) ✅ ✅
BTreeMap ✅ ✅
HashSet ✅ ✅
Nested HashMap (depth 1) ✅ ✅
Integer types (i8, i16, i32, i64, i128, u8, u16, u32, u64, u128, isize, usize) ✅ ✅
include! directive ✅ ✅
Error codes (E001–E008) ✅ ✅
#[terminates] checking ✅ ✅
async free functions ✅ (synchronous) ✅
async &self impl methods ✅ ✅
cross-thread await with Send captures n/a ✅
drop-based cancellation n/a ✅
polling timeout helper n/a ✅
Box patterns ✅ ⬜ Deferred
Payload enum slices ✅ ⬜ Deferred

BTreeMap iteration is sorted by key in both backends. HashSet iteration order is unspecified; sort before comparing. get_mut on BTreeMap/HashSet returns an owned copy at the native Python boundary (see SPEC 8.10.2 and 10.20).

Features

  • Shared front end: Turns source text into tokens, a parsed program, and AST nodes.
  • Python backend: Translates the Rust-like AST to a Python AST, then compiles and executes it.
  • Native backend: Generates Rust source with PyO3, builds it with Cargo, and loads the resulting Python extension (.so on macOS and Linux, .pyd on Windows).
  • Common backend interface: Select a backend with get_backend("python") or get_backend("native"). Calling backend.compile(program) returns a namespace dictionary.
  • CLI: polyglot run, polyglot build, and polyglot check for file-based workflows.
  • N5: Option and Result construction and combinators, the ? operator, if let, async functions, match exhaustiveness warnings, and Python sandboxing.
  • N6: Additional Vec methods (first, last, get, truncate, rotate_*, fill, swap, resize, split_off, partition, sort_*, is_sorted, and others) and String methods (find, rfind, strip_*, split_once, lines, as_bytes, is_char_boundary, and others).
  • N7: Distinct usize/isize types, explicit as boundaries, range-loop variable typing, and selectable Python target-width simulation.
  • N8: HashMap<K, V> construction and supported operations, typed usize length, dict conversion, key/value/iteration views, mutable iterators, entry insertion, and mutable access.
  • N9: Dedicated Python/native coverage for Platform-defined Result/Option combinators and native Result::and_then/or_else.
  • M10: #[terminates] enables conservative static termination checks for bounded ranges, decreasing while variants, and recursive descent.
  • M5: Send-aware PyFuture, async &self impl methods, drop-based cancellation, and a polling timeout helper for the native backend.
  • Module isolation: Every Python compilation registers under a unique generated module name, so separate programs do not clobber each other in sys.modules.

CLI

polyglot run <file> [--backend python|native] [--target-width N]
polyglot build <file> [--backend native] [--target <triple>] [--out-dir <dir>]
polyglot check <file>
  • run compiles and executes a file and prints the result of main() when it returns a value.
  • build compiles a file. The native backend produces an extension module and prints its path.
  • check parses, translates (Python backend), and runs termination checks without executing.
  • Errors use the toolchain five-line format, or six lines when a span is available. Pass --verbose to include a traceback.
  • --error-format text|json selects the diagnostic format on run, build, check, and compile. JSON output is a single object with code, message, line, col, span, and source.

The Python backend has no native dependencies. The native backend requires a Rust toolchain (cargo and rustc) and PyO3-compatible build tooling.

Include directive

include!("path.poly"); splices the items of another .poly file at that point. Paths are resolved relative to the including file, and includes may nest.

// math.poly
fn add(a: i64, b: i64) -> i64 { a + b }

// main.poly
include!("math.poly");
fn main() -> i64 { add(2, 3) }
  • Includes are one-way; the included file does not see the includer's items.
  • Cycles are rejected with E005, and missing files with E006.
  • Including the same file twice is allowed and splices its items twice. This is textual inclusion: duplicate item definitions are the author's responsibility.
  • include! is only valid at item position. Using it inside a block is a parse error.
  • Absolute paths and .. segments are allowed. There is no repository-containment policy.

The directive is expanded before code generation, so both backends see the same spliced program.

Examples

The examples/ directory contains runnable programs:

  • hello.poly — function call and return value
  • fib.poly — recursive #[terminates] function
  • vec_ops.poly — Vec mutation, length, and sorting
  • string_ops.poly — split, join, and replace
  • hashmap.poly — HashMap insert/get/iteration
  • option_result.poly — Option, Result, ?, and if let
  • terminates.poly — #[terminates] decreasing loop

Run them with:

polyglot check examples/hello.poly
polyglot run examples/hello.poly

Primitive semantics

Explicit numeric types follow Rust where Python's defaults differ:

  • as casts truncate and reinterpret integer values at the target width; isize/usize are 64-bit, f32 rounds through single precision, and char uses Unicode code points.
  • / truncates toward zero for integers; % keeps the dividend's sign. Float % uses fmod.
  • Annotated numeric bindings, assignments, compound assignments, and returns wrap at the declared width, matching native release builds.

Overflow parity is limited to explicitly typed bindings; unannotated inference still uses Python's unbounded integers. These semantics are covered by tests/test_native_n8_types.py.

Async

Native async functions can be awaited from Python with asyncio.run:

import asyncio

from backends import get_backend
from lexer import lex
from parser import parse

source = """
async fn add(a: i64, b: i64) -> i64 {
    a + b
}
"""
namespace = get_backend("native").compile(parse(lex(source)))
assert asyncio.run(namespace["add"](2, 3)) == 5

The native backend uses a hand-written PyFuture runtime, not pyo3-asyncio. Futures are Send-aware: a future created on one thread can be awaited on another when its captured values are Send. Async &self impl methods also return PyFuture; referenced fields are cloned into owned locals before the future is created, so the method body does not capture self. Dropping a future or its iterator before completion cancels it by drop semantics.

The internal await_with_timeout(future, seconds) helper polls until the future is ready or the timeout expires, sleeping with the GIL released. It is polling-based and does not integrate with an event loop. Async &mut self methods are rejected by codegen, as are async methods that use bare self or capture non-Clone fields.

Sandbox

PythonBackend(sandbox=True) restricts __builtins__ to a safe subset, independently of the trusted flag. It blocks __import__, eval, open, compile, and globals. The safe subset includes abs, all, any, bool, dict, enumerate, filter, float, format, frozenset, int, isinstance, issubclass, iter, len, list, map, max, min, next, object, print, range, repr, reversed, round, set, slice, sorted, str, sum, tuple, type, zip, the True/False/None constants, standard exceptions, and __build_class__.

from backends.python.translator import PythonBackend

namespace = PythonBackend(sandbox=True).compile(program)

For isolation levels A/B/C and the X-Trusted-Token requirement, see the Security section in NOTES.md.

trusted=False is not a sandbox; it is a speed bump. The sandbox restricts __builtins__ but does not provide process isolation. Treat untrusted input as requiring OS-level isolation, such as the subprocess or Docker levels described in NOTES.md (isolation B or C), with network access disabled and a read-only filesystem where appropriate.

Performance

Native builds use sccache through RUSTC_WRAPPER when it is available. Cargo's target directory stays within each build directory. After clearing the cache, the first build takes about two minutes; a warm second run takes about three seconds. Four slow tests, including source-line mapping, are deselected by default.

See NOTES.md for detailed build-performance notes.

Relationship to PyO3 and maturin

Quick decision: If you want full Rust control, wheel packaging, and cross-platform builds, use maturin. If you want to write one Rust-like file and get either a Python module or a native extension without writing Cargo.toml or PyO3 scaffolding by hand, use polyglot-lang. Native builds still run Cargo internally, and the current single-file model does not yet expose arbitrary crate dependencies.

Error codes

Diagnostics include a stable error code. The table below maps each code to its class and meaning.

Code Class Meaning
E001 TypeErrorForBoundary An implicit conversion between different integer types was attempted. Add an explicit as cast. See SPEC 2.4.6–2.4.7.
E002 TerminationError A function annotated #[terminates] could not be proven to terminate. See SPEC 2.8.
E003 TranslateError A generic AST-to-backend translation failure (unsupported construct, invalid pattern, etc.).
E004 RustCompileError The generated Rust failed to compile. The message includes the original source location when available.
E005 IncludeCycleError A cyclic include! chain was detected. The message prints the full chain. See SPEC 9.2.
E006 IncludeNotFoundError An include! target file does not exist or is unreadable. See SPEC 9.2.
E007 UnsupportedNativePatternError A source pattern cannot be represented by stable Rust. See SPEC 8.3.
E008 InvalidIntegerParseLiteralError A literal integer could not be parsed for the target type.

Known limitations

  • is_ok, is_err, map_err, ok_or, and_then, and or_else are Python-only inside let-bindings.
  • while let Some(x) = iter.next() is out of scope because Python has no Rust-style Option-returning next().
  • Box patterns, enum payload slices, and nested Vec slice patterns are Python-only.
  • Async &mut self methods are rejected by the native codegen. Async &self methods may only read fields (self.<field>); bare self use and non-Clone fields such as payload enums are rejected.
  • The async timeout helper uses polling with std::thread::sleep and does not integrate with an event loop. External wakeups are not supported: futures are polled only when Python calls into PyFutureIterator.__next__.
  • Option map is intentionally unsupported because method dispatch is not yet fully type-aware; Vec::map and Option::map collide at the call site. Vec::filter and Option::filter are currently distinguished by a receiver-type check, which is not sufficient for map. Both backends reject it with a clear error. Use match or if let instead. See SPEC 5.7.
  • HashMap supports only the implemented entry subset (or_insert, or_insert_with, or_insert_with_key, and_modify, key); other entry combinators are not yet lowered.
  • vec![...] macro syntax is parsed as a token stream and is not lowered; use [...].to_vec() instead.
  • String::from(...) is not lowered; use "...".to_string() instead.
  • The #[terminates] checker accepts a single recursive call that strictly decreases one argument. Multiple recursive calls (e.g. classic fib(n-1) + fib(n-2)) are deferred and therefore rejected even though they terminate. Mutual recursion, loops with break, complex arithmetic, non-recursive calls, and async suspension are also deferred. See SPEC 10.22.
  • if let is supported as a statement and as the final expression of a block (e.g. the tail of main()), but not as a sub-expression inside a let binding or a function argument. Use match when an if let expression is required inside another expression.
  • BTreeMap and HashSet are available in both backends for the type sets in SPEC 7.2.6. Deeper composition and unsupported key/value combinations remain deferred. See SPEC 10.25 and 10.27.
  • Nested HashMap (HashMap<K, HashMap<K2, V2>>) is supported at depth 1 in both backends. Deeper nesting is deferred. See SPEC 10.25.
  • get_mut on BTreeMap and HashSet follows the same Python-boundary copy semantics as HashMap (Python: mutable view; native: owned copy that cannot mutate the input). See SPEC 8.10.2 and 10.20.

Running tests

Run the test suite from the project root:

pytest

The native backend caches build artifacts under ~/.cache/polyglot-native. To clear the cache:

rm -rf ~/.cache/polyglot-native

Architecture

source → lexer → parser → AST → (Python AST → compile/exec | Rust + PyO3 → Cargo build → extension) → Python namespace

Roadmap

N0 through N10, M10, F1, and M5 are complete. Remaining known limitations are listed under Known limitations. The main out-of-scope items are while let Some(x) = iter.next(), async &mut self methods, and real wakeups/event-loop integration.

Notes

See NOTES.md for design decisions, per-batch history, and the lessons recorded for each fixed bug.

License

MIT License — see LICENSE for the full text.

About

A Rust-like language with a Python backend and a native PyO3 extension backend.

Topics

Resources

Contributing

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages