Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions doc/observability/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,30 @@ with tracer.span("downstream_step", context=ctx) as span:

Call `tracer.shutdown()` (done automatically by `Xcore.shutdown()`) before process exit — otherwise spans still sitting in the `BatchSpanProcessor` buffer are dropped, never exported.

### What each layer costs per call

Measured on a trivial trusted plugin (best of 7 rounds × 2000 calls, Python 3.14, 8 cores — see
`scripts/profile_runtime.py`):

| Layer | p50 | Delta |
|---|---|---|
| none | 38.5 µs | — |
| logging at DEBUG | 40.0 µs | +1.5 µs |
| metrics (Prometheus) | 43.4 µs | +5 µs |
| tracing, OTel SDK with a no-op exporter | 95.5 µs | **+57 µs (×2.5)** |
| tracing, OTel console exporter | 181 µs | +143 µs |

Logging and metrics are cheap. **The OTel span is the only significant cost on the call path** — budget
for it before enabling tracing on a hot service, and prefer sampling at the collector.

> **The console exporter is for development only.** It uses a synchronous `SimpleSpanProcessor`, so
> every span is written out on the call path (+143 µs per call measured above). In production, set an
> `endpoint`: the OTLP exporter batches. Not measured here: OTLP + `BatchSpanProcessor` under load.

The in-memory history behind `tracer.export()` is capped by `observability.tracing.max_local_spans`
(default 1000, `0` to keep nothing). It is a debugging aid, not an export pipeline — it keeps the most
recent spans and nothing older.

---

## 4. Health Checks
Expand Down
68 changes: 68 additions & 0 deletions tests/unit/kernel/test_tracing_local_history.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
"""The local span history is bounded (issue #305, memory leak).

`TracingMiddleware` wraps every plugin call, so `Tracer.span()` is on the hot path even when tracing
is disabled. `Tracer._spans` was a plain list that nothing ever trimmed, so a `Span` leaked per call:
~562 B/call measured, about 4.9 GB/day per worker at 100 req/s.

What these tests defend:
- the history stops growing past its bound, whatever the number of calls;
- `export()` keeps working (it is the introspection/debug use case the history exists for) and shows
the MOST RECENT spans, not the first ones;
- the bound is configurable, including `0` (keep nothing), and an absurd setting cannot turn the
bound off — `deque(maxlen=None)` would be an unbounded list again, i.e. the leak.
"""

import pytest

from xcore.configurations.sections import TracingConfig
from xcore.kernel.observability.tracing import DEFAULT_MAX_LOCAL_SPANS, Tracer


def _run(tracer, count, name="op"):
for i in range(count):
with tracer.span(f"{name}-{i}"):
pass


class TestLocalHistoryIsBounded:
def test_history_stops_growing_past_the_bound(self):
tracer = Tracer("svc", TracingConfig(max_local_spans=50))
_run(tracer, 5_000)
assert len(tracer._spans) == 50

def test_default_bound_applies_without_any_configuration(self):
tracer = Tracer("svc")
_run(tracer, DEFAULT_MAX_LOCAL_SPANS * 2)
assert len(tracer._spans) == DEFAULT_MAX_LOCAL_SPANS

def test_export_keeps_working_and_shows_the_most_recent_spans(self):
tracer = Tracer("svc", TracingConfig(max_local_spans=3))
_run(tracer, 10)
exported = tracer.export()
assert [s["name"] for s in exported] == ["op-7", "op-8", "op-9"]
assert all(
s["duration_ms"] is not None and s["status"] == "ok" for s in exported
)

def test_zero_keeps_nothing_at_all(self):
tracer = Tracer("svc", TracingConfig(max_local_spans=0))
_run(tracer, 100)
assert tracer.export() == []

@pytest.mark.parametrize("value", [-1, None, "beaucoup", True, 2.5])
def test_an_absurd_setting_never_removes_the_bound(self, value):
"""`deque(maxlen=None)` is an unbounded list — the leak itself. A bad setting must not reach it."""
tracer = Tracer("svc", TracingConfig(max_local_spans=value))
assert tracer._spans.maxlen is not None
_run(tracer, DEFAULT_MAX_LOCAL_SPANS + 100)
assert len(tracer._spans) <= DEFAULT_MAX_LOCAL_SPANS

def test_a_span_still_carries_what_callers_read_from_it(self):
"""The bound must not change the span API: a caller sets attributes and reads ids/duration."""
tracer = Tracer("svc", TracingConfig(max_local_spans=10))
with tracer.span("op", tenant="acme") as span:
span.set_attribute("rows", 3)
assert span.attributes == {"tenant": "acme", "rows": 3}
assert len(span.trace_id) == 32 and len(span.span_id) == 16
assert span.duration_ms is not None
assert tracer.export()[-1]["attributes"] == {"tenant": "acme", "rows": 3}
72 changes: 72 additions & 0 deletions tests/unit/kernel/test_yamlio.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
"""The kernel reads YAML through the C loader when it is available (issue #305, point 4).

`yaml.safe_load` uses the pure-Python loader even when PyYAML is built against libyaml. The kernel
reads one manifest per plugin at boot, and the sandbox worker re-reads its own at every recycle, so
this is pure startup time: measured 3.5 ms -> 0.37 ms on a real manifest.

What these tests defend:
- `yamlio.safe_load` parses exactly what `yaml.safe_load` parses (same result, same types);
- it is just as SAFE: an arbitrary-object tag is refused, like `SafeLoader` (the whole point of not
reaching for `yaml.load` with the full loader);
- it works whether or not the C loader is present, so an installation without libyaml is slower and
nothing more.
"""

import io

import pytest
import yaml

from xcore.kernel import yamlio

DOC = """
name: te
version: 1.0.0
enabled: true
retries: 3
ratio: 1.5
nothing: null
permissions:
- resource: "db.*"
actions: [read, write]
tags: {a: 1, b: [2, 3]}
"""


class TestParity:
def test_it_parses_what_safe_load_parses(self):
assert yamlio.safe_load(DOC) == yaml.safe_load(DOC)

def test_types_are_the_same_not_just_the_shape(self):
parsed = yamlio.safe_load(DOC)
assert parsed["enabled"] is True
assert isinstance(parsed["retries"], int) and isinstance(parsed["ratio"], float)
assert parsed["nothing"] is None
assert parsed["tags"] == {"a": 1, "b": [2, 3]}

def test_it_reads_a_stream_as_well_as_a_string(self):
assert yamlio.safe_load(io.StringIO(DOC)) == yaml.safe_load(DOC)

@pytest.mark.parametrize("empty", ["", "\n", "---\n"])
def test_an_empty_document_is_none_as_everywhere(self, empty):
assert yamlio.safe_load(empty) is None


class TestSafety:
def test_an_arbitrary_object_tag_is_refused(self):
"""Same safety as SafeLoader: a manifest must never be able to build a Python object."""
with pytest.raises(yaml.YAMLError):
yamlio.safe_load("!!python/object/apply:os.system ['echo nope']")

def test_a_syntax_error_is_a_yaml_error_not_a_crash(self):
with pytest.raises(yaml.YAMLError):
yamlio.safe_load("a: [1, 2\nb: 3")


def test_the_fast_flag_says_which_loader_is_in_use():
"""Informational, and it is what a diagnostic reads: it must stay a boolean, never raise."""
assert isinstance(yamlio.FAST, bool)
if yaml.__with_libyaml__: # pragma: no branch - dépend de l'installation
assert (
yamlio.FAST
), "PyYAML est compilé avec libyaml : le loader C devrait être pris"
74 changes: 74 additions & 0 deletions tests/unit/sdk/test_lazy_imports.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
"""`import xcore` does not pay for celery or SQLAlchemy (issues #305 point 2, #306).

`xcore/sdk/__init__.py` imported `..services.xworker` (celery) eagerly, and the external `xcoresdk`
package (which drags SQLAlchemy through its auth module). Both were paid by every application that
does `import xcore` — and by EVERY sandbox worker, at every recycle, even with no worker service and
no database configured. Measured: `import xcore` 594 ms -> 294 ms once both are lazy.

What these tests defend, each in a FRESH interpreter (`sys.modules` is global, so an earlier test
importing celery would hide the regression):
- after `import xcore`, neither celery nor sqlalchemy is in `sys.modules`;
- the names are still reachable, and `from xcore.sdk import *` still exposes them;
- the first access does the import, so nothing is silently broken for the plugins that use them.
"""

import subprocess
import sys

import pytest


def _run(code: str) -> str:
"""Run `code` in a fresh interpreter and return its stdout (fails the test on a non-zero exit)."""
done = subprocess.run([sys.executable, "-c", code], capture_output=True, text=True)
assert done.returncode == 0, done.stderr
return done.stdout.strip()


class TestNothingHeavyAtImportTime:
@pytest.mark.parametrize("heavy", ["celery", "sqlalchemy"])
def test_import_xcore_does_not_import_it(self, heavy):
loaded = _run(f"import sys, xcore; print({heavy!r} in sys.modules)")
assert loaded == "False"

@pytest.mark.parametrize("heavy", ["celery", "sqlalchemy"])
def test_importing_the_sdk_itself_does_not_import_it_either(self, heavy):
loaded = _run(f"import sys, xcore.sdk; print({heavy!r} in sys.modules)")
assert loaded == "False"


class TestTheNamesStillWork:
def test_the_worker_service_resolves_and_only_then_pulls_celery(self):
out = _run(
"import sys, xcore.sdk as s\n"
"before = 'celery' in sys.modules\n"
"name = s.WorkerService.__name__\n"
"print(before, name, 'celery' in sys.modules)"
)
assert out == "False WorkerService True"

def test_task_and_task_registry_resolve(self):
out = _run(
"from xcore.sdk import task, task_registry\n"
"print(callable(task), task_registry is not None)"
)
assert out == "True True"

def test_a_star_import_still_exposes_the_lazy_names(self):
"""`__all__` must not shrink: a plugin doing `from xcore.sdk import *` kept working."""
out = _run(
"import xcore.sdk as s\n"
"names = set(s.__all__)\n"
"print(all(n in names for n in ('WorkerService', 'task', 'task_registry')))"
)
Comment on lines +61 to +63

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Star import can still fail

When an export included in __all__ beyond the three checked worker names is unavailable or mis-mapped, the test checks only that three worker names appear in __all__; it never executes from xcore.sdk import *. When an available conditional lazy export, such as a SQL adapter, is listed but cannot be resolved, the star import raises while this assertion still passes.

Run an actual star import in a fresh interpreter and verify the expected names are bound.

Prompt for AI agents
In `tests/unit/sdk/test_lazy_imports.py` at lines 61-63:

**Star import can still fail**

When an export included in `__all__` beyond the three checked worker names is unavailable or mis-mapped, the test checks only that three worker names appear in `__all__`; it never executes `from xcore.sdk import *`. When an available conditional lazy export, such as a SQL adapter, is listed but cannot be resolved, the star import raises while this assertion still passes.

Run an actual star import in a fresh interpreter and verify the expected names are bound.

assert out == "True"

def test_a_name_that_does_not_exist_still_raises_attribute_error(self):
out = _run(
"import xcore.sdk as s\n"
"try:\n"
" s.PasDuTout\n"
"except AttributeError as e:\n"
" print('AttributeError', 'PasDuTout' in str(e))"
)
assert out == "AttributeError True"
9 changes: 7 additions & 2 deletions tests/unit/security/test_ast_scanner.py
Original file line number Diff line number Diff line change
Expand Up @@ -388,8 +388,13 @@ def test_inject_dotenv_path_traversal(self, validator, tmp_path):
assert "traversal" in str(exc_info.value)

def test_yaml_import_error(self, validator, tmp_path):
"""Test _yaml raises error when pyyaml not installed."""
with patch.dict("sys.modules", {"yaml": None}):
"""Test _yaml raises error when pyyaml not installed.

`_yaml` passe par `xcore.kernel.yamlio` (loader C quand il est là, #305) : c'est donc
CE module qu'il faut rendre introuvable pour simuler l'absence de PyYAML — rendre
`yaml` seul inutilisable ne change rien quand `yamlio` est déjà importé.
"""
with patch.dict("sys.modules", {"yaml": None, "xcore.kernel.yamlio": None}):
(tmp_path / "test.yaml").touch()
with pytest.raises(ManifestError) as exc_info:
ManifestValidator._yaml(tmp_path / "test.yaml")
Expand Down
61 changes: 61 additions & 0 deletions tests/unit/security/test_scanner_fails_closed.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
"""A security scanner must fail CLOSED: a file it cannot parse is not a file it found clean.

`ASTScanner.scan` has two import phases: the compiled `scanner_core` extension when it is built, the
Python AST otherwise. Only the Python one reported a syntax error — so with the compiled scanner
installed (the optimised, shipped configuration) a plugin whose source does not parse came back
`passed=True`, and none of its imports, builtins or forbidden attributes had been examined.

`_check_builtins_and_attrs` parses every file in BOTH phases, so that is where the parse failure is
now reported, and only when the import phase could not do it (no duplicate message).
"""

from pathlib import Path

import pytest

from xcore.kernel.security.validation import ASTScanner


def _plugin(tmp_path: Path, source: str) -> Path:
src = tmp_path / "src"
src.mkdir()
(src / "main.py").write_text(source)
return tmp_path


class TestUnparsableIsRefused:
def test_a_file_that_does_not_parse_fails_the_scan(self, tmp_path):
result = ASTScanner().scan(_plugin(tmp_path, "def broken(\n pass"))
assert result.passed is False
assert any("syntaxe" in e for e in result.errors)

def test_it_is_said_once_not_twice(self, tmp_path):
"""Both phases parse the file; the operator must read one message, not two."""
result = ASTScanner().scan(_plugin(tmp_path, "def broken(\n pass"))
assert len([e for e in result.errors if "syntaxe" in e]) == 1

def test_the_verdict_does_not_depend_on_the_compiled_scanner(
self, tmp_path, monkeypatch
):
"""The whole point: the C++ path and the Python path must agree on an unparsable file."""
plugin = _plugin(tmp_path, "def broken(\n pass")
with_cpp = ASTScanner()
without_cpp = ASTScanner()
monkeypatch.setattr(without_cpp, "_cpp", None)
Comment on lines +42 to +44

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Compiled scanner path goes untested

When the test suite runs without the optional C++ scanner extension, both ASTScanner instances use the Python fallback, so the assertion compares that path with itself and the compiled scanner’s fail-closed behavior is not tested.

Require the compiled scanner for this test and verify that with_cpp uses it before asserting its behavior.

Prompt for AI agents
In `tests/unit/security/test_scanner_fails_closed.py` at lines 42-44:

**Compiled scanner path goes untested**

When the test suite runs without the optional C++ scanner extension, both `ASTScanner` instances use the Python fallback, so the assertion compares that path with itself and the compiled scanner’s fail-closed behavior is not tested.

Require the compiled scanner for this test and verify that `with_cpp` uses it before asserting its behavior.

assert with_cpp.scan(plugin).passed is without_cpp.scan(plugin).passed is False

def test_a_sound_file_still_passes(self, tmp_path):
result = ASTScanner().scan(
_plugin(tmp_path, "import json\n\nVALUE = json.dumps({'a': 1})\n")
)
assert result.passed is True, result.errors
assert result.errors == []

@pytest.mark.parametrize("source", ["eval('1+1')", "exec('x = 1')"])
def test_a_forbidden_builtin_in_a_sound_file_is_still_caught(
self, tmp_path, source
):
"""The regression to avoid: reporting parse errors must not shadow the real checks."""
result = ASTScanner().scan(_plugin(tmp_path, source))
assert result.passed is False
assert any("built-in interdit" in e for e in result.errors)
4 changes: 2 additions & 2 deletions xcore/configurations/loader.py
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,10 @@ def _read(cls, path: str | Path | None) -> dict[str, Any]:
suffix = candidate.suffix.lower()
try:
if suffix in (".yaml", ".yml"):
import yaml
from ..kernel import yamlio

with open(candidate, encoding="utf-8") as f:
data = yaml.safe_load(f) or {}
data = yamlio.safe_load(f) or {}
else:
import json

Expand Down
5 changes: 5 additions & 0 deletions xcore/configurations/sections.py
Original file line number Diff line number Diff line change
Expand Up @@ -315,6 +315,11 @@ class TracingConfig:
backend: str = "noop" # noop | opentelemetry | jaeger
service_name: str = "xcore"
endpoint: str | None = None
# Nombre de spans gardés EN MÉMOIRE pour l'introspection (`Tracer.export()`), les plus
# récents. Ce n'est pas le pipeline d'export OTel : c'est un historique de débogage, et
# il doit être borné — une liste sans limite faisait ~562 octets par appel, soit près de
# 5 Go par jour et par worker à 100 req/s (#305). 0 = ne rien garder.
max_local_spans: int = 1000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Configured span history is ignored

When observability.tracing.max_local_spans is set in application configuration, ConfigLoader._parse_observability does not pass max_local_spans into TracingConfig, so the default limit is used and the configured history size is ignored.

Pass the configured max_local_spans value into TracingConfig in ConfigLoader._parse_observability.

Also at xcore/configurations/sections.py:323.

Prompt for AI agents
In `xcore/configurations/sections.py` at line 322:

**Configured span history is ignored**

When `observability.tracing.max_local_spans` is set in application configuration, `ConfigLoader._parse_observability` does not pass `max_local_spans` into `TracingConfig`, so the default limit is used and the configured history size is ignored.

Pass the configured `max_local_spans` value into `TracingConfig` in `ConfigLoader._parse_observability`.

Also at `xcore/configurations/sections.py:323`.



@dataclass
Expand Down
20 changes: 19 additions & 1 deletion xcore/kernel/observability/tracing.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

import time
import uuid
from collections import deque
from contextlib import contextmanager
from dataclasses import dataclass, field
from typing import TYPE_CHECKING, Any, Generator, Mapping, MutableMapping
Expand Down Expand Up @@ -40,6 +41,19 @@
_provider_configured = False


DEFAULT_MAX_LOCAL_SPANS = 1000


def _max_local_spans(config: "TracingConfig | None") -> int:
"""La taille de l'historique local, prise dans la configuration et bornée à zéro : un réglage
négatif ou aberrant ne doit pas rendre `deque(maxlen=…)` illimité (`maxlen=None`).
"""
value = getattr(config, "max_local_spans", DEFAULT_MAX_LOCAL_SPANS)
if isinstance(value, bool) or not isinstance(value, int):
return DEFAULT_MAX_LOCAL_SPANS
return max(value, 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Tracer startup fails on large limits

When max_local_spans exceeds the platform’s Py_ssize_t range, _max_local_spans returns the value unchanged, so deque(maxlen=...) raises OverflowError during Tracer construction and prevents startup.

Clamp or reject values that cannot be represented as a Py_ssize_t before passing them to deque.

Prompt for AI agents
In `xcore/kernel/observability/tracing.py` at line 54:

**Tracer startup fails on large limits**

When `max_local_spans` exceeds the platform’s `Py_ssize_t` range, `_max_local_spans` returns the value unchanged, so `deque(maxlen=...)` raises `OverflowError` during `Tracer` construction and prevents startup.

Clamp or reject values that cannot be represented as a `Py_ssize_t` before passing them to `deque`.



@dataclass
class Span:
"""
Expand Down Expand Up @@ -134,7 +148,11 @@ def __init__(
config: "TracingConfig | None" = None,
) -> None:
self.service_name = service_name
self._spans: list[Span] = []
# BORNÉ : `span()` est traversé par chaque appel de plugin (TracingMiddleware enveloppe
# tout, même tracing désactivé). Une liste sans limite gardait donc un Span par appel
# pour toujours — une fuite mesurée à ~562 octets par appel (#305). Un deque garde les
# N plus récents, ce qui est tout ce que `export()` sert à montrer.
self._spans: deque[Span] = deque(maxlen=_max_local_spans(config))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Concurrent exports fail

When Tracer.export() runs while another thread completes a span, Tracer.export() iterates self._spans while span completion appends to the deque; the iterator raises RuntimeError and the caller gets no history.

Synchronize deque access so Tracer.export() iterates a stable snapshot of the spans.

Prompt for AI agents
In `xcore/kernel/observability/tracing.py` at line 155:

**Concurrent exports fail**

When `Tracer.export()` runs while another thread completes a span, `Tracer.export()` iterates `self._spans` while span completion appends to the deque; the iterator raises `RuntimeError` and the caller gets no history.

Synchronize deque access so `Tracer.export()` iterates a stable snapshot of the spans.

self._otel_tracer = None

if (
Expand Down
Loading
Loading