Skip to content

perf(kernel): borne la fuite de spans, -300 ms d'imports, YAML x24 - #314

Merged
traoreera merged 1 commit into
mainfrom
perf-305-310
Nov 11, 2026
Merged

traoreera merged 1 commit into
mainfrom
perf-305-310

Conversation

@traoreera

@traoreera traoreera commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Traite #305 points 1, 2, 3 et 4, la moitié mesurable de #306, le fond de #310, et un défaut de sécurité trouvé en route. Chaque gain est mesuré avant/après sur cette machine (Python 3.14.8, 8 cœurs).

#305.1 — La fuite (bug)

Tracer._spans était une list que rien ne vidait, alors que TracingMiddleware enveloppe chaque appel de plugin, tracing désactivé compris : un Span fuyait par appel (~562 o, près de 5 Go/jour/worker à 100 req/s). C'est maintenant un deque(maxlen=…), réglé par observability.tracing.max_local_spans (défaut 1000, 0 pour ne rien garder).

Critères d'acceptation de l'issue : len(tracer._spans) reste borné après N ≫ maxlen appels ✅, export() fonctionne toujours ✅ (et rend les spans les plus récents). Un réglage aberrant (négatif, None, texte) ne peut pas redonner maxlen=None, qui serait la fuite — c'est testé.

Non fait, et l'issue le posait comme un « considérer » : ne pas construire le Span du tout quand personne ne le lit. span() doit yielder un objet, et rendre trace_id/span_id paresseux changerait la forme publique d'une dataclass pour ~2 µs sur un appel de 38 µs. À trancher par le mainteneur.

#305.2 + #306 — Les imports

xcore/sdk/__init__.py importait ..services.xworker (celery) et le package externe xcoresdk, qui tire SQLAlchemy par son module d'authentification. Payé par toute application qui fait import xcore, et par chaque worker sandbox à chaque recyclage, sans service de tâches ni base configurés.

import xcore : 593,9 ms  →  294,1 ms
  dont celery     :  52,1 ms  →  0,0 ms
  dont sqlalchemy : 102,0 ms  →  0,0 ms
  dont fastapi    : 231,5 ms  →  187,4 ms  (le socle, irréductible)

Critère d'acceptation : python -X importtime -c "import xcore" ne montre plus celery ni sqlalchemy ✅.

Résolus à la demande (PEP 562). find_spec résout sans exécuter (0,05 ms), donc __all__ ne bouge pas et from xcore.sdk import * expose les mêmes noms qu'avant — c'est testé dans un interpréteur neuf par cas, parce que sys.modules est global et qu'un test qui aurait importé celery avant masquerait la régression.

Le gain de RSS par worker annoncé dans #306 (79 Mo) n'est pas vérifié ici : il demande un hôte avec des plugins sandboxés en vrai. Seul le temps d'import est mesuré.

#305.4 — Le YAML

Les trois lecteurs du noyau passaient par le loader Python de PyYAML alors que libyaml est installé. xcore/kernel/yamlio.py choisit CSafeLoader, avec repli silencieux sur SafeLoader (une installation sans libyaml est juste plus lente, ce n'est pas une anomalie). Module feuille, sans aucun import xcore : le worker sandbox garde son jeu d'imports minimal, comme #306 le demande.

Mesuré sur un manifeste : 33,97 ms → 1,39 ms (×24). Même sûreté que safe_load — un tag !!python/object/apply reste refusé, et c'est testé.

#305.3 — La doc

doc/observability/observability.md donne le coût par appel de chaque couche (le span OTel est le seul significatif : +57 µs, ×2,5) et dit que l'exportateur console est synchrone, donc réservé au développement.

#310 — L'ordonnanceur

Sans Redis il n'y a aucun verrou et chaque processus exécute chaque job : deux workers, deux paies. Une bibliothèque ne peut pas savoir combien de processus tournent (--workers est décidé dehors, et chaque worker se croit seul), donc pas de refus — un avertissement unique au démarrage, qui nomme les jobs concernés.

En revanche, deux replis silencieux deviennent des erreurs, parce qu'ils trompaient l'opérateur :

  • backend: redis sans les paquets redis → il croyait ses jobs protégés et n'avait rien ;
  • un backend inconnu (database est documenté dans le commentaire du champ mais pas implémenté) → il recevait « mémoire » sans un mot.

Les deux marquent le service DEGRADED. Le vrai verrou sans Redis (un bail en base) reste à faire : c'était l'option 1 de l'issue et c'est une décision d'architecture, pas un correctif.

Trouvé en route : le scan de sécurité échouait en mode OUVERT

ASTScanner.scan a deux phases d'imports — l'extension compilée scanner_core quand elle est là, l'AST Python sinon. Seule la seconde signalait une erreur de syntaxe. Avec le scanner compilé — la configuration optimisée, celle qui est livrée — un plugin dont une source ne se parse pas revenait donc passed=True, alors qu'aucun de ses imports, builtins ou attributs interdits n'avait pu être examiné. Ne pas savoir lire un fichier n'est pas l'avoir trouvé sain.

_check_builtins_and_attrs parse de toute façon dans les deux phases : c'est là que le défaut de parsing est signalé, une fois et une seule. Les deux chemins rendent maintenant le même verdict, ce qu'un test vérifie explicitement.

C'est ce qui faisait échouer test_scan_syntax_error dans test_ast_scanner.py et test_validation.py — avant comme après ce travail : vérifié en remisant mes modifications et en relançant sur l'arbre d'origine (2 échecs). Ils passent maintenant.

Vérification

1516 passed, 15 skipped   (avant : 1514 passed, 2 failed)
black / isort / flake8    muets

Un seul test a dû être adapté : test_yaml_import_error simulait l'absence de PyYAML en neutralisant sys.modules["yaml"] ; il neutralise aussi xcore.kernel.yamlio, puisque c'est par là que passe désormais la lecture. Le contrat testé est identique (ManifestError « pyyaml non installé »).

🤖 Generated with Claude Code

Summary by Sourcery

Reduce startup and YAML parsing overhead, bound tracing memory usage, and make scheduler and security failures explicit and safe.

Bug Fixes:

  • Make the security scanner fail closed when plugin source cannot be parsed, consistently across compiled and Python scanning paths.
  • Prevent scheduler misconfiguration from silently falling back to an unprotected memory backend and report degraded status when distributed locking is unavailable.

Enhancements:

  • Bound local tracing span history with configurable retention while preserving recent-span export behavior.
  • Defer optional SDK, worker, and database-related imports until their APIs are accessed to reduce startup time.
  • Use PyYAML's C safe loader when available while retaining safe-loader compatibility and fallback behavior.
  • Document observability overhead and the production limitations of synchronous console tracing.

Documentation:

  • Document per-layer observability costs, tracing history limits, and console exporter usage.

Tests:

  • Add coverage for bounded tracing history, safe YAML loader parity and safety, lazy imports, and fail-closed scanner behavior.

… échouaient en mode ouvert

Les trois premiers points de #305 dans l'ordre que l'issue conseille, la moitié
mesurable de #306, le fond de #310, et un défaut trouvé en route.

#305.1 — FUITE : `Tracer._spans` était une liste que rien ne vidait, alors que
`TracingMiddleware` enveloppe CHAQUE appel de plugin, tracing désactivé compris.
Un Span fuyait donc par appel : ~562 octets, près de 5 Go par jour et par worker à
100 req/s. C'est un `deque(maxlen=…)`, réglable par
`observability.tracing.max_local_spans` (défaut 1000, 0 pour ne rien garder) ; un
réglage aberrant ne peut pas redonner `maxlen=None`, qui serait la fuite d'origine.
`export()` rend les spans les plus RÉCENTS, ce qui est tout ce qu'un historique de
débogage doit montrer.

Non fait, et c'était un « considérer » de l'issue : ne pas construire le Span du tout
quand personne ne le lit. `span()` doit yielder un objet, et rendre `trace_id` /
`span_id` paresseux changerait la forme publique d'une dataclass pour ~2 µs sur un
appel de 38 µs.

#305.2 et #306 — IMPORTS : `xcore/sdk/__init__.py` importait `..services.xworker`
(celery) et le package externe `xcoresdk` (qui tire SQLAlchemy par son module
d'authentification). Les deux étaient payés par toute application qui fait
`import xcore`, et par CHAQUE worker sandbox à chaque recyclage, même sans service
de tâches ni base configurés. Résolus à la demande (PEP 562) :

    import xcore : 593,9 ms → 294,1 ms   (celery et sqlalchemy : 0,0 ms)

`find_spec` résout sans exécuter (0,05 ms), donc `__all__` ne change pas et
`from xcore.sdk import *` expose les mêmes noms qu'avant. Le premier accès paie
l'import, et lui seul.

#305.4 — YAML : les trois lecteurs du noyau passaient par le loader PYTHON de PyYAML
alors que libyaml est là. Un module feuille `xcore/kernel/yamlio.py` (aucun import
xcore : le worker sandbox garde son jeu minimal) choisit `CSafeLoader`, avec repli
silencieux sur `SafeLoader`. Même sûreté — un tag d'objet arbitraire reste refusé,
c'est testé. Mesuré sur un manifeste : 33,97 ms → 1,39 ms (×24).

#305.3 — DOC : le coût par appel de chaque couche d'observabilité, et le fait que
l'exportateur console est synchrone, donc réservé au développement.

#310 — ORDONNANCEUR : sans Redis il n'y a AUCUN verrou, et chaque processus exécute
chaque job — deux workers, deux paies. Une bibliothèque ne peut pas savoir combien de
processus tournent (uvicorn `--workers` est décidé dehors), donc pas de refus : un
avertissement unique au démarrage, qui NOMME les jobs concernés. Deux replis
silencieux deviennent en revanche des erreurs, parce qu'ils trompaient l'opérateur :
`backend: redis` sans les paquets redis (il croyait ses jobs protégés) et un backend
inconnu comme `database`, documenté mais pas implémenté. Les deux marquent le service
DEGRADED au lieu de se taire.

TROUVÉ EN ROUTE — le scan de sécurité échouait en mode OUVERT. `ASTScanner.scan` a
deux phases d'imports : l'extension compilée `scanner_core` quand elle est là, l'AST
Python sinon. Seule la seconde signalait une erreur de syntaxe. Avec le scanner
compilé — la configuration optimisée, celle qui est livrée — un plugin dont une source
ne se parse pas revenait donc `passed=True`, sans qu'aucun de ses imports, builtins ou
attributs interdits n'ait pu être examiné. Ne pas savoir lire un fichier n'est pas
l'avoir trouvé sain : `_check_builtins_and_attrs`, qui parse dans les DEUX phases, le
dit maintenant, une fois et une seule. C'est ce qui faisait échouer
`test_scan_syntax_error` dans deux fichiers de tests, avant comme après ce travail
(vérifié sur l'arbre d'origine).

`test_yaml_import_error` simulait l'absence de PyYAML en neutralisant `sys.modules["yaml"]` :
il neutralise aussi `xcore.kernel.yamlio`, puisque c'est par là que passe désormais la
lecture. Le contrat testé est le même (ManifestError « pyyaml non installé »).

Tests : 1516 passent, 0 échec (dont 2 qui échouaient avant). black, isort et flake8
sont muets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@sourcery-ai

sourcery-ai Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Reviewer's Guide

Data structures

Bound the tracer’s in-memory span history and add the configurable limit that controls its size.

  • xcore/configurations/sections.py L318-322
  • xcore/kernel/observability/tracing.py L15, L44-56, L151-155

API

Make SDK exports for worker, SQL, and external SDK features lazy while preserving their public names and optional-dependency errors.

  • xcore/sdk/__init__.py L118-134, L139-166, L207-235 (+3 small edits)

Provide a leaf YAML-loading interface that uses PyYAML’s safe C loader when available and safely falls back otherwise.

  • xcore/kernel/yamlio.py L1-37

Uses of the API

Route configuration, sandbox-manifest, and security-manifest YAML reads through the new shared loader.

  • xcore/configurations/loader.py L77-80
  • xcore/kernel/sandbox/worker.py L804-807
  • xcore/kernel/security/validation.py L159-162

Implementation

Make syntax and file-read failures fail the security scan rather than allowing unparsable files to pass.

  • xcore/kernel/security/validation.py L448-454, L556-581

Report missing or unsupported scheduler locks as degraded operation and warn once with the affected jobs when running unlocked.

  • xcore/services/scheduler/service.py L219-238, L259-284 (+2 small edits)

Documentation

Document measured per-call observability costs, console-exporter limitations, and the bounded local span history.

  • doc/observability/observability.md L155-178

Tests

Cover bounded span retention, recent-span export, invalid limits, and span API behavior.

  • tests/unit/kernel/test_tracing_local_history.py L1-68

Check YAML loader parity, safe rejection, stream handling, and C-loader selection.

  • tests/unit/kernel/test_yamlio.py L1-72

Verify lazy SDK imports and preserved exports in fresh interpreters.

  • tests/unit/sdk/test_lazy_imports.py L1-74

Adapt the missing-PyYAML test to the new loader module and cover fail-closed scanning across scanner implementations.

  • tests/unit/security/test_ast_scanner.py L391-397
  • tests/unit/security/test_scanner_fails_closed.py L1-61

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@github-actions

Copy link
Copy Markdown
Contributor

@traoreera traoreera changed the title perf: fuite de spans bornée, ~300 ms d'imports en moins, YAML ×24 — et le scan de sécurité qui échouait en mode ouvert perf(kernel): borne la fuite de spans, -300 ms d'imports, YAML x24 Nov 11, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 9 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="xcore/services/scheduler/service.py" line_range="277" />
<code_context>
+        """
+        if _REDIS_LOCK_CLIENT is not None:
+            return
+        jobs = [str(j.get("id") or j.get("func") or "?") for j in self._config.jobs]
+        if not jobs and not _JOB_REGISTRY:
+            return  # aucun job : rien à dire
</code_context>
<issue_to_address>
**Malformed job config aborts boot**

When `services.scheduler.jobs` contains a non-mapping entry, such as `null`, `_add_job_from_config` catches and logs invalid job entries, but `_warn_if_unlocked` then calls `.get()` on each entry without checking its type. A non-mapping entry raises `AttributeError` during scheduler initialization and aborts application boot instead of skipping the bad job.

Filter or safely normalize job entries before reading `id` and `func` in `_warn_if_unlocked`.
</issue_to_address>

### Comment 2
<location path="xcore/services/scheduler/service.py" line_range="254" />
<code_context>

         self._scheduler.start()
-        self._status = ServiceStatus.READY
+        self._status = ServiceStatus.DEGRADED if degraded else ServiceStatus.READY
         logger.info(
             "scheduler started",
</code_context>
<issue_to_address>
**Unavailable Redis still reports ready**

When redis packages are installed but the configured Redis service is down or unreachable, `SchedulerService.init` treats successful Redis client construction as proof that the distributed lock works, although `from_url` does not verify connectivity; it can set the service to `READY` and report `distributed_lock=True` while `_dispatch_job` later fails at `client.set` and the job does not run.

Validate Redis connectivity during initialization, mark the service degraded when it fails, and handle lock-acquisition failures explicitly so job failures are visible and do not masquerade as a healthy scheduler.

Also at `xcore/services/scheduler/service.py:259`.
</issue_to_address>

### Comment 3
<location path="xcore/configurations/sections.py" line_range="322" />
<code_context>
+    # récents. Ce n'est pas le pipeline d'export OTel : c'est un historique de débogage, et
+    # il doit être borné — une liste sans limite faisait ~562 octets par appel, soit près de
+    # 5 Go par jour et par worker à 100 req/s (#305). 0 = ne rien garder.
+    max_local_spans: int = 1000


</code_context>
<issue_to_address>
**Configured span history is ignored**

When `observability.tracing.max_local_spans` is set in application configuration, `ConfigLoader._parse_observability` does not pass `max_local_spans` into `TracingConfig`, so the default limit is used and the configured history size is ignored.

Pass the configured `max_local_spans` value into `TracingConfig` in `ConfigLoader._parse_observability`.

Also at `xcore/configurations/sections.py:323`.
</issue_to_address>

### Comment 4
<location path="xcore/kernel/observability/tracing.py" line_range="54" />
<code_context>
+    value = getattr(config, "max_local_spans", DEFAULT_MAX_LOCAL_SPANS)
+    if isinstance(value, bool) or not isinstance(value, int):
+        return DEFAULT_MAX_LOCAL_SPANS
+    return max(value, 0)
+
+
</code_context>
<issue_to_address>
**Tracer startup fails on large limits**

When `max_local_spans` exceeds the platform’s `Py_ssize_t` range, `_max_local_spans` returns the value unchanged, so `deque(maxlen=...)` raises `OverflowError` during `Tracer` construction and prevents startup.

Clamp or reject values that cannot be represented as a `Py_ssize_t` before passing them to `deque`.
</issue_to_address>

### Comment 5
<location path="xcore/sdk/__init__.py" line_range="196" />
<code_context>
+# `find_spec` RÉSOUT sans exécuter (mesuré : 0,05 ms) : `from xcore.sdk import *` ne doit
+# pas lever pour une dépendance que l'utilisateur n'a pas, et ne doit pas non plus importer
+# celery ou SQLAlchemy juste pour savoir s'ils sont là.
+__all__ += list(_WORKER)
+if importlib.util.find_spec("sqlalchemy") is not None:
+    __all__ += list(_SQL_ADAPTERS)
</code_context>
<issue_to_address>
**Wildcard imports require Celery**

When an application without the optional Celery dependency uses a wildcard import from `xcore.sdk`, `__all__` always includes the worker names, so `from xcore.sdk import *` resolves `WorkerService` and triggers the lazy Celery import. Without the optional worker dependency, the wildcard import raises `ImportError` even for callers that do not use worker functionality.

Only include worker names in `__all__` when Celery is available, or otherwise ensure wildcard import does not resolve unavailable optional names.
</issue_to_address>

### Comment 6
<location path="xcore/services/scheduler/service.py" line_range="279" />
<code_context>
+            return
+        jobs = [str(j.get("id") or j.get("func") or "?") for j in self._config.jobs]
+        if not jobs and not _JOB_REGISTRY:
+            return  # aucun job : rien à dire
+        logger.warning(
+            "scheduler sans verrou : en multi-processus, CHAQUE processus exécutera ces jobs",
</code_context>
<issue_to_address>
**Unprotected jobs go unwarned**

When the scheduler starts with no jobs and jobs are registered afterward, `SchedulerService.init` calls `_warn_if_unlocked` before plugin or caller registrations, and the empty-job return suppresses the warning permanently. The later jobs run in every process without operators being warned about duplicate executions.

Emit the unlocked-scheduler warning when jobs are registered after initialization, or otherwise defer it until those registrations are available.

Also at `xcore/services/scheduler/service.py:261`, `xcore/services/scheduler/service.py:278`.
</issue_to_address>

### Comment 7
<location path="xcore/kernel/observability/tracing.py" line_range="155" />
<code_context>
+        # tout, même tracing désactivé). Une liste sans limite gardait donc un Span par appel
+        # pour toujours — une fuite mesurée à ~562 octets par appel (#305). Un deque garde les
+        # N plus récents, ce qui est tout ce que `export()` sert à montrer.
+        self._spans: deque[Span] = deque(maxlen=_max_local_spans(config))
         self._otel_tracer = None

</code_context>
<issue_to_address>
**Concurrent exports fail**

When `Tracer.export()` runs while another thread completes a span, `Tracer.export()` iterates `self._spans` while span completion appends to the deque; the iterator raises `RuntimeError` and the caller gets no history.

Synchronize deque access so `Tracer.export()` iterates a stable snapshot of the spans.
</issue_to_address>

### Comment 8
<location path="tests/unit/security/test_scanner_fails_closed.py" line_range="42-44" />
<code_context>
+    ):
+        """The whole point: the C++ path and the Python path must agree on an unparsable file."""
+        plugin = _plugin(tmp_path, "def broken(\n  pass")
+        with_cpp = ASTScanner()
+        without_cpp = ASTScanner()
+        monkeypatch.setattr(without_cpp, "_cpp", None)
+        assert with_cpp.scan(plugin).passed is without_cpp.scan(plugin).passed is False
+
</code_context>
<issue_to_address>
**Compiled scanner path goes untested**

When the test suite runs without the optional C++ scanner extension, both `ASTScanner` instances use the Python fallback, so the assertion compares that path with itself and the compiled scanner’s fail-closed behavior is not tested.

Require the compiled scanner for this test and verify that `with_cpp` uses it before asserting its behavior.
</issue_to_address>

### Comment 9
<location path="tests/unit/sdk/test_lazy_imports.py" line_range="61-63" />
<code_context>
+        """`__all__` must not shrink: a plugin doing `from xcore.sdk import *` kept working."""
+        out = _run(
+            "import xcore.sdk as s\n"
+            "names = set(s.__all__)\n"
+            "print(all(n in names for n in ('WorkerService', 'task', 'task_registry')))"
+        )
+        assert out == "True"
+
</code_context>
<issue_to_address>
**Star import can still fail**

When an export included in `__all__` beyond the three checked worker names is unavailable or mis-mapped, the test checks only that three worker names appear in `__all__`; it never executes `from xcore.sdk import *`. When an available conditional lazy export, such as a SQL adapter, is listed but cannot be resolved, the star import raises while this assertion still passes.

Run an actual star import in a fresh interpreter and verify the expected names are bound.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 9 findings to address first, and the bounded deque permanently drops older spans from Tracer.export(); reverting restores future history collection, but cannot recover spans already evicted. That loss is limited to in-memory debugging data, while the scanner and scheduler changes alter reversible runtime behavior.

Blocking findings: xcore/services/scheduler/service.py:277, xcore/services/scheduler/service.py:254, xcore/configurations/sections.py:322, xcore/kernel/observability/tracing.py:54, xcore/sdk/__init__.py:196, and 4 more


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

"""
if _REDIS_LOCK_CLIENT is not None:
return
jobs = [str(j.get("id") or j.get("func") or "?") for j in self._config.jobs]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Malformed job config aborts boot

When services.scheduler.jobs contains a non-mapping entry, such as null, _add_job_from_config catches and logs invalid job entries, but _warn_if_unlocked then calls .get() on each entry without checking its type. A non-mapping entry raises AttributeError during scheduler initialization and aborts application boot instead of skipping the bad job.

Filter or safely normalize job entries before reading id and func in _warn_if_unlocked.

Prompt for AI agents
In `xcore/services/scheduler/service.py` at line 277:

**Malformed job config aborts boot**

When `services.scheduler.jobs` contains a non-mapping entry, such as `null`, `_add_job_from_config` catches and logs invalid job entries, but `_warn_if_unlocked` then calls `.get()` on each entry without checking its type. A non-mapping entry raises `AttributeError` during scheduler initialization and aborts application boot instead of skipping the bad job.

Filter or safely normalize job entries before reading `id` and `func` in `_warn_if_unlocked`.


self._scheduler.start()
self._status = ServiceStatus.READY
self._status = ServiceStatus.DEGRADED if degraded else ServiceStatus.READY

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 High · Unavailable Redis still reports ready

When redis packages are installed but the configured Redis service is down or unreachable, SchedulerService.init treats successful Redis client construction as proof that the distributed lock works, although from_url does not verify connectivity; it can set the service to READY and report distributed_lock=True while _dispatch_job later fails at client.set and the job does not run.

Validate Redis connectivity during initialization, mark the service degraded when it fails, and handle lock-acquisition failures explicitly so job failures are visible and do not masquerade as a healthy scheduler.

Also at xcore/services/scheduler/service.py:259.

Prompt for AI agents
In `xcore/services/scheduler/service.py` at line 254:

**Unavailable Redis still reports ready**

When redis packages are installed but the configured Redis service is down or unreachable, `SchedulerService.init` treats successful Redis client construction as proof that the distributed lock works, although `from_url` does not verify connectivity; it can set the service to `READY` and report `distributed_lock=True` while `_dispatch_job` later fails at `client.set` and the job does not run.

Validate Redis connectivity during initialization, mark the service degraded when it fails, and handle lock-acquisition failures explicitly so job failures are visible and do not masquerade as a healthy scheduler.

Also at `xcore/services/scheduler/service.py:259`.

# récents. Ce n'est pas le pipeline d'export OTel : c'est un historique de débogage, et
# il doit être borné — une liste sans limite faisait ~562 octets par appel, soit près de
# 5 Go par jour et par worker à 100 req/s (#305). 0 = ne rien garder.
max_local_spans: int = 1000

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Configured span history is ignored

When observability.tracing.max_local_spans is set in application configuration, ConfigLoader._parse_observability does not pass max_local_spans into TracingConfig, so the default limit is used and the configured history size is ignored.

Pass the configured max_local_spans value into TracingConfig in ConfigLoader._parse_observability.

Also at xcore/configurations/sections.py:323.

Prompt for AI agents
In `xcore/configurations/sections.py` at line 322:

**Configured span history is ignored**

When `observability.tracing.max_local_spans` is set in application configuration, `ConfigLoader._parse_observability` does not pass `max_local_spans` into `TracingConfig`, so the default limit is used and the configured history size is ignored.

Pass the configured `max_local_spans` value into `TracingConfig` in `ConfigLoader._parse_observability`.

Also at `xcore/configurations/sections.py:323`.

value = getattr(config, "max_local_spans", DEFAULT_MAX_LOCAL_SPANS)
if isinstance(value, bool) or not isinstance(value, int):
return DEFAULT_MAX_LOCAL_SPANS
return max(value, 0)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Tracer startup fails on large limits

When max_local_spans exceeds the platform’s Py_ssize_t range, _max_local_spans returns the value unchanged, so deque(maxlen=...) raises OverflowError during Tracer construction and prevents startup.

Clamp or reject values that cannot be represented as a Py_ssize_t before passing them to deque.

Prompt for AI agents
In `xcore/kernel/observability/tracing.py` at line 54:

**Tracer startup fails on large limits**

When `max_local_spans` exceeds the platform’s `Py_ssize_t` range, `_max_local_spans` returns the value unchanged, so `deque(maxlen=...)` raises `OverflowError` during `Tracer` construction and prevents startup.

Clamp or reject values that cannot be represented as a `Py_ssize_t` before passing them to `deque`.

Comment thread xcore/sdk/__init__.py
# `find_spec` RÉSOUT sans exécuter (mesuré : 0,05 ms) : `from xcore.sdk import *` ne doit
# pas lever pour une dépendance que l'utilisateur n'a pas, et ne doit pas non plus importer
# celery ou SQLAlchemy juste pour savoir s'ils sont là.
__all__ += list(_WORKER)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Wildcard imports require Celery

When an application without the optional Celery dependency uses a wildcard import from xcore.sdk, __all__ always includes the worker names, so from xcore.sdk import * resolves WorkerService and triggers the lazy Celery import. Without the optional worker dependency, the wildcard import raises ImportError even for callers that do not use worker functionality.

Only include worker names in __all__ when Celery is available, or otherwise ensure wildcard import does not resolve unavailable optional names.

Prompt for AI agents
In `xcore/sdk/__init__.py` at line 196:

**Wildcard imports require Celery**

When an application without the optional Celery dependency uses a wildcard import from `xcore.sdk`, `__all__` always includes the worker names, so `from xcore.sdk import *` resolves `WorkerService` and triggers the lazy Celery import. Without the optional worker dependency, the wildcard import raises `ImportError` even for callers that do not use worker functionality.

Only include worker names in `__all__` when Celery is available, or otherwise ensure wildcard import does not resolve unavailable optional names.

return
jobs = [str(j.get("id") or j.get("func") or "?") for j in self._config.jobs]
if not jobs and not _JOB_REGISTRY:
return # aucun job : rien à dire

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Unprotected jobs go unwarned

When the scheduler starts with no jobs and jobs are registered afterward, SchedulerService.init calls _warn_if_unlocked before plugin or caller registrations, and the empty-job return suppresses the warning permanently. The later jobs run in every process without operators being warned about duplicate executions.

Emit the unlocked-scheduler warning when jobs are registered after initialization, or otherwise defer it until those registrations are available.

Also at xcore/services/scheduler/service.py:261, xcore/services/scheduler/service.py:278.

Prompt for AI agents
In `xcore/services/scheduler/service.py` at line 279:

**Unprotected jobs go unwarned**

When the scheduler starts with no jobs and jobs are registered afterward, `SchedulerService.init` calls `_warn_if_unlocked` before plugin or caller registrations, and the empty-job return suppresses the warning permanently. The later jobs run in every process without operators being warned about duplicate executions.

Emit the unlocked-scheduler warning when jobs are registered after initialization, or otherwise defer it until those registrations are available.

Also at `xcore/services/scheduler/service.py:261`, `xcore/services/scheduler/service.py:278`.

# tout, même tracing désactivé). Une liste sans limite gardait donc un Span par appel
# pour toujours — une fuite mesurée à ~562 octets par appel (#305). Un deque garde les
# N plus récents, ce qui est tout ce que `export()` sert à montrer.
self._spans: deque[Span] = deque(maxlen=_max_local_spans(config))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Concurrent exports fail

When Tracer.export() runs while another thread completes a span, Tracer.export() iterates self._spans while span completion appends to the deque; the iterator raises RuntimeError and the caller gets no history.

Synchronize deque access so Tracer.export() iterates a stable snapshot of the spans.

Prompt for AI agents
In `xcore/kernel/observability/tracing.py` at line 155:

**Concurrent exports fail**

When `Tracer.export()` runs while another thread completes a span, `Tracer.export()` iterates `self._spans` while span completion appends to the deque; the iterator raises `RuntimeError` and the caller gets no history.

Synchronize deque access so `Tracer.export()` iterates a stable snapshot of the spans.

Comment on lines +42 to +44
with_cpp = ASTScanner()
without_cpp = ASTScanner()
monkeypatch.setattr(without_cpp, "_cpp", None)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Compiled scanner path goes untested

When the test suite runs without the optional C++ scanner extension, both ASTScanner instances use the Python fallback, so the assertion compares that path with itself and the compiled scanner’s fail-closed behavior is not tested.

Require the compiled scanner for this test and verify that with_cpp uses it before asserting its behavior.

Prompt for AI agents
In `tests/unit/security/test_scanner_fails_closed.py` at lines 42-44:

**Compiled scanner path goes untested**

When the test suite runs without the optional C++ scanner extension, both `ASTScanner` instances use the Python fallback, so the assertion compares that path with itself and the compiled scanner’s fail-closed behavior is not tested.

Require the compiled scanner for this test and verify that `with_cpp` uses it before asserting its behavior.

Comment on lines +61 to +63
"names = set(s.__all__)\n"
"print(all(n in names for n in ('WorkerService', 'task', 'task_registry')))"
)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Star import can still fail

When an export included in __all__ beyond the three checked worker names is unavailable or mis-mapped, the test checks only that three worker names appear in __all__; it never executes from xcore.sdk import *. When an available conditional lazy export, such as a SQL adapter, is listed but cannot be resolved, the star import raises while this assertion still passes.

Run an actual star import in a fresh interpreter and verify the expected names are bound.

Prompt for AI agents
In `tests/unit/sdk/test_lazy_imports.py` at lines 61-63:

**Star import can still fail**

When an export included in `__all__` beyond the three checked worker names is unavailable or mis-mapped, the test checks only that three worker names appear in `__all__`; it never executes `from xcore.sdk import *`. When an available conditional lazy export, such as a SQL adapter, is listed but cannot be resolved, the star import raises while this assertion still passes.

Run an actual star import in a fresh interpreter and verify the expected names are bound.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core documentation Improvements or additions to documentation tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant