Skip to content

Commit c4caa99

Browse files
authored
fix(telemetry): emit error events at ERROR severity and declutter slash menu (#129)
* docs: add CLAUDE.md importing AGENTS.md and AGENTS.local Claude Code does not read AGENTS.md automatically. Add a CLAUDE.md that imports the tracked repo rules (AGENTS.md) and the machine-local overlay (AGENTS.local) so Claude Code sessions get the same guidance Pythinker injects via PYTHINKER_AGENTS_MD. * fix(telemetry): emit error events at ERROR severity with canonical attributes track("error"/"crash"/"api_error") forwarded every event to OTel logs at the emit_log() default of INFO, so error/crash telemetry was indistinguishable from product analytics — SigNoz severity filters and the error saved views found nothing, and the views' error_type filter never matched (call sites flatten it to property.error_type / property.exc_class). - Map known error event names to severity in EventSink: error/crash/api_error -> ERROR, session_load_failed -> WARN; everything else stays INFO. - Add canonical error.type/error.site/error.expected/error.kind attributes for error-like events so dashboards query one set of keys regardless of whether the call site emitted error_type or exc_class. Original property.* preserved. - Factor OTel emission into emit_events_to_otel() and have flush_sync() drain the pre-sink _event_queue, so a startup crash before attach_sink() still reaches SigNoz (Bugsink already captured it). - Docs: correct the stale claim that report_handled_error always calls Sentry; expected errors are withheld from Bugsink. Tests cover severity mapping, canonical attributes, and crash-safe flush. * feat(ui/shell): declutter and space out the slash command menu Add a blank gap line between the input row and the slash command popup, drop the redundant [command]/[shell] tag (keeping the distinguishing [skill]/[flow] tags), and add a persistent footer legend set off by its own separator line. When the list overflows, the footer folds in a '+N more' count instead of silently hiding entries; the menu height adapts to the terminal and is capped to leave room for the chrome rows. * fix(telemetry): surface swallowed errors in crash-safe flush Address CodeRabbit review feedback on PR #129: - flush_sync(): replace the blanket `suppress(Exception)` around the pre-sink queue drain with an explicit try/except that logs at debug and only clears `_event_queue` after a successful emit, so a failed hand-off no longer silently drops startup-crash telemetry. - emit_events_to_otel(): log the OTel import failure (with event count) before dropping events instead of returning silently. - test_slash_completer: build the selection marker via chr(0x276F) instead of the literal glyph to satisfy the ambiguous-character lint. - Add a test asserting the queue is retained when the crash-safe emit raises.
1 parent 58c710c commit c4caa99

10 files changed

Lines changed: 490 additions & 97 deletions

File tree

CHANGELOG.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ GitHub Releases page; `0.8.0` is the new starting line.
2424
- **Login selector polish.** Configured `/login` providers render with distinct success/state styling; the background working indicator uses the braille spinner, and working tips wrap with a hanging indent under the verb.
2525
- **Scratch cleanup on exit.** Sessions that end via an exception now clean up their scratch files instead of orphaning them.
2626
- **Readable diff context.** Unchanged context lines in file-edit diff snippets now render in the normal body-text color instead of muted grey, so edited-file previews are easier to read; added/removed lines are unchanged.
27+
- **Cleaner slash command menu.** The slash command popup now has a blank line separating it from the input row, drops the repetitive `[command]`/`[shell]` tag (keeping the distinguishing `[skill]`/`[flow]` ones), and gains a persistent footer (`Enter to select · ↑/↓ to navigate · Esc to cancel`) set off by its own separator line. When the list scrolls, the footer folds in a `+N more` count instead of silently hiding entries, and the menu height adapts to the terminal.
2728

2829
## 0.42.0 (2026-06-12)
2930

CLAUDE.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# CLAUDE.md
2+
3+
This repository's agent guidance lives in `AGENTS.md` (the portable, tracked standard injected
4+
into Pythinker sessions via `PYTHINKER_AGENTS_MD`). Claude Code does not read `AGENTS.md`
5+
automatically, so this file imports it — plus the machine-local overlay — to keep a single source
6+
of truth.
7+
8+
Read both, in order:
9+
10+
1. **`AGENTS.md`** — non-negotiable repository rules. Always applies.
11+
2. **`AGENTS.local`** — machine-specific / private local instructions (gitignored). Read it after
12+
`AGENTS.md`. It may add workflow detail (e.g. the code-graph / graphify workflow) but must not
13+
weaken or override the rules in `AGENTS.md`.
14+
15+
@AGENTS.md
16+
17+
@AGENTS.local

docs/en/reference/telemetry.md

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -63,8 +63,14 @@ warning), and currently has no monitoring visibility.
6363

6464
It:
6565

66-
1. Emits an OTel `error` event with `{site, exc_class, tool, **attrs}`.
67-
2. Calls `sentry.capture_exception(exc)`.
66+
1. Emits an OTel `error` event with `{site, exc_class, expected, tool, **attrs}`.
67+
2. Calls `sentry.capture_exception(exc)` **only when the error is not expected**.
68+
Expected user-environment failures — bad/expired credentials, exhausted
69+
quotas, rate limits, request timeouts, offline network, abandoned OAuth
70+
flows, MCP servers lacking an optional capability (see
71+
`errors.is_expected_error`) — still flow to the OTel `error` stream with
72+
`expected=True`, but are withheld from Sentry/Bugsink, which is reserved for
73+
actionable defects.
6874

6975
Both calls are wrapped in `contextlib.suppress(Exception)` so monitoring can
7076
never break the host program.

src/pythinker_code/telemetry/__init__.py

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -202,6 +202,26 @@ def flush_sync() -> None:
202202
"""
203203
if _sink is not None:
204204
_sink.flush_sync()
205+
elif _event_queue:
206+
# No sink was ever attached — e.g. a crash during startup, before
207+
# attach_sink() runs. Best-effort direct emit so the crash/error event
208+
# still reaches SigNoz (otel.emit_log no-ops if OTel was never inited).
209+
try:
210+
from pythinker_code.telemetry.sink import emit_events_to_otel
211+
212+
for event in _event_queue:
213+
if event.get("device_id") is None:
214+
event["device_id"] = _device_id
215+
if event.get("session_id") is None:
216+
event["session_id"] = _session_id
217+
emit_events_to_otel(list(_event_queue))
218+
# Only drop the buffer once the events have been handed off, so a
219+
# failed emit leaves them intact for the vendor-SDK flush below.
220+
_event_queue.clear()
221+
except Exception as exc:
222+
from pythinker_code.utils.logging import logger
223+
224+
logger.debug("Crash-safe telemetry flush failed: {err}", err=exc)
205225
# Flush vendor SDKs last — they take the network hit.
206226
try:
207227
from pythinker_code.telemetry import otel as _otel

src/pythinker_code/telemetry/sink.py

Lines changed: 88 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,93 @@ def _flatten_event(event: dict[str, Any]) -> dict[str, Any]:
4949
return out
5050

5151

52+
# Event names whose telemetry must be recorded at ERROR severity, so SigNoz
53+
# severity filters and error dashboards can find them. Without this, track()
54+
# forwards everything at the emit_log() default of INFO, leaving crashes and
55+
# handled errors indistinguishable from product-analytics events.
56+
_ERROR_EVENTS = frozenset({"error", "crash", "api_error"})
57+
58+
# Telemetry event name -> OTel severity. Names not listed stay INFO.
59+
_EVENT_SEVERITY: dict[str, str] = {
60+
**dict.fromkeys(_ERROR_EVENTS, "error"),
61+
# A session that failed to load but fell back to a fresh state is degraded,
62+
# not broken — surface it above INFO without crying ERROR.
63+
"session_load_failed": "warning",
64+
}
65+
66+
67+
def _event_severity(event_name: str) -> str:
68+
"""Map a telemetry event name to an OTel severity (defaults to ``info``)."""
69+
return _EVENT_SEVERITY.get(event_name, "info")
70+
71+
72+
def _apply_canonical_error_attrs(event_name: str, attrs: dict[str, Any]) -> None:
73+
"""Add stable, queryable ``error.*`` attributes for error-like events.
74+
75+
Call sites are inconsistent: crashes and API errors carry ``error_type``
76+
while handled errors carry ``exc_class`` — which, after flattening, become
77+
``property.error_type`` / ``property.exc_class``. Dashboards shouldn't have
78+
to know which. Mirror the discriminator into canonical top-level keys while
79+
leaving the original ``property.*`` values untouched. Mutates ``attrs``.
80+
"""
81+
if event_name not in _ERROR_EVENTS:
82+
return
83+
error_type = attrs.get("property.error_type") or attrs.get("property.exc_class")
84+
if error_type is not None:
85+
attrs.setdefault("error.type", error_type)
86+
site = attrs.get("property.site")
87+
if site is not None:
88+
attrs.setdefault("error.site", site)
89+
if "property.expected" in attrs:
90+
attrs.setdefault("error.expected", attrs["property.expected"])
91+
# 'error' (handled) vs 'crash' (uncaught) vs 'api_error' (provider call).
92+
attrs.setdefault("error.kind", event_name)
93+
94+
95+
def emit_events_to_otel(events: list[dict[str, Any]]) -> None:
96+
"""Forward telemetry events to the OTel logs pipeline.
97+
98+
Shared by :meth:`EventSink._emit_to_otel` and the crash-safe queue drain in
99+
:func:`pythinker_code.telemetry.flush_sync`, so a startup crash that occurs
100+
before any sink is attached still reaches SigNoz. ``otel.emit_log`` is a
101+
no-op when the SDK was never initialized, so this is always safe to call.
102+
"""
103+
if not events:
104+
return
105+
try:
106+
from pythinker_code.telemetry import otel as _otel
107+
except Exception as exc:
108+
logger.debug(
109+
"Telemetry OTel import failed; dropping {n} events: {err}",
110+
n=len(events),
111+
err=exc,
112+
)
113+
return
114+
115+
for event in events:
116+
event_name = str(event.get("event") or "event")
117+
ts = event.get("timestamp")
118+
ts_ns = int(ts * 1_000_000_000) if isinstance(ts, (int, float)) else None
119+
try:
120+
attrs = _flatten_event(event)
121+
except TypeError as exc:
122+
# Schema violation — drop, never retry.
123+
logger.debug("Telemetry event dropped (non-primitive attr): {err}", err=exc)
124+
continue
125+
attrs.pop("event", None)
126+
attrs.pop("timestamp", None)
127+
_apply_canonical_error_attrs(event_name, attrs)
128+
try:
129+
_otel.emit_log(
130+
name=event_name,
131+
attributes=attrs,
132+
severity=_event_severity(event_name),
133+
timestamp_ns=ts_ns,
134+
)
135+
except Exception:
136+
logger.debug("OTel emit failed; event dropped")
137+
138+
52139
class EventSink:
53140
"""Buffers telemetry events and flushes them in batches to OTel logs."""
54141

@@ -155,29 +242,7 @@ async def _flush_async(self) -> None:
155242
self._emit_to_otel(events)
156243

157244
def _emit_to_otel(self, events: list[dict[str, Any]]) -> None:
158-
if not events:
159-
return
160-
try:
161-
from pythinker_code.telemetry import otel as _otel
162-
163-
for event in events:
164-
ts = event.get("timestamp")
165-
ts_ns = int(ts * 1_000_000_000) if isinstance(ts, (int, float)) else None
166-
try:
167-
attrs = _flatten_event(event)
168-
except TypeError as exc:
169-
# Schema violation — drop, never retry.
170-
logger.debug("Telemetry event dropped (non-primitive attr): {err}", err=exc)
171-
continue
172-
attrs.pop("event", None)
173-
attrs.pop("timestamp", None)
174-
_otel.emit_log(
175-
name=str(event.get("event") or "event"),
176-
attributes=attrs,
177-
timestamp_ns=ts_ns,
178-
)
179-
except Exception:
180-
logger.debug("OTel flush failed; events dropped")
245+
emit_events_to_otel(events)
181246

182247
def _schedule_async_flush(self) -> None:
183248
"""Schedule an async flush from any thread."""

0 commit comments

Comments
 (0)