Distributed tracing with OpenTelemetry

[greatest0fallt1me]

Description

• Export to OTLP; propagate context through jobs and RPC clients.

Requirements and context

• Must be secure, tested, and documented for production operation.
• Should be efficient, observable, and easy to review in PRs.
• Scope is Predictify backend / off-chain services only (no Soroban contract changes, no frontend UI in this issue).

Suggested execution

• Fork the repository and create a branch:
  • git checkout -b feature/otel-tracing
• Implement changes:
  1. Service / module: implement in src/telemetry/otel.rs and related packages as needed.
  2. Tests: add or extend trace propagation tests (unit + integration where applicable).
  3. Documentation: update or add docs/backend/OBSERVABILITY.md (architecture notes, OpenAPI, or runbooks).
  4. Comments & types: document public APIs, config knobs, and failure modes clearly.

Primary touchpoints: src/telemetry/otel.rs

Test and commit

• Run the project test command (e.g. pytest, cargo test, or npm test — follow repo conventions).
• Cover edge cases listed in the description; add regression tests for any bug found.
• In the PR, include summarized test output and security / ops notes (authn/z, data handling, rate limits).

Example commit message

feat(obs): opentelemetry tracing

Guidelines

• Target ≥ 95% coverage on new or materially changed modules (per language/tooling configured in CI).
• Documentation must allow a new engineer to operate and verify the feature locally.
• Timeframe: 96 hours from assignment.

[System625]

@System625 has applied to work on this issue as part of the Stellar Wave Program's 4th wave.

Hello there, I would like to work on this issue.
I will implement OpenTelemetry tracing in src/telemetry/otel.rs, focusing on OTLP export and context propagation across jobs and RPC clients. I’ll ensure 95%+ test coverage, update the observability docs, and deliver a production-ready setup within the 96-hour timeframe, strictly following the backend monorepo standards.
ETA: 24 hours

ℹ️ Repo Maintainers: To accept this application, review their application or assign @System625 to this issue.