Skip to content

feat: add OpenTelemetry instrumentation and improve tool lifecycle hooks #83

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ssdeanx merged 1 commit into from
Jan 9, 2026
Merged

feat: add OpenTelemetry instrumentation and improve tool lifecycle hooks #83

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions instrumentation.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { W3CTraceContextPropagator } from "@opentelemetry/core";

const sdk = new NodeSDK({
serviceName: "my-service",

@gemini-code-assist gemini-code-assist Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The serviceName is hardcoded. It's a best practice to make this configurable via an environment variable to easily identify the service in different observability environments (e.g., development, staging, production).

Suggested change
serviceName: "my-service",
serviceName: process.env.OTEL_SERVICE_NAME ?? "agentstack",

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hardcoded service name "my-service" in instrumentation.ts should be configurable. The service name should be read from an environment variable (e.g., process.env.OTEL_SERVICE_NAME) or a configuration file to allow different service names in different environments without code changes.

Suggested change
serviceName: "my-service",
serviceName: process.env.OTEL_SERVICE_NAME ?? "my-service",

Copilot uses AI. Check for mistakes.
spanProcessors: [
new BatchSpanProcessor(
new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT ?? "http://localhost:4318/v1/traces",
})
),
],
contextManager: undefined,

@gemini-code-assist gemini-code-assist Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

critical

Setting contextManager to undefined disables automatic asynchronous context propagation. This means that trace.getActiveSpan() and otelContext.active() will not work correctly across async/await calls, which will break the tracing instrumentation implemented in logger.ts and e2b.ts. To fix this, you should remove this line entirely to let the SDK use the default AsyncLocalStorageContextManager.

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The contextManager: undefined configuration at line 16 in instrumentation.ts is concerning. Explicitly setting the context manager to undefined may cause issues with context propagation in async operations. Either remove this line to use the default context manager, or explicitly configure an appropriate context manager (e.g., AsyncHooksContextManager) for proper trace context propagation.

Suggested change
contextManager: undefined,

Copilot uses AI. Check for mistakes.
autoDetectResources: true,
instrumentations: [getNodeAutoInstrumentations()],
textMapPropagator: new W3CTraceContextPropagator(),
});

sdk.start();

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The instrumentation file exports the SDK instance but never calls sdk.shutdown(). The NodeSDK should be properly shut down when the application terminates to flush pending telemetry data. Consider adding a shutdown handler or documenting that the consuming application must handle SDK lifecycle management.

Suggested change
const shutdown = () => {
sdk
.shutdown()
.catch((error) => {
console.error("Error shutting down OpenTelemetry SDK", error);
});
};
process.once("SIGINT", shutdown);
process.once("SIGTERM", shutdown);
process.once("beforeExit", shutdown);

Copilot uses AI. Check for mistakes.
export { sdk };
2,225 changes: 1,762 additions & 463 deletions package-lock.json
View file
Open in desktop

Large diffs are not rendered by default.

29 changes: 17 additions & 12 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "agentstack",
"version": "1.1.1",
"version": "1.1.2",
"description": "Multi-agent frameworks and tools for building AI applications with Mastra.",
"main": "index.js",
"scripts": {
Expand Down Expand Up @@ -46,29 +46,31 @@
"@ai-sdk/google-vertex": "^4.0.11",
"@ai-sdk/openai": "^3.0.7",
"@ai-sdk/openai-compatible": "^2.0.4",
"@ai-sdk/react": "^3.0.21",
"@ai-sdk/react": "^3.0.25",
"@dotenvx/dotenvx": "^1.51.4",
"@e2b/code-interpreter": "^2.3.3",
"@emotion/react": "^11.14.0",
"@gsap/react": "^2.1.2",
"@mastra/ai-sdk": "^1.0.0-beta.13",
"@mastra/auth-supabase": "^1.0.0-beta.1",
"@mastra/client-js": "^1.0.0-beta.20",
"@mastra/client-js": "^1.0.0-beta.21",
"@mastra/cloudflare-d1": "^1.0.0-beta.9",
"@mastra/core": "^1.0.0-beta.20",
"@mastra/core": "^1.0.0-beta.21",
"@mastra/deployer": "^1.0.0-beta.20",
"@mastra/evals": "^1.0.0-beta.4",
"@mastra/lance": "^1.0.0-beta.10",
"@mastra/libsql": "^1.0.0-beta.11",
"@mastra/loggers": "^1.0.0-beta.4",
"@mastra/mcp": "^1.0.0-beta.8",
"@mastra/memory": "^1.0.0-beta.11",
"@mastra/mcp": "^1.0.0-beta.9",
"@mastra/memory": "^1.0.0-beta.12",
"@mastra/mongodb": "^1.0.0-beta.10",
"@mastra/observability": "^1.0.0-beta.10",
"@mastra/otel-bridge": "^1.0.0-beta.10",
"@mastra/otel-exporter": "^1.0.0-beta.11",
"@mastra/pg": "^1.0.0-beta.12",
"@mastra/qdrant": "^1.0.0-beta.3",
"@mastra/rag": "^2.0.0-beta.5",
"@mastra/react": "^0.1.0-beta.20",
"@mastra/rag": "^2.0.0-beta.6",
"@mastra/react": "^0.1.0-beta.21",
"@mastra/upstash": "^1.0.0-beta.10",
"@mastra/vectorize": "^1.0.0-beta.3",
"@mastra/voice-google": "^0.12.0-beta.2",
Expand All @@ -81,6 +83,7 @@
"@octokit/plugin-retry": "^8.0.3",
"@openrouter/ai-sdk-provider": "^1.5.4",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/auto-instrumentations-node": "^0.67.3",
"@opentelemetry/exporter-trace-otlp-proto": "^0.208.0",
"@opeoginni/github-copilot-openai-compatible": "^1.0.0",
"@radix-ui/react-accordion": "^1.2.12",
Expand Down Expand Up @@ -128,21 +131,23 @@
"excalidraw-to-svg": "^3.1.0",
"fast-glob": "^3.3.3",
"fast-xml-parser": "^5.3.3",
"framer-motion": "^12.24.12",
"framer-motion": "^12.25.0",
"gray-matter": "^4.0.3",
"gsap": "^3.14.2",
"husky": "^9.1.7",
"input-otp": "^1.4.2",
"isolated-vm": "^6.0.2",
"jose": "^6.1.3",
"jsdom": "^27.4.0",
"langfuse-vercel": "^3.38.6",
"lenis": "^1.3.17",
"lucide-react": "^0.562.0",
"marked": "^17.0.1",
"monaco-editor": "^0.55.1",
"monaco-editor-webpack-plugin": "^7.1.1",
"monaco-languages": "^2.11.1",
"monaco-themes": "^0.4.8",
"motion": "^12.24.12",
"motion": "^12.25.0",
"nanoid": "^5.1.6",
"next": "^16.1.1",
"next-themes": "^0.4.6",
Expand All @@ -154,7 +159,7 @@
"re2": "^1.23.0",
"react": "^19.2.3",
"react-dom": "^19.2.3",
"react-resizable-panels": "^4.3.0",
"react-resizable-panels": "^4.3.3",
"recharts": "^2.15.4",
"rehype": "^13.0.2",
"rehype-class-names": "^2.0.0",
Expand Down Expand Up @@ -184,7 +189,7 @@
"v0-sdk": "^0.15.3",
"vscode-languageserver": "^10.0.0-next.16",
"vscode-languageserver-textdocument": "^1.0.12",
"zod": "^4.1.13"
"zod": "^4.2.1"
},
"devDependencies": {
"@mdx-js/loader": "^3.1.1",
Expand Down
96 changes: 86 additions & 10 deletions src/mastra/config/logger.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
import { PinoLogger } from '@mastra/loggers'
import { PinoLogger } from "@mastra/loggers";
import { FileTransport } from "@mastra/loggers/file";

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

'FileTransport' is defined but never used.

Suggested change
import { FileTransport } from "@mastra/loggers/file";

Copilot uses AI. Check for mistakes.
import * as fs from 'node:fs'
import * as path from 'node:path'
import { trace, propagation, context as otelContext } from '@opentelemetry/api'

// Use __dirname directly for CommonJS
//const __dirname: string = path.resolve(path.dirname(''));
Expand All @@ -11,10 +13,60 @@ if (!fs.existsSync(logsDir)) {
fs.mkdirSync(logsDir, { recursive: true })
}

// OpenTelemetry helpers to enrich logs with trace context and Baggage
const getActiveSpanContextInfo = () => {
const span = trace.getActiveSpan()
if (!span) {
return { traceId: undefined, spanId: undefined }
}
const sc = (span as any).spanContext ? (span as any).spanContext() : undefined

@gemini-code-assist gemini-code-assist Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The cast (span as any) is unnecessary and bypasses type safety. The spanContext() method is a standard part of the Span interface from @opentelemetry/api. You can call it directly on the span object.

Suggested change
const sc = (span as any).spanContext ? (span as any).spanContext() : undefined
const sc = span.spanContext();

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected any. Specify a different type.

Copilot uses AI. Check for mistakes.
return { traceId: sc?.traceId, spanId: sc?.spanId }
Comment on lines +22 to +23

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unexpected any. Specify a different type.

Suggested change
const sc = (span as any).spanContext ? (span as any).spanContext() : undefined
return { traceId: sc?.traceId, spanId: sc?.spanId }
const sc = span.spanContext()
return { traceId: sc.traceId, spanId: sc.spanId }

Copilot uses AI. Check for mistakes.
}

const BaggageToObject = (b: unknown) => {
if (b === null || b === undefined) {
return {}
}
const obj: Record<string, string> = {}
for (const [k, v] of (b as any).getAllEntries()) {
Comment on lines +30 to +31

Copilot AI Jan 9, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The BaggageToObject helper in logger.ts uses (b as any) type assertion without proper type guards. At line 31, the code casts an unknown type to any and calls .getAllEntries() without verifying that the object has this method. This can cause runtime errors if baggage is null, undefined, or an unexpected type. The function should include proper type checking before accessing methods.

Suggested change
const obj: Record<string, string> = {}
for (const [k, v] of (b as any).getAllEntries()) {
// Ensure the value has a callable getAllEntries method before using it
const maybeHasEntries = b as { getAllEntries?: unknown }
if (typeof maybeHasEntries.getAllEntries !== 'function') {
return {}
}
const baggageWithEntries = maybeHasEntries as {
getAllEntries: () => Iterable<[string, { value: unknown }]>
}
const obj: Record<string, string> = {}
for (const [k, v] of baggageWithEntries.getAllEntries()) {

Copilot uses AI. Check for mistakes.
obj[k] = String(v.value)
}
return obj
}
Comment on lines +26 to +35

@gemini-code-assist gemini-code-assist Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The BaggageToObject function takes b as unknown and then casts it to any. This can be made more type-safe by typing b as Baggage | undefined | null and avoiding the any cast, as getAllEntries() is a standard method on the Baggage interface.

Suggested change
const BaggageToObject = (b: unknown) => {
if (b === null || b === undefined) {
return {}
}
const obj: Record<string, string> = {}
for (const [k, v] of (b as any).getAllEntries()) {
obj[k] = String(v.value)
}
return obj
}
const BaggageToObject = (b: Baggage | undefined | null) => {
if (!b) {
return {}
}
const obj: Record<string, string> = {}
for (const [k, v] of b.getAllEntries()) {
obj[k] = String(v.value)
}
return obj
}


const attachTraceAndBaggage = (data: Record<string, unknown> = {}) => {
try {
const traceInfo = getActiveSpanContextInfo()
const bag = propagation.getBaggage(otelContext.active())
return { ...data, _trace: traceInfo, _Baggage: BaggageToObject(bag) }
} catch {
// Don't throw from logging helpers
return data
}
}

// Enhanced PinoLogger with full tracing integration
export const log = new PinoLogger({
name: 'MastraLogger',
level: 'debug',
// formatters: {level: (label: string) => ({ level: label.toUpperCase() }), bindings: (bindings: object) => bindings as Record<string, unknown>, log: (logObj: object) => logObj as Record<string, unknown>},
// Enable pretty printing in development
...(process.env.NODE_ENV === 'development' && {
transport: {
target: 'pino-pretty',
options: {
colorize: true,
translateTime: 'SYS:standard',
ignore: 'pid,hostname',
},
},
}),
formatters: {
// Add custom formatter to include trace and baggage info
// log(object) {
// return attachTraceAndBaggage(object)
// },
Comment on lines +65 to +67

@gemini-code-assist gemini-code-assist Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The log formatter, which is responsible for calling attachTraceAndBaggage to enrich all logs with trace context, is commented out. This means that no logs will be automatically correlated with traces, which undermines a key benefit of this instrumentation. Please uncomment this block to enable automatic log enrichment.

Suggested change
// log(object) {
// return attachTraceAndBaggage(object)
// },
log(object) {
return attachTraceAndBaggage(object)
},

},

})

// Create a simple file logger wrapper
Expand All @@ -35,15 +87,18 @@ export const logWorkflowStart = (
input: Record<string, unknown>
) => {
const message = `🚀 Starting workflow: ${workflowId}`
const data: {
let data: {
workflowId: string
input: Record<string, unknown>
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
workflowId,
input,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
Comment on lines +90 to +101

@gemini-code-assist gemini-code-assist Bot Jan 9, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

There is significant code duplication across the logWorkflowStart, logWorkflowEnd, logStepStart, and other logging functions. Each one re-defines a data type and manually calls attachTraceAndBaggage. If you enable the Pino log formatter (as suggested in another comment), it will handle this automatically for all logs, and you can remove the manual enrichment from each of these functions, greatly simplifying the code.

log.info(message, data)
logToFile(message, data)
}
Expand All @@ -54,17 +109,20 @@ export const logWorkflowEnd = (
duration: number
) => {
const message = `✅ Workflow completed: ${workflowId}`
const data: {
let data: {
workflowId: string
output: Record<string, unknown>
duration: string
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
workflowId,
output,
duration: `${duration}ms`,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.info(message, data)
logToFile(message, data)
}
Expand All @@ -74,15 +132,18 @@ export const logStepStart = (
input: Record<string, unknown>
) => {
const message = `📋 Starting step: ${stepId}`
const data: {
let data: {
stepId: string
input: Record<string, unknown>
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
stepId,
input,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.info(message, data)
logToFile(message, data)
}
Expand All @@ -93,17 +154,20 @@ export const logStepEnd = (
duration: number
) => {
const message = `✓ Step completed: ${stepId}`
const data: {
let data: {
stepId: string
output: Record<string, unknown>
duration: string
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
stepId,
output,
duration: `${duration}ms`,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.info(message, data)
logToFile(message, data)
}
Expand All @@ -114,17 +178,20 @@ export const logToolExecution = (
output?: Record<string, unknown>
) => {
const message = `🔧 Tool execution: ${toolId}`
const data: {
let data: {
toolId: string
input: Record<string, unknown>
output?: Record<string, unknown>
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
toolId,
input,
output,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.info(message, data)
logToFile(message, data)
}
Expand All @@ -135,17 +202,20 @@ export const logAgentActivity = (
details: Record<string, unknown>
) => {
const message = `🤖 Agent activity: ${agentId} - ${action}`
const data: {
let data: {
agentId: string
action: string
details: Record<string, unknown>
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
agentId,
action,
details,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.info(message, data)
logToFile(message, data)
}
Expand All @@ -156,19 +226,22 @@ export const logError = (
context?: Record<string, unknown>
) => {
const message = `❌ Error in ${component}`
const data: {
let data: {
component: string
error: string
stack?: string
context?: Record<string, unknown>
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
component,
error: error instanceof Error ? error.message : String(error),
stack: error instanceof Error ? error.stack : undefined,
context,
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.error(message, data)
logToFile(message, data)
}
Expand All @@ -179,19 +252,22 @@ export const logProgress = (
total: number
) => {
const logMessage = `📊 Progress: ${message} (${progress}/${total})`
const data: {
let data: {
message: string
progress: number
total: number
percentage: number
timestamp: string
_trace?: Record<string, unknown>
_Baggage?: Record<string, string>
} = {
message,
progress,
total,
percentage: Math.round((progress / total) * 100),
timestamp: new Date().toISOString(),
}
data = attachTraceAndBaggage(data) as typeof data
log.info(logMessage, data)
logToFile(logMessage, data)
}
Loading
Loading