Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions deploy/helm/nvca-operator/nvca-operator/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,21 @@ agentConfig:
# mergeConfig: |
# agent:
# logLevel: debug
#
# BYOO metric subset example:
#
# mergeConfig: |
# agent:
# byooMetricSubset:
# enabled: true
# filterConfig: |
# error_mode: ignore
# metric_conditions:
# - 'metric.name != "BpsInstrument"'
# byooWorkloadMetrics:
# dropLabels:
# - metric_subset_enabled
# - custom_label
## @section resources Resource requests and limits for the nvca-operator container
## @param resources.limits.cpu CPU limit for the nvca-operator container
## @param resources.limits.memory Memory limit for the nvca-operator container
Expand Down
18 changes: 18 additions & 0 deletions docs/ngc-managed/cluster-management/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -733,6 +733,24 @@ pod and merged with the generated config at runtime.
logLevel: debug
```

BYOO metric subset example:

```yaml
agentConfig:
mergeConfig: |
agent:
byooMetricSubset:
enabled: true
filterConfig: |
error_mode: ignore
metric_conditions:
- 'metric.name != "BpsInstrument"'
byooWorkloadMetrics:
dropLabels:
- metric_subset_enabled
- custom_label
```

**Apply via Helm:**

```bash
Expand Down
18 changes: 18 additions & 0 deletions docs/user/cluster-management/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -854,6 +854,24 @@ isolated test clusters that run the LLM request router tunnel without TLS. For
the full LLM addon setup, see
[LLM Function Enablement](../llm-function-enablement.md).

BYOO metric subset example:

```yaml
agentConfig:
mergeConfig: |
agent:
byooMetricSubset:
enabled: true
filterConfig: |
error_mode: ignore
metric_conditions:
- 'metric.name != "BpsInstrument"'
byooWorkloadMetrics:
dropLabels:
- metric_subset_enabled
- custom_label
```

Apply via Helm:

```bash
Expand Down
8 changes: 4 additions & 4 deletions src/compute-plane-services/byoo-otel-collector/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ Exposed Ports:
- 14358: OTLP HTTP receiver
- 13133: `/health?verbose` endpoint to get detailed health status of collector (healthcheck v2 extension)
- 19090: `/metrics` endpoint for the byoo-otel-collector metrics
- 19091: `/metrics` endpoint for optional SRE user metrics
- 19091: `/metrics` endpoint for optional metric subset user metrics

### nvcf-otel-collector Image

Expand All @@ -129,9 +129,9 @@ Chunking is disabled by default. Configure it with:
- `BYOO_LOG_CHUNK_MAX_BODY_BYTES`: maximum log body size in bytes before chunking. `0` disables the processor. Enabled values must be at least `4` bytes so chunks can preserve UTF-8 rune boundaries. Use `983040` bytes for normal BYOO deployments to leave room for log attributes and exporter envelope overhead under a `1000000` byte backend entry limit.
- `BYOO_LOG_CHUNK_DRY_RUN`: records oversized-log metrics and warnings without mutating log payloads. Dry-run metric datapoints use `mode=dry_run`.
- `BYOO_LOG_EXPORTER_BATCH_MAX_SIZE_BYTES`: serialized log export request batch size used for exporterhelper byte splitting. `0` or unset uses the default `1000000` bytes.
- `BYOO_SRE_METRICS_ENABLED`: enables an additional OTLP-only metrics pipeline that exposes filtered user metrics through a Prometheus exporter on port `19091`. Disabled by default.
- `BYOO_SRE_METRICS_FILTER_CONFIG`: optional YAML filter processor config for the SRE metrics pipeline. If unset, the default drops every metric except `BpsInstrument`, `FpsInstrument`, `RtdInstrument`, and `StageOpenDuration`, and drops datapoints/resources explicitly labeled `sre_metrics_enabled=false` or `sre_enabled_metrics=false`.
- `BYOO_CUSTOMER_METRICS_DROP_LABELS`: comma-separated resource attribute names removed from the customer metrics pipeline when SRE metrics are enabled. If unset, defaults to `sre_metrics_enabled,sre_enabled_metrics`.
- `BYOO_METRIC_SUBSET_ENABLED`: enables an additional OTLP-only metrics pipeline that exposes filtered user metrics through a Prometheus exporter on port `19091`. Disabled by default.
- `BYOO_METRIC_SUBSET_FILTER_CONFIG`: optional YAML filter processor config for the metric subset pipeline. If unset, the default drops every metric except `BpsInstrument`, `FpsInstrument`, `RtdInstrument`, and `StageOpenDuration`, and drops datapoints/resources explicitly labeled `metric_subset_enabled=false`.
- `BYOO_WORKLOAD_METRICS_DROP_LABELS`: comma-separated resource attribute names removed from the generated workload `metrics` pipeline. If unset, defaults to `metric_subset_enabled` only when the metric subset pipeline is enabled.

When chunking is enabled, each emitted chunk preserves the original log metadata and adds these attributes so chunks can be grouped in the backend:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,11 @@ exporters:
resource_to_telemetry_conversion:
enabled: true
send_timestamps: true
prometheusremotewrite/PROMETHEUS-customer-metrics-metrics:
endpoint: https://customer-metrics.example.invalid/api/v1/write
prometheusremotewrite/PROMETHEUS-workload-metrics-metrics:
endpoint: https://workload-metrics.example.invalid/api/v1/write
tls:
cert_file: /etc/byoo-otel-collector/secrets/customer-metrics-clientCert
key_file: /etc/byoo-otel-collector/secrets/customer-metrics-clientKey
cert_file: /etc/byoo-otel-collector/secrets/workload-metrics-clientCert
key_file: /etc/byoo-otel-collector/secrets/workload-metrics-clientKey
processors:
attributes/add-metadata:
actions:
Expand Down Expand Up @@ -149,10 +149,16 @@ processors:
send_batch_max_size: 8192
send_batch_size: 4096
timeout: 400ms
batch/sre_metrics:
batch/metric_subset:
send_batch_max_size: 8192
send_batch_size: 4096
timeout: 400ms
filter/metric_subset:
error_mode: ignore
metric_conditions:
- metric.name != "BpsInstrument" and metric.name != "FpsInstrument" and metric.name != "RtdInstrument" and metric.name != "StageOpenDuration"
- resource.attributes["metric_subset_enabled"] == "false"
- datapoint.attributes["metric_subset_enabled"] == "false"
filter/metrics:
metrics:
exclude:
Expand All @@ -162,14 +168,6 @@ processors:
- scrape_samples_post_metric_relabeling
- scrape_samples_scraped
- scrape_series_added
filter/sre_metrics:
error_mode: ignore
metric_conditions:
- metric.name != "BpsInstrument" and metric.name != "FpsInstrument" and metric.name != "RtdInstrument" and metric.name != "StageOpenDuration"
- resource.attributes["sre_metrics_enabled"] == "false"
- resource.attributes["sre_enabled_metrics"] == "false"
- datapoint.attributes["sre_metrics_enabled"] == "false"
- datapoint.attributes["sre_enabled_metrics"] == "false"
memory_limiter:
check_interval: 1s
limit_percentage: 80
Expand Down Expand Up @@ -208,12 +206,10 @@ processors:
attributes:
- action: delete
key: service.instance.id
resource/customer_metrics:
resource/workload_metrics_drop_labels:
attributes:
- action: delete
key: sre_metrics_enabled
- action: delete
key: sre_enabled_metrics
key: metric_subset_enabled
extensions:
cgroup_runtime:
gomaxprocs:
Expand Down Expand Up @@ -274,25 +270,25 @@ service:
- otlp
- prometheus
exporters:
- prometheusremotewrite/PROMETHEUS-customer-metrics-metrics
- prometheusremotewrite/PROMETHEUS-workload-metrics-metrics
processors:
- memory_limiter
- filter/metrics
- resource
- resource/customer_metrics
- resource/workload_metrics_drop_labels
- metrics_transform
- batch
metrics/sre:
metrics/metric_subset:
receivers:
- otlp
exporters:
- prometheus/user-metrics
processors:
- memory_limiter
- filter/sre_metrics
- filter/metric_subset
- resource
- metrics_transform
- batch/sre_metrics
- batch/metric_subset
extensions:
- healthcheckv2
- cgroup_runtime
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,8 @@ type TemplateConfig struct {
InstanceID string
ZoneName string
LogChunking LogChunkingConfig
SREMetrics SREMetricsConfig
MetricSubset MetricSubsetConfig
WorkloadMetrics WorkloadMetricsConfig
// LogExporterBatchMaxSizeBytes configures exporterhelper byte batching for logs.
// Zero uses the default selected for BYOO.
LogExporterBatchMaxSizeBytes int
Expand All @@ -58,10 +59,13 @@ type LogChunkingConfig struct {
DryRun bool
}

type SREMetricsConfig struct {
Enabled bool
FilterConfig map[string]interface{}
CustomerMetricsDropLabels []string
type WorkloadMetricsConfig struct {
DropLabels []string
}

type MetricSubsetConfig struct {
Enabled bool
FilterConfig map[string]interface{}
}

func ExecuteTemplate(w io.Writer, tcfg TemplateConfig) error {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,9 +42,9 @@ type envConfig struct {
ByooLogChunkMaxBodyBytes int `split_words:"true"`
ByooLogChunkDryRun bool `split_words:"true"`
ByooLogExporterBatchMaxSizeBytes int `split_words:"true"`
ByooSreMetricsEnabled bool `split_words:"true"`
ByooSreMetricsFilterConfig string `split_words:"true"`
ByooCustomerMetricsDropLabels string `split_words:"true"`
ByooMetricSubsetEnabled bool `split_words:"true"`
ByooMetricSubsetFilterConfig string `split_words:"true"`
ByooWorkloadMetricsDropLabels string `split_words:"true"`
}

func processEnvConfig(env *envConfig) error {
Expand Down Expand Up @@ -79,15 +79,15 @@ func getTemplateConfig() (TemplateConfig, error) {
return TemplateConfig{}, fmt.Errorf("BYOO_LOG_EXPORTER_BATCH_MAX_SIZE_BYTES: %w", err)
}
tcgf.LogExporterBatchMaxSizeBytes = logExporterBatchMaxSizeBytes
sreMetricsFilterConfig, err := resolvedSREMetricsFilterConfig(env.ByooSreMetricsFilterConfig)
metricSubsetFilterConfig, err := resolvedMetricSubsetFilterConfig(env.ByooMetricSubsetFilterConfig)
if err != nil {
return TemplateConfig{}, fmt.Errorf("BYOO_SRE_METRICS_FILTER_CONFIG: %w", err)
return TemplateConfig{}, fmt.Errorf("BYOO_METRIC_SUBSET_FILTER_CONFIG: %w", err)
}
tcgf.SREMetrics = SREMetricsConfig{
Enabled: env.ByooSreMetricsEnabled,
FilterConfig: sreMetricsFilterConfig,
CustomerMetricsDropLabels: resolvedCustomerMetricsDropLabels(env.ByooCustomerMetricsDropLabels),
tcgf.MetricSubset = MetricSubsetConfig{
Enabled: env.ByooMetricSubsetEnabled,
FilterConfig: metricSubsetFilterConfig,
}
tcgf.WorkloadMetrics.DropLabels = resolvedWorkloadMetricsDropLabels(env.ByooWorkloadMetricsDropLabels, env.ByooMetricSubsetEnabled)

functionID := env.NvcfFunctionID
functionVersionID := env.NvcfFunctionVersionID
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,40 +117,40 @@ func TestGetTemplateConfig(t *testing.T) {
},
},
{
name: "custom SRE metrics config",
name: "custom metric subset config",
env: map[string]string{
"NVCF_BACKEND_TYPE": "gfn",
"NVCF_INSTANCE_ID": "test-instance",
"NVCF_NAMESPACE": "test-ns",
"NVCF_WORKLOAD_TYPE": "function",
"NVCT_TASK_ID": "task-123",
"NVCF_ZONE_NAME": "zone-1",
"BYOO_SRE_METRICS_ENABLED": "true",
"BYOO_SRE_METRICS_FILTER_CONFIG": "error_mode: ignore\nmetric_conditions:\n - 'metric.name == \"drop\"'\n",
"BYOO_CUSTOMER_METRICS_DROP_LABELS": "sre_metrics_enabled, custom_label, sre_metrics_enabled",
"BYOO_METRIC_SUBSET_ENABLED": "true",
"BYOO_METRIC_SUBSET_FILTER_CONFIG": "error_mode: ignore\nmetric_conditions:\n - 'metric.name == \"drop\"'\n",
"BYOO_WORKLOAD_METRICS_DROP_LABELS": "metric_subset_enabled, custom_label, metric_subset_enabled",
},
expectErr: false,
expect: func(t *testing.T, cfg TemplateConfig) {
assert.True(t, cfg.SREMetrics.Enabled)
assert.True(t, cfg.MetricSubset.Enabled)
assert.Equal(t, map[string]interface{}{
"error_mode": "ignore",
"metric_conditions": []interface{}{
`metric.name == "drop"`,
},
}, cfg.SREMetrics.FilterConfig)
assert.Equal(t, []string{"sre_metrics_enabled", "custom_label"}, cfg.SREMetrics.CustomerMetricsDropLabels)
}, cfg.MetricSubset.FilterConfig)
assert.Equal(t, []string{"metric_subset_enabled", "custom_label"}, cfg.WorkloadMetrics.DropLabels)
},
},
{
name: "invalid SRE metrics filter config",
name: "invalid metric subset filter config",
env: map[string]string{
"NVCF_BACKEND_TYPE": "gfn",
"NVCF_INSTANCE_ID": "test-instance",
"NVCF_NAMESPACE": "test-ns",
"NVCF_WORKLOAD_TYPE": "function",
"NVCT_TASK_ID": "task-123",
"NVCF_ZONE_NAME": "zone-1",
"BYOO_SRE_METRICS_FILTER_CONFIG": "processors: []",
"NVCF_BACKEND_TYPE": "gfn",
"NVCF_INSTANCE_ID": "test-instance",
"NVCF_NAMESPACE": "test-ns",
"NVCF_WORKLOAD_TYPE": "function",
"NVCT_TASK_ID": "task-123",
"NVCF_ZONE_NAME": "zone-1",
"BYOO_METRIC_SUBSET_FILTER_CONFIG": "processors: []",
},
expectErr: true,
},
Expand Down Expand Up @@ -240,9 +240,9 @@ func TestGetTemplateConfig(t *testing.T) {
"BYOO_LOG_CHUNK_MAX_BODY_BYTES",
"BYOO_LOG_CHUNK_DRY_RUN",
"BYOO_LOG_EXPORTER_BATCH_MAX_SIZE_BYTES",
"BYOO_SRE_METRICS_ENABLED",
"BYOO_SRE_METRICS_FILTER_CONFIG",
"BYOO_CUSTOMER_METRICS_DROP_LABELS",
"BYOO_METRIC_SUBSET_ENABLED",
"BYOO_METRIC_SUBSET_FILTER_CONFIG",
"BYOO_WORKLOAD_METRICS_DROP_LABELS",
}
backup := map[string]*string{}
for _, k := range envKeys {
Expand Down
Loading
Loading