Add OTel Span Attribute Filter

cf-java-logging-support-opentelemetry-agent-extension/README.md

@@ -13,6 +13,7 @@ The extension provides the following main features:
 * additional exporters for logs, metrics and traces for [SAP Cloud Logging](https://discovery-center.cloud.sap/serviceCatalog/cloud-logging)
 * additional exporter for metrics for [Dynatrace](https://docs.dynatrace.com/docs/setup-and-configuration/setup-on-container-platforms/cloud-foundry/deploy-oneagent-on-sap-cloud-platform-for-application-only-monitoring)
 * adding resource attributes describing the CF application
+* filtering span attributes by name before export
 
 See the section on [configuration](#configuration) for further details.
 
@@ -151,6 +152,22 @@ Note, that the `include` filter is applied before the `exclude` filter.
 That means, if a metric matches both filters, it will be excluded.
 The configuration applies to both the `cloud-logging` and `dynatrace` exporters independently.
 
+### Filtering Span Attributes
+
+_This feature was introduced with version 4.3.0 of the extension._
+
+You can filter which span attributes are exported by name using the following properties:
+
+| Property                                                                              | Description                                                                                                            |
+|---------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------|
+| `sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.include.names`    | A comma-separated list of span attribute name patterns to be included. This may include a wildcard "*" at the end of the name. |
+| `sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.exclude.names`    | A comma-separated list of span attribute name patterns to be excluded. This may include a wildcard "*" at the end of the name. |
+
+The filter is only active when the sanitizer is enabled (`sap.cf.integration.otel.extension.sanitizer.enabled=true`) and at least one of the two properties is set.
+
+The `include` filter is applied before the `exclude` filter.
+That means, if a span attribute matches both filters, it will be excluded.
+
 ### Configuration Properties Summary
 
 The following table summarizes all configuration properties provided by the extension:
@@ -175,7 +192,9 @@ The following table summarizes all configuration properties provided by the exte
 | `otel.exporter.dynatrace.metrics.include.names`                     | A comma-separated list of metric name patterns to be included when exporting metrics to Dynatrace. Wildcard "\*" is only supported at the end of the name. If not set, all metrics are exported.                                                                                                                                              |                                                         |
 | `otel.exporter.dynatrace.metrics.temporality.preference`            | The default histogram aggregation for metrics exported to Dynatrace. Delegates to the underlying OTLP exporter, supporting all its configurations. The Dynatrace metrics exporter provides an additional option `always_delta` which always uses delta aggregation temporality. This is also the default behavior if the property is not set. | `always_delta`                                          |
 | `otel.exporter.dynatrace.metrics.timeout`                           | The maximum duration to wait for Dynatrace when exporting metrics.                                                                                                                                                                                                                                                                            | `10000` (from OTel SDK)                                 |
-| `sap.cf.integration.otel.extension.sanitizer.enabled`               | Enables or disables the sanitizer.                                                                                                                                                                                                                                                                                                            | `true`                                                  |
+| `sap.cf.integration.otel.extension.sanitizer.enabled`                                    | Enables or disables the sanitizer.                                                                                                                                                                                                                                                                                                            | `true`                                                  |
+| `sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.exclude.names`        | A comma-separated list of span attribute name patterns to be excluded. Wildcard "\*" is only supported at the end of the name. If not set, no span attributes are excluded. Requires the sanitizer to be enabled and at least one filter property to be set.                                                                                  |                                                         |
+| `sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.include.names`        | A comma-separated list of span attribute name patterns to be included. Wildcard "\*" is only supported at the end of the name. If not set, all span attributes are included. Requires the sanitizer to be enabled and at least one filter property to be set.                                                                                 |                                                         |
 | `sap.cloudfoundry.otel.resources.enabled`                           | Should Cloud Foundry resource attributes be added to the OpenTelemetry resource?                                                                                                                                                                                                                                                              | `true`                                                  |
 | `sap.cloudfoundry.otel.resources.format`                            | Determines the semantic convention used for Cloud Foundry resource attributes names. `SAP` - use SAP specific attribute names (default). `OTEL` - use OpenTelemetry semantic convention attribute names.                                                                                                                                      | `SAP`                                                   |
 | `sap.cloud-logging.cf.binding.label.value`                          | The label value used to identify managed Cloud Logging service bindings.                                                                                                                                                                                                                                                                      | `cloud-logging`                                         |

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/config/ExtensionConfigurations.java

@@ -185,6 +185,33 @@ interface SANITIZER {
              */
             ConfigProperty<Boolean> ENABLED =
                     booleanValued("sap.cf.integration.otel.extension.sanitizer.enabled").withDefaultValue(true).build();
+
+            interface SPAN {
+                interface ATTRIBUTE {
+                    interface FILTER {
+                        /**
+                         * <p>Parses
+                         * {@code sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.exclude.names}.</p>
+                         * <p>A comma-seperated list of span attribute name patterns to be excluded when sanitizing
+                         * span
+                         * attributes. Wildcard "*" is only supported at the end of the name. If not set, no span
+                         * attributes are excluded.</p>
+                         */
+                        ConfigProperty<List<String>> EXCLUDE_NAMES = listValued(
+                                "sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.exclude.names").build();
+
+                        /**
+                         * <p>Parses
+                         * {@code sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.include.names}.</p>
+                         * <p>A comma-seperated list of span attribute name patterns to be included when sanitizing span
+                         * attributes. Wildcard "*" is only supported at the end of the name. If not set, all span
+                         * attributes are included.</p>
+                         */
+                        ConfigProperty<List<String>> INCLUDE_NAMES = listValued(
+                                "sap.cf.integration.otel.extension.sanitizer.span.attribute.filter.include.names").build();
+                    }
+                }
+            }
         }
     }
 

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/SanitizeSpanExporterCustomizer.java

@@ -1,7 +1,8 @@
 package com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter;
 
-import com.sap.hcf.cf.logging.opentelemetry.agent.ext.config.ExtensionConfigurations.EXTENSION;
-import io.opentelemetry.api.common.AttributeKey;
+import com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter.customizer.DbConnectStatementCustomizer;
+import com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter.customizer.SpanAttributeCustomizer;
+import com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter.customizer.SpanAttributeNameFilterCustomizer;
 import io.opentelemetry.api.common.Attributes;
 import io.opentelemetry.api.common.AttributesBuilder;
 import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
@@ -11,52 +12,62 @@
 import io.opentelemetry.sdk.trace.export.SpanExporter;
 
 import java.util.Collection;
+import java.util.List;
 import java.util.function.BiFunction;
-import java.util.stream.Collectors;
 
-import static io.opentelemetry.api.common.AttributeKey.stringKey;
+import static java.util.stream.Collectors.toList;
 
 public class SanitizeSpanExporterCustomizer implements BiFunction<SpanExporter, ConfigProperties, SpanExporter> {
 
-    private static final AttributeKey<String> DB_QUERY_TEXT = stringKey("db.query.text");
-    //@Deprecated
-    private static final AttributeKey<String> DB_STATEMENT = stringKey("db.statement");
+    private final List<SpanAttributeCustomizer> customizers;
+
+    public SanitizeSpanExporterCustomizer() {
+        this(List.of(new DbConnectStatementCustomizer(), new SpanAttributeNameFilterCustomizer()));
+    }
+
+    SanitizeSpanExporterCustomizer(List<SpanAttributeCustomizer> customizers) {
+        this.customizers = customizers;
+    }
 
     @Override
     public SpanExporter apply(SpanExporter delegate, ConfigProperties config) {
-        if (EXTENSION.SANITIZER.ENABLED.getValue(config) != Boolean.TRUE) {
+        // Keep delegate exporter unwrapped if no customizers are provided.
+        if (customizers == null || customizers.isEmpty()) {
+            return delegate;
+        }
+        // Keep delegate exporter unwrapped if no customizers are enabled.
+        final List<SpanAttributeCustomizer> enabledCustomizers =
+                customizers.stream().filter(c -> c.isEnabled(config)).collect(toList());
+        if (enabledCustomizers.isEmpty()) {
             return delegate;
         }
         return new SpanExporter() {
             @Override
             public CompletableResultCode export(Collection<SpanData> spans) {
-                return delegate.export(spans.stream().map(this::sanitizeSpanData).collect(Collectors.toList()));
+                return delegate.export(spans.stream().map(this::sanitizeSpanData).collect(toList()));
             }
 
             private SpanData sanitizeSpanData(SpanData spanData) {
                 Attributes attributes = spanData.getAttributes();
                 if (attributes == null) {
                     return spanData;
                 }
-                String dbQueryText = attributes.get(DB_QUERY_TEXT);
-                String dbStatement = attributes.get(DB_STATEMENT);
-                if (isClean(dbQueryText) && isClean(dbStatement)) {
-                    return spanData;
-                }
-                AttributesBuilder sanitized = attributes.toBuilder();
-                if (!isClean(dbQueryText)) {
-                    sanitized.put(DB_QUERY_TEXT, dbQueryText.substring(0, 7) + " [REDACTED]");
+                // Only create a new AttributesBuilder if at least one customizer is applicable to the attributes.
+                AttributesBuilder sanitized = null;
+                for (SpanAttributeCustomizer customizer: enabledCustomizers) {
+                    if (customizer.isApplicable(attributes)) {
+                        if (sanitized == null) {
+                            sanitized = attributes.toBuilder();
+                        }
+                        customizer.customize(sanitized, attributes);
+                    }
                 }
-                if (!isClean(dbStatement)) {
-                    sanitized.put(DB_STATEMENT, dbStatement.substring(0, 7) + " [REDACTED]");
+                if (sanitized == null) {
+                    return spanData;
                 }
                 return new SanitizedSpanData(spanData, sanitized.build());
             }
 
-            private boolean isClean(String query) {
-                return query == null || !query.toLowerCase().startsWith("connect");
-            }
-
             @Override
             public CompletableResultCode flush() {
                 return delegate.flush();

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/customizer/DbConnectStatementCustomizer.java

@@ -0,0 +1,52 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter.customizer;
+
+import com.sap.hcf.cf.logging.opentelemetry.agent.ext.config.ExtensionConfigurations;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+
+import static io.opentelemetry.api.common.AttributeKey.stringKey;
+
+/**
+ * A customizer that redacts the text of database connect statements in span attributes. This is to avoid leaking
+ * sensitive information in traces. The customizer checks for the presence of the "db.query.text" and "db.statement"
+ * attributes, and if they start with "connect", it replaces the rest of the string with "[REDACTED]". The customizer
+ * can be enabled or disabled via the configuration property "sap.cf.integration.otel.extension.sanitizer.enabled". By
+ * default, it is enabled.
+ */
+public class DbConnectStatementCustomizer implements SpanAttributeCustomizer {
+
+    private static final AttributeKey<String> DB_QUERY_TEXT = stringKey("db.query.text");
+    //@Deprecated
+    private static final AttributeKey<String> DB_STATEMENT = stringKey("db.statement");
+    private static final String REDACTED = " [REDACTED]";
+
+    @Override
+    public boolean isEnabled(ConfigProperties config) {
+        return ExtensionConfigurations.EXTENSION.SANITIZER.ENABLED.getValue(config);
+    }
+
+    @Override
+    public boolean isApplicable(Attributes original) {
+        String dbQueryText = original.get(DB_QUERY_TEXT);
+        String dbStatement = original.get(DB_STATEMENT);
+        return isCritical(dbQueryText) || isCritical(dbStatement);
+    }
+
+    private boolean isCritical(String query) {
+        return query != null && query.toLowerCase().startsWith("connect");
+    }
+
+    @Override
+    public void customize(AttributesBuilder builder, Attributes original) {
+        String dbQueryText = original.get(DB_QUERY_TEXT);
+        String dbStatement = original.get(DB_STATEMENT);
+        if (isCritical(dbQueryText)) {
+            builder.put(DB_QUERY_TEXT, dbQueryText.substring(0, 7) + REDACTED);
+        }
+        if (isCritical(dbStatement)) {
+            builder.put(DB_STATEMENT, dbStatement.substring(0, 7) + REDACTED);
+        }
+    }
+}

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/customizer/SpanAttributeCustomizer.java

@@ -0,0 +1,42 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter.customizer;
+
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+
+public interface SpanAttributeCustomizer {
+
+    /**
+     * Returns true if this customizer is enabled based on the given configuration.
+     *
+     * @param config
+     *         the configuration properties
+     * @return true if this customizer is enabled, false otherwise
+     */
+    default boolean isEnabled(ConfigProperties config) {
+        return true;
+    }
+
+    /**
+     * Returns true if this customizer is applicable to the given attributes. This avoids object allocation for
+     * attributes that are not relevant to this customizer.
+     *
+     * @param attributes
+     *         the attributes to check
+     * @return true if this customizer is applicable, false otherwise
+     */
+    default boolean isApplicable(Attributes attributes) {
+        return false;
+    }
+
+    /**
+     * Customizes the given attributes builder based on the original attributes. This should be a no-op if the
+     * customizer is not applicable to the given attributes.
+     *
+     * @param attributesBuilder
+     *         the attributes builder to customize
+     * @param original
+     *         the original attributes
+     */
+    void customize(AttributesBuilder attributesBuilder, Attributes original);
+}

cf-java-logging-support-opentelemetry-agent-extension/src/main/java/com/sap/hcf/cf/logging/opentelemetry/agent/ext/exporter/customizer/SpanAttributeNameFilterCustomizer.java

@@ -0,0 +1,57 @@
+package com.sap.hcf.cf.logging.opentelemetry.agent.ext.exporter.customizer;
+
+import com.sap.hcf.cf.logging.opentelemetry.agent.ext.config.ExtensionConfigurations.EXTENSION.SANITIZER;
+import io.opentelemetry.api.common.AttributeKey;
+import io.opentelemetry.api.common.Attributes;
+import io.opentelemetry.api.common.AttributesBuilder;
+import io.opentelemetry.sdk.autoconfigure.spi.ConfigProperties;
+
+import java.util.List;
+import java.util.function.Predicate;
+import java.util.stream.Collectors;
+
+import static java.util.function.Predicate.not;
+
+public class SpanAttributeNameFilterCustomizer implements SpanAttributeCustomizer {
+
+    private Predicate<AttributeKey<?>> rejected = k -> false;
+
+    @Override
+    public boolean isEnabled(ConfigProperties config) {
+        List<String> included = SANITIZER.SPAN.ATTRIBUTE.FILTER.INCLUDE_NAMES.getValue(config);
+        List<String> excluded = SANITIZER.SPAN.ATTRIBUTE.FILTER.EXCLUDE_NAMES.getValue(config);
+
+        List<String> includedNames = getNames(included);
+        List<String> includedPrefixes = getPrefixes(included);
+        List<String> excludedNames = getNames(excluded);
+        List<String> excludedPrefixes = getPrefixes(excluded);
+        this.rejected = k -> {
+            String name = k.getKey();
+            boolean isIncluded = (includedNames.isEmpty() && includedPrefixes.isEmpty()) || includedNames.contains(
+                    name) || includedPrefixes.stream().anyMatch(name::startsWith);
+            boolean isExcluded = excludedNames.contains(name) || excludedPrefixes.stream().anyMatch(name::startsWith);
+            return !isIncluded || isExcluded;
+        };
+
+        return SANITIZER.ENABLED.getValue(config) && (!included.isEmpty() || !excluded.isEmpty());
+    }
+
+    private static List<String> getPrefixes(List<String> included) {
+        return included.stream().filter(s -> s.endsWith("*")).map(s -> s.substring(0, s.length() - 1))
+                       .collect(Collectors.toList());
+    }
+
+    private static List<String> getNames(List<String> included) {
+        return included.stream().filter(not(s -> s.endsWith("*"))).collect(Collectors.toList());
+    }
+
+    @Override
+    public boolean isApplicable(Attributes attributes) {
+        return attributes.asMap().keySet().stream().anyMatch(rejected);
+    }
+
+    @Override
+    public void customize(AttributesBuilder attributesBuilder, Attributes original) {
+        attributesBuilder.removeIf(rejected);
+    }
+}