[SCI2-5924] Add Iru (Kandji) OCSF pipeline - #24918
Open
cepolation-datadog wants to merge 8 commits into
Open
Conversation
Adds OCSF support to the Iru (formerly Kandji) integration, which previously had no isOcsf blocks at all. Mappings were generated from real production log samples (27 us1.prod orgs, 30-day window, 1040 events) rather than only the 6 committed test fixtures, which surfaced 26 distinct log shapes -- including target_type: library_item, which had no sub-pipeline at all. Six OCSF classes across seven sub-pipelines: Vulnerability Finding [2002] target_type:vulnerability + service:detection Detection Finding [2004] target_type:file_detection Account Change [3001] target_type:admin Entity Management [3004] target_type:device/library_item/blueprint Remediation Activity [7001] remediation / first_runs compliance runs Base Event [0] catch-all Entity Management [3004] is used for the device bucket rather than Device Config State Change [5019]: 5019's schema is a narrow security-toggle enum (Disabled/Enabled/Unknown/Other), while 3004's enum includes Enroll/Unenroll which map cleanly onto device enrollment and deletion. Local OCSF validator reports 6/6 test logs valid at 100% required-field coverage. Arrays (vulnerabilities, malware.classification_ids, resources, cve.cvss) are built with array-processor and their generated contents were inspected by hand, since the local validator currently no-ops array validation. Facets are the ocsf-facet-generator output plus the conventional classification facets (activity_id, class_uid, category_uid, status_id, severity_id, type_uid) that the generator does not emit. Co-Authored-By: Claude <noreply@anthropic.com>
cepolation-datadog
added
assets/no-dry-run
evalya-impact-summaryevalya impact analysis |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 57e28b460e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The legacy admin audit sub-pipeline moved new_state.email to usr.email
with preserveSource: false. Since that pipeline runs before the OCSF
sub-pipelines, new_state.email no longer existed by the time the
Account Change [3001] schema-processor ran, so ocsf.user.email_addr was
always omitted -- visible in the fixture, which supplies an email yet
produced no email_addr.
Flips that remapper to preserveSource: true, per style guide 7.1
("if the upstream processors delete the original source field you need,
restore it by changing preserveSource: false to preserveSource: true")
rather than sourcing from usr.email, which FIELD-1 disallows.
Change is additive: usr.email is still populated for existing
consumers, and new_state.email is now retained alongside it, which 7.2
explicitly permits.
Audited the other three legacy preserveSource: false remappers
(new_state.event.outcome, new_state.user.name, new_state.user.id) --
none are sourced by any OCSF mapper, so this was the only instance.
Co-Authored-By: Claude <noreply@anthropic.com>
CI validate-logs flagged: "Attributes for the facet 'Event Time' (path: ocsf.time) are not properly alphabetically sorted". When reindenting the ocsf-facet-generator output I moved `groups` to the front for visual consistency with the other facet entries, which pushed `facetType` out of alphabetical order. The generator emits facetType first, as does the linux_audit_logs precedent. ocsf.time was the only affected facet -- verified all 42 entries now have alphabetically sorted keys. Facet list order itself is not enforced (anthropic_compliance_logs and linux_audit_logs are both unsorted by path and pass CI). Co-Authored-By: Claude <noreply@anthropic.com>
The staging backend rejected the pipeline with six errors the local
validator could not see. Root cause for five of them: Remediation
Activity [7001] extends base_event, which has no `device` field at all
(unlike the Finding classes, which inherit `device` from `finding`), so
all five ocsf.device.* mappers targeted fields that do not exist.
Rather than delete the device context, these events are remapped to
Compliance Finding [2003], which fits them better on the merits:
- OCSF describes 2003 as "results of evaluations performed against
resources, to check compliance with Industry Frameworks or Security
Standards", which is exactly what Kandji parameters are -- the
sampled parameter names are verbatim CIS macOS controls ("Enable
System Integrity Protection (SIP)", "Disable File Sharing").
- 2003 extends finding, so `device` is inherited and device context
survives.
- 7001's activity_id enum is D3FEND threat-response tactics
(Isolate/Evict/Restore/Harden/Detect); everything was being forced
to Harden. 2003 uses the finding lifecycle instead.
- 7001 has no way to express a check result. 2003 has
compliance.control, compliance.standards and compliance.status_id,
so the parameter name and pass/fail state now have real homes.
Sixth backend error: Base Event [0] was missing the required
activity_id. Added, restricted to the only values base_event permits
(Unknown 0 / Other 99) with a fallback carrying the vendor's literal
action.
Also fixed while verifying against the schema:
- compliance.status_id enum is Pass 1 / Warning 2 / Fail 3; a first
draft used a non-existent "Fixed 5" and misused 2 for Fail.
- compliance.checks is an array of `check` objects, not strings, so
the parameter-name array mapping was dropped.
- Top-level status_id on a finding subclass is the finding lifecycle
(New/In Progress/Suppressed/Resolved/Archived), not base_event's
Success/Failure. Now New on evaluation, Resolved on remediation.
- Three category mappers repeated a name/id pair, which CAT-3
forbids; merged into OR queries (actor_type device/kandji -> System
in [3001] and [3004], and the [3004] status_id Success catch-all).
- A staging field had been named ocsf.compliance_standards_str and
was being picked up as a phantom OCSF facet; renamed out of the
ocsf namespace.
- Dropped the now-unused ocsf.command_uid facet, added
ocsf.compliance.control and ocsf.compliance.standards.
- Sub-pipelines reordered so 2003 sits between 2002 and 2004.
Added three test fixtures (remediation, first_runs, and an unhandled
target_type that exercises the Base Event catch-all). No fixture
covered [7001] or [0] before, which is precisely why local validation
passed while the backend rejected the pipeline -- the local validator
checks emitted documents, so sub-pipelines no fixture reaches are never
exercised.
Local validator: 9/9 valid, 100% required-field coverage, no warnings.
Staging upsert now reports "Pipeline created successfully".
Co-Authored-By: Claude <noreply@anthropic.com>
CI validate-logs reported the expected output did not match actual for the three fixtures added in the previous commit. A field-by-field diff of CI's reported actual against the committed file showed the entire result.custom block -- including all OCSF output -- was already byte-identical. The only differences were `sample` and `result.message` (which echoes the sample), and only in the ordering of keys within the raw JSON. CI canonicalizes key order in the sample and expects the committed file to match, so the three hand-written samples were rewritten into CI's order and the file regenerated so result.message follows. Worth noting for future pipelines: the local ocsf-validator does not compare sample/message text, so `--check-all` passes on a file CI will reject. Verified here by diffing directly against CI's reported output; all three fixtures now match exactly. Local validator: 9/9 valid, 100% required-field coverage, no warnings. Co-Authored-By: Claude <noreply@anthropic.com>
cepolation-datadog
removed
the
assets/no-dry-run
…iptive
Two fixes from a semantic review of all 40 OCSF targets against the
vendored dictionary.json and schema.ocsf.io/api/1.5.0.
ocsf.user.type had two writers in Account Change [3001]: a
schema-remapper emitting the raw Iru role ("admin") and a
schema-category-mapper emitting the normalized caption ("Admin"). OCSF
requires `type` to be "normalized to the caption of the type_id value",
so the raw value was not valid -- correct output only happened because
the category mapper ran later and overwrote it. Also violated FIELD-7
(one mapper per target) and CAT-6 (enums belong to the category mapper).
Removed the remapper; the category mapper already handled both halves.
finding_info.title carried a bare identifier or object name, where OCSF
asks for "a title or a brief phrase summarizing the reported finding".
It was also inconsistent across the two Vulnerability Finding
sub-pipelines (CVE id vs application name). Now built per sub-pipeline:
2002 audit CVE-2024-44187 in Safari
2002 detections CVE-2025-10892 in Google Chrome
2004 SpyDok detected in AppStore
2003 remediation Disable the ability to login to another user's ...
2003 first_runs Compliance run for blueprint DDS-Test
The staging field is consumed with preserveSource: false so it does not
leak into customer logs as an extra top-level attribute.
Local validator: 9/9 valid, 100% required-field coverage.
Co-Authored-By: Claude <noreply@anthropic.com>
finding_info.uid was mapped from cve_id, which is not unique: GET /api/v1/vulnerability-management/detections returns one row per (device, application, CVE) across the fleet, so a single CVE repeats on every affected device and all of those findings shared one uid. The sibling /vulnerabilities endpoint is explicitly the CVE-grouped view, so cve_id is the grouping key rather than a finding identifier. Checked the vendor's published API collection for a better field: the detections response has no id, detection_id or uuid, and its only id-shaped fields (device_id, bundle_id, cve_id, blueprint_id) are all dimension keys. There is no single-field option, so the uid is composed from device_id + bundle_id + cve_id. This sub-pipeline also had no ocsf.metadata.uid mapper at all -- the other six map `id`, which this stream does not return -- so metadata.uid came out empty. The composite now feeds both, which is reasonable here because a snapshot-list endpoint has no event identity distinct from the finding itself. cve_id remains mapped to vulnerabilities[].cve.uid, so CVE-based correlation is unaffected. The staging attribute is consumed with preserveSource: false and does not reach the log. Local validator: 9/9 valid, 100% required-field coverage. Co-Authored-By: Claude <noreply@anthropic.com>
metadata.threat_id is the detected file's SHA256 -- byte-identical to new_state.file.hash.sha256 -- so using it for ocsf.finding_info.uid gave every device that the same malware was found on an identical finding uid, and left finding_info.uid and malware.uid holding the same value. target_id on this shape is `<host uuid>.<device>.<inode>`, which identifies the file instance on a specific host and is therefore unique per finding. malware.uid keeps the SHA256, which is the correct place for an identifier of the malware itself. Same defect class as the cve_id finding uid fixed in 12e2687; found by auditing every ocsf *uid/*id target against its source rather than trusting the earlier per-field review. Local validator: 9/9 valid, 100% required-field coverage. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Validation ReportAll 21 validations passed. Show details
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds OCSF support to the Iru (formerly Kandji) integration. The integration previously had no
isOcsfblocks at all, so this is a greenfield OCSF build.Jira: SCI2-5924
Motivation
Mappings were generated with Fleak from real production log samples rather than only the 6 committed test fixtures. Sampling ran via NBT across 27 real us1.prod customer orgs (30-day window, 1040 events), excluding every org flagged
IS_AI_TRAINING_DATA_RESTRICTED,IS_CUSTOMER_DATA_RESTRICTEDorIS_HIPAA_ORGand intersecting withnbt_session.approved_orgs. Samples were sanitized on-pod before leaving Datadog infrastructure.That surfaced 26 distinct log shapes, several not represented in the test fixtures at all — most notably
target_type: library_item, which had no sub-pipeline.Classes
Fleak segregated the logs into 17 groups; these were consolidated to 6 classes / 7 sub-pipelines after checking each suggested class against the vendored OCSF 1.5.0 schema in
logs-backend.target_type:vulnerability(audit) +service:detection— 2 sub-pipelines, different source shapestarget_type:file_detection(malware/threat)target_type:admintarget_type:device/library_item/blueprintremediation/first_runscompliance runsOn [3004] vs [5019]: Fleak suggested
Device Config State Change [5019]for much of thedevicebucket, but its actual schema is a narrow security-toggle enum (Disabled/Enabled/Unknown/Other) that doesn't fit general device lifecycle events.Entity Management [3004]is used instead — its enum includesEnroll/Unenroll, which map cleanly onto device enrollment and deletion.Validation
Local OCSF validator: 6/6 test logs valid, 100% required-field coverage.
Issues found and fixed during validation:
device.os.type_idin 3 sub-pipelinesfileis not a valid top-level attribute on Detection Finding [2004] — moved intoresources[]cve.cvss_base_scorefield — replaced with a properly constructed nestedcve.cvss[]arrayseverity_idresolving to Unknown on the olderservice:detectionshape — added a fallback across bothcvss_severityandseverityArrays (
vulnerabilities,malware.classification_ids,resources,cve.cvss) are all built witharray-processor. Because the local validator currently no-ops array validation, the generated array contents were inspected by hand inkandji_tests.yaml.Facets are the
ocsf-facet-generatoroutput (29) plus the conventional classification facets (activity_id,class_uid,category_uid,status_id,severity_id,type_uid, …) that the generator doesn't emit but ~79% of shippedintegrations-coreOCSF pipelines declare — 42 total.The existing non-OCSF pipeline is untouched; OCSF is purely additive and no
preserveSourcewas flipped tofalseon any original field.Reviewer notes
target_type: adminandtarget_type: file_detectionproduced zero events across the 27-org 30-day sample, so those two sub-pipelines are exercised only by the committed test fixtures, not by observed traffic.schema-category-mapperis first-match-wins, so hoisting the wildcard would collapse every event to the lowest severity.Additional Notes
Sub-pipelines are ordered ascending by
class_uidwithBase Event [0]last, per the style guide.🤖 Generated with Claude Code