Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
tag:
description: Git tag to build and push
required: true
default: v0.4.0
default: v0.5.0
type: string

permissions:
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ On a PR you can comment `@coderabbitai review` (incremental) or `@coderabbitai f

## Development guidelines

- Keep collectors **read-only** toward cloud accounts by default. `om scan workload aws`, `om scan workload azure`, and `om scan workload gcp` are the opt-in exception: they may create and delete snapshots in the account, and they store package identifiers only
- Keep collectors **read-only** toward cloud accounts by default. `om scan workload` is the opt-in exception: the root-volume commands may create and delete snapshots, the boot-image commands may create and delete a volume or disk from an image, and both store package identifiers only
- Document graph schema changes in `docs/`
- Prefer small, reviewable PRs
- Add tests for graph queries and normalization logic
Expand Down
24 changes: 15 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ OpenSourceOM Core is the platform behind [opensourceom.org](https://opensourceom

Traditional scanners flood you with CVEs and misconfigurations. OpenSourceOM connects the dots — showing which findings sit on paths from the internet to your sensitive data and privileged identities.

> **Status:** Phase 3 has shipped. Phase 4 is open: `om scan workload aws`, `om scan workload azure`, and `om scan workload gcp` inventory Linux root volumes. Phase 5 (secret and malware findings on that storage) is open. Further rule packs stay open for contributors ([#10](https://github.com/OpenSourceOM/core/issues/10)). See the [roadmap](./docs/ROADMAP.md).
> **Status:** Phase 3 has shipped. Phase 4 is open: `om scan workload` inventories Linux root volumes and boot images. Phase 5 (secret and malware findings on that storage) is open. Further rule packs stay open for contributors ([#10](https://github.com/OpenSourceOM/core/issues/10)). See the [roadmap](./docs/ROADMAP.md).

## Why this exists

Expand Down Expand Up @@ -95,9 +95,12 @@ export AWS_REGION=us-east-1
./om scan aws
# On an EC2 instance in that account, as root, after the API scan:
# sudo -E ./om scan workload aws
# sudo -E ./om scan workload image aws
# On an Azure VM or a GCE instance in that region, as root:
# sudo -E ./om scan workload azure
# sudo -E ./om scan workload image azure
# sudo -E ./om scan workload gcp
# sudo -E ./om scan workload image gcp
./om scan k8s # requires kubeconfig

# Or ingest a custom collector (stdout is a graph batch)
Expand Down Expand Up @@ -138,11 +141,14 @@ An API token is created only when an admin asks for one, from the console or wit
```bash
./om scan demo # sample environment, no credentials
./om scan aws
./om scan workload aws # root on an EC2 worker; Linux root-volume packages
./om scan azure # requires AZURE_SUBSCRIPTION_ID + az login
./om scan workload azure # root on an Azure VM; Linux OS-disk packages
./om scan gcp # requires GCP_PROJECT_ID + ADC
./om scan workload gcp # root on a GCE instance; Linux boot-disk packages
./om scan workload aws # root on an EC2 worker; Linux root-volume packages
./om scan workload image aws # root on an EC2 worker; AMI packages, applied to instances
./om scan azure # requires AZURE_SUBSCRIPTION_ID + az login
./om scan workload azure # root on an Azure VM; Linux OS-disk packages
./om scan workload image azure # root on an Azure VM; boot-image packages, applied to VMs
./om scan gcp # requires GCP_PROJECT_ID + ADC
./om scan workload gcp # root on a GCE instance; Linux boot-disk packages
./om scan workload image gcp # root on a GCE instance; image packages, applied to instances
./om scan k8s # requires kubeconfig or in-cluster credentials
./om scan plugin -- ./my-collector
```
Expand All @@ -152,10 +158,10 @@ The CLI does not read `.env`. It defaults `POSTGRES_HOST` to `localhost`, which
**Kubernetes:**

```bash
docker build -t ghcr.io/opensourceom/core:0.4.0 .
docker build -t ghcr.io/opensourceom/core:0.5.0 .
helm install om deploy/helm/opensourceom \
--set postgres.password='change-me' \
--set image.tag=0.4.0
--set image.tag=0.5.0
```

Collectors stay off until you enable one and supply credentials, an existing Secret, or `serviceAccountAuth`. The CronJob runs `om scan` against the same Postgres the API reads. `om scan demo` stays a one-shot command. See `scan` in `deploy/helm/opensourceom/values.yaml`.
Expand All @@ -170,7 +176,7 @@ Full documentation: [opensourceom.org](https://opensourceom.org) (docs at [opens
| **1** | Attack path queries, CVE enrichment, web UI, Azure/GCP collectors |
| **2** | CSPM rules, blast radius, K8s connector, exports |
| **3** | Plugin SDK, Helm, embedded rule pack, demo graph, inventory-backed CVEs, crown-jewel datastores, attack-path findings, and cloud audit context (CloudTrail, Activity Log, Admin Activity). Shipped. |
| **4** | Package inventory from persistent workload storage. AWS, Azure, and GCE Linux root volumes ship with `om scan workload aws`, `om scan workload azure`, and `om scan workload gcp`. Boot images, container layers, function bundles, and shared filesystems are open. |
| **4** | Package inventory from persistent workload storage. Linux root volumes and boot images ship with `om scan workload`. Container layers, function bundles, and shared filesystems are open. |
| **5** | Secret and malware findings on storage Phase 4 already opens. Kind and path only. Open. |

Details: [docs/ROADMAP.md](./docs/ROADMAP.md)
Expand Down
4 changes: 2 additions & 2 deletions collectors/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@ Cloud and platform ingestion plugins. Each collector normalizes provider APIs in

The **demo** collector loads a fixed environment (internet-exposed web tier, production database, public/private buckets, admin vs app identities, public Kubernetes service). Use it to exercise CSPM packs without cloud credentials.

The **AWS** collector emits properties the CIS pack matches on: `open_ingress`, `imdsv2`, `public_ip`, S3 `encryption` / `versioning` / `public_access_block`, and IAM user `mfa` / `unused_access_keys`. It also records RDS and Aurora instances as datastores. S3 bucket tags and the RDS `TagList` copy `sensitivity` or `data-class` onto the datastore. `sensitivity` wins when both are set. The same scan reads CloudTrail management events from the last 24 hours and stores the ones that name an identity and a resource on an exposed path. A failed lookup omits those events. `om scan aws` does not read instance disks. `om scan workload aws` is the separate command that inventories packages on Linux root volumes.
The **AWS** collector emits properties the CIS pack matches on: `open_ingress`, `imdsv2`, `public_ip`, S3 `encryption` / `versioning` / `public_access_block`, and IAM user `mfa` / `unused_access_keys`. It also records RDS and Aurora instances as datastores. S3 bucket tags and the RDS `TagList` copy `sensitivity` or `data-class` onto the datastore. `sensitivity` wins when both are set. The same scan reads CloudTrail management events from the last 24 hours and stores the ones that name an identity and a resource on an exposed path. A failed lookup omits those events. `om scan aws` does not read instance disks. `om scan workload aws` inventories packages on Linux root volumes. `om scan workload image aws` reads each boot image once and applies those packages to instances launched from it.

The **Azure** collector records logical SQL servers. `om scan azure` and `om scan gcp` do not read OS disks. `om scan workload azure` and `om scan workload gcp` are the separate commands that inventory packages on those disks. The same scan reads administrative Activity Log events from the last 24 hours and stores the ones that name an identity and a resource on an exposed path. A failed lookup omits those events. Entra ID sign-in logs are not part of this scan. The **GCP** collector records Cloud SQL instances. The same scan reads Admin Activity audit logs from the last 24 hours and stores the ones that name an identity and a resource on an exposed path. A failed lookup omits those events. Data Access logs are not part of this scan. A workload gets `CAN_ACCESS` only when a security group, firewall, or VPC path allows it. Azure copies the crown-jewel mark from storage-account and SQL-server tags. GCP copies it from a bucket label or a Cloud SQL user label. The keys are `sensitivity` and `data-class`. A plugin may set `sensitivity` on a datastore directly.
The **Azure** collector records logical SQL servers. `om scan azure` and `om scan gcp` do not read OS disks. `om scan workload azure` and `om scan workload gcp` inventory packages on those disks. `om scan workload image azure` and `om scan workload image gcp` read each boot image once. The same scan reads administrative Activity Log events from the last 24 hours and stores the ones that name an identity and a resource on an exposed path. A failed lookup omits those events. Entra ID sign-in logs are not part of this scan. The **GCP** collector records Cloud SQL instances. The same scan reads Admin Activity audit logs from the last 24 hours and stores the ones that name an identity and a resource on an exposed path. A failed lookup omits those events. Data Access logs are not part of this scan. A workload gets `CAN_ACCESS` only when a security group, firewall, or VPC path allows it. Azure copies the crown-jewel mark from storage-account and SQL-server tags. GCP copies it from a bucket label or a Cloud SQL user label. The keys are `sensitivity` and `data-class`. A plugin may set `sensitivity` on a datastore directly.

The **Kubernetes** collector records Ingress objects and whether a NetworkPolicy selects each pod. A pod is internet-reachable from a LoadBalancer or an Ingress backend unless that policy governs ingress and does not allow the world. NodePort alone is not exposure.

Expand Down
4 changes: 2 additions & 2 deletions deploy/helm/opensourceom/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,5 @@ apiVersion: v2
name: opensourceom
description: Self-hosted OpenSourceOM Core API, optional Postgres, and optional scheduled collectors
type: application
version: 0.4.0
appVersion: "0.4.0"
version: 0.5.0
appVersion: "0.5.0"
4 changes: 2 additions & 2 deletions deploy/helm/opensourceom/templates/NOTES.txt
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
1. Build and load an image (the default repository is a placeholder until you publish one):

docker build -t ghcr.io/opensourceom/core:0.4.0 .
docker build -t ghcr.io/opensourceom/core:0.5.0 .

2. Install with required secrets:

helm install om deploy/helm/opensourceom \
--set api.secret='change-me' \
--set postgres.password='change-me' \
--set image.repository=ghcr.io/opensourceom/core \
--set image.tag=0.4.0
--set image.tag=0.5.0

3. Open the console via the ClusterIP service or `kubectl port-forward svc/om-opensourceom 8080:8080`.

Expand Down
5 changes: 3 additions & 2 deletions deploy/helm/opensourceom/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -60,8 +60,9 @@ resources:
# a kubeconfig Secret the chart creates a read-only ServiceAccount for this
# cluster. `om scan demo` and the workload scans are not scheduled.
# The workload scan mounts disks and runs on a VM in the account: EC2 for
# `om scan workload aws`, an Azure VM for `om scan workload azure`, and a
# GCE instance for `om scan workload gcp`.
# `om scan workload aws` and `om scan workload image aws`, an Azure VM for
# `om scan workload azure` and `om scan workload image azure`, and a GCE
# instance for `om scan workload gcp` and `om scan workload image gcp`.
scan:
schedule: "0 */6 * * *"
successfulJobsHistoryLimit: 3
Expand Down
8 changes: 4 additions & 4 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ SPDX-License-Identifier: Apache-2.0

# Architecture

> **Status:** Phase 3 has shipped. The collector plugin SDK, Helm, the embedded rule pack, and cloud audit context from CloudTrail, Activity Log, and Admin Activity logs are in the tree, alongside Phase 2 CSPM rules, blast radius, Kubernetes ingest, and exports. Phase 4 is open: `om scan workload aws`, `om scan workload azure`, and `om scan workload gcp` inventory Linux root volumes. Phase 5 (secret and malware findings on that storage) is open. Further rule packs stay open for contributors.
> **Status:** Phase 3 has shipped. The collector plugin SDK, Helm, the embedded rule pack, and cloud audit context from CloudTrail, Activity Log, and Admin Activity logs are in the tree, alongside Phase 2 CSPM rules, blast radius, Kubernetes ingest, and exports. Phase 4 is open: `om scan workload` inventories Linux root volumes and boot images. Phase 5 (secret and malware findings on that storage) is open. Further rule packs stay open for contributors.

## Overview

Expand All @@ -32,7 +32,7 @@ SPDX-License-Identifier: Apache-2.0
| Component | Location | Notes |
|-----------|----------|-------|
| **Collectors** | `internal/collectors/` | AWS, Azure, GCP, Kubernetes, demo; RDS, Azure SQL, and Cloud SQL are datastores; AWS emits CIS pack properties and recent CloudTrail management events; Azure emits recent Activity Log events; GCP emits recent Admin Activity audit logs |
| **Workload packages** | `internal/workload/` | `om scan workload aws`, `om scan workload azure`, and `om scan workload gcp` set `packages` from Linux root volumes |
| **Workload packages** | `internal/workload/` | `om scan workload` sets `packages` from Linux root volumes and from boot images |
| **Plugin SDK** | `sdk/collector`, `internal/plugins/` | External executables; `om scan plugin` |
| **Graph store** | `internal/graph/`, `migrations/` | PostgreSQL `nodes` + `edges` |
| **Path queries** | `internal/graph/query.go` | Named queries including `internet-to-datastore` and `internet-to-sensitive-datastore` |
Expand All @@ -52,7 +52,7 @@ SPDX-License-Identifier: Apache-2.0

Findings link to affected resources via `VIOLATES` edges. CSPM and CVE findings share the same `Finding` node type but use `finding_type` and `rule_id` / `cve_id` properties for provenance.

See [ADR 001](./adr/001-graph-schema-v0.md), [ADR 002](./adr/002-phase1-findings-ui.md), [ADR 003](./adr/003-phase2-cspm-rbac.md), [ADR 004](./adr/004-collector-plugin-sdk.md), [ADR 005](./adr/005-console-admins.md), [ADR 006](./adr/006-aws-root-volume-packages.md), and [ADR 007](./adr/007-azure-gce-root-volume-packages.md).
See [ADR 001](./adr/001-graph-schema-v0.md), [ADR 002](./adr/002-phase1-findings-ui.md), [ADR 003](./adr/003-phase2-cspm-rbac.md), [ADR 004](./adr/004-collector-plugin-sdk.md), [ADR 005](./adr/005-console-admins.md), [ADR 006](./adr/006-aws-root-volume-packages.md), [ADR 007](./adr/007-azure-gce-root-volume-packages.md), and [ADR 008](./adr/008-boot-image-packages.md).

## Technology choices

Expand Down Expand Up @@ -87,7 +87,7 @@ Further community rule packs (PCI and additional CIS mappings) stay open for con

Phase 4 writes `packages` (CPE 2.3 names and package URLs) from persistent workload storage onto the workload node that enrichment already matches. The sources, in order, are AWS Linux root volumes, Azure VM and GCE root volumes, boot images, container image layers and function bundles, and shared filesystems (EFS, Azure Files, Filestore). The scan is opt-in. `om scan aws`, `om scan azure`, and `om scan gcp` stay read-only API collectors.

`om scan workload aws` is the AWS root-volume scan. `om scan workload azure` and `om scan workload gcp` are the same walk for Azure OS disks and GCE boot disks. Each runs as root on a worker in that account and region, snapshots the root disk, mounts the copy read-only, and stores package identifiers only. A later API scan keeps `packages`. Windows, LVM, and non-root disks are skipped. Boot images, container layers, function bundles, and shared filesystems are still open. See [ADR 006](./adr/006-aws-root-volume-packages.md) and [ADR 007](./adr/007-azure-gce-root-volume-packages.md).
`om scan workload aws` is the AWS root-volume scan. `om scan workload azure` and `om scan workload gcp` are the same walk for Azure OS disks and GCE boot disks. `om scan workload image aws`, `om scan workload image azure`, and `om scan workload image gcp` read each boot image once and add those package identifiers to instances launched from it. A package that exists only on an instance disk stays on that instance. Each command runs as root on a worker in that account and region, mounts a copy read-only, and stores package identifiers only. A later API scan keeps `packages`. Windows, LVM, and non-root disks are skipped. Container layers, function bundles, and shared filesystems are still open. See [ADR 006](./adr/006-aws-root-volume-packages.md), [ADR 007](./adr/007-azure-gce-root-volume-packages.md), and [ADR 008](./adr/008-boot-image-packages.md).

Phase 5 adds a finding for a secret pattern or a malware signature on storage that Phase 4 reader already opened. The finding stores the kind and the path. Object bodies and database rows are in neither phase.

Expand Down
4 changes: 3 additions & 1 deletion docs/ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,12 +70,14 @@ The path, in order:

- [x] **AWS Linux root volumes** — [#102](https://github.com/OpenSourceOM/core/issues/102)
- [x] **Azure VM and GCE root volumes** — the same walker, those snapshot APIs — [#103](https://github.com/OpenSourceOM/core/issues/103)
- [ ] **Boot images** — an AMI and the Azure and GCE equivalents, scanned once and applied to instances launched from that image — [#104](https://github.com/OpenSourceOM/core/issues/104)
- [x] **Boot images** — an AMI and the Azure and GCE equivalents, scanned once and applied to instances launched from that image — [#104](https://github.com/OpenSourceOM/core/issues/104)
- [ ] **Container image layers and function bundles** — layers for images already stored on Kubernetes workloads, plus Lambda deployment packages and the Azure and GCP equivalents. No volume attach — [#105](https://github.com/OpenSourceOM/core/issues/105)
- [ ] **Shared filesystems** — EFS, Azure Files, and Filestore, when the application is installed there and a root-volume scan would miss it — [#106](https://github.com/OpenSourceOM/core/issues/106)

`om scan workload aws` reads Linux root volumes in the current AWS account. It runs as root on an EC2 instance in that account, snapshots the root volume, mounts the copy read-only, and writes CPE 2.3 names and package URLs onto `packages`. `om scan workload azure` does the same for managed OS disks from an Azure VM in that region, and `om scan workload gcp` does it for GCE boot disks from an instance in that region. A later `om scan aws`, `om scan azure`, or `om scan gcp` leaves `packages` in place. Windows, LVM, and disks other than the root volume are skipped.

`om scan workload image aws` reads each AMI once, including an image no instance is running, and adds those package identifiers to every instance launched from it. `om scan workload image azure` and `om scan workload image gcp` do the same for Azure and GCE images. A package found only on an instance's own disk stays on that instance. Marketplace images the account cannot snapshot are skipped. The volume or disk created for the read is deleted. The image's own snapshot is not.

Object bodies and database rows stay out. They are customer data, not the installed software. A public bucket or database is already an exposure finding from the API scan.

## Phase 5 — Findings on inventoried storage
Expand Down
2 changes: 1 addition & 1 deletion docs/adr/006-aws-root-volume-packages.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ An inventory upsert that omits `packages` keeps the list already stored. `om sca

The IAM actions are `ec2:CreateSnapshot`, `ec2:AttachVolume`, `ec2:DeleteSnapshot`, `ec2:DeleteVolume`, the describe and tag calls around that lifecycle, and `kms:CreateGrant` plus `kms:Decrypt` (with `kms:DescribeKey`, `kms:GenerateDataKeyWithoutPlaintext`, and `kms:ReEncrypt*`) for an encrypted root volume. `om scan aws` does not gain these actions.

Azure, GCE, boot images, container layers, function bundles, and shared filesystems stay on later Phase 4 issues. Secret and malware findings stay on Phase 5, after this reader exists.
Azure and GCE root volumes are [ADR 007](./007-azure-gce-root-volume-packages.md). Boot images are [ADR 008](./008-boot-image-packages.md). Container layers, function bundles, and shared filesystems stay on later Phase 4 issues. Secret and malware findings stay on Phase 5, after this reader exists.

## Consequences

Expand Down
Loading
Loading