diff --git a/.claude/commands/seed-ea.md b/.claude/commands/seed-ea.md new file mode 100644 index 0000000..54014fc --- /dev/null +++ b/.claude/commands/seed-ea.md @@ -0,0 +1,86 @@ +--- +name: seed-ea +description: Populate Entity Analytics pages using security-documents-generator. Helps pick the right commands for the EA pages or data you want to seed. +--- + +# Seed EA Data + +Use `security-documents-generator` to populate Entity Analytics pages for development or testing. + +Run commands from the repo root with: `yarn start ` + +Before running any command, ensure the correct Node version is active. The repo includes an `.nvmrc` pinned to the required version — if you have nvm installed, run `nvm use` from the repo root first to avoid version mismatch errors. + +## Input + +The user will either: +- Name a specific EA page or feature they want to populate (e.g. "home page", "flyouts", "PUM", "AI summary") +- Ask for a full end-to-end seed of all EA pages +- Provide flags or modifications (e.g. "more hosts", "non-default space", "skip setup") +- Ask what command to use for something specific +- Specify a target cluster (e.g. "seed into my BC environment" or "use the cloud deployment in config.bc.json") + +If no input is given, ask what they want to populate. If a target cluster is mentioned, remind the user that the generator reads from `config.json` in the repo root — they should ensure that file points to the right cluster before running. + +## Page → Command Mapping + +Use this to recommend the right command(s): + +| Page / Feature | Command(s) | +|---|---| +| Home page — entities table, risk KPI/history, watchlists | `risk-score-v2` | +| Entity store management page | `risk-score-v2` (sets up entity store) | +| Asset criticality | `risk-score-v2` (includes criticality) or `generate-asset-criticality` separately | +| Privileged user monitoring | `privmon-quick` or `privileged-user-monitoring` (interactive) | +| Threat hunting leads | `leads` (interactive, needs inference connector pre-configured) | +| AI summary / anomalies panel | `generate-entity-ai-insights --v2 --correlate-with-entity-store` | +| Entity flyouts — host/user/service right panels | `generate-entity-maintainers-data --quick` + `generate-entity-ai-insights --v2` | +| Entity flyout — generic right panel | `quick-entity-store` (includes generic; `risk-score-v2` does not) | +| Explore pages (hosts/users/network) | Covered by `risk-score-v2` alerts; top up with `generate-alerts` if empty | +| Risk score history snapshots | `generate-entity-maintainers-data --quick` | +| CSP / cloud posture findings | `csp --data-sources elastic_all --csp-scores` | +| All EA pages end-to-end | See full sequence below | + +## Full End-to-End Sequence + +When the user wants everything populated, recommend these in order: + +```bash +# 1. Core: entity store, risk engine, criticality, watchlists +yarn start risk-score-v2 --entity-kinds host,idp_user,local_user,service --hosts 20 --users 20 --services 10 --alerts-per-entity 10 + +# 2. Risk history + relationships + anomaly behaviours (flyouts, home history panel) +yarn start generate-entity-maintainers-data --space default --quick + +# 3. Privileged user monitoring +yarn start privmon-quick --space default + +# 4. AI insights + anomaly records (ai_summary, flyout panels) +yarn start generate-entity-ai-insights --v2 --correlate-with-entity-store -h 20 -u 20 -s default + +# 5. Threat hunting leads (requires inference connector pre-configured in Kibana) +yarn start leads --space default +# → choose "Generate leads now" + +# 6. Generic entities (for generic flyout — not covered by risk-score-v2) +yarn start quick-entity-store --space default +``` + +## Key Flags + +- `--no-setup` — skip entity store installation (use when already installed) +- `--space ` — target a non-default space +- `--dangerous-clean` — wipe existing data before seeding (use with care) +- `--perf` — scale preset: 1000 users, 1000 hosts, 50 alerts each +- `--hosts ` / `--users ` / `--services ` — control entity counts +- `--alerts-per-entity ` — controls how many alerts drive risk scoring + +## Known Gaps + +- **Generic flyout**: `risk-score-v2` doesn't seed generic entities — use `quick-entity-store` instead +- **Threat hunting leads**: requires an inference connector to be set up in Kibana first +- **Anomalies panel**: `generate-entity-ai-insights` seeds anomaly *records* but doesn't run actual ML jobs + +## Output + +Give the user the exact `yarn start` command(s) to run, with a note on what each one populates. If they want to run them, offer to do so using Bash from the repo root. Always confirm before running any command with `--dangerous-clean`. diff --git a/.gitignore b/.gitignore index 5b9bf1c..7d80f9d 100644 --- a/.gitignore +++ b/.gitignore @@ -53,4 +53,6 @@ Thumbs.db !.cursor/rules/ .cursorworkspace .claude/ +!.claude/commands/ +!.claude/commands/** .claude.json diff --git a/README.md b/README.md index 76ca97b..63893c9 100644 --- a/README.md +++ b/README.md @@ -16,7 +16,7 @@ Generate synthetic Security data for Elasticsearch and Kibana development, demos ## Requirements -- Node.js `24.13.1` +- Node.js `24.18.0` - Yarn `^1.22.22` - Access to Elasticsearch and Kibana @@ -193,49 +193,52 @@ Each prompt is skipped individually when its flag is present. Omit any flag to b Detailed command documentation is colocated with command code under `src/commands`. -| Command | Summary | Details | -| ---------------------------------------------------------------- | -------------------------------------------------------------------------------- | -------------------------------------------------------- | -| `generate-alerts` | Generate synthetic detection alerts | `src/commands/documents/README.md` | -| `generate-events` | Generate synthetic events in `eventIndex` | `src/commands/documents/README.md` | -| `generate-graph` | Generate fake graph data | `src/commands/documents/README.md` | -| `delete-alerts` | Delete all generated alerts | `src/commands/documents/README.md` | -| `delete-events` | Delete all generated events | `src/commands/documents/README.md` | -| `entity-resolution-demo` | Load entity resolution demo dataset | `src/commands/entity_store/README.md` | -| `entity-store` | Interactive Entity Store generation flow | `src/commands/entity_store/README.md` | -| `quick-entity-store` | Quick non-interactive Entity Store setup | `src/commands/entity_store/README.md` | -| `clean-entity-store` | Clean Entity Store data | `src/commands/entity_store/README.md` | -| `test-risk-score` | Run risk score API test call | `src/commands/misc/README.md` | -| `generate-entity-ai-insights` | Generate vulnerabilities, misconfigurations and anomalous behavior for entities. | `src/commands/misc/README.md` | -| `generate-asset-criticality` | Generate asset criticality assignments | `src/commands/misc/README.md` | -| `generate-legacy-risk-score` | Install and generate legacy risk score data | `src/commands/misc/README.md` | -| `single-entity` | Create one entity with optional setup flows | `src/commands/misc/README.md` | -| `privileged-user-monitoring` | Interactive privileged user monitoring dataset generation | `src/commands/privileged_user_monitoring/README.md` | -| `privmon-quick` | Fast privileged user monitoring generation | `src/commands/privileged_user_monitoring/README.md` | -| `rules` | Generate detection rules and events | `src/commands/rules/README.md` | -| `delete-rules` | Delete detection rules | `src/commands/rules/README.md` | -| `risk-engine ingest` | Generate and ingest risk-engine data in batches | `src/commands/risk_engine/README.md` | -| `esql-stress-test` | Stress test ESQL queries | `src/commands/risk_engine/README.md` | -| `painless-stress-test` | Stress test scripted metric risk scoring | `src/commands/risk_engine/README.md` | -| `create-risk-engine-data` | Build risk engine perf data file | `src/commands/risk_engine/README.md` | -| `create-risk-engine-dataset` | Build named risk engine perf datasets | `src/commands/risk_engine/README.md` | -| `upload-risk-engine-dataset` | Upload all files from a perf dataset directory | `src/commands/risk_engine/README.md` | -| `upload-risk-engine-data-interval` | Repeatedly upload risk engine data file | `src/commands/risk_engine/README.md` | -| `create-perf-data` | Create Entity Store perf JSONL data file | `src/commands/entity_store_perf/README.md` | -| `upload-perf-data` | Upload perf data once | `src/commands/entity_store_perf/README.md` | -| `upload-perf-data-interval` | Upload perf data repeatedly at intervals | `src/commands/entity_store_perf/README.md` | -| `create-baseline` | Extract and save baseline metrics from logs | `src/commands/baseline_metrics/README.md` | -| `list-baselines` | List saved baseline metric files | `src/commands/baseline_metrics/README.md` | -| `compare-metrics` | Compare a run against baseline metrics | `src/commands/baseline_metrics/README.md` | -| `generate-cloud-security-posture` (`csp`) | Generate CSP findings across sources | `src/commands/generate_cloud_security_posture/README.md` | -| `generate-correlated-organization-data` (`org-data`) | Generate correlated organization security integration data | `src/commands/org_data/` | -| `generate-correlated-organization-data-quick` (`org-data-quick`) | Quick correlated organization data generation with defaults | `src/commands/org_data/` | +| Command | Summary | Details | +| ---------------------------------------------------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------- | +| `generate-alerts` | Generate synthetic detection alerts | `src/commands/documents/README.md` | +| `generate-events` | Generate synthetic events in `eventIndex` | `src/commands/documents/README.md` | +| `generate-graph` | Generate fake graph data | `src/commands/documents/README.md` | +| `delete-alerts` | Delete all generated alerts | `src/commands/documents/README.md` | +| `delete-events` | Delete all generated events | `src/commands/documents/README.md` | +| `entity-resolution-demo` | Load entity resolution demo dataset | `src/commands/entity_store/README.md` | +| `entity-store` | Interactive Entity Store generation flow | `src/commands/entity_store/README.md` | +| `quick-entity-store` | Quick non-interactive Entity Store setup | `src/commands/entity_store/README.md` | +| `clean-entity-store` | Clean Entity Store data | `src/commands/entity_store/README.md` | +| `risk-score-v2` | End-to-end Entity Store V2: entities, alerts, risk engine, criticality, watchlists | `src/commands/entity_store/README.md` | +| `generate-entity-maintainers-data` | Entity Store V2 maintainer data: risk history, anomaly behaviours, relationships | `src/commands/entity_store/README.md` | +| `leads` | Generate AI-driven threat hunting leads (interactive, requires inference connector) | `src/commands/lead_generation/` | +| `test-risk-score` | Run risk score API test call | `src/commands/misc/README.md` | +| `generate-entity-ai-insights` | Generate vulnerabilities, misconfigurations and anomalous behavior for entities. | `src/commands/misc/README.md` | +| `generate-asset-criticality` | Generate asset criticality assignments | `src/commands/misc/README.md` | +| `generate-legacy-risk-score` | Install and generate legacy risk score data | `src/commands/misc/README.md` | +| `single-entity` | Create one entity with optional setup flows | `src/commands/misc/README.md` | +| `privileged-user-monitoring` | Interactive privileged user monitoring dataset generation | `src/commands/privileged_user_monitoring/README.md` | +| `privmon-quick` | Fast privileged user monitoring generation | `src/commands/privileged_user_monitoring/README.md` | +| `rules` | Generate detection rules and events | `src/commands/rules/README.md` | +| `delete-rules` | Delete detection rules | `src/commands/rules/README.md` | +| `risk-engine ingest` | Generate and ingest risk-engine data in batches | `src/commands/risk_engine/README.md` | +| `esql-stress-test` | Stress test ESQL queries | `src/commands/risk_engine/README.md` | +| `painless-stress-test` | Stress test scripted metric risk scoring | `src/commands/risk_engine/README.md` | +| `create-risk-engine-data` | Build risk engine perf data file | `src/commands/risk_engine/README.md` | +| `create-risk-engine-dataset` | Build named risk engine perf datasets | `src/commands/risk_engine/README.md` | +| `upload-risk-engine-dataset` | Upload all files from a perf dataset directory | `src/commands/risk_engine/README.md` | +| `upload-risk-engine-data-interval` | Repeatedly upload risk engine data file | `src/commands/risk_engine/README.md` | +| `create-perf-data` | Create Entity Store perf JSONL data file | `src/commands/entity_store_perf/README.md` | +| `upload-perf-data` | Upload perf data once | `src/commands/entity_store_perf/README.md` | +| `upload-perf-data-interval` | Upload perf data repeatedly at intervals | `src/commands/entity_store_perf/README.md` | +| `create-baseline` | Extract and save baseline metrics from logs | `src/commands/baseline_metrics/README.md` | +| `list-baselines` | List saved baseline metric files | `src/commands/baseline_metrics/README.md` | +| `compare-metrics` | Compare a run against baseline metrics | `src/commands/baseline_metrics/README.md` | +| `generate-cloud-security-posture` (`csp`) | Generate CSP findings across sources | `src/commands/generate_cloud_security_posture/README.md` | +| `generate-correlated-organization-data` (`org-data`) | Generate correlated organization security integration data | `src/commands/org_data/` | +| `generate-correlated-organization-data-quick` (`org-data-quick`) | Quick correlated organization data generation with defaults | `src/commands/org_data/` | ### Quick command list - **Documents** - `generate-alerts`, `generate-events`, `generate-graph`, `delete-alerts`, `delete-events` - **Entity Store** - - `entity-resolution-demo`, `entity-store`, `quick-entity-store`, `clean-entity-store` + - `entity-resolution-demo`, `entity-store`, `quick-entity-store`, `clean-entity-store`, `risk-score-v2`, `generate-entity-maintainers-data` - **Risk and Security utilities** - `test-risk-score`, `generate-entity-ai-insights`, `generate-asset-criticality`, `generate-legacy-risk-score`, `single-entity` - **Privileged User Monitoring** @@ -252,10 +255,25 @@ Detailed command documentation is colocated with command code under `src/command - `generate-cloud-security-posture` (`csp`) - **Correlated Organization Data** - `generate-correlated-organization-data` (`org-data`), `generate-correlated-organization-data-quick` (`org-data-quick`) +- **Threat hunting leads** + - `leads` -## Agent skills +## AI assistant skills -### `/update-org-data-integrations` +Skills are instruction files that give an AI assistant focused context for a specific task in this repo. Two formats are included: + +- **Claude Code skills** (`.claude/commands/`) — slash commands auto-discovered when the repo is open in [Claude Code](https://claude.ai/code). Type `/skill-name` to invoke. +- **Agent skills** (`.agents/skills/`) — reusable instruction files you can reference manually in any AI assistant that supports attaching context files. + +### `/seed-ea` (Claude Code) + +Helps you populate every Entity Analytics page in Kibana without needing to remember which command covers which page. Describe what you want to populate and the skill gives you the exact `yarn start` command — or runs it for you. + +Requires [Claude Code](https://claude.ai/code). See [SEED_EA_SKILL.md](SEED_EA_SKILL.md) for full documentation. + +Located at `.claude/commands/seed-ea.md`. + +### `/update-org-data-integrations` (agent skill) Updates existing integrations or creates new ones for the `org-data` command, using upstream field definitions and reference sample events from Elastic repos as the source of truth. Recommended to diff --git a/SEED_EA_SKILL.md b/SEED_EA_SKILL.md new file mode 100644 index 0000000..54776d9 --- /dev/null +++ b/SEED_EA_SKILL.md @@ -0,0 +1,108 @@ +# `/seed-ea` + +A [Claude Code](https://claude.ai/code) skill that helps you populate every Entity Analytics page in Kibana using `security-documents-generator` commands. + +Instead of remembering which command covers which page, you describe what you want and the skill gives you the exact `yarn start` command to run — or runs it for you. + +--- + +## Requirements + +- [Claude Code](https://claude.ai/code) installed — this skill uses the `.claude/commands/` format and is **Claude Code only**. It will not work in Cursor or other AI editors. +- This repo cloned locally and opened in Claude Code (the skill is auto-discovered from `.claude/commands/seed-ea.md`) +- A running Kibana instance with a valid `config.json` in the repo root ([see setup](README.md#configuration)) +- The correct Node version active — run `nvm use` from the repo root before invoking the skill. The repo includes an `.nvmrc` that pins the required version, so this avoids version mismatch errors at runtime. + +--- + +## Usage + +Type `/seed-ea` followed by what you want to populate. Claude Code will respond with the exact command(s) to run. + +### Examples + +``` +/seed-ea populate all EA pages end-to-end +``` + +``` +/seed-ea I want to populate the privileged user monitoring page +``` + +``` +/seed-ea seed host and user flyouts in a non-default space called my-space +``` + +``` +/seed-ea what command do I need for the AI summary panel? +``` + +``` +/seed-ea give me a quick host-only seed, entity store is already set up +``` + +``` +/seed-ea seed into my BC environment +``` + +> **Targeting a specific cluster:** the generator reads connection details from `config.json` in the repo root. If you want to seed a different environment, update `config.json` to point at the right cluster before running — or keep separate config files (e.g. `config.bc.json`) and swap them in as needed. + +--- + +## What it covers + +| Page | Commands used | +| -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | +| Home page (entities table, risk KPI/history, watchlists) | `risk-score-v2` | +| Entity store management | `risk-score-v2` | +| Asset criticality | `risk-score-v2` or `generate-asset-criticality` | +| Privileged user monitoring | `privmon-quick` | +| Threat hunting leads | `leads` | +| AI summary / anomalies | `generate-entity-ai-insights --v2 --correlate-with-entity-store` | +| Entity flyouts — host/user/service | `generate-entity-maintainers-data --quick` + `generate-entity-ai-insights --v2 --correlate-with-entity-store` | +| Entity flyout — generic | `quick-entity-store` | +| Explore pages (hosts/users/network) | `risk-score-v2` alerts + `generate-alerts` | +| CSP / cloud posture | `csp --data-sources elastic_all --csp-scores` | + +### Known gaps + +- **Generic flyout**: `risk-score-v2` doesn't seed generic entities — the skill will direct you to `quick-entity-store` instead +- **Threat hunting leads**: requires an inference connector configured in Kibana before the `leads` command can generate anything +- **Anomalies panel**: `generate-entity-ai-insights` seeds anomaly records for the UI but does not run real ML jobs + +--- + +## Full end-to-end sequence + +If you ask to populate everything, the skill recommends these commands in order: + +```bash +# 1. Core — entity store, risk engine, criticality, watchlists +yarn start risk-score-v2 --entity-kinds host,idp_user,local_user,service \ + --hosts 20 --users 20 --services 10 --alerts-per-entity 10 + +# 2. Risk history, relationships, anomaly behaviours (flyouts + home history panel) +yarn start generate-entity-maintainers-data --space default --quick + +# 3. Privileged user monitoring +yarn start privmon-quick --space default + +# 4. AI insights + anomaly records (ai_summary, entity flyout panels) +yarn start generate-entity-ai-insights --v2 --correlate-with-entity-store \ + -h 20 -u 20 -s default + +# 5. Threat hunting leads (inference connector required) +yarn start leads --space default +# → choose "Generate leads now" + +# 6. Generic entities (for the generic entity flyout) +yarn start quick-entity-store --space default +``` + +--- + +## How it works + +The skill is defined in `.claude/commands/seed-ea.md`. When Claude Code opens this repo, it auto-discovers the file and registers `/seed-ea` as an available slash command. The skill instructs Claude to map your request to the right generator command, explain what it populates, and optionally execute it for you from the repo root. + +No additional setup is needed beyond having Claude Code installed and the repo open.