The onion-validator repository implements validation logic that checks raw observations from upstream components (crawler, fingerprint, extractor, etc.) for correctness, completeness, and security policy compliance before they are stored or processed further.
Planning – only a placeholder README existed. No code has been added yet.
Separating validation from data collection enables a clean pipeline where each stage can focus on its core responsibility. Validators can be updated independently without impacting crawlers or the intelligence engine.
- Library – imported by
onion-crawler,tech-fingerprint,entity-extractor,change-detector, and ultimately byonion-intelligencevia the SDK. - Provides functions such as
ValidateObservation(obs OnionObservation) errorandSanitizeHeaders(headers map[string]string) map[string]string.
- Schema validation for
OnionObservation(required fields, data types). - Security‑policy checks (e.g., disallowing known vulnerable TLS ciphers, leaking IPs).
- Normalisation helpers (canonical URL, timestamp formatting).
- Configurable rule sets via a YAML file.
# Go (primary)
go get github.com/FLATLINEDSTAR/onion-validatorThe package will be published after the MVP is complete.
- Language: Go.
- Follow contribution standards in the organization
.githubfolder.
- Unit tests for each validator function using table‑driven test cases.
- Fuzz tests for the sanitisation utilities.
Refer to the organization‑wide CONTRIBUTING.md.
- Phase 1 – Define validation interfaces in
onion-sdkand implement core validators. - Phase 2 – Add configurable rule engine and integration tests with upstream components.
- Phase 3 – Publish the package and add CI checks.
Validators are a critical middleware that ensures data quality before it reaches the intelligence engine (onion-intelligence).
MIT – see LICENSE in the repository root.