breaking/refactor: Split read/write tenancy model

[saswatamcode]

This refactor reimagines the gateway's authentication and authorization model, replacing a complex RBAC/OPA system with a simpler, more flexible approach centered on mTLS authentication and tenant-based query filtering.

What changed:

• RBAC/OPA authorization removed entirely
• Label enforcers reinstated but driven by tenant headers instead of RBAC roles
• Write authentication: mTLS with tenant extracted from certificate OU field
• Read authentication: SSO/OIDC or mTLS with tenant specified via HTTP headers
• Multi-tenant queries now supported (e.g., query team-a|team-b|team-c data simultaneously)
• URL structure simplified (tenant removed from paths)

What it enables:

• Machine writers: Certificate-based authentication with automatic tenant extraction
• Human readers: Grafana-driven queries across multiple tenants with single datasource
• Simplified operations: No RBAC YAML configuration required
• Query-time tenant isolation: Labels automatically injected based on tenant headers

Motivation

The previous RBAC model required:

• Complex per-tenant YAML configuration with role bindings and permissions
• OPA policy engine for authorization decisions at query time
• Tenant name embedded in URL paths (/api/{signal}/v1/{tenant}/...)
• Label enforcers that consulted OPA to determine which labels to inject
• Separate authorization checks for each request

The new model simplifies this to:

• Machine writers: mTLS with tenant in certificate OU field
• Human readers: SSO/mTLS with tenant(s) in HTTP headers (set by Grafana)
• Label enforcers driven directly by tenant headers (no OPA lookup required)
• No RBAC or OPA required
• Cleaner URLs without tenant in path

Architecture Changes

Before (Old RBAC Model)

Request with tenant in URL: /api/logs/v1/team-alpha/query
  ↓
Extract tenant from URL path
  ↓
Authenticate user (SSO/mTLS)
  ↓
Load RBAC roles for user + tenant
  ↓
Call OPA with roles to get label matchers
  ↓
Label enforcer injects matchers based on OPA response
  ↓
Query modified: {app="web"} → {app="web",namespace="prod"}
  ↓
Proxy to upstream

After (New Tenant-Based Model)

Request with tenant in header: X-Scope-OrgID: team-alpha
  ↓
Extract tenant from header → context
  ↓
Authenticate user (SSO/mTLS)
  ↓
Convert tenant to label matcher (no OPA)
  ↓
Label enforcer injects tenant matcher
  ↓
Query modified: {app="web"} → {app="web",tenant_id="team-alpha"}
  ↓
Proxy to upstream

Multi-Tenant Support (New Feature)

Request with multiple tenants: X-Scope-OrgID: team-a|team-b|team-c
  ↓
Extract tenant string → context
  ↓
Authenticate user (SSO/mTLS)
  ↓
Convert to regex matcher: {tenant_id=~"team-a|team-b|team-c"}
  ↓
Label enforcer injects matcher
  ↓
Query modified: {app="web"} →
{app="web",tenant_id=~"team-a|team-b|team-c"}
  ↓
Proxy to upstream (returns data for all 3 tenants)

[squat]

I think there is a bit of a convenient misrepresentation of the complexity of the current state of authz in Observatorium. Notably, RBAC has never required OPA and in fact using OPA for RBAC is kind of an advanced feature where you can encode the concept of roles into policies that OPA can enforce for you, bypassing Observatorium's native RBAC handling entirely. They are two distinct packages with complete and independent implementations of the Observatorium authorization interfaces. In other words, the Before graph is wrong.

I'm also suspicious of the claim that mTLS-based authorization is in any way "simple": TLS certificates are incredibly easy to get wrong and implementing authn, let alone authz, on top of TLS is a very common source of security holes.

[saswatamcode]

@squat apologies, I misread GitHub UI, did not mean to raise this upstream or "slopbomb" reviewers. 😓

This branch was meant for an internal impl, with certain specialized RBAC needs, not the general case.
I agree it sort of misrepresents things in terms of simplicity. For usual deployments of Observatorium API, I think the current model actually works really well and is convenient to admin. This branch is more meant for a multicluster product, where admins won't necessarily be aware of tenancy primitives, as they'll be more automatic in nature.

I'll close this and raise it on appropriate fork. Sorry for the noise