You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Don't serve the unauthenticated REST API, including job cancel, on the scheduler's gRPC port by default #2746
Is your feature request related to a problem or challenge?
With the default rest-api feature, start_grpc_service merges the REST routes into the same axum router as the scheduler's gRPC services, so they are served on the scheduler's gRPC port (bind_host:bind_port). The API has no authentication, and disable_rest_api defaults to false. Besides read-only routes such as /api/jobs, /api/executors and the job plans, it serves PATCH /api/job/{job_id}, which cancels a job.
Every executor has to reach that port to register and report task status, so the port is normally open to the whole cluster network, and often to more than that in Kubernetes. Anything that can reach it can list every job with its plan, and cancel any job, over plain HTTP. Apache DataFusion Comet hit this in review of apache/datafusion-comet#7420 and now sets disable_rest_api: true in its scheduler.
Describe the solution you'd like
Some options, which could be combined:
Serve the REST API on its own listener, with its own rest_api_bind_host / rest_api_port. The bind host would default to 127.0.0.1, so exposing it is an explicit choice and it can be firewalled separately from the gRPC port that executors need.
Make the job-cancel route opt-in, for example rest_api_allow_cancel, so the default API is read-only.
Default disable_rest_api to true for the library's SchedulerConfig::default(), keeping it enabled in the ballista-scheduler binary through its CLI default, so embedders opt in.
Allow an authentication hook (for example a tower layer supplied through SchedulerConfig) for deployments that do expose it.
Option 1 alone keeps today's behavior for anyone running the scheduler locally, and removes the exposure on the port executors use.
Describe alternatives you've considered
Documenting that rest-api should be disabled, or the port firewalled, in untrusted networks. That doesn't help the common case, because executors need the same port.
Additional context
Route setup: ballista/scheduler/src/scheduler_process.rs, start_grpc_service. Routes: ballista/scheduler/src/api/routes.rs. The executor serves only /healthz and /readyz, so this concerns only the scheduler.
Activity