You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Scheduler runs without event logs, and only warns, when it can't write event_log_dir #2766
When the scheduler can't write to event_log_dir, it starts and runs queries normally, writes no event logs, and only logs warnings. EventLogWriter::new (ballista/history/src/writer.rs) creates the directory in run, on the task it spawns. When that fails, run logs a warning and returns, which closes the writer's channel, so every later event is dropped with another warning. A directory that exists but isn't writable, such as a volume mounted with another owner, gets past create_dir_all, and then each job's log fails to open with event-log writer: cannot open ....
To Reproduce
Set event_log_dir (--event-log-dir) to a path inside a regular file and run a query. The query succeeds, and the scheduler logs (path shortened):
WARN ballista_history::writer] event-log writer: cannot create /var/folders/.../comet-scratch-80543/logs: Not a directory (os error 20)
WARN ballista_history::writer] event-log writer: dropping event for 038D3AT922000 (channel full or closed)
WARN ballista_history::writer] event-log writer: dropping event for 038D3AT922000 (channel full or closed)
WARN ballista_history::writer] event-log writer: failed to enqueue terminal event for 038D3AT922000 (channel closed)
WARN ballista_history::writer] event-log writer: failed to enqueue finish for 038D3AT922000 (channel closed)
Expected behavior
The scheduler fails to start, with an error naming the directory: an operator who sets event_log_dir expects logs. For example, SchedulerServer could create the directory and check that it can create a file there before it starts the writer, or EventLogWriter::new could do that and return a Result.
Additional context
Seen with 55.0.0, through Apache DataFusion Comet's scheduler binary, which embeds Ballista's scheduler; main at 70a0743 has the same code. Comet's binary now checks the directory itself before it starts the scheduler (apache/datafusion-comet#7420), and could drop that check once Ballista does it.
Activity