Skip to content

Scheduler runs without event logs, and only warns, when it can't write event_log_dir #2766

Closed

Description

@andygrove

Describe the bug

When the scheduler can't write to event_log_dir, it starts and runs queries normally, writes no event logs, and only logs warnings. EventLogWriter::new (ballista/history/src/writer.rs) creates the directory in run, on the task it spawns. When that fails, run logs a warning and returns, which closes the writer's channel, so every later event is dropped with another warning. A directory that exists but isn't writable, such as a volume mounted with another owner, gets past create_dir_all, and then each job's log fails to open with event-log writer: cannot open ....

To Reproduce

Set event_log_dir (--event-log-dir) to a path inside a regular file and run a query. The query succeeds, and the scheduler logs (path shortened):

WARN  ballista_history::writer] event-log writer: cannot create /var/folders/.../comet-scratch-80543/logs: Not a directory (os error 20)
WARN  ballista_history::writer] event-log writer: dropping event for 038D3AT922000 (channel full or closed)
WARN  ballista_history::writer] event-log writer: dropping event for 038D3AT922000 (channel full or closed)
WARN  ballista_history::writer] event-log writer: failed to enqueue terminal event for 038D3AT922000 (channel closed)
WARN  ballista_history::writer] event-log writer: failed to enqueue finish for 038D3AT922000 (channel closed)

Expected behavior

The scheduler fails to start, with an error naming the directory: an operator who sets event_log_dir expects logs. For example, SchedulerServer could create the directory and check that it can create a file there before it starts the writer, or EventLogWriter::new could do that and return a Result.

Additional context

Seen with 55.0.0, through Apache DataFusion Comet's scheduler binary, which embeds Ballista's scheduler; main at 70a0743 has the same code. Comet's binary now checks the directory itself before it starts the scheduler (apache/datafusion-comet#7420), and could drop that check once Ballista does it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions