If the directory "spark.history.fs.logDirectory" does not exist, the history server will fail to start. Automatically creating this directory can reduce errors.
Similarly, if the spark.eventLog.dir does not exist, the operation will also fail, even if it is written to S3.
If the directory "spark.history.fs.logDirectory" does not exist, the history server will fail to start. Automatically creating this directory can reduce errors.
Similarly, if the spark.eventLog.dir does not exist, the operation will also fail, even if it is written to S3.