Spark eks job cancel + small janitor query modification

[wlggraham]

PR Details

Description

This PR enables cleanup in sparkeks plugin. Because the Cleanup() function does not include a Runtime object in the signature, some of the functions had to be modified slightly, the main one being where the kubeconfig file would be created.

The stale query modification prevents a race condition where the janitor can actually move a job from 'canceling' to 'canceled' before the go context actually cancels context. This can lead to the new "failed" status overwriting the 'canceling' status. The query was tested and still only takes ~2-3ms.

Types of changes

• Docs change / refactoring / dependency upgrade
• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation and I have updated the documentation accordingly.
• I have added tests to cover my changes.

[hladush]

Add metrics and LGTM

[wlggraham]

Add metrics and LGTM

The janitor process has a bunch of metrics that will report any failures that happen within the cleanup function, so I think we're good.