bypass java

[martindurant]

The YARN REST API appears complete enough that, if we can use hdfs3 to manage file-based resources (i.e., a minimal python) and create tokens, the client and AM could both be completely python, but still be able to successfully manage delegation tokens (including their renewal)

[mrocklin]

This would be great. Was this added recently?

[martindurant]

Yes, this is a new idea. I don't yet know how much effort it would be, but certainly a rewrite of the current code. For the client portion, probably not too hard, and the AppMaster can maybe stay scala, since it necessarily lives on a cluster node.
The critical question is whether the kerberos support in hdfs3 is good enough for all circumstances. It might be simpler than trying to write a token-renewer service in scala. Simpler for me, that is.
If doable, it would also enable starting yarn jobs from off-cluster (if hdfs/kerberos is configured to allow that). Containers could probably not reach a dask scheduler running outside the cluster, so this idea raises a few open questions.

I should probably mark this issue as "abstract thought"

[mrocklin]

Lets ping @quasiben and @mariusvniekerk

[martindurant]

The protobuf rcp defs that would need to be used by a python AM.

[martindurant]

More details on the phases of this general goal, each with increasing uncertainty:

• the current implementation calls the YARN CLI executable in a few places; these can certainly be replaced by HTTP calls, or, for logs-after-completion, directly on hdfs
• the client java process currently launches and stays active; it should be closed after launching the AM, since the functions can either be done by HTTP (get status, kill) or by talking to the AM (add/remove containers)
• the client process basically just instructs YARN to launch the AM; this can be done by HTTP with a JSON blob giving the AM spec, and hdfs3 upload of files. As I understand, this will work on unsecured clusters, or if writable HTTP access has been enabled (don't know if this is typical). Note that, if authorised, can access tokens, including renewal, over HTTP, and tokens give access to HDFS. Also, HDFS tokens probably give same access and hdfs3 can make them if its own SASL problems are ironed out.
• the AM could be python rather than java, but it need to talk protobuf RCP as opposed to HTTP: so the SASL mechanism on the RCP channel needs to implement all possibly security that we might come across. If that auth mechanism is "token", perhaps it's simple?