Worker managed cpu utilization

[Dym03]

This PR adds a option to show utilization with distinguished cpus that are used by running pinned tasks that are assigned to a worker. Without the pinning the feature doesn't work well now.
Currently the change of color scheme is used to distinguish the cpus. But maybe the graying out of the other cpus might be better.

Keybindings added is 'c' that enables the user to toggle between global and worker specific cpu usage.

Features that might be connected with this:

• Add this cpu info into the hq worker info command
• If multiple tasks are runned, there could be a specific mode to filter the running tasks further to show the cpus only for specific task/job
• Update the cpu utilization history, to work with the global / worker specific information

This is part of the feature requested in #1016

[Kobzol]

Hi, thank you for the PR! I haven't checked the code in detail yet, but at a glance it looks fine.

I'll answer the questions from #1016 (comment) here:

1. I don't think we need to deal with pinning in any special way. One of the use-cases for this feature is actually to let users (visually) know that their programs might not be using the right cores. If they don't use pinning properly, they will see that the CPUs being utilized do not correspond to the ones assigned to the task. So it is more of a "detect that I do pinning wrong" feature rather than "only show this if pinning is enabled" feature. The visualization not corresponding to task utilization is a feature, not a bug :)

2. I don't think so.

3. Good question :) This is related to what resources do we actually show; global ones or only those assigned to the worker (see below for more discussion)? To avoid dealing with too many things at once, let's assume that we only deal with "all workers CPUs" vs "all CPUs assigned to tasks" for the moment. I think that we have a few options there:

   • Show only the assigned task CPU utilization
   • Show both assigned and total CPU utilization (but here we might have to rethink the UI layout, the chart title is becoming a bit cramped)
   • Switch between total worker and assigned CPU utilization with a button, as you have now

   Not sure if I like the button switch. While it is useful to see both utilization regimes, it is also not very discoverable; when I tried your PR, I was confused by why doesn't it react to assigned tasks, before I realized that I have to press C. At the very least, the assigned task utilization should be the default, IMO.

4. I would personally prefer graying out the unassigned tasks (or having three levels - color for unmanaged CPUs, color for unassigned CPUs, and then green/yellow/red for assigned CPUs). In the current scheme it's not immediately obvious to me that the blue parts are not that important. But I'd have to see how it looks :)

There is also kind of a second level of utilization that we could somehow show visually - which CPUs are assigned to the given worker. Because right now we show all CPUs of the given PC, and the CPUs assigned to tasks, but in theory the worker might be managing only a subset of the CPUs (though that is probably relative rare? hard to tell). In fact, maybe we should just get rid of "global" CPU utilization and always just deal with the assigned resources (at least for CPUs, for memory it's more complicated).

It could be useful follow-up work, to 1) only show the worker assigned CPUs here, and also to only gather utilization for those CPUs in the worker collection loop.

Another follow-up could be to separate memory utilization across tasks (based on the memory utilization of their processes), but that can be very tricky to pull off, as tasks can spawn subprocesses, etc.

CC @spirali about the color scheme (blue vs graying out unassigned CPUs) and if you think that we should show all CPUs on the worker, or only those managed by the worker. The latter would be more consistent with how we treat non-CPU resources.

[spirali]

Majority of our users probably have whole node, but I am aware of some users that do subnode SLURM allocation. AFAIK they do not usually have more workers on a single node, they just have a single worker on a node that manages just subset of the node.

Graying out non-managed resources seems good to me. But it should be always possible to see non-managed CPUs (or ideally all node resources). There are usually two use cases:

• "I want to know what is utilization of my tasks", then I usually want to know only managed resources.
• "I want to know what is happening with the HW/node", then I usually want to see all information.

Let us say that will eventually have some "top"-like utility. For the first question, we want to only processes spawned by our task. For the second question we want to see all processes.

So we should probably support both views. I have no strong opinion on what is a good default.

[Kobzol]

Well, the thing is that knowing all resources of the node is not really something that we can robustly know. Currently, we sorta try to guess it for CPUs, but even there it might not be accurate. And for GPUs and other resources, we often might only have access to a subset of the resources.

I think that from our point of view, we cannot reliably saw what is the "whole node", and should mostly talk only about resources managed by HQ workers. What we could do though is to say something like "we detected N additional CPUs on the node" or something.

[spirali]

I wrote in "ideal case"; I know that we cannot provide "generic HW monitoring tool", but other CPUs are quite easy, so if we can provide them than I think that we should show them.

[Kobzol]

but other CPUs are quite easy

I guess that depends on how SLURM is configured, it is definitely possible to hide some CPUs from HyperQueue.

But yeah, I guess that we can show all CPUs (that are visible to us), especially since that we already do this today 😆 Just that I would treat it more as an auxiliary information rather than the main thing that we want to present.

[Dym03]

I do agree, that not showing the worker assigned cpus utilization as default is not ideal, but i wanted to keep the original intention, and this as an added feature. But as you mention above, I agree that showing it as a default makes more sense, and opting out to see "all cpus" could be a alternative.

Also thinking about it now, if the user wants to see how the whole node/(all cpus in the list) are doing, they surely can do it without the fancy colors. So maybe even keep it simple without the switch might be a good option.

So i will change the color scheme to classic green->red to assigned cpus, and graying out the other cpus. Set it as a default. If you come to a final decision about the possibility to switch between all cpus <> assigned cpus, i can remove it or keep it as is.

I can try to brainstorm a bit to come up with new layout to represent the usage statistics. Alternatively we can discuss it when your schedule frees up.

[Kobzol]

👍 on all you said. Regarding

to switch between all cpus <> assigned cpus, i can remove it or keep it as is.

you can keep it, but please make the assigned CPUs view be the default.

[spirali]

I vote for keeping "assigned only" view, or at least sorting assigned cpus at the beginning. The original motivation for all this was when I need to find utilization of "my" 4 cpus on 256+ cpus machine.