Runtime CPU+GPU monitoring, multi-config benchmark matrix, GPU selection, presets, and MPS control#61
Conversation
|
I tested the script on my side and it seems to work as expected.
|
|
There seems to be an issue when running with multiple configurations. In my case: correctly processes the first configuration, but fails before starting the second: Running the second configuration separately works as expected. EDIT: Running the configuration a second time I get a different kind of error once the second configuration starts: |
MPS is used for cases where the jobs are not evenly distributed on GPUs. For example, if 7 jobs are sent to 2 GPUs, one of them will end up with 4 jobs (MPS 25%) and the other with 3 (MPS 33%). In this case the automatic label will be
The reported value is correct in the sense that the same rounding is applied to the value set internally. The result of the division is floored to the nearest integer (e.g. 6.25 -> 6) since I've not found references to MPS supporting floating point values and the actual SM attribution is quantized in any case (updated the PR description to include this info)
Looking into it |
Parsifal-2045
force-pushed
the
devMonitorGPU
branch
from
bc49938 to
2426d81
Compare
The latest force-push should have fixed all these errors. The root cause was in the way the parsing of each config was done.
I have folded the fix in the first commit, where multi-config parsing was introduced.
|
|
@Parsifal-2045 I confirm that the new changes solve the issue. |
|
@Parsifal-2045 thanks for all the developments and improvements ! For now I have a few comments about the interface, later I'll try to go through the changes.
I have a mild preference for a syntax that is a bit more explicit, for example
Is this the original file given on the command line, or the preprocessed file passed to I guess both could have a use: what one is asking to run, and what is effectively running.
I would prefer to keep the name
How about
Does anything rely on the old behaviour?
I have a mild preference for What happens if the job does not use any NVIDIA GPUs ? Also for completeness can you add a
?
I think we should round to the ceiling, e.g. 100/16 = 6.25 -> 7.
I like the idea, with some suggestions:
Please keep the old default, one can use |
|
Thanks for the comments. I'll address most of the interface-related ones in follow-up commits. Let me also answer to the more direct questions.
Right now the original file is copied, to preserve exactly the configuration that was passed as argument to
Not that I know of, but I stayed on the side of caution here. I have no issue in using basic monitoring for both CPU and GPU as default everywhere
Will change the name of the parameter and include the corresponding then the job goes on as normal using the available GPUs
Fair point, I was undecided on whether to floor or ceil the division, will implement ceiling :)
I will take another look at naming and expansion of parameters to try and incorporate all the suggestions. Currently any job that does no use MPS still gets a noMPS tag at the end (so the name of the directory either ends with
Just to be clear, you'd prefer the following behaviour:
Do I understand correctly? |
The downside of this approach is that it may not capture the whole configuration. |
Unless there is any evidence that it adversely affect performance, I would say let's do that. |
I see, that's something I had not considered. We can maybe copy both the original input and the fully customized dump to cover all cases? |
That would make sense. Maybe keep the original name for the file passed as an argument, and use something like |
I think that's what I wrote, but not what I meant :-D Let me amend my proposal to
|
OK... if you manage, I would suggest that a job that does not use NVIDIA GPUs (either CPU only or using only AMD GPUs) should not get any MPS field at all. |
I wanted to do something similar for CPU-only jobs not getting the |
…port, templated logs
- accept several config files and a --setup list (explicit "j=J,t=T,s=S"
fields, any order, omitted fields auto-derived), running the configs x
setups matrix with unique log directories and CSVs
- add --gpus to restrict the run to a subset of GPUs (plain list or
per-vendor form): it filters the visible devices so the automatic
affinity still assigns one GPU per job, only across the selection
- make --logdir templated and on by default: no --logdir (or a bare
--logdir) uses the automatic template, --logdir DIR uses DIR, and
--no-logdir (or --logdir '') disables logs. The placeholders
%config/%j/%t/%s/%gpj/%gpus are documented in a dedicated "LOGDIR
TEMPLATE" help section; %gpus is optional and drops out (with its
separator) when no GPU is used (also parsing the configuration's
accelerators)
- add --output-log to tee the full console output (from the system
overview and input job report onward) to <logdir>/output.log
- generate the input job report automatically with --input-xml auto and
build the trimmed input-only process once per configuration
- copy each run's configuration file into its log directory, together
with the fully-expanded dump that is actually run (<config>_dump.py)
- auto-merge Phase2Timing_resources.json
- parse each configuration in its own subprocess and load back only its
fully-expanded dumpPython(): HLT menus apply their era / ProcessModifier
customisations at import time, and CMSSW forbids a later cms.Process
from choosing modifiers the first one did not, so parsing several
configs in a single interpreter would abort ("tried to redefine which
Modifiers to use after another Process was already started") or leak the
first config's import-time modifier state into the next;
loadModuleFromFile moves to common.py as the single loader used by both
the subprocess and the parent
- sample aggregate host-memory RSS and per-GPU utilization and memory (optionally power and temperature) at one sample per second, through auto-selected nvidia-smi / amd-smi backends - monitor the GPUs the jobs can use: detect the devices and restrict them to the CUDA_VISIBLE_DEVICES / HIP_VISIBLE_DEVICES selection, honoring --gpus - control the detail with --monitor-gpu and --monitor-host (none/basic/full, both default basic) - write the full-run cpu_monitor.csv and gpu_monitor.csv at <logdir> top level and fold each step's numpy arrays into its monit.py (drained per step, so even an indefinite run stays memory-bounded), extending the per-process host-memory monitoring already there - provide a line-by-line CSV streamer and a shared RunMonitor (start/finish plus per-step snapshot) so a caller can span several runs into one continuous series that survives an interruption or crash
- apply a named set of default options before parsing, so any preset value can still be overridden explicitly; the presets live in an extensible registry - include hltRun3 and hltPhase2 for the "standard" HLT timing setups
- with --nvidia-mps, start the MPS control daemon (or attach to a running one) and set each job's active-thread percentage from how many jobs share its GPU. By default it rounds up to the nearest integer: ceil(100 / jobs-per-GPU). Otherwise the acrive thread percentage can be set to a specific value by passing --nvidia-mps VALUE. Restore or stop the daemon on exit and on Ctrl+C - derive the per-slot percentages from the actual job-to-GPU assignment, so it works with both --slot and the automatic affinity - start MPS only when an NVIDIA GPU is actually used; otherwise warn and skip it. Add --no-nvidia-mps to require MPS off: never start it, and exit with an error if a control daemon is already running - record the applied MPS setting in each run's automatic log directory and CSV name through a %mps placeholder (e.g. ..._MPS50, ..._MPS25-34 when different GPUs run at different percentages, or ..._noMPS when off); the MPS tag drops out entirely when no NVIDIA GPU is used - share the multiCmsRun options through a single build_options helper used by benchmark, scan and multirun - also give actionable errors when a variadic option swallows the config, close per-run output files on every exit path, and fix a makedirs typo on the EvFDaqDirector path
Parsifal-2045
force-pushed
the
devMonitorGPU
branch
from
2426d81 to
956ad40
Compare
|
The latest force push implements all the changes discussed. In particular (again, ~ per commit):
|
Extend the host monitoring added in "Add unified multi-vendor CPU+GPU runtime monitoring" to report CPU utilization alongside memory, and make it independent of GPU monitoring. - rename HostMemoryInfo to HostMonitorInfo: BASIC now samples CPU utilization (psutil cpu_percent), VSS, and RSS; FULL adds USS and PSS. Both the per-process (singleCmsRun) and the aggregate (monitorResources) samplers carry the new column as well as the cpu_monitor.csv, per-step monit.py and hardware summary - fixed a bug where cpu_monitor.csv and the hardware summary were coupled to GPU monitoring. Now cpu_monitor.csv and the CPU hardware summary are produced whenever --monitor-host is on, even with --monitor-gpu none or on a GPU-less node. --monitor-gpu now governs only the GPU metrics, and gpu_monitor.csv is written only when --monitor-gpu is set to basic or full
|
The latest commit adds CPU frequency monitoring via |
3 participants
Overview
This PR reworks the benchmark/scan workflow and adds device-level runtime monitoring so a single benchmark invocation produces the throughput, resource-usage, and log artifacts previously only available via external scripts.
The developments build on the input-only benchmark (#57) and configurable host-memory monitoring (#58) and, although the scope of this PR is quite large, I've tried to keep the changes organized as 4 self-contained commits.
Edited 07/07 addressing the first batch of comments #61 (comment)
Few more features and fixes introduced in a follow-up commit, see #61 (comment)
Main developments (more or less per commit)
Benchmark workflow: multi-config matrix,
--gpus, input auto-report, templated logs, several config files--setup J,T,S--setup j=J,t=T,s=S(in any order and with omitted fields automatically deduced using the existing logic) in conjunction with the possibility to list multiple configs to benchmark, runs the full configs x setup matrix saving the results of each run under a unique log dir / CSV (deduped, never overwriting)--gpusrestricts to a GPU subset (plain list ornvidia=..:amd=..) by filtering the visible devices so affinity still assigns one GPU per job across the selection (warns if the selection matches no device)--logdirwithout any additional arguments now automatically produces a directory for the output logs created following the default templatelogs.%config.%jobs.%threads.%streams.%gpuslogs_%config_%jobs_%threads_%streams(_%gpus)with the%gpustag omitted in case the job does not use any GPU. The user can also specify custom names for the output directory taking advantage of the same expansions (documented in the--helpmessage)--output-logtees the full console output to<logdir>/output.log--input-xml auto(+--input-xml-events) generates the job report and builds the trimmed input-only process once per configAuto-merged resources (resources.json/Phase2Timing_resources.json) are named after the run's log directory (e.g.gpu_16j_16t_16s_allGPUs.json)Unified multi-vendor CPU+GPU runtime monitoring
nvidia-smi/amd-smi, for the GPUs the jobs can use (detected devices restricted toCUDA_VISIBLE_DEVICES/HIP_VISIBLE_DEVICES, honoring--gpus)--gpu-monitoringand--host-memory-monitoring--monitor-hostand--monitor-gpu(none/basic/full, both basic by default). A single monitor spans a whole benchmark run streaming one continuouscpu_monitor.csv+gpu_monitor.csvto<logdir>line by line (so they survive an interrupt and a hard crash/SIGKILL), and folding each step's numpy arrays into<logdir>/step*/monit.py(extending the pre-existing per-process host-memory arrays). Finally a new "hardware usage summary" is printed to standard output once the benchmark finishes processing (CPU/GPU max/mean for all reported metrics). The new GPU monitor reuses the CPU-monitoring pattern. Indefinite runs (-r 0) fall back to each run's own monitor, drained per step intomonit.pyso memory stays bounded. AcmsRunjob failure is handled gracefully (measurement skipped) and each run's own monitor is flushed, so no termination loses the collected data. The CPU-only path remains identical to the previous implementation when (--gpu-monitoring none)--preset NAMEhltRun3(10300 ev, skip 300, res 100, 8j/32t/24s, wait 0,--input-collections rawDataCollector) andhltPhase2(1000 ev, skip 100, res 25, 16/16/16, wait 30,--input-xml auto). Applied before parsing so any value can be overridden explicitly.Automatic MPS setup/usage
--nvidia-mpsstarts/attaches the NVIDIA MPS daemon and set each job's active-thread % from how many jobs share its GPU (or an explicit value)flooringceiling100 / number_of_jobs_per_gputo the nearest integer (e.g. 100/16 = 6.25 ->67)--slotand the automatic affinity<logdir>, csv and merged resources) to include the MPS status or active thread percentage at the end (noMPS,MPS<VALUE>orMPS<RANGE>).MPS<RANGE>is used in cases where jobs are not equally distributed across GPUs and provides the min-max range of active thread percentages set (e.g. 7 jobs split across 2 GPUs -> 4 jobs on one GPU (MPS 25%), 3 jobs on the other (34%),MPS<RANGE>will beMPS25-34). The%mpstag is added to the expandable parameters that can be passed to--logdirand is automatically dropped when no NVIDIA GPU is available--no-nvidia-mpsflag that does not start the MPS daemon and errors out of the job in case a daemon is running alreadybuild_optionshelper (single source for themultiCmsRunoption dict across benchmark/scan/multirun), actionable errors when a variadic option swallows the config, per-run output-file closing, and a couple of typo fixesExamples
1. Phase-2 timing job (with reduced setup and full GPU monitoring)
Console output:
Resulting log directory:
gpu_monitor.csvformat2. A configurations × setups matrix, restricted to a subset of GPUs
runs the 2 configs × 2 setups matrix on GPUs 0 and 1, producing four independent log directories (each with its own CSVs,
output.log, config copy and per-step data, no%gpusand%mpstags in the logs forcpu_config.py); the monitor reports only GPUs 0 and 1:3. Sharing one GPU between two slots via MPS
Both jobs share GPU 0, so each is capped at 50%; with one job per GPU it would be 100%.
4. MPS with uneven job division across GPUs
Behaviour changes and backward compatibility
--logdiror--logdirwith no argument now creates a<logdir>named using the default patternlogs_%config_%jobs_%threads_%streams(_%gpus)(_%mps) (waslogs, still disabled by passing either--logdir ''or--no-logdir). Users can also pass a custom name--logdir DIRusing the same expandable parameters (documented in a new section at the end of the--help` message--monitor-gpudefaults to basic: every GPU-node run starts the monitor and prints the summary (--monitor-gpu noneopts out)Validation
I have run extensive tests on both NVIDIA and AMD hardware, trying to stress-test the new configuration options and setup. I was able to reproduce results obtained with an external monitoring script. The logging behaviour seems to be consistent regardless of the available hardware/setup and the automatic MPS setup sets the active thread percentage to the expected value regardless of the job division across GPUs.