You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With this PR, it is possible to compile concurrently the MadMatrix libraries across several subprocesses.
The number of concurrent processes is taken from the value of cpu_thread_pool_size in the run card.
Each subprocess is compiled using misc.compile with default nb_core=1.
Backend autodetection (in case of cppauto) and commonlib targets are still realised synchronously to avoid race conditions.
Logging happens in two ways on the basis of the value of verbosity from the run card:
pretty: re-uses the PrettyBox from MadSpace, updating it every time a new subprocess has finished compilation, and prevents ugly box updates if the number of subprocesses makes the box height larger than the viewport; if that happens, it will automatically contract into a summary box;
I like it ... but we need to define a strategy for this.
This is unfortunate because I was going in the opposite direction ...
I did split get_amp into multiple file to speed up the compilation (especially in presence of helicity recycling where that part is huge) and also because crossing is decreasing the number of sub-processes...
Are we really going to reach a point in which CPPProcess.cc (or whatever file contains the HELAS) is going to not be the compilation bottleneck?
Because the idea behind this PR is that even when using make -j, at some point you have only one core dealing with CPPProcess.cc, so we can speed up a bit the compilation (especially GPU builds) by compiling all of these CPPProcess.cc together across different subprocesses.
I'm convinced that this would be useful even for O(10) subprocesses.
However, if there is no bottleneck anymore, then it's just better to use make -j and that's it, I suppose.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
With this PR, it is possible to compile concurrently the MadMatrix libraries across several subprocesses.
The number of concurrent processes is taken from the value of
cpu_thread_pool_sizein the run card.Each subprocess is compiled using
misc.compilewith defaultnb_core=1.Backend autodetection (in case of
cppauto) andcommonlibtargets are still realised synchronously to avoid race conditions.Logging happens in two ways on the basis of the value of
verbosityfrom the run card:pretty: re-uses thePrettyBoxfrom MadSpace, updating it every time a new subprocess has finished compilation, and prevents ugly box updates if the number of subprocesses makes the box height larger than the viewport; if that happens, it will automatically contract into a summary box;log: prints messages one after the other