Skip to content

Implement concurrent MadMatrix builds across subprocesses - #61

Open
Qubitol wants to merge 4 commits into
MadGraphTeam:mainfrom
Qubitol:madmatrix-multiprocess-build
Open

Implement concurrent MadMatrix builds across subprocesses#61
Qubitol wants to merge 4 commits into
MadGraphTeam:mainfrom
Qubitol:madmatrix-multiprocess-build

Conversation

@Qubitol

@Qubitol Qubitol commented Aug 10, 2026

Copy link
Copy Markdown
Member

With this PR, it is possible to compile concurrently the MadMatrix libraries across several subprocesses.

The number of concurrent processes is taken from the value of cpu_thread_pool_size in the run card.

Each subprocess is compiled using misc.compile with default nb_core=1.

Backend autodetection (in case of cppauto) and commonlib targets are still realised synchronously to avoid race conditions.

Logging happens in two ways on the basis of the value of verbosity from the run card:

  • pretty: re-uses the PrettyBox from MadSpace, updating it every time a new subprocess has finished compilation, and prevents ugly box updates if the number of subprocesses makes the box height larger than the viewport; if that happens, it will automatically contract into a summary box;
  • log: prints messages one after the other

@oliviermattelaer

Copy link
Copy Markdown
Contributor

I like it ... but we need to define a strategy for this.

This is unfortunate because I was going in the opposite direction ...
I did split get_amp into multiple file to speed up the compilation (especially in presence of helicity recycling where that part is huge) and also because crossing is decreasing the number of sub-processes...

@Qubitol

Qubitol commented Aug 11, 2026

Copy link
Copy Markdown
Member Author

Yeah, fair point.

Are we really going to reach a point in which CPPProcess.cc (or whatever file contains the HELAS) is going to not be the compilation bottleneck?
Because the idea behind this PR is that even when using make -j, at some point you have only one core dealing with CPPProcess.cc, so we can speed up a bit the compilation (especially GPU builds) by compiling all of these CPPProcess.cc together across different subprocesses.

I'm convinced that this would be useful even for O(10) subprocesses.

However, if there is no bottleneck anymore, then it's just better to use make -j and that's it, I suppose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants