Run spec tests in parallel to reduce the execution time#8088
Conversation
stevenfontanella
force-pushed
the
multithread
branch
3 times, most recently
from
44c5834 to
62b613b
Compare
stevenfontanella
changed the title
stevenfontanella
force-pushed
the
multithread
branch
from
62b613b to
bb0d612
Compare
|
The alpine builder has been running for over three hours now, so I think there's a problem here. Looking at the log, it looks much more verbose than before because the stdout from the wasm-shell commands is being printed. Maybe fixing that will make the builder go faster? |
|
Re:
I changed |
|
I think the CI problem is that this member isn't initialized in the default constructor of SmallVector since std::array is an aggregate type. I'm not sure why we see the breakage here and not in main or in other architectures. I'll try to re-run for now and send a PR to fix separately. |
|
About the |
…8088) Reduces runtime from ~15 minutes to 1.5 minutes on my machine * Run tests through a thread pool with `os.cpu_count()` threads * `os.cpu_count() * 4` shows no benefit. `os.cpu_count() // 2` also shows no regression in runtime, but might be worse for machines with less cores. There are currently 315 spec tests total for reference. * Prefixes round-trip file name tests with their test name to avoid clobbering the a.wasm / ab.wast files during tests * Add stdout and stderr params to functions that print so that lines can be captured by each thread and not interleaved * Note that we pass stdout as the stderr param in practice so that they are interleaved, otherwise all stdout lines and stderr lines will be outputted together in each test.
sbc100
left a comment
sbc100
left a comment
There was a problem hiding this comment.
I just came here to say things change is awesome. Why didn't we do this earlier!
| # Hack to allow subprocess.Popen with stdout/stderr to StringIO, which doesn't have a fileno and doesn't work otherwise | ||
| def _process_communicate(*args, **kwargs): | ||
| overwrite_stderr = "stderr" in kwargs and isinstance(kwargs["stderr"], io.StringIO) | ||
| overwrite_stdout = "stdout" in kwargs and isinstance(kwargs["stdout"], io.StringIO) |
There was a problem hiding this comment.
Ha, this looks very similar to the code we have in emscripten test suite for this same purpose: https://github.com/emscripten-core/emscripten/blob/cb4450380c27e9be81e7c239ec7af5005ad283cd/test/common.py#L1153-L1181
Great minds think alike (or use the same AI :)
|
Thank you Sam! |
4 participants
Reduces runtime from ~15 minutes to 1.5 minutes on my machine
Before:
After:
Failed test example (stdout and stderr isn't ordered with the exception unfortunately, but it's easy to re-run the particular test):
os.cpu_count() * 2threadsos.cpu_count() * 4shows no benefit.os.cpu_count()oros.cpu_count() // 2also show no regression in runtime, but might be worse for machines with less cores. There are currently 315 spec tests total for reference.