WIP: Parallelize passes using rayon

[Zoxc]

This builds on #46193 and #45912 and actually makes code run in parallel. This is not quite ready yet since rustc is not yet completely thread safe. It also uses a rough fork of rayon which uses fibers/stackful coroutines.

[Zoxc]

@bors try

[bors]

⌛ Trying commit c3e21ab7c517fd303077036df3b211cd28bf7b0e with merge 4f1b9ac80d435420ca47d60344b959db6e8e4f51...

[bors]

💔 Test failed - status-travis

[bors]

☔ The latest upstream changes (presumably #46509) made this pull request unmergeable. Please resolve the merge conflicts.

[Zoxc]

@bors try

[bors]

⌛ Trying commit 086a5fb with merge 463435e9a0879e3db40c9ed64f12a146f4937b16...

[bors]

💔 Test failed - status-travis

[Zoxc]

@bors try

[bors]

⌛ Trying commit 26a0515 with merge 8ad7c34...

[Zoxc]

@Mark-Simulacrum I'd also like a perf run here when the try build is complete

[bors]

☀️ Test successful - status-travis
State: approved= try=True

[Mark-Simulacrum]

Perf run queued.

[kennytm]

Perf result FYI: http://perf.rust-lang.org/compare.html?start=2537a499c2f1983011ad04ba2e59e5ff2e2989b6&end=8ad7c343c0a178cc8cf419f4fe41cf46c0bc84f4&stat=wall-time

[mark-i-m]

What are the specs of the machine on which this is running? Does it have the cores to actually take advantage of this PR?

[Mark-Simulacrum]

In general, yes, it should. I don't have the exact specs right now, though -- it has ~8 cores IIRC.

[Zoxc]

@Mark-Simulacrum I'd also like to know the specs of the machine and also I'd like to know how it is configured; does the CPU run with constant frequency? are there any background services running?

Note that the try build is with thread safety and the rayon thread pool disabled.

[mark-i-m]

Note that the try build is with thread safety and the rayon thread pool disabled.

Ah, that makes more sense... We were measuring the overhead of thread-safety.

[Zoxc]

Lets try that again.

Note that the try build is with (thread safety and the rayon thread pool) disabled.

It is just measuring what the cost of merging this with everything disabled is.

[bors]

☔ The latest upstream changes (presumably #46335) made this pull request unmergeable. Please resolve the merge conflicts.

[bors]

☔ The latest upstream changes (presumably #50235) made this pull request unmergeable. Please resolve the merge conflicts.

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

[00:32:05]    Compiling quick-error v1.2.1
[00:32:05]    Compiling rustc_driver v0.0.0 (file:///checkout/src/librustc_driver)
[00:32:06]    Compiling ar v0.3.1
[00:32:07]    Compiling arrayvec v0.4.7
[00:32:07] thread '<unnamed>' panicked at 'called `Option::unwrap()` on a `None` value', libcore/option.rs:335:21
[00:32:08]    Compiling owning_ref v0.3.3
[00:32:09]    Compiling crossbeam-utils v0.2.2
[00:32:10]    Compiling log v0.4.1
[00:32:10]    Compiling log_settings v0.1.1
[00:32:10]    Compiling log_settings v0.1.1
[00:32:10] 
[00:32:10] error: internal compiler error: unexpected panic
[00:32:10] 
[00:32:10] note: the compiler unexpectedly panicked. this is a bug.
[00:32:10] 
[00:32:10] note: we would appreciate a bug report: https://github.com/rust-lang/rust/blob/master/CONTRIBUTING.md#bug-reports
[00:32:10] note: rustc 1.27.0-dev running on x86_64-unknown-linux-gnu
[00:32:10] 
[00:32:10] 
[00:32:10] note: compiler flags: -Z force-unstable-if-unmarked -C prefer-dynamic -C opt-level=3 -C prefer-dynamic -C debug-assertions=y -C link-args=-Wl,-rpath,$ORIGIN/../lib --crate-type dylib
[00:32:10] 
[00:32:10] note: some of the compiler flags provided by cargo are hidden
[00:32:10] error: Could not compile `rustc_platform_intrinsics`.
[00:32:10] 
[00:32:10] Caused by:
[00:32:10] Caused by:
[00:32:10]   process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustc --crate-name rustc_platform_intrinsics librustc_platform_intrinsics/lib.rs --color always --error-format json --crate-type dylib --emit=dep-info,link -C prefer-dynamic -C opt-level=3 -C metadata=f84136b70dcfe48e -C extra-filename=-f84136b70dcfe48e --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-rustc/release/deps` (exit code: 101)
[00:32:15] error: build failed
[00:32:15] error: build failed
[00:32:15] command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/bin/cargo" "build" "--target" "x86_64-unknown-linux-gnu" "--release" "--locked" "--color" "always" "--features" " jemalloc" "--manifest-path" "/checkout/src/rustc/Cargo.toml" "--message-format" "json"
[00:32:15] expected success, got: exit code: 101
[00:32:15] thread 'main' panicked at 'cargo must succeed', bootstrap/compile.rs:1091:9
[00:32:15] travis_fold:end:stage1-rustc

[00:32:15] travis_time:end:stage1-rustc:start=1526230742463667857,finish=1526230768271878763,duration=25808210906


[00:32:15] failed to run: /checkout/obj/build/bootstrap/debug/bootstrap build
[00:32:15] Build completed unsuccessfully in 0:25:55
[00:32:15] make: *** [all] Error 1
[00:32:15] Makefile:28: recipe for target 'all' failed
315012 ./src/llvm
256636 ./obj/build/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/lib
241192 ./src/llvm-emscripten
210200 ./src/llvm/test
---
149732 ./.git/modules/src
149128 ./src/llvm-emscripten/test
148820 ./obj/build/bootstrap/debug/incremental
124220 ./obj/build/bootstrap/debug/incremental/bootstrap-182x3aewwy26b
124216 ./obj/build/bootstrap/debug/incremental/bootstrap-182x3aewwy26b/s-f103x0i68q-1uevj8y-34xtuyxu3kvz1
102820 ./obj/build/x86_64-unknown-linux-gnu/stage0/lib/rustlib/x86_64-unknown-linux-gnu/codegen-backends
97088 ./obj/build/x86_64-unknown-linux-gnu/stage1
97064 ./obj/build/x86_64-unknown-linux-gnu/stage1/lib
89816 ./src/llvm/test/CodeGen

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[bors]

☔ The latest upstream changes (presumably #48523) made this pull request unmergeable. Please resolve the merge conflicts.

[emilyalbini]

Ping from triage @Zoxc! What should we do with this PR?

[rust-highfive]

The job x86_64-gnu-llvm-3.9 of your PR failed on Travis (raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @TimNN. (Feature Requests)

[bors]

☔ The latest upstream changes (presumably #50699) made this pull request unmergeable. Please resolve the merge conflicts.

[mrhota]

@Zoxc what's the story with this PR? is it dead?

[Zoxc]

An alternative approach which doesn't use fibers is merged in master. This still parallelizes more code though. That should be cherry picked out and analyzed (it may not be beneficial to parallelize all passes).

[mikumikuchan]

sorry I mis clicked "approve" and I don't know why some how it succeeded. please discard my approval.

[mark-i-m]

@mikumikuchan No worries! That's the Github code review feature. The rust project instead uses @bors commands, which have an ACL (which I'm not on :P ).

[TimNN]

Ping from triage, @Zoxc: What is the status of this?

[TimNN]

Ping from triage, @Zoxc: we haven't heard from you for a while, so we are closing this PR. Feel free to re-open it in the future. Thanks for your contribution!