Introduce `run-make` V2 infrastructure, a `run_make_support` library and port over 2 tests as example by jieyouxu · Pull Request #113026 · rust-lang/rust

jieyouxu · 2023-06-25T09:48:09Z

Preface

See issue #40713: Switch run-make tests from Makefiles to rust for more context.

Basic Description of `run-make` V2

run-make V2 aims to eliminate the dependency on make and Makefiles for building run-make-style tests. Makefiles are replaced by recipes (rmake.rs). The current implementation runs run-make V2 tests in 3 steps:

We build the support library run_make_support which the rmake.rs recipes depend on as a tool lib.
We build the recipe rmake.rs and link in the support library.
We run the recipe to build and run the tests.

rmake.rs is basically a replacement for Makefile, and allows running arbitrary Rust code. The support library is built using cargo, and so can depend on external crates if desired.

The infrastructure implemented by this PR is very barebones, and is the minimally required infrastructure needed to build, run and pass the two example run-make tests ported over to the new infrastructure.

Example `run-make` V2 test

// ignore-tidy-linelength

extern crate run_make_support;

use std::path::PathBuf;

use run_make_support::{aux_build, rustc};

fn main() {
    aux_build()
        .arg("--emit=metadata")
        .arg("stable.rs")
        .run();
    let mut stable_path = PathBuf::from(env!("TMPDIR"));
    stable_path.push("libstable.rmeta");
    let output = rustc()
        .arg("--emit=metadata")
        .arg("--extern")
        .arg(&format!("stable={}", &stable_path.to_string_lossy()))
        .arg("main.rs")
        .run();

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}

Follow Up Work

Adjust rustc-dev-guide docs

rustbot · 2023-06-25T09:48:17Z

r? @Mark-Simulacrum

(rustbot has picked a reviewer for you, use r? to override)

bjorn3 · 2023-06-26T09:54:35Z

src/tools/run-make-support/src/lib.rs

Maybe do:

Suggested change

let mut cmd = $crate::xshell::cmd!(sh, "{rustc}")

.arg("--out-dir")

.arg($scx.tmpdir())

.arg("-L")

.arg($scx.tmpdir());

for arg in args.split_whitespace() {

cmd = cmd.arg(arg);

}

let mut cmd = $crate::xshell::cmd!(sh, concat_str!("{rustc} ", $args))

.arg("--out-dir")

.arg($scx.tmpdir())

.arg("-L")

.arg($scx.tmpdir());

This should allow interpolations the same way as xshell itself and should correctly handle quoting arguments. I haven't tested it though.

A bit unfortunate but cmd!() only accepts a literal, can concat_str! output a literal? I don't think so right? And yeah quoting arguments is a problem with the current incorrect behavior of split_whitespace.

I could force the input to be a &[], since

let args = ["hello", "world"]; let c = cmd!(sh, "echo {args...}");

works I think. This however forces the use site to become something like

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]); let output = rustc!( scx, [ "--emit=metadata", "--extern", &format!("stable={}/libstable.rmeta", scx.tmpdir()), "main.rs" ] );

Of course. cmd!() would see the concat_str!() before expanding. @matklad would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance? Such that this could be cmd!(sh, $args) where sh is Shell::new().wrapper_command(rustc_path).

While a proper fix to this requires a change to xshell, I will temporarily use the iterable form for the recipes since they're at least more correct than split_whitespace.

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance?

This would technically work, but I don't think that's an obvious API with a clear use-case. My answer would be, if you want to control the first arg, to use something like

cmd!(sh, "{rustc} --emit=metadata --crate-type=lib")

Like shells, xshell supports interpolation of the first argumetn (or at least it should support it).

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

(Note that passing-as-an-array here is a temporary band-aid fix, the intention is to avoid needing an array literal.) That is, ideally, the API would look something more like

let _output = rustc!("--emit=metadata --crate-type=lib stable.rs");

so the test writer doesn't have to bother with an array literal.

src/tools/run-make-support/src/lib.rs

src/bootstrap/tool.rs

src/tools/compiletest/src/runtest.rs

src/tools/run-make-support/src/lib.rs

Cargo.lock

matklad · 2023-06-26T15:03:45Z

src/tools/run-make-support/src/lib.rs

let _output = rustc!(scx, ["--emit=metadata", "--crate-type=lib", "stable.rs"]);

We are passing argument as an array here, so I don't think we even need xshell? The whole purpose of xshell is to avoid needing an array literal, but, if you are okay with that, I don't think xshell brings much to the table here?

would you accept adding something like a method on Shell that defines a wrapper program for all commands spawned using this Shell instance?

This would technically work, but I don't think that's an obvious API with a clear use-case. My answer would be, if you want to control the first arg, to use something like

cmd!(sh, "{rustc} --emit=metadata --crate-type=lib")

Like shells, xshell supports interpolation of the first argumetn (or at least it should support it).

jieyouxu · 2023-06-27T03:29:04Z

I changed the implementation to rely on shell-words instead to split the input string into words that can be passed to Command::arg. The API now looks something like

extern crate run_make_support;

use run_make_support::{rustc, run, run_fail};

fn main() {
    let _output = rustc("a.rs --cfg x -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = rustc("b.rs -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = run("b");
    let _output = rustc("a.rs --cfg y -C prefer-dynamic -Z unstable-options -C symbol-mangling-version=legacy");
    let _output = run_fail("b");
}

This unfortunately isn't as nice as having straight up string interpolation like xshell does, e.g. in CURRENT_RUSTC_VERSION:

extern crate run_make_support;

use run_make_support::{rustc, aux_build};

fn main() {
    let _output = aux_build("stable.rs");
    let output = rustc(format!("--emit=metadata --extern stable={}/libstable.rmeta main.rs", env!("TMPDIR")));

    let stderr = String::from_utf8_lossy(&output.stderr);
    let version = include_str!(concat!(env!("S"), "/src/version"));
    let expected_string = format!("stable since {}", version.trim());
    assert!(stderr.contains(&expected_string));
}

rustbot · 2023-06-27T05:40:22Z

These commits modify the Cargo.lock file. Unintentional changes to Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

matklad · 2023-06-27T12:26:07Z

It might be interesting to combine "split-on-whitespace" idea with builder-style interface. This is the style of testing employed by Cargo:

https://github.com/rust-lang/cargo/blob/5febbe5587b74108165f748e79a4f8badbdf5e0e/tests/testsuite/build.rs#L4168-L4169

So,

let tmpdir = env!("TMPDIR");
let output = rustc("main.rs --emit=metadata")
    .arg(format!("--extern=stable={tempdir}/libstable.rmeta"))
    .stderr_utf8_lossy()?
;

Thinking more about this, I think "specific builders" are probably a better fit here than "generic shell".

I expect these tests to be relatively few, so we don't have to optimize suuuper hard for readability.

At the same time, I expect most of these tests to be similar, and to benefit from abstracting common functionality. Abstracting through a builder is easier than abstracting through macro-based DSL.

bors · 2023-06-30T06:56:49Z

☔ The latest upstream changes (presumably #113162) made this pull request unmergeable. Please resolve the merge conflicts.

Mark-Simulacrum

r? @bjorn3 - this looks good to me in broad strokes, but looks like you've done a bunch of review here already.

src/bootstrap/tool.rs

src/tools/compiletest/src/lib.rs

bjorn3 · 2024-02-28T18:30:56Z

@bors r+

bors · 2024-02-28T18:30:59Z

📌 Commit 5727711 has been approved by bjorn3

It is now in the queue for this repository.

bors · 2024-02-28T23:15:48Z

⌛ Testing commit 5727711 with merge 6475c81...

bors · 2024-02-29T00:52:33Z

💔 Test failed - checks-actions

jieyouxu · 2024-02-29T09:18:13Z

Gonna edit ci.yml to try to test this on msvc beforehand.
@rustbot author

Zalathar · 2024-02-29T09:35:46Z

Oh, if you edit ci.yml, you also need to do x run src/tools/expand-yaml-anchors to update the real file that is generated from it.

(If you forget, x test tidy will complain and tell you to run that command.)

EDIT: Never mind, you figured it out 👍

jieyouxu · 2024-02-29T09:40:04Z

Oh, if you edit ci.yml, you also need to do x run src/tools/expand-yaml-anchors to update the real file that is generated from it.

Yeah, great advice on editing ci.yml, thank you so much 😄 Would waste so much more time trying to debug why this PR always die on MSVC otherwise.

jieyouxu · 2024-02-29T11:05:06Z

I fixed the rmake executable missing a .exe on Windows, and fixed the a-b-a-linker-guard test accidentally having an extra --cfg x passed in to the second rustc() invocation. Please, msvc, please work now.

jieyouxu · 2024-02-29T13:12:10Z

tools.mk special cases Windows and uses PATH=PATH:$TARGET_RPATH_DIR for executables instead of just using $TARGET_RPATH_ENV for non-Windows platforms...

jieyouxu · 2024-02-29T14:00:24Z

DLL_NOT_FOUND but am not sure which DLL is not found... One definite bug is PATH=$1;$2 seperator is ; on Windows and not : like *nixes.

bjorn3 · 2024-02-29T14:51:34Z

src/tools/run-make-support/src/lib.rs

+        target_rpath_env_path.push(path_sep(&target));
        target_rpath_env_path.push_str(&std::env::var("TARGET_RPATH_ENV").unwrap());
-        target_rpath_env_path.push(':');
+        target_rpath_env_path.push(path_sep(&target));


It is the host that matters here, right? You can probably use std::env::join_paths here.

Yep, that did it!

jieyouxu · 2024-02-29T16:32:45Z

Seems to finally work for msvc!
@rustbot ready

bjorn3 · 2024-03-01T15:18:47Z

@bors r+

bors · 2024-03-01T15:18:50Z

📌 Commit 48e9f92 has been approved by bjorn3

It is now in the queue for this repository.

bors · 2024-03-01T16:44:00Z

⌛ Testing commit 48e9f92 with merge 17edace...

bors · 2024-03-01T19:05:33Z

☀️ Test successful - checks-actions
Approved by: bjorn3
Pushing 17edace to master...

rust-timer · 2024-03-01T20:57:06Z

Finished benchmarking commit (17edace): comparison URL.

Overall result: no relevant changes - no action needed

@rustbot label: -perf-regression

Instruction count

This benchmark run did not return any relevant results for this metric.

Max RSS (memory usage)

This benchmark run did not return any relevant results for this metric.

Cycles

Results

This is a less reliable metric that may be of interest but was not used to determine the overall result at the top of this comment.

	mean	range	count
Regressions ❌ (primary)	-	-	0
Regressions ❌ (secondary)	2.7%	[1.7%, 3.5%]	9
Improvements ✅ (primary)	-	-	0
Improvements ✅ (secondary)	-	-	0
All ❌✅ (primary)	-	-	0

Binary size

This benchmark run did not return any relevant results for this metric.

Bootstrap: 651.67s -> 651.533s (-0.02%)
Artifact size: 311.10 MiB -> 311.14 MiB (0.01%)

rustbot assigned Mark-Simulacrum Jun 25, 2023

rustbot added A-testsuite Area: The testsuite used to check the correctness of rustc S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jun 25, 2023

This comment has been minimized.

Sign in to view