Implement token-based handling of attributes during expansion

[Aaron1011]

This PR modifies the macro expansion infrastructure to handle attributes
in a fully token-based manner. As a result:

• Derives macros no longer lose spans when their input is modified
  by eager cfg-expansion. This is accomplished by performing eager
  cfg-expansion on the token stream that we pass to the derive
  proc-macro
• Inner attributes now preserve spans in all cases, including when we
  have multiple inner attributes in a row.

This is accomplished through the following changes:

• New structs AttrAnnotatedTokenStream and AttrAnnotatedTokenTree are introduced.
  These are very similar to a normal TokenTree, but they also track
  the position of attributes and attribute targets within the stream.
  They are built when we collect tokens during parsing.
  An AttrAnnotatedTokenStream is converted to a regular TokenStream when
  we invoke a macro.
• Token capturing and LazyTokenStream are modified to work with
  AttrAnnotatedTokenStream. A new ReplaceRange type is introduced, which
  is created during the parsing of a nested AST node to make the 'outer'
  AST node aware of the attributes and attribute target stored deeper in the token stream.
• When we need to perform eager cfg-expansion (either due to #[derive] or #[cfg_eval]), we tokenize and reparse our target, capturing additional information about the locations of #[cfg] and #[cfg_attr] attributes at any depth within the target. This is a performance optimization, allowing us to perform less work in the typical case where captured tokens never have eager cfg-expansion run.

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit ac71b40b84add10df1cc6b4394ca53f5c3bfe16d with merge b86108850e24b4dd26ad05cb34d04ef489f2385a...

[bors]

☀️ Try build successful - checks-actions
Build commit: b86108850e24b4dd26ad05cb34d04ef489f2385a (b86108850e24b4dd26ad05cb34d04ef489f2385a)

[rust-timer]

Queued b86108850e24b4dd26ad05cb34d04ef489f2385a with parent ec7f8d9, future comparison URL.

[rust-timer]

Finished benchmarking try commit (b86108850e24b4dd26ad05cb34d04ef489f2385a): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[petrochenkov]

@Aaron1011
Could you move tests to a separate PR or commit? I'm interested in diffs in test outputs before and after the compiler changes.

[Aaron1011]

@petrochenkov Sure

[Aaron1011]

Opened #82643

[petrochenkov]

(Still need to review changes in rustc_parse.)

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

@rustbot label: +S-waiting-on-perf

[bors]

⌛ Trying commit 87977c6a776ca0b4075f0998bde21267b3addcd6 with merge f1c431c58af7789983cbdfd61c470034c37eb631...

[bors]

☀️ Try build successful - checks-actions
Build commit: f1c431c58af7789983cbdfd61c470034c37eb631 (f1c431c58af7789983cbdfd61c470034c37eb631)

[rust-timer]

Queued f1c431c58af7789983cbdfd61c470034c37eb631 with parent 573a697, future comparison URL.

[rust-timer]

Finished benchmarking try commit (f1c431c58af7789983cbdfd61c470034c37eb631): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[Aaron1011]

There appears to be a significant hit from being unable to bail out early from collect_tokens_trailing_token when parsing an expression. We have several different options:

1. Accept the performance hit. This is a pretty bad option, as I suspect this could be a measurable slowdown on large, complex projects.
2. Refactor expression parsing to determine in advance if the specific type of expression we are parsing (e.g. an if expression) supports inner attributes. However, the expression parsing code is already quite complicated, and this would require introducing even more complexity.
3. Speed up the slower path of collect_tokens_trailing_token. Given the current design of the parser, I think this will prove somewhat difficult. We need to clone the current token and the TokenCursor, which I think is causing most of the slowdown
4. Try to detect if inner attributes are definitely not present in the token stream. This would be kind of a hack, since we'd be effectively re-implementing part of the parser in collect_tokens_trailing_token. If inner attributes are present, then we will have the tokens <open_delimiter> # ! ... <close_delimiter> somewhere in the input. However, src/test/ui/proc-macro/weird-braces.rs shows that finding where this occurs is very tricky without actually parsing all of the tokens.

[Aaron1011]

I've rebased against master - this should now be ready to merge.

[petrochenkov]

r=me with commits squashed.

[Aaron1011]

@bors r=petrochenkov

[bors]

📌 Commit a93c4f0 has been approved by petrochenkov

[bors]

⌛ Testing commit a93c4f0 with merge ba6275b...

[bors]

☀️ Test successful - checks-actions
Approved by: petrochenkov
Pushing ba6275b to master...

[EliaGeretto]

I believe this PR introduced a regression, breaking the clang-sys crate. I performed a bisection with cargo-bisect-rustc and this change seems to be the culprit. I opened an issue in the clang-sys repo because I am not sure where the problem is. It should be reproducible with cargo build --features runtime in that repo.

[Aaron1011]

@EliaGeretto: Thanks for finding that! I've opened #84130 to fix it