[WIP] Token-based outer attributes handling

[Aaron1011]

Makes progress towards #43081

We now capture tokens for attributes, and remove them from the tokenstream when applying #[cfg] attributes (in addition to modifying the AST). As a result, derive-proc-macros now receive the exact input (instead of the pretty-printed/retokenized input), even when fields/variants get #[cfg]-stripped.

Several changes are made in order to accomplish this:

• New structs PreexpTokenStream and PreexpTokenTree are added. These are identical to TokenStream and PreexpTokenstream, with the exception of an OuterAttributes variant in PreexpTokenTree. This is used to represent captured attributes, allowing the target tokens to be removed by #[cfg]-stripping
  (and when invoking attribute macros).
• Tokens are now attached to Attribute. This allows us to remove prepend_attributes, which required pretty-printing/retokenizing attributes in certain cases.
• collect_tokens was rewritten. The implementation is now much simpler - instead of keeping track of nested TokenTree::Delimited at various deps, we collect all tokens into a flat buffer (e.g. we push TokenKind::OpenDelim and TokenKind::CloseDelim. After capturing, we losslessly re-package these tokens back into the normal TokenTree::Delimited structure.
• We now store a PreexpTokenStream on AST s structs instead of a plain TokenStream. This contains both the attributes and the target itself.
• parse_outer_attributes now passes the parsed attributes to a closure, instead of simply returning them. The closure is responsible for parsing the attribute target, and allows us to automatically construct the proper PreexpTokenStream.
• HasAttrs now has a visit_tokens method. This is used during #[cfg]-stripping to allow us to remove attribute targets from the PreexpTokenStream.

This PR is quite large (though ~1000 lines are tests). I opened it mainly to show how all of the individual pieces fit together, since some of them (e.g. the parse_outer_attributes change) don't make sense if we're not going to land this PR. However, many pieces are mergeable on their own - I plan to split them into their own PRs.

TODO:

• Any errors for malformed #[cfg]/#[cfg_attr] are duplicated. This is because we run the same code for both the AST-based attributes and the token-based attributes. We could possibly skip validating the token-based attributes, but I was worried about accidentally skipping gating if the pretty-print/reparse check ever fails. These are deduplicated by the diagnostic infrastructure outside of compiletest, but it would be nice to fix this.
• Inner attributes are partially handled - we store them in the PreexpTokenStream alongside outer attributes, which allows us to handle #![cfg(FALSE)] and remove its parent. This should be good enough to handle all stable uses of inner attributes that are observable by proc-macros (currently, just cfg-stripping before #[derive] macros are invoked). Custom inner attributes are not handled.
• Parser::parse_stmt_without_recovery does not eat a trailing semicolon, which means we will not capture it in the tokenstream. I need to either refactor statement parsing to eat the semicolon inside the parse_outer_attributes_with_tokens, or manually adjust the captured tokens.
• I currently only check for the presence/absence of attributes when determining whether to run token-collection logic. If this causes performance problems, we could look for specific attributes (cfg, cfg_attr, derive, or any non-builtin attribute). There are many scenarios where we can skip token collection for some/all ast structs (no derive on an an item, no cfg inside an item, no custom attributes on an item, etc.)

[rust-highfive]

r? @nikomatsakis

(rust_highfive has picked a reviewer for you, use r? to override)

[Aaron1011]

r? @petrochenkov

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion

[bors]

⌛ Trying commit 083792bcf67d1e50afc9ad50500ab011d34b0bee with merge 30ddab741f8c02392c4383445436138f5a6ec53d...

[bors]

☀️ Try build successful - checks-actions, checks-azure
Build commit: 30ddab741f8c02392c4383445436138f5a6ec53d (30ddab741f8c02392c4383445436138f5a6ec53d)

[rust-timer]

Queued 30ddab741f8c02392c4383445436138f5a6ec53d with parent 80cacd7, future comparison URL.

[rust-timer]

Finished benchmarking try commit (30ddab741f8c02392c4383445436138f5a6ec53d): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never

[Aaron1011]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion

[bors]

⌛ Trying commit 23fd9fbd9d07096ca254592013a42b4cae12ba29 with merge b3535fba76a3fc7202284b7699afff8f8831e461...

[bors]

☀️ Try build successful - checks-actions, checks-azure
Build commit: b3535fba76a3fc7202284b7699afff8f8831e461 (b3535fba76a3fc7202284b7699afff8f8831e461)

[rust-timer]

Queued b3535fba76a3fc7202284b7699afff8f8831e461 with parent 42d896a, future comparison URL.

[rust-timer]

Finished benchmarking try commit (b3535fba76a3fc7202284b7699afff8f8831e461): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never

[petrochenkov]

@Aaron1011
Apologies, I spent all the weekend time on testing LLD, so I didn't have chance to look a this.
I'll try to do it during the week.

[petrochenkov]

We need to trim this PR a bit first.
Right now I don't really understand what happens here because it's too large.

The next changes are improvements on their own and can be landed in separate PRs:

• Newly added tests and test cases.
• Adding tokens to statements.
• Adding tokens to ast::Attributes and eliminating prepend_attrs (at least partially).
• Token collection refactoring.

Additionally:

• Field / variant / etc span changes are responsible for the majority of the diff in test outputs. Can we keep the old spans in AST and not extend them to comma separators?
• parse_outer_attributes and similar changes involving self -> this dominate the parser diff. Can they be factored out into a separate commit (the first one). Other commits contain some significant back and forth, so I ended up reading the diff for the whole PR instead of separate commits, so it would be better to squash them.

[petrochenkov]

Inlining AttributeData will improve readability here, IMO.

[Aaron1011]

Field / variant / etc span changes are responsible for the majority of the diff in test outputs. Can we keep the old spans in AST and not extend them to comma separators?

We could, but that seems weirdly inconsistent - we would then have a span that doesn't include some of the tokens we capture. However, I could split this out into a separate PR to minimize the diff.

[Aaron1011]

I'll work on splitting things up. I opened this PR mainly to do a Crater run to check that everything fit together.

[petrochenkov]

With this PR we perform transformations like cfg-expansion on both tokens and AST simultaneously, and then make sure that they are still in sync with the pretty-print-reparse hack.

In the end state, I think, we should treat the AST part as a cache (used for performance only) and always reparse it from tokens after any transformations (which will only be performed on tokens). The reparse hack will then be eliminated.

This is kinda opposite to what we do now by treating the tokens as a "cache" and regenerating them from AST when necessary (via pretty-printing), but unlike the "AST -> tokens", the "tokens -> AST" conversion is always lossless.

Having only tokens without any AST cache should be functionally correct, but it's pretty reasonable to predict that it will be a performance regression because we'll have to parse same tokens at least twice and maybe more.

[Dylan-DPC-zz]

Blocked on #77250

[crlf0710]

Triage: A reminder that #77250 has been merged, so it seems maybe this can continue...

[Aaron1011]

This is currently blocked on further work on statement attributes

[Aaron1011]

Closing in favor of #80689