Add support for repetition to `proc_macro::quote`

[tgross35]

Part 1: complete in #141608

I'm opening a standalone issue to have something with "help wanted" labels in case anyone is interested in picking this up.

Our proc_macro::quote does not support repetition, unlike quote from the quote crate. As mentioned many times on the tracking issue, this is something we should support or at least account for before proc_macro::quote can be stabilized.

This should use the syntax:

proc_macro::quote! {
    $( CONTENTS )SEP*
}

Where CONTENTS is the thing to be repeated and SEP is an optional single-character separator. Expansion should work for anything that implements IntoIterator. This matches the quote crate's logic (except quote::quote uses # rather than $).

It's probably easiest to just copy quote's logic here, which uses an extension trait to facilitate this.

• quote crate source repo: https://github.com/dtolnay/quote.
• Link to source for our quote that needs to be updated: https://github.com/rust-lang/rust/blob/fa58ce343ad498196d799a7381869e79938e952a/library/proc_macro/src/quote.rs
• Link to tests that should be updated: https://github.com/rust-lang/rust/blob/c02a4f0852e6665cf3df3867982021383f5615df/tests/ui/proc-macro/quote/auxiliary/basic.rs (plus any new tests in tests/ui/proc-macro/quote for things we should reject)
• Tracking issue: Tracking issue for the quote! macro in proc_macro #54722

Part 2: incomplete

From @dtolnay's comment at #141608 (comment), there are some more minor issues to be resolved before stabilization:

#![feature(proc_macro_quote)]

use proc_macro::TokenStream;

macro_rules! decl {
    ($($iter:tt)*) => {
        stringify!($($iter) << *)
    };
}

#[proc_macro]
pub fn repro(input: TokenStream) -> TokenStream {
    // macro_rules macro
    let tokens = decl!(a b c);
    eprintln!("{}", tokens);

    // quote crate
    let input2 = proc_macro2::TokenStream::from(input.clone());
    let iter2 = input2.into_iter();
    let tokens = quote::quote!(#(#iter2) << *);
    eprintln!("{}", tokens);

    // libproc_macro
    let iter = input.into_iter();
    let tokens = proc_macro::quote!($($iter) << *);
    eprintln!("{}", tokens);

    TokenStream::new()
}

Macro_rules macro: a << b << c
Quote crate: a << b << c
Libproc_macro:

error: proc macro panicked
  --> src/lib.rs:25:18
   |
25 |     let tokens = proc_macro::quote!($($iter) << *);
   |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   |
   = help: message: `$(...)` must be followed by `*` in `quote!`

Another example:

error[E0425]: cannot find value `j` in this scope
  --> src/lib.rs:25:47
   |
25 |     let tokens = proc_macro::quote!($$ j $($$ j $iter)*);
   |                                               ^ not found in this scope

The parsing logic will need some more scrutiny in followup PRs before the macro can be stabilized.

[tgross35]

Cc @dtolnay in case you have any notes here

[tgross35]

Also fyi @SpriteOvO if you might be interested in this, since you have done a lot of other work with quote!

[dtolnay]

I recommend copying from the quote crate. It is tricky to get something that has all the behaviors that people expect are "obvious". If you see $($thing)*,—

• You cannot turn it into for x in $thing {...} because this is incompatible with $($thing)* $($thing)* where thing is something non-copyable like a Vec, a ubiquitous use case.

• You cannot turn it into for x in &$thing {...} because this is incompatible with thing being an Iterator, a ubiquitous use case.

• You cannot turn it into for x in &mut $thing {...} because this requires let mut thing in the caller's code which is highly unexpected.

• On top of all the above, in $($thing1 $thing2)* the correct behavior depends on which one of these is an iterator/collection and which one is a single token (Ident) or TokenStream. For a TokenStream, even though TokenStream has an IntoIterator impl, using that here produces an extremely unexpected result.

[moatom]

Hi there!

Excuse me for interrupting, but I’d like to try working on this PR when I have time — as long as it’s not an urgent task. (I’m not very familiar with Rust macros, but I have solid experience with meta-programming in OCaml and Scala, so I believe I can handle it given enough time.)

The main task seems to be porting the logic from declarative macros to procedural macros:
https://github.com/dtolnay/quote/blob/ab1e92c27a492e6077e203a6f85496bb4b1522e8/src/lib.rs#L888-L937

The most challenging part for me is likely the parsing.
In the declarative macro logic, parsing is done using a sliding window with 3-token(-tree) lookahead and lookback. To replicate this in procedural macros, I'll need to implement the logic in a stateful way.
However, I don’t have much experience with hand-written stateful parsers, so it may take me some time. (More importantly, I’ll first need to fully understand the overall BNF.)

[ora-0]

@moatom I don't see why you have to replicate the parsing logic of the crate. I'm sure it won't interfere with the quoting behavior. The code for the quote crate I feel is unnecessarily complex by virtue of it being a declarative macro, with one of the comments also saying that it is written that way to avoid using a muncher. I think recursive descent (or a simpler method) would suffice.

As for the grammar, I think I have got it here, though I'm not 100% sure I got everything correct. It is quite simple:

<content> ::= <tokentree> | <tokentree> <quote> <content> | ε
<quote> ::= "$" <ident> | "$" <repetition>
<repetition> ::= "(" <content> ")" <quantifier>
<quantifier> ::= "*" | <punct> "*"