Convert string into TokenStream

Question

Given a string (str), how can one convert that into a TokenStream in Rust?

I've tried using the quote! macro.

let str = "4";
let tokens = quote! { let num = #str; }; // #str is a str not i32

The goal here is to generate tokens for some unknown string of code.

let thing = "4";
let tokens = quote! { let thing = #thing }; // i32

or

let thing = ""4"";
let tokens = quote! { let thing = #thing }; // str

Lucrative

Your question doesn't make sense. In the title you want to know, how to convert a String into a TokenStream, but in your text, you are complaining, that str is not an i32, which is a complet different question. – Olivaolivaceous

Not just that, what else would it be? You define it as a string literal by quoting it. try let str = 4 as i32 and watch it turn into an i32 literal in quote!. – Landgrabber

This certainly might not be possible. I'll update the question with more details to clarify. – Lucrative

Your title still does not match your question. – Olivaolivaceous

I honestly feel like you're both using the wrong tool, and trying to solve a problem different to the one you have. Why are you trying to get a TokenStream from code you already know the structure of, and where does the additional data on types come from (even in your example, the type of thing could very well be an u8, or an i16, or anything else that 4 conforms to). What are you actually trying to parse? – Landgrabber

@Olivaolivaceous I disagree. I’d like to know how to convert strings such as “(1,2)”, “func(1)”, “4” to TokenStreams. – Lucrative

I do think I’m going about this the wrong way though. I’m writing a macro and I’ve been managing the attributes to the macro by using regexes. I then want to go back to code during the generation step. It’s seeming like I should just work with the attributes I’m the TokenSteam that is provided to me. – Lucrative

What is the actual purpose of the macro? From this and your other question, it feels like the examples you give have very little to do with what you're actually attempting to do. – Landgrabber

You haven't even specified where this TokenStream comes from. Do you mean proc_macro::TokenStream or proc_macro2::TokenStream, or some other one? – Stevenage

Answer 1

how can one convert [a string] into a TokenStream

Rust has a common trait for converting strings into values when that conversion might fail: FromStr. This is usually accessed via the parse method on &str.

proc_macro2::TokenStream

use proc_macro2; // 0.4.24

fn example(s: &str) {
    let stream: proc_macro2::TokenStream = s.parse().unwrap();
}

proc_macro::TokenStream

extern crate proc_macro;

fn example(s: &str) {
    let stream: proc_macro::TokenStream = s.parse().unwrap();
}

You should be aware that this code cannot be run outside of the invocation of an actual procedural macro.

Stevenage