Flan T5 - How to give the correct prompt/question?

Question

Giving the right kind of prompt to Flan T5 Language model in order to get the correct/accurate responses for a chatbot/option matching use case.

I am trying to use a Flan T5 model for the following task. Given a chatbot that presents the user with a list of options, the model has to do semantic option matching. For instance, if the options are "Barbeque Chicken, Smoked Salmon", if the user says "I want fish", the model should select smoked salmon. Another use case could be "The first one" in which case the model should select Barbeque Chicken. A third use case could be "The BBQ one" in which case the model should select Barbeque chicken.

I am using some code from the huggingface docs to play around with flan-t5 but I did not get the correct output.

model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-small")
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-small")

inputs = tokenizer('''Q:Select from the following options 
(a) Quinoa Salad 
(b) Kale Smoothie 
A:Select the first one
''', return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

The output is

['(b) Kale Smoothie']

How should I give the correct prompt/question to elicit the correct response from Flan t5 ?

Heal

Answer 1

A recent paper goes into detail about how the Flan Collection was created ("The Flan Collection: Designing Data and Methods for Effective Instruction Tuning") and points to a GitHub repo with the templates used for creating the training data for it.

Some examples:

"Write a short summary for this text: {text}"

"Context: {context}\n\nQuestion: {question}\n\nAnswer:"

"Who is {pronoun} in the following sentence?\n\n{sentence}\n\n{options_}"

For picking from a list of options, it looks like the code generates a newline/hyphen-separated list or prefaces the answers with capitalized letters in parentheses:

OPTIONS:
- first thing
- second thing
- third thing

or

OPTIONS:
(A) first thing
(B) second thing
(C) third thing

Gurglet

Answer 2

The original paper shows an example in the format "Question: abc Context: xyz", which seems to work well. I get more accurate results with the larger models like flan-t5-xl. Here is an example with flan-t5-base, illustrating mostly good matches, but a few spurious results:

Be careful: Concatenating user-generated input with a fixed template like this opens up the possibility of a "prompt injection" attack. Treat the output of the model as untrusted or potentially hostile user-generated input; for example, do not echo it back to the user as un-escaped HTML.

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")
model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")

def query_from_list(query, options):
    t5query = f"""Question: Select the item from this list which is "{query}". Context: * {" * ".join(options)}"""
    inputs = tokenizer(t5query, return_tensors="pt")
    outputs = model.generate(**inputs, max_new_tokens=20)
    return tokenizer.batch_decode(outputs, skip_special_tokens=True)

tests = ["the first one", "the fish", "the chicken", "2nd", "bbq", "salmon", "roasted turkey", "dried halibut"]
options = ["Barbecue Chicken", "Smoked Salmon"]
for t in tests:
    result = query_from_list(t, options)
    print(f"{t:<24} {result[0]}")

returns:

the first one            Barbecue Chicken
the fish                 Smoked Salmon
the chicken              Barbecue Chicken
2nd                      Barbecue Chicken
bbq                      Barbecue Chicken
salmon                   salmon
roasted turkey           Barbecue Chicken
dried halibut            Smoked Salmon

Flexile