Indefinite wait while using Langchain and HuggingFaceHub in python

Question

from langchain import PromptTemplate, HuggingFaceHub, LLMChain
import os

os.environ['HUGGINGFACEHUB_API_TOKEN'] = 'token'

# initialize HF LLM
flan_t5 = HuggingFaceHub(
    repo_id="google/flan-t5-xl",
    model_kwargs={"temperature": 1e-10}
)

multi_template = """Answer the following questions one at a time.

Questions:
{questions}

Answers:
"""
long_prompt = PromptTemplate(
    template=multi_template,
    input_variables=["questions"]
)

llm_chain = LLMChain(
    prompt=long_prompt,
    llm=flan_t5
)

qs_str = (
    "Which NFL team won the Super Bowl in the 2010 season?\n" +
    "If I am 6 ft 4 inches, how tall am I in centimeters?\n" +
    "Who was the 12th person on the moon?" +
    "How many eyes does a blade of grass have?"
)

print(llm_chain.run(qs_str))

I am learning langchain, on running above code, there has been indefinite halt and no response for minutes,

Can anyone tell why is it? and what is to be corrected.

I expected that it will come up with answers to 4 questions asked, but there has been indefinite waiting to it.

Lowbrow

I get this error when running this: ValueError: Error raised by inference API: Model google/flan-t5-xl time out Likely related to this discussion: huggingface.co/google/flan-t5-xxl/discussions/43 – Adown

Answer 1

As an alternative, you may use google/flan-t5-xxl with a positive temperature:

# initialize HF LLM
flan_t5 = HuggingFaceHub(
    repo_id="google/flan-t5-xxl",
    model_kwargs={"temperature": 0.5}
)

Charisecharisma