How to store chat history using langchain conversationalRetrievalQA chain in a Next JS app?

Question

Im creating a text document QA chatbot, Im using Langchainjs along with OpenAI LLM for creating embeddings and Chat and Pinecone as my vector Store.

See Diagram:

After successfully uploading embeddings and creating an index on pinecone. I am using Next JS app to communicate with OpenAI and Pinecone.

The current structure of my App is like so:

1: Frontend -> user inputs a question and makes a POST call to a NEXT js server API route /ask

2: The server func looks like the following:

const vectorStore = await PineconeStore.fromExistingIndex(
        new OpenAIEmbeddings(),
        { pineconeIndex })

const model = new ChatOpenAI({ temperature: 0.5, modelName: 'gpt-3.5-turbo' })

const memory = new ConversationSummaryMemory({
    memoryKey: 'chat_history',
    llm: new ChatOpenAI({ modelName: 'gpt-3.5-turbo', temperature: 0 }),
})

const chain = ConversationalRetrievalQAChain.fromLLM(
    model,
    vectorStore.asRetriever(),
    {
        memory,
    }
)

const result = await chain.call({
    question,
})

return NextResponse.json({ data: result.text })

The ISSUE: The Chatbot never has access to any history because the memory always ONLY has ONLY the latest message in storage.

console.log('memory:', await memory.loadMemoryVariables({}))

I also tried BufferMemory and same Issue, the memory buffer only has ONLY the message just asked, if new query comes in the buffer is cleared and the new query is the only message in memory.

I may be unclear on how exactly to correctly store the history so the const result = await chain.call({question,}) call has access to the previous info

UPDATE: I have successfully used Upstash Redis-Backed Chat Memory with langchain, however still curious if it is possible to store messages using User Browser?

Stout

The related documentation talks about external memory for history : await chain.call({question: "Was that nice?",chat_history: chatHistory,}); but tbh I didn't try it out: js.langchain.com/docs/modules/chains/index_related_chains/… – Microsome

We managed to store the history with RedisChatMessageHistory in BufferMemory. Does Pinecone have a similar solution? – Magdau

Answer 1

I am using same exact stack as you (Next.js, Pinecone, Langchain ConversationalRetrievalQAChain) and I fought with this exact issue for a while.

Eventually I resorted to the below 'hack' which does work, basically capturing the history in a messages array which I shove into the end of the prompt. To my knowledge this is not all that different than how LangChain's 'Memory' components are supposed to work so I don't feel terrible doing this.

Here is a trimmed down version of my main handler function. Take note of line starting with queryText+='\nIf necessary ...

export async function POST(req: Request) {
    const { messages } = await req.json();
    let queryText = messages[messages.length - 1].content;
    
    queryText+='\nIf necessary, utilize the below chat history as additional context:'+JSON.stringify(messages);
    
    const chain = ConversationalRetrievalQAChain.fromLLM(
            streamingModel,
            vectorStore.asRetriever(),
            {
                returnSourceDocuments: true,
                questionGeneratorChainOptions: {
                    llm: nonStreamingModel,
                },
            }

    const { stream, handlers } = LangChainStream();
    chain.call({ question: queryText }, [handlers]).catch(console.error)

    return new StreamingTextResponse(stream);
}

Stimulative

Answer 2

I think you should use returnMessages:true property of ConversationSummaryMemory:

const memory = new ConversationSummaryMemory({
  memoryKey: 'chat_history',
  llm: new ChatOpenAI({ modelName: 'gpt-3.5-turbo', temperature: 0 }),
  returnMessages:true
})

this is the code for loadMemoryVariables from

async loadMemoryVariables(_) {
        if (this.returnMessages) {
            const result = {
                [this.memoryKey]: [new this.summaryChatMessageClass(this.buffer)],
            };
            return result;
        }
        const result = { [this.memoryKey]: this.buffer };
        return result;
    }

if you set returnMessages:true it will return this.summaryChatMessageClass(this.buffer) object otherwise it just returns a string.

Cucumber