OpenAI APIs rely on SSE (Server Side Events) to stream the response back to you. If you pass the stream parameter in your API request, you will receive chunks of data when they are calculated by OpenAI.
This creates the illusion of a real-time response that mimics someone typing.
The hardest part to figure out might be how to connect your frontend with your backend. Every-time the backend receives a new chunk you want to display it in the frontend.
I created a simple NextJs project on Replit that demonstrates just that. Live demo
you will need to install better-sse package
npm install better-sse
Server side
In an API route file
import {createSession} from "better-sse";
const session = await createSession(req, res);
if (!session.isConnected) throw new Error('Not connected');
const { data } = await openai.createCompletion({
model: 'text-davinci-003',
n: 1,
max_tokens: 2048,
temperature: 0.3,
stream: true,
prompt: `CHANGE TO YOUR OWN PROMPTS`
}, {
timeout: 1000 * 60 * 2,
responseType: 'stream'
});
//what to do when receiving data from the API
data.on('data', text => {
const lines = text.toString().split('\n').filter(line => line.trim() !== '');
for (const line of lines) {
const message = line.replace(/^data: /, '');
if (message === '[DONE]') { //OpenAI sends [DONE] to say it's over
session.push('DONE', 'error');
return;
}
try {
const { choices } = JSON.parse(message);
session.push({text:choices[0].text});
} catch (err) {
console.log(err);
}
}
});
//connection is close
data.on('close', () => {
console.log("close")
res.end();
});
data.on('error', (err) => {
console.error(err);
});
On your front end you can now call this API route
let [result, setResult] = useState("");
//create the sse connection
const sse = new EventSource(`/api/completion?prompt=${inputPrompt}`);
//listen to incoming messages
sse.addEventListener("message", ({ data }) => {
let msgObj = JSON.parse(data)
setResult((r) => r + msgObj.text)
});
Hope this makes sense and help others with similar issue.
