How do I Stream OpenAI's completion API?

Question

I want to stream the results of a completion via OpenAI's API.

The doc's mention using server-sent events - it seems like this isn't handled out of the box for flask so I was trying to do it client side (I know this exposes API keys). However, because the OpenAI API requires it post it seems like it isn't compatible with the eventSource API. I tried doing it via a fetch (Using readable streams) but when I try to convert to JSON via the example I get the following error: Uncaught (in promise) SyntaxError: Unexpected token 'd', "data: {"id"... is not valid JSON (I know this isn't valid JSON). It seems like it is parsing the entire result not each individual stream.

data: {"id": "cmpl-5l11I1kS2n99uzNiNVpTjHi3kyied", "object": "text_completion", "created": 1661887020, "choices": [{"text": " to", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-002"}

data: {"id": "cmpl-5l11I1kS2n99uzNiNVpTjHi3kyied", "object": "text_completion", "created": 1661887020, "choices": [{"text": " AL", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-002"}

data: {"id": "cmpl-5l11I1kS2n99uzNiNVpTjHi3kyied", "object": "text_completion", "created": 1661887020, "choices": [{"text": "I", "index": 0, "logprobs": null, "finish_reason": null}], "model": "text-davinci-002"}

Would love some pointers or a simple code example of how to do this because I've been banging my head against it for a while. Thanks!

Brammer

Hi there, first remove "data:" javascript const data = bytes.toString(); const message = data.replace(/^data: /, ""); console.log(JSON.parse(message)); and that's it. – Ahoufe

Answer 1

Finally got this working code:

import { Configuration, OpenAIApi } from "openai";
import dotenv from "dotenv";
dotenv.config({ override: true });

const openai = new OpenAIApi(new Configuration({ apiKey: process.env.OPENAI_KEY }));

const getText = async (prompt, callback) => {
    const completion = await openai.createCompletion(
        {
            model: "text-davinci-003",
            prompt: prompt,
            max_tokens: 1000,
            stream: true,
        },
        { responseType: "stream" }
    );
    return new Promise((resolve) => {
        let result = "";
        completion.data.on("data", (data) => {
            const lines = data
                ?.toString()
                ?.split("\n")
                .filter((line) => line.trim() !== "");
            for (const line of lines) {
                const message = line.replace(/^data: /, "");
                if (message == "[DONE]") {
                    resolve(result);
                } else {
                    let token;
                    try {
                        token = JSON.parse(message)?.choices?.[0]?.text;
                    } catch {
                        console.log("ERROR", json);
                    }
                    result += token;
                    if (token) {
                        callback(token);
                    }
                }
            }
        });
    });
};
    
console.log(await getText("Who was the latest president of USA?", (c) => process.stdout.write(c)));

Gaynell

I've got an error, completion.data.on is not a function – Mohandis

Use Node v18 please – Gaynell

I am not sure why, but i think it's an issue on OpenAI's end. With this code and even with native CURL call, the stream doesn't work at all. It does break the response into chunks, but still responds after the entire call finishes, not in parts. – Straightlaced

It will call char callback and type in stream mode; I just need also full answer at the end so I return and console.log() it too – Gaynell

Answer 2

in browser, you can use fetch API, example:

const response = await fetch('https://api.openai.com/v1/completions', {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${config.apiKey}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        model: 'text-davinci-003',
        prompt: input,
        stream: true,
      }),
    });
    const reader = response.body?.pipeThrough(new TextDecoderStream()).getReader();
    if (!reader) return;
    // eslint-disable-next-line no-constant-condition
    while (true) {
      // eslint-disable-next-line no-await-in-loop
      const { value, done } = await reader.read();
      if (done) break;
      let dataDone = false;
      const arr = value.split('\n');
      arr.forEach((data) => {
        if (data.length === 0) return; // ignore empty message
        if (data.startsWith(':')) return; // ignore sse comment message
        if (data === 'data: [DONE]') {
          dataDone = true;
          return;
        }
        const json = JSON.parse(data.substring(6));
        console.log(json);
      });
      if (dataDone) break;
    }

Fadil

You really shouldn't be doing this in the browser. People will steal your API keys. – Fulks

Answer 3

This code works.

    import { Configuration, OpenAIApi } from "openai";
    
    const configuration = new Configuration({
          apiKey: process.env.OPENAI_KEY,
        });
    
    const openai = new OpenAIApi(configuration);
    
    const completion = await openai.createCompletion(
            {
                model: "text-davinci-003",
                prompt: prompt,
                max_tokens: 100,
                stream: true,
            },
            { responseType: "stream" }
        );
       
    completion.data.on("data", (data) => {
          const lines = data
            ?.toString()
            ?.split("\n")
            .filter((line) => line.trim() !== "");
          for (const line of lines) {
            const message = line.replace(/^data: /, "");
            if (message === "[DONE]") {
              break; // Stream finished
            }
            try {
              const parsed = JSON.parse(message);
              console.log(parsed.choices[0].text);
            } catch (error) {
              console.error("Could not JSON parse stream message", message, error);
            }
          }
        });

Thereupon

Answer 4

Streaming completions are not natively supported by the openai package. There is a workaround though, see https://github.com/openai/openai-node/issues/18#issuecomment-1369996933. Copy-pasting the code below for convenience.

try {
    const res = await openai.createCompletion({
        model: "text-davinci-002",
        prompt: "It was the best of times",
        max_tokens: 100,
        temperature: 0,
        stream: true,
    }, { responseType: 'stream' });
    
    res.data.on('data', data => {
        const lines = data.toString().split('\n').filter(line => line.trim() !== '');
        for (const line of lines) {
            const message = line.replace(/^data: /, '');
            if (message === '[DONE]') {
                return; // Stream finished
            }
            try {
                const parsed = JSON.parse(message);
                console.log(parsed.choices[0].text);
            } catch(error) {
                console.error('Could not JSON parse stream message', message, error);
            }
        }
    });
} catch (error) {
    if (error.response?.status) {
        console.error(error.response.status, error.message);
        error.response.data.on('data', data => {
            const message = data.toString();
            try {
                const parsed = JSON.parse(message);
                console.error('An error occurred during OpenAI request: ', parsed);
            } catch(error) {
                console.error('An error occurred during OpenAI request: ', message);
            }
        });
    } else {
        console.error('An error occurred during OpenAI request', error);
    }
}

Jacklynjackman

Answer 5

I used the onDownloadProgress option that the openai library provides.

async generateTextStream(
  req: OpenaiCompletionRequest,
  onDownloadProgressCallback: (pe: ProgressEvent) => void
): Promise<any> {
  return await this.openai.createCompletion(
    {
      model: req.model,
      prompt: req.prompt,
      max_tokens: req.max_tokens,
      temperature: 0.2,
      stream: true
    },
    {
      responseType: 'stream',
      onDownloadProgress: onDownloadProgressCallback,
    },
  );
}

And then the data is is in the progress event.

const result = await this.openAiService
  .generateTextStream(
    { model: 'text-davinci-003', prompt: this.input, max_tokens: 1000 },
    (pe: ProgressEvent) => {
      const res: string = pe.target['response'];
      console.log(pe);
      const newReturnedData = res.split('data:');
      let lastPart = newReturnedData[newReturnedData.length - 1];
      lastPart = lastPart.trim()
      console.log(lastPart);
    },
  );

You will also have to parse the string to an object.(I have created the respone class for better intellisense)

const data: OpenaiEditResponseData = JSON.parse(lastPart);
console.log(data.choices[0].text);

Jourdan

Answer 6

I've had good success using https://github.com/SpellcraftAI/openai-streams to abstract the streaming details away from my application. It also appears there is a streaming implementation in Langchain respository, documented here: https://js.langchain.com/docs/getting-started/guide-llm#streaming.

Their implementation is viewable here: https://github.com/SpellcraftAI/openai-streams/blob/40e1fd03788327ae3d63dfe1ffb2432087fb0921/src/lib/streaming/streams.ts#L43.

Copied for convenience:

async start(controller) {
      const parser = createParser(async (event) => {
        if (event.type === "event") {
          const { data } = event;
          /**
           * Break if event stream finished.
           */
          if (data === "[DONE]") {
            const controllerIsClosed = controller.desiredSize === null;
            if (!controllerIsClosed) {
              controller.close();
            }

            await onDone?.();
            return;
          }
          /**
           * Verify we have a valid JSON object and then enqueue it.
           */
          try {
            const parsed = JSON.parse(data);
            controller.enqueue(ENCODER.encode(data));

            /**
             * In `tokens` mode, if the user runs out of tokens and the stream
             * does not complete, we will throw a MAX_TOKENS error. In `raw`
             * mode, we leave it up to the user to handle this.
             *
             * This requires iterating over result.choices[] and throwing an
             * error if any of them have `{ finish_reason: "length" }`.
             */
            if (mode === "tokens" && parsed?.choices) {
              const { choices } = parsed;
              for (const choice of choices) {
                if (choice?.finish_reason === "length") {
                  throw new OpenAIError("MAX_TOKENS");
                }
              }
            }
          } catch (e) {
            controller.error(e);
          }
        }
      });
      /**
       * Feed the parser with decoded chunks from the raw stream.
       */
      for await (const chunk of yieldStream(stream)) {
        const decoded = DECODER.decode(chunk);

        try {
          const parsed = JSON.parse(decoded);

          if (parsed.hasOwnProperty("error"))
            controller.error(new Error(parsed.error.message));
        } catch (e) {}

        parser.feed(decoded);
      }
    },
  });

Greasewood

Answer 7

This is the new way (2024):

import OpenAI from 'openai';

const openai = new OpenAI();

async function main() {
  const stream = await openai.chat.completions.create({
    model: 'gpt-4',
    messages: [{ role: 'user', content: 'Say this is a test' }],
    stream: true,
  });
  for await (const part of stream) {
    process.stdout.write(part.choices[0]?.delta?.content || '');
  }
}

main();

Source.

Lakshmi

Answer 8

Use this code:

const { Configuration, OpenAIApi } = require("openai");

const configuration = new Configuration({
    apiKey: process.env.REACT_APP_APIKEY,// your api key
  });
const openai = new OpenAIApi(configuration);
let fetchData = async () => {
        await openai
          .createCompletion({
            model: "text-davinci-002",
            prompt: `hello i am searched text`,
            max_tokens: 500,
            temperature: 0,
          })
          .then(response => {
         
            console.log(response.data.choices[0].text);
          })
          .catch(err => console.log(err));
      };
      fetchData();

You will receive data, in object -> data -> choices[0] -> text.

Krak

This snippet doesn't show the usage of the "stream" parameter. I'm also looking for a valid example but it seems that the OpenAI NPM module is not the place to look as it doesn't support streaming at the moment. – Koestler

This code doesn't answer the question, no usage of streams – Promotive