How to use the OpenAI stream=true property with a Django Rest Framework response, and still save the content returned?

Question

I'm trying to use the stream=true property as follows.

completion = openai.Completion.create(
            model="text-davinci-003",
            prompt="Write me a story about dogs.",
            temperature=0.7,
            max_tokens=MAX_TOKENS,
            frequency_penalty=1.0,
            presence_penalty=1.0,
            stream=True,
        )

Unfortunately, I don't know what to do from here to return it to my React frontend. Typically, I've used standard response objects, setting a status and the serializer.data as the data. From my readings online, it seems I have to use the StreamingHttpResponse, but I'm not sure how to integrate that with the iterator object of completion, and actually save the outputted data once it is done streaming, as the view will end after returning the iterator to the endpoint. Any help?

Meda

Answer 1

You can use StreamingHttpResponse. Note that you can't see it stream live for most api clients like postman, but you can see it on your terminal. If you'd like to use it for react, you'll have to use fetch api.

@api_view(["POST"])
def generate_names(request):
    if request.method == 'POST':
        # Parse the request body and extract the prompt
        prompt = request.data.get('prompt')
        
        # Set up the OpenAI API client
        openai.api_key = OPENAI_API_KEY
        
        # Define a generator function to stream the response
        def generate_response():
            for chunk in openai.ChatCompletion.create(
                model="gpt-3.5-turbo",
                messages=[{
                    "role": "user",
                    "content": prompt
                }],
                stream=True,
            ):
                content = chunk["choices"][0].get("delta", {}).get("content")
                if content is not None:
                    
                    yield content
        
        # Return a streaming response to the client
        return StreamingHttpResponse(generate_response(), content_type='text/event-stream')
    
    # Return a JSON error if the request method is not POST
    return JsonResponse({'error': 'Method not allowed.'}, status=405)

To see it in your terminal use this Curl command http://127.0.0.1:8000/api/v1/askkk --header "Content-Type: application/json" --data '{"prompt": "How do I set up payment invoices?"}'

Montagnard

This line has too many closing parentheses (can't edit it b/c SO forces you to change >6 chars): return StreamingHttpResponse(generate_response(), content_type='text/event-stream')) – Cleanse

How does this allow you to capture the value of content when the entire chat is complete? – Cleanse

Hey, great answer @Montagnard - would you happen to know how to do it in react, and through an AWS Lambda? – Edda

@Edda i had one hack for it, i doubt its correct way at all. I have not used lambda before. – Montagnard

not working it give full response at once. – Balaton

Is it better idea to use Django channel? – Fraga

Answer 2

Answer ended up being I was thinking of approaching it incorrectly. Use websockets! Not StreamingHttpResponse.

Meda

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. – Opsonize

Answer 3

Here is a 2024 version:

@csrf_exempt
def generate_names_view(request):
    print(request.method)
    if request.method == 'POST':
        # Parse the request body and extract the prompt
        data = json.loads(request.body)  # Use json.loads to parse request.body
        prompt = data.get('prompt')
        
        # Set up the OpenAI API client
        openai = OpenAI(api_key=settings.OPENAI_API_KEY)
        
        # Define a generator function to stream the response
        def generate_response():

            stream = openai.chat.completions.create(
                model="gpt-4",
                messages=[{"role": "user", "content": prompt}],
                stream=True,
            )
            for chunk in stream:
                if chunk.choices[0].delta.content is not None:
                    content = chunk.choices[0].delta.content
                    print(content)
                    yield content
        
        # Return a streaming response to the client
        return StreamingHttpResponse(generate_response(), content_type='text/event-stream')
    
    # Return a JSON error if the request method is not POST
    return JsonResponse({'error': 'Method not allowed.'}, status=405)

Here is the curl command to test it:

curl -X POST http://localhost:8000/test/ -H "Content-Type: application/json" -d '{"prompt":"Write the name of 10 dogs"}'

Here is how to use it from the browser:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Generate Names</title>

    <button onclick="fetchData()">Fetch Data</button>
    <div id="output"></div>
</head>
<body>
    <script>
    async function fetchData() {
        const url = '/test/' // Replace with your URL
        const data = {
        prompt: 'Write a short story about a robot who is trying to learn how to love.'
        }
        
        const response = await fetch(url, {
        method: 'POST', // Set the method to POST
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify(data) // Convert the JavaScript object to a JSON string
        })
        
        const reader = response.body.getReader();
        const decoder = new TextDecoder();
        while (true) {
        const { value, done } = await reader.read();
        if (done) {
            break;
        }
        document.getElementById('output').innerHTML += decoder.decode(value).replace(/\n/g, '<br>');
        }
    }

    {% comment %} fetchData().catch(console.error); {% endcomment %}
    </script>

</body>
</html>

Illinois

This works, but gives me an error at the end of the stream: "(56) OpenSSL SSL_read: error:0A000126:SSL routines::unexpected eof while reading, errno 0" It seems the stream is continuing to be read after eof. – Lysis