How to render Streamable image on React coming from FastAPI server?

Question

I would like to render an image on React returned from FastAPI backend using StreamingResponse. The image is in the form of a numpy array, which is of cv2 type of object.

@app.post("/predict")
async def root(file: UploadFile = File(...)):
    global model
    global store_coordinates
    global store_faces
    global store_mesh

    content = await file.read()
    nparr = np.fromstring(content, np.uint8)
    bg_img = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
    ......................
    for i in range(len(store_coordinates)):
            x, y, w, h = store_coordinates[i]
            bg_img [b:b + d, a:a + c] = store_mesh[i]
    
    res,im_png = cv2.imencode(".png", bg_img)
    return StreamingResponse(io.BytesIO(im_png.tobytes()), media_type="image/png")

Here, I have created an API endpoint in which the uploaded image is received using POST request, and a StreamableResponse(Image) is returned. How can I render this returned response on React frontend?

React Code:

import React, { Component } from "react";
import axios from "axios";

class Detect extends Component {
  state = {
    title: "",
    content: "",
    image: null,
  };

  handleChange = (e) => {
    this.setState({
      [e.target.id]: e.target.value,
    });
  };

  handleImageChange = (e) => {
    this.setState({
      image: e.target.files[0],
    });
  };

  
  handleSubmit = (e) => {
    e.preventDefault();
    console.log(this.state);
    let form_data = new FormData();
    form_data.append("image", this.state.image, this.state.image.name);
    let url = "http://127.0.0.1:8000/predict";
    axios
      .post(url, form_data, {
        headers: {
          "content-type": "multipart/form-data",
        },
      })
      .then((res) => {
        console.log(res.data);
      })
      .catch((err) => console.log(err));
  };

  render() {
    return (
      <div className="App">
        <form onSubmit={this.handleSubmit}>
          <p>
            <input
              type="file"
              id="image"
              accept="image/png, image/jpeg"
              onChange={this.handleImageChange}
              required
            />
          </p>
          <input type="submit" />
        </form>
        <div id="image-render">
          <img></img>
        </div>
      </div>
    );
  }
}

export default Detect;

I would like to render the returned image in the div tag which has the id of image-render.

Edit - This is the response I get from my backend.

Bullet

You can set the src of the img tag to data:image/png;base64,<base 64 encoded img data> - either return base64 data directly from your API or convert it in Javascript by handling the response from the server. – Exploit

Answer 1

There are two options:

1. encode the image data into Base64 format on server side and return the base64-encoded string, as shown in this answer and this answer, for example:

   base64_string= base64.b64encode(image_bytes).decode('utf-8')
   return base64_string
   

   which can then be used to display the image in an HTML page as shown here, for instance:

   <img src="data:image/png;base64,ADD_BASE64_STRING_HERE>
   

2. or, send the raw bytes from the server, as you currently do. However, it might be best not to use a StreamingResponse (as shown in the example you provided) for sending the raw bytes, as, in your case, the entire image bytes are already loaded into memory. Thus, you should rather use a custom Response, as explained in this answer and this answer, for example:

   return Response(image_bytes, media_type='image/jpeg')
   

   On client side (using JavaScript), convert the raw bytes into either a base64-encoded string (using btoa(), String.fromCharCode() and Uint8Array), or a Blob object (and then call URL.createObjectURL() to create a URL representing the Blob object).

The examples below demonstrate how to implement the two methods explained in Option 2 above (i.e., having the server sending raw image bytes and, on client side, converting the image bytes to either a base64-encoded string or Blob object), using either Fetch API or Axios library.

Using Fetch API

Option 1 - Convert raw image bytes into Blob object

fetch('/predict', {
        method: 'POST',
        body: formData,
    })
    .then(response => response.blob())
    .then(blob => {
        var blobURL = URL.createObjectURL(blob);
        var image = document.getElementById("myImage");
        image.onload = function(){
            URL.revokeObjectURL(this.src); // release the blob URL once the image is loaded
        }
        image.src = blobURL;
    })
    .catch(error => {
        console.error(error);
    });

Option 2 - Convert raw image bytes into base64-encoded string

fetch('/predict', {
        method: 'POST',
        body: formData,
    })
    .then(response => {
        contentType = response.headers.get('content-type')
        return response.arrayBuffer();
    })
    .then(arrayBuffer => {
        base64string = btoa(String.fromCharCode(...new Uint8Array(arrayBuffer)))
        var image = document.getElementById("myImage");
        image.src = "data:" + contentType + ";base64," + base64string;
    })
    .catch(error => {
        console.error(error);
    });

Remember to define an <img> tag in your HTML file, where you wish to display the image:

<img id="myImage" src="">

Using Axios

Option 1 - Convert raw image bytes into Blob object

axios({
        method: 'POST',
        url: '/upload',
        data: formData,
        headers: {
            'Content-Type': 'multipart/form-data'
        },
        responseType: "blob"
    })
    .then(response => {
        var blobURL = URL.createObjectURL(response.data);
        var image = document.getElementById("myImage");
        image.onload = function(){
            URL.revokeObjectURL(this.src); // release the blob URL once the image is loaded
        }
        image.src = blobURL;
    })
    .catch(error => {
        console.error(error);
    });

Option 2 - Convert raw image bytes into base64-encoded string

axios({
        method: 'POST',
        url: '/predict',
        data: formData,
        headers: {
            'Content-Type': 'multipart/form-data'
        },
        responseType: "arraybuffer"
    })
    .then(response => {
        base64string = btoa(String.fromCharCode(...new Uint8Array(response.data)))
        contentType = response.headers['content-type']
        return base64string;
    })
    .then(base64string => {
        var image = document.getElementById("myImage");
        image.src = "data:" + contentType + ";base64," + base64string;
    })
    .catch(error => {
        console.error(error);
    });

Remember to define an <img> tag in your HTML file, where you wish to display the image:

<img id="myImage" src="">

Housley

Please look at the edit part I am getting this message in the console while implementing the solution – Bullet