Downloading File using Axios then uploading to Amazon S3

Question

I've seen various questions on SO about downloading files using Axios and uploading them to S3 but none that tie it all together and I'm getting confused with streams, blobs, multi-part forms etc. Here's my code so far.

Downloading the file.

const downloadResponse = await axios({
    url: `https://example.com/test.jpg`,
    method: 'GET',
    responseType: 'stream'   // Should this be blob, stream or arraybuffer?
})

Not sure what is contained within "downloadResponse.data" at this point, the typeof suggests it's an object, not a stream

Getting Signed response (this is done through Storyblok CMS not Amazon)

const signedResponse = await axios.post(`https://api.storyblok.com/v1/spaces/xxx/assets`, {
    filename: 'test.jpg',
}, {
    headers: {'Authorization': 'xxxxx'}
})

Creating the form data using form-data package

let form = new FormData()

for (var key in signedResponse.fields) {
    form.append(key, signedResponse.fields[key])
}

try {
    form.append('file', fs.createReadStream(downloadResponse.data))
} catch (error) {
    console.log("Error Creating Data", error)
}

Uploading to S3

try {
    
    const uploadResponse = await axios({
        method: 'post',
        url: signedResponse .data.post_url,
        data: form.getBuffer(),
        headers: form.getHeaders()
    })

} catch (error) {
    console.log("Error Uploading", error)
}

At the moment this is causing the following error when creating the stream..

Error Creating Data TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be one of type string, Buffer, or URL. Received type object.

I have tried taking the data directly from the first request and not using fs.createReadStream() on it as it should already be a stream but that gives the same error.

I have also tried returning an arraybuffer from the first request and using Buffer.from() and then attaching the buffer but that didn't work either, I think Amazon gave me a 400 error from that so I'm lost.

Any help would be really appreciated.

Ribband

This is what worked for me - #61605578 – Tertia

Answer 1

This code uses v3 version of AWS JS SDK. Using Node v16 (ES16 Module)

1. Initialize S3 Client

// Ref: https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/clients/client-s3/index.html
import { S3Client } from '@aws-sdk/client-s3';

// Constants
const accessKeyId = process.env.AWS_ACCESS_KEY
const secretAccessKey = process.env.AWS_SECRET_KEY
const awsRegion = 'ap-south-1';
export const awsBucket = process.env.AWS_BUCKET;

export const S3 = new S3Client({
  credentials: {
    accessKeyId,
    secretAccessKey,
  },
  region: awsRegion,
});

2. Download Using Axios.
   The most important part is using responseType: 'arraybuffer'

const resp = await axios.get('https://domain/image/imagename.png', {   
  decompress: false,
  // Ref: https://mcmap.net/q/1136765/-axios-get-a-file-from-url-and-upload-to-s3
  responseType: 'arraybuffer',
})

3. Upload Using S3 Client

import { PutObjectCommand } from '@aws-sdk/client-s3';
import { awsBucket, S3 } from '../lib/aws.js'; // Defined in Point 1 above

await S3.send(
  new PutObjectCommand({
    Bucket: awsBucket,
    Key: `foldername/filename`,
    Body: resp.data,
  })
)

Daiseydaisi

very good, thanks! – Mignonmignonette