ffmpeg won't execute properly in google app engine standard nodejs

Question

I have tried for three full days to get GAE (standard - nodejs) to run a simple video transcoder from MOV to MP4 using ffmpeg. I have tried using ffluent-ffmpeg, kicking off a child process (e.g. spawn), and nothing works. As soon as it hits the call to the executable it always errors. I have confirmed ffmpeg is installed and even tried using ffmpeg-static. Moreover, I have it working on my local machine with no problems (using all of the aforementioned ways).

I have also tried logging the errors and nothing is really all that helpful. I can see its working through any installed package including ffmpeg (system package).

Below is the pseudo code...step three is where the problem occurs.

1. Send file name to GAE endpoint
2. Download the file from google cloud storage to a temp file
3. Transcode using ffmpeg
4. Upload temp file to google cloud storage
5. Remove old google cloud storage file
6. Remove temp file

The file I am using to test is 6MB...a 5 second video I took on my iPhone. Thank you in advance.

UPDATE: I successfully deployed the exact same code to Node Flex environment and everything works great. I wasn't able to get any errors in the standard environment that directed me where to look but my guess is it has something to do with how it stores the file I pipe into FFMPEG on GAE Node Standard. The docs say its a virtual file system that uses RAM. I'd love to hear if anybody managed to get it working in the standard environment.

Keijo

If we can't see the error it produces, how can we know what is going wrong? – Dissimulation

@dKorosec, I already mentioned I tried logging the errors but they didn't offer much otherwise I would have posted them. My question is more geared toward using FFMPEG in GAE Standard Node runtime. Its the first package listed in the "included packages" section of the documentation. Seems odd they would include a package that isn't useable in the runtime configuration. – Keijo

Hey Tommy, are you spawning new VM instances to process each video or do you have a single VM dedicated to the encoding? If it is the former, are you managing that programmatically or is there a GCP API which makes processing jobs like that safer? I'm leery about managing VM spawning as I don't want to end up in a scenario where I have runaway VM instances being spawned. Thanks. – Dulla

@Chance, sorry this comment is late. I have a queue that my VM monitors as well as code that sends out alerts when the queue gets too big, in which case I will start up a new VM. Transcoding is super expensive if you are processing a lot of data so I try to be mindful and use AWS Media Convert as a backup until I manually get a VM up and running. – Keijo

Answer 1

After a long battle, I finally figured out what was going on. I did not have enough compute resources. If anyone out there is going to build a transcoding service for images and videos, be sure you up your cores to at least 4 out of the gate. My jobs were randomly failing (but not repeatable for processing the same files), web sockets were disconnecting and reconnecting, etc.

To the person who downgraded my question because I did not post an error (which I stated I did not really have)...well, there isn't going to necessarily be an error in the logs when your CPU starts dropping jobs because it can't keep up with the load. Like I mentioned in my question, I would get errors but nothing meaningful.

Keijo

Was there anything more to this; we got 4 cores but it's still dropping? – Troublesome

Are you running it in Docker? – Keijo

Also, some filters can eat up a ton of memory. – Keijo

Answer 2

You're right, ffmpeg is listed in the pre-installed packages for the Node.js Runtime.

However they don't mention which ffmpeg version is installed.

I looked into the fluent-ffmeg prerequisites and it requires ffmpeg >= 0.9 to work.

Try to update your ffmpeg version running the command:

apt-get update ffmpeg 

in your instance's console. Tell us how it goes.

Smashed

I thought about that which is why I tried FFmpeg-static but did not work. – Keijo

Answer 3

const outFile = bucket.file(`${storagePath}`)
//create a referance to your storage bucket path
const outStream = outFile.createWriteStream()

You can always put a on 'stderr' listener to your ffmpeg command. I had similar problems transcoding on google app engine so fluent ffmpeg stderr listener helped me a lot debugging it.

ffmpeg.addInput(`tmp/${app_engine_filepath})
.format('mp3')
.on('stderr', function(stderrLine) {
  console.log('Stderr output: ' + stderrLine);
})
.on('error, (error) => {
  console.log(error)
})
.pipe(outStream)
.on('end', () => {
   fs.remove(`tmp/${app_engine_filepath}`)
})

You might also want to check your ffmpeg version on standart environment. (that should also be viewable through stderrlogs)

Wandawander

Thank you for your comment. I tried your above code and it didn't work along with a number of other variations and it did not work in the STANDARD environment. – Keijo

which part do you have trouble with? can you make a call to ffmpeg at all ? do you have trouble with installing ffmpeg to standart environment? – Wandawander