AWS Rekognition InvalidImageFormatException when jpeg source is mjpeg stream

Question

I'm pulling jpg frames out of a mjpg stream. These are valid jpg files and work in any image tool I've tried; however, Rekognition will not accept them either when sending it as Bytes, or when I move them to S3 and try that route.

I've made a few versions (attached here), all from the same source jpg (I would include them inline but I don't want image optimization code to alter them)

test.jpg - original frame

test-photoshop.jpg - opened in Photoshop, "save for web"d

test-imageoptim.jpg - run through ImageOptim (which I believe compresses with jpegtran)

Looking at these in a hex editor, the only difference I can't see is more exif data (using exiftool). When I run exiftool on the original, it still reports back all the basic details of the frame.

I'm assuming this is a bug with Rekognition, or there is some specific exif bit it's looking for that my mjpeg stream extraction is omitting. Maybe someone has information on why pulling jpeg frames from mjpeg isn't possible by just attaching the right start and end frame bytes.

Restive

Just a guess, try chopping off the last 6 bytes, after the final FF D9 marker, perhaps those bytes are confusing Amazon. – Crin

@ChrisO this works! So maybe my method of extracting the frame from the mjpeg stream is padding the end with zero bytes.. If you put this in an answer I'll accept it – Restive

Answer 1

From comment:

Just a guess, try chopping off the last 6 bytes, after the final FF D9 marker, perhaps those bytes are confusing Amazon.

Here's a snap from a hex viewer:

Crin

This was the correct answer. I am watching the mjpg stream for frames and extracting them, prepending and appending the sof and eof markers, but in some cases it was adding this zero byte tail after the eof, which amazon didn't like. It would read fine in other applications, I'm guessing because jpg doesn't require the eof marker – Restive

JPEG does require the EOF marker. However, many decoders will just stop when they hit it. – Drove

Answer 2

I can only make an educated guess because I do not have access to the application that is causing the problem. However, my guess is that it does not recognize the file format (or lack thereof) of the JPEG stream.

You have a Start of Frame Marker right after the Start of Image marker. That is legal according to the JPEG standard. However, there is nothing to indicate a file format. This is not strictly necessary to interpret the JPEG stream (assuming a 3 component scan is YCbCr encoded) but your application may be looking for some format specific marker (e.g., JFIF, EXIF).

Drove

Answer 3

Your images are fine. Your code is buggy or your permissions are not right. I used https://github.com/yourtion/Rekognition_API_NODEJS_SDK with your test.jpg and here is the result:

{
   "url":"POST",
   "face_detection":[
      {
         "boundingbox":{
            "tl":{
               "x":470.77,
               "y":109.23
            },
            "size":{
               "width":53.85,
               "height":53.85
            }
         },
         "confidence":0.25,
         "eye_left":{
            "x":489.3,
            "y":132.4
         },
         "eye_right":{
            "x":498,
            "y":128.9
         },
         "nose":{
            "x":504.1,
            "y":142.5
         },
         "mouth l":{
            "x":497.7,
            "y":152.1
         },
         "mouth_l":{
            "x":497.7,
            "y":152.1
         },
         "mouth r":{
            "x":502.4,
            "y":151.9
         },
         "mouth_r":{
            "x":502.4,
            "y":151.9
         },
         "pose":{
            "roll":-27.01,
            "yaw":86.01,
            "pitch":-12.56
         },
         "quality":{
            "brn":0.22,
            "shn":0.1
         },
         "emotion":{
            "happy":0.25,
            "surprised":0.09,
            "sad":0.07
         },
         "age":30.11,
         "smile":0.71,
         "glasses":0.79,
         "sunglasses":0.87,
         "beard":0.96,
         "mustache":0.52,
         "eye_closed":1,
         "mouth_open_wide":0.04,
         "beauty":0.5041,
         "sex":1
      }
   ],
   "ori_img_size":{
      "width":640,
      "height":480
   },
   "usage":{
      "quota":98,
      "status":"Succeed.",
      "api_id":"XXXXXXXXXXXXXXXXXXXX"
   }
}

Labanna

I don't know how that's possible. I tried uploading the test.jpg directly to the Rekognition console on AWS console.aws.amazon.com/rekognition/home?region=us-east-1#/… and it fails with the same error. I've also uploaded a valid image from node.js through the sdk and it works. @chris_o 's comment above does work stripping everything after the eof marker. Can you really upload the test.jpg through the api and get a valid result? – Restive

Tried the console. Got "Bad Request (400) - InvalidImageFormatException Invalid image encoding". My guess is console does some sort of preliminary check/validation. – Labanna