How to Determine and Auto-Rotate Images?

Question

I have bunch of images, among them some of the images have to be rotated.

Sample:

I want to rotate this image 90° counter-clockwise.

I Googled to know how can I rotate an image and found many links and SO threads. But how can I determine if the image needs to be rotated? Picasa has a Auto-Rotating feature. I want to have similar functionality.

Any pointer would be very helpful to me.

I have found a link but it is related to Android.

Scribe

Very tricky. What if she was really laying down? – Premonitory

@RogerRowland good one :D. Maybe if it is possible to determine whether or not an image which is supposed to be portrait but placed as landscape then only we can rotate it! I am not sure if I have thought this right way. – Scribe

Maybe some image formats have a clue in meta-data? I seem to recall something in EXIF (thinking out loud) ... – Premonitory

@RogerRowland I have found this link #12727360 it deals with the EXIF, but it is about Android. – Scribe

If you're looking for some java code, maybe this would be useful? Or perhaps this? – Premonitory

@RogerRowland thank you very much. Your pointer solved the problem. If you please provide the pointer as answer to this question, I can accept it. – Scribe

No problem, you've already posted the answer, just accept that - I'm not worried about the rep, just as long as you're sorted :-) – Premonitory

If the image has the EXIF orientation tag, then using ImageMagick will automatically re-orient the image. convert image -auto-orient result. see imagemagick.org/script/command-line-options.php#auto-orient – Mating

The image you posted does not have EXIF data. But that may be due to the upload process to this forum. In Imagemagick, with current versions, you can use convert image -format "%[EXIF:orientation]" info: to see if your image has such. Alternately, you can do identify -verbose image and look at the long listing of information for the EXIF tags, if they exist. If the imaged does not have EXIF orientation, your problem becomes much more difficult. You would likely need some deep learning code, but I have not seen anything like that, though I have not searched for such. – Mating

For a Python solution, see: #4229030 – Proceleusmatic

Answer 1

The pointer of metadata-extractor which Roger Rowland has provided solved the problem. I am posting it here for future reference:

import java.awt.geom.AffineTransform;
import java.awt.image.AffineTransformOp;
import java.awt.image.BufferedImage;
import java.io.File;

import javax.imageio.ImageIO;

import com.drew.imaging.ImageMetadataReader;
import com.drew.metadata.Metadata;
import com.drew.metadata.exif.ExifIFD0Directory;
import com.drew.metadata.jpeg.JpegDirectory;

public class Main {

    private static String inFilePath = "C:\\Users\\TapasB\\Desktop\\MHIS031522.jpg";
    private static String outFilePath = "C:\\Users\\TapasB\\Desktop\\MHIS031522-rotated.jpg";

    public static void main(String[] args) throws Exception {
        File imageFile = new File(inFilePath);
        BufferedImage originalImage = ImageIO.read(imageFile);

        Metadata metadata = ImageMetadataReader.readMetadata(imageFile);
        ExifIFD0Directory exifIFD0Directory = metadata.getDirectory(ExifIFD0Directory.class);
        JpegDirectory jpegDirectory = (JpegDirectory) metadata.getDirectory(JpegDirectory.class);

        int orientation = 1;
        try {
            orientation = exifIFD0Directory.getInt(ExifIFD0Directory.TAG_ORIENTATION);
        } catch (Exception ex) {
            ex.printStackTrace();
        }

        int width = jpegDirectory.getImageWidth();
        int height = jpegDirectory.getImageHeight();

        AffineTransform affineTransform = new AffineTransform();

        switch (orientation) {
        case 1:
            break;
        case 2: // Flip X
            affineTransform.scale(-1.0, 1.0);
            affineTransform.translate(-width, 0);
            break;
        case 3: // PI rotation
            affineTransform.translate(width, height);
            affineTransform.rotate(Math.PI);
            break;
        case 4: // Flip Y
            affineTransform.scale(1.0, -1.0);
            affineTransform.translate(0, -height);
            break;
        case 5: // - PI/2 and Flip X
            affineTransform.rotate(-Math.PI / 2);
            affineTransform.scale(-1.0, 1.0);
            break;
        case 6: // -PI/2 and -width
            affineTransform.translate(height, 0);
            affineTransform.rotate(Math.PI / 2);
            break;
        case 7: // PI/2 and Flip
            affineTransform.scale(-1.0, 1.0);
            affineTransform.translate(-height, 0);
            affineTransform.translate(0, width);
            affineTransform.rotate(3 * Math.PI / 2);
            break;
        case 8: // PI / 2
            affineTransform.translate(0, width);
            affineTransform.rotate(3 * Math.PI / 2);
            break;
        default:
            break;
        }       

        AffineTransformOp affineTransformOp = new AffineTransformOp(affineTransform, AffineTransformOp.TYPE_BILINEAR);  
        BufferedImage destinationImage = new BufferedImage(originalImage.getHeight(), originalImage.getWidth(), originalImage.getType());
        destinationImage = affineTransformOp.filter(originalImage, destinationImage);
        ImageIO.write(destinationImage, "jpg", new File(outFilePath));
    }
}

Scribe

I think the API has changed in the current version (2.8.1 at time of writing) of metadata-extractor: metadata.getDirectory is now metadata.getFirstDirectoryOfType. – Barfuss

Answer 2

I had some issues getting some of the switch cases to work. Even when there was no rotation to be done, the AffineTransform would create a new image with black space in the image and would chop off some of the dimensions. Piggy backing off of the accepted answer here, I used the metadata-extractor class to determine what the orientation should be. Then I used the Imgscalr library for scaling and rotation.

The complete solution that worked for me can be seen below. Thank you Tapas Bose for the original solution. I hope that this helps anyone!

BufferedImage originalImage = Utils.prepareBufferedImage(fileUpload.getFile_data(), fileUpload.getFile_type());
                    BufferedImage scaledImg = Scalr.resize(originalImage, 200);

                    // ---- Begin orientation handling ----
                    Metadata metadata = ImageMetadataReader.readMetadata(fileUpload.getFile_data());
                    ExifIFD0Directory exifIFD0Directory = metadata.getFirstDirectoryOfType(ExifIFD0Directory.class);

                    int orientation = Integer.parseInt(id);
                    try {
                        orientation = exifIFD0Directory.getInt(ExifIFD0Directory.TAG_ORIENTATION);
                    } catch (Exception ex) {
                        logger.debug("No EXIF information found for image: " + fileUpload.getFile_name());
                    }

                    switch (orientation) {
                    case 1:
                        break;
                    case 2: // Flip X
                        scaledImg = Scalr.rotate(scaledImg, Rotation.FLIP_HORZ);
                        break;
                    case 3: // PI rotation
                        scaledImg = Scalr.rotate(scaledImg, Rotation.CW_180);
                        break;
                    case 4: // Flip Y
                        scaledImg = Scalr.rotate(scaledImg, Rotation.FLIP_VERT);
                        break;
                    case 5: // - PI/2 and Flip X
                        scaledImg = Scalr.rotate(scaledImg, Rotation.CW_90);
                        scaledImg = Scalr.rotate(scaledImg, Rotation.FLIP_HORZ);
                        break;
                    case 6: // -PI/2 and -width
                        scaledImg = Scalr.rotate(scaledImg, Rotation.CW_90);
                        break;
                    case 7: // PI/2 and Flip
                        scaledImg = Scalr.rotate(scaledImg, Rotation.CW_90);
                        scaledImg = Scalr.rotate(scaledImg, Rotation.FLIP_VERT);
                        break;
                    case 8: // PI / 2
                        scaledImg = Scalr.rotate(scaledImg, Rotation.CW_270);
                        break;
                    default:
                        break;
                    }       
                    // ---- End orientation handling ----

                    if(fileUpload.getFile_type().toLowerCase().contains("jpeg")){
                        ImageIO.write(scaledImg, "jpeg", fileUpload.getFile_data());
                        user.setProfile_picture_ext("jpg");
                    }
                    else{
                        Sanselan.writeImage(scaledImg, fileUpload.getFile_data(), ImageFormat.IMAGE_FORMAT_PNG, null);
                        user.setProfile_picture_ext("png");
                    }

Upton

what is this garbage code? Integer.parseInt(id) ??? – Jerusalem

Answer 3

Sometimes you aren't working from images captured by a camera and don't have EXIF data to work with. In my case, I have a project to scan and import into our digital repository tens of thousands of vintage postcards most of which are landscape on the front, and only a small percentage of which are portrait (the backs remain landscape even on these). To minimize the time spent scanning, deskewing, and cropping these they are done three-per-scan and landscape (always).

I came to this question looking for an answer to automating the detection and rotation of these portrait orientation card scans.

There are many discussions about doing this based on camera metadata. There are some examples of how to use machine learning to automatically level a photo where the camera was not held plum to the ground/horizon. I haven't found any that would help in my situation (which is not to say there aren't any, but if there are they are hard to find because of the other situations)...

EDIT 3/22/2019: Here's one https://d4nst.github.io/2017/01/12/image-orientation/

..., so I did come up with an answer that I'm going to try:

For each card front (batch processing using ImageMagick and simple scripts):

1. Make a smaller jpeg version of the image (because we don't want to 200MB images to work with)
2. Make three more copies of that smaller jpeg, each with an additional 90 degrees of rotation applied to it
3. Submit each of these four orientations to a cloud machine learning API (I've had good luck with Microsoft's in the past)
4. Analyze the responses. Chose the most detailed and/or with the highest confidence as the correct orientation
5. Rotate the original full-size scan the appropriate amount and delete the four smaller jpegs.

I've tested this with one scan and in my n=1 case the API had a much longer list of tags and a better (and longer) suggested caption for the correct orientation.

Potential problems:

1. The cloud provider may discontinue the API or start charging more than we can afford (when I scripted a test of metadata creation using a batch of these cards the level of use stayed in the free category).
2. I think Microsoft may already rotate the image you send it for OCR purposes (to catch words written in any orientation) if they start applying the more generalized metadata AI to all the orientations as well then this could stop working (although, one would hope they would add a key in the response for best orientation guess).
3. (In my situation) postcards often have writing on them at orientations not true to the image (photography studio names, etc.). If they aren't doing the above as I suspect then better OCR at one rotation could fool the script. One might have to ignore OCR results if that proves to be a problem.
4. Uses bandwidth.

Choiseul

Great thinking. You could also try with Amazon Rekognition to analyze the images. – Scribe

@TapasBose Thanks! I haven't tried that one yet. I've tested Microsoft's, Google's, and IBM's offerings. Microsoft's seems to the best of those at the moment. – Choiseul

@TapasBose I wrote this up in Powershell: gist.github.com/pr3sidentspence/… it gets most of them right. – Choiseul

Forgot to mention that it requires ImageMagick 7+ (less if you change magick.exe to IMPATH\convert.exe) – Choiseul

Thank you for the effort :) – Scribe

It was for selfish reasons, It's a hassle to check each postcard manually. :) – Choiseul

It failed a lot on "word-art" postcards (not our image, but this is an ex. manitobaphotos.com/Postcards/top.jpg ) so I'm adding in some checks that skip cards like that (I have to use keywords like "text", "sign", and "book" in the tags and captions to weed them out. I've got it skipping most of them, I hope to have it fail over to best OCR results in the future (OCR has difficulty with the word art, too, but there's often accompanying text that's more straightforward). – Choiseul

Is it possible to apply machine learning, where the system can learn how to distinguish between "word-art" and "non-word-art"? – Scribe

@TapasBose I would think so, but you'd need training sets in the thousands, or so I've been told by someone who worked on Amazon's quick-deploy machine learning service, "Deep Learning AMIs" ( aws.amazon.com/machine-learning/amis ). I think it's beyond me right now. – Choiseul

@TapasBose And now I'm installing tensorflow... thanks :) – Choiseul

I'm really glad 🙂 – Scribe

Please let me know – Scribe

It's strange, I swear this is the article I referenced when I mentioned straightening slightly skewed images, but it does discuss how to apply machine learning to this exact problem: d4nst.github.io/2017/01/12/image-orientation – Choiseul