ios Vision VNImageRequestHandler orientation issue

Question

I am trying to detect faces via camera using VNImageRequestHandler (iOS Vision). When I point on the photo by the camera in landscape mode it detects faces but with opposite orientation mode.

  let detectFaceRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, options: [:])

Lavation

Answer 1

Have you tried to play with the VNImageRequestHandler orientation property?

let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right, options: [:])

I had to set it to .right while reading video input from back camera in portrait mode.

Headstone

Answer 2

Convert the image to CIImage and apply the orientation property like below and pass it to the imagerequest handler

let orientation = CGImagePropertyOrientation(uiImage.imageOrientation)
   let imageElement = ciImage.applyingOrientation(Int32(orientation.rawValue))

        // Show the image in the UI.
        originalImage.image = uiImage

also check https://github.com/gunapandianraj/iOS-Vision code for converting Vision rect to UIKit rect

Hoofbeat

on trying to work with the above code for a I get Use of unresolved identifier 'uiImage'as soon as I attempt to declare the variable orientation in the first line. Is there a specific library that needs to be imported for this to work? – Haddix

sorry for the naming convention . uiImage is the local variable of type UIImage which will be assigned with image i picked from photo library or taking photo or downloaded image ... – Hoofbeat

what is imageElement? and where is used? – Katinka

Answer 3

I always got flipped points (bounding boxes) vertically. I fixed this totally with helper method:

private static func translateVisionToNormalBoundingBox(bb: CGRect, imageFullRect: CGRect) -> CGRect
{
    let renormalized = VNImageRectForNormalizedRect(bb, Int(imageFullRect.width), Int(imageFullRect.height))
    // Vertically translate origin !!!
    // Vertically translate origin !!!
    // Vertically translate origin !!!
    return CGRect(
        origin: CGPoint(
            x: renormalized.origin.x,
            y: imageFullRect.maxY - renormalized.origin.y - renormalized.size.height
        ),
        size: renormalized.size
    )
}

For cgImage orientation I use this StackOverflow's extension:

extension UIImage {

var cgImagePropertyOrientation: CGImagePropertyOrientation {
    switch imageOrientation
    {
    case .down: return .down
    case .left: return .left
    case .right: return .right
    case .up: return .up
    case .downMirrored: return .downMirrored
    case .leftMirrored: return .leftMirrored
    case .rightMirrored: return .rightMirrored
    case .upMirrored: return .upMirrored
        
        // TWEAK FOR NEW CASES !!!
        // TWEAK FOR NEW CASES !!!
        // TWEAK FOR NEW CASES !!!
    @unknown default:
        return .down
    }
}

}

Surefooted