Colour image file IO with Haskell repa array library

Question

I'm exploring the Haskell repa library by trying out the multitude of programming examples. I'm aiming to implementation common image processing algorithms using repa.

Repa examples

There are some helpful code examples in the repa repository. They all operate on images of type Array U DIM2 a or Array DIM2 Float or Array U DIM2 Double.

-- three image types used below
type Image = Array U DIM2 Double
type Image = Array DIM2 Float
type Image = Array U DIM2 (Word8, Word8, Word8)

-- examples/Blur/src-repa/Main.hs
blur :: Monad m => Int -> Array U DIM2 Double -> m (Array U DIM2 Double)

-- examples/Laplace/src-repa/SolverStencil.hs
solveLaplace :: Monad m => Int -> Array U DIM2 Double -> Array U DIM2 Double -> Array U DIM2 Double -> m (Array U DIM2 Double)

-- examples/Sobel/src-repa/SolverSeparated.hs
type Image = Array DIM2 Float
gradientX_sep :: Image -> Image
gradientX1    :: Image -> Image
gradientX2    :: Image -> Image
gradientY_sep :: Image -> Image
gradientY2    :: Image -> Image

-- examples/Canny/src-repa/Main.hs
type Image a = Array U DIM2 a
toGreyScale  :: Image (Word8, Word8, Word8) -> IO (Image Float)
blurSepX     :: Image Float -> IO (Image Float)
blurSepY     :: Image Float -> IO (Image Float)
gradientX    :: Image Float -> IO (Image Float)
gradientY    :: Image Float -> IO (Image Float)
suppress     :: Float -> Float -> Image (Float, Word8) -> IO (Image Word8)
wildfire     :: Image Word8 -> Array U DIM1 Int -> IO (Image Word8)
selectStrong :: Image Word8 -> IO (Array U DIM1 Int)
gradientMagOrient :: Float -> Image Float -> Image Float -> IO (Image (Float, Word8))

Image file IO

There are two options for image file IO:

1. repa-devil package that supports PNG, BMP, JPG, TIF. Unfortunately, they are parsed into an array type that does not conform to the repa examples above, as confirmed by the repa-devil maintainer here.
2. repa-io package more closely corresponds to array type parameters for images in repa-examples, but only supports BMP files.

repa-devil (not compatible with repa-examples)

Images in the repa-examples package are of type Array F DIM3 Word8, or Array F DIM2 Word8 if it's a greyscale image. This means that repa-devil cannot be used to read images to be processed with the examples in repa-examples, because images in repa-examples are two dimensional arrays whilst images in repa-devil are three dimensional arrays.

readImage :: FilePath -> IL Image
writeImage :: FilePath -> Image -> IL ()
data Image = RGBA (Array F DIM3 Word8)
           | RGB  (Array F DIM3 Word8)
           | BGRA (Array F DIM3 Word8)
           | BGR  (Array F DIM3 Word8)
           | Grey (Array F DIM2 Word8)

repa-io (some compatibility with repa-examples)

There is a closer correspondence between repa-examples and repa-io.

readImageFromBMP :: FilePath -> IO (Either Error (Array U DIM2 (Word8,Word8, Word8)))
writeImageToBMP  :: FilePath -> Array U DIM2 (Word8, Word8, Word8) -> IO ()

This time, a BMP image file is being parsed into a two dimensional array with elements of type (Word8,Word8,Word8), presumably to represent R, G and B values. Even so, the only compatible function from the repa-examples package is toGreyScale from above. All other functions operate on values of type Array U DIM2 Float or Array DIM2 Float or Array U DIM2 Double.

Questions

1. With the exception of toGreyScale, are all examples in repa-examples only suitable for use on greyscale images? Whilst this would make sense looking at the types, it comes as a surprise that there are no repa examples for colour images. For example, why is the type for blur not instead: blur :: Monad m => Int -> Array U DIM2 (Word8, Word8, Word8) -> m (Array U DIM2 (Word8, Word8, Word8))
2. What value is the float capturing in Array U DIM2 Float? Is it a greyscale value between 0 and 255?
3. Has the been any work on adding JPG/PNG/TIF IO support in the repa-io package?

Solangesolano

Juicy Pixels also has a repa library and I believe the author just finished adding support for transitional jpgs – Mensa

@Mensa Thanks. It looks like this JuicyPixels repa library has adopted the same types as repa-devil, rather than those used in repa-examples i.e. with DIM3 type parameter hackage.haskell.org/package/JuicyPixels-repa-0.7/docs/… . – Solangesolano

The 'blur' example uses readImageFromBMP which returns a 3D array, the upper indices are pixel positions and the lower index is the value of a single channel. So if there are 3 channels, you will have 3 values for each (x,y) pair, I imagine. You could probably use readComponentsFromBMP which instead returns a 3-tuple of arrays, one for each colour component, if this is easier. The blur example simple converts back and forth Word8 to Double using fromIntegral and truncate. – Aretina

Answer 1

1. No reason. You could apply the single channel blur function to each RGB channel of a colour image.
2. With an image represented as an Array U DIM2 Float the elements typically have the range 0.0 - 1.0.
3. I don't think loading of JPG/PNG images should go in the repa-io package because this would induce a dependency on a foreign codec library. Use one of the other packages like repa-devil to load images.

The repa-devil package wraps the external DevIL library, so loaded images end up in foreign memory -- hence the F index in Array F DIM3 Word8. The DevIL library itself doesn't know how to build an unboxed U array in the Haskell heap.

The examples are only examples, and I wasn't intending repa-examples to be a fully featured image manipulation library. The fact that some arrays use a foreign F representation, and some use an unboxed U representation just reflects the standard issue with wrapping foreign code. If you want a uniform image manipulation API then you either need to change image representations at the boundary (which can introduce redundant copying), make the functions more polymorphic (which complicates their types), or hide the problem some how (which makes the cost model non-obvious). No matter which option you choose someone will complain about it.

Crossfade

Answer 2

For example, why is the type for blur not instead: blur :: Monad m => Int -> Array U DIM2 (Word8, Word8, Word8) -> m (Array U DIM2 (Word8, Word8, Word8))

1. Permament work with tuples instead of simple numbers is boilerplate, unless you use fixed-vector

2. Resulting code isn't reusable

3. You cannot parallelize computation by color channels

Take a look at yarr, specially designed for RGB-image processing. For example, you can define blur :: (Num v) => Array Dim2 v -> Array Dim2 v (approximate signature) and then apply it to Grayscale image, or colour:

let blurred = fromSlices $ map blur $ slices image

See https://github.com/leventov/yarr/blob/master/tests/blur.hs

Exact