Java swt - Get actual x,y coordinates from Image after scaling and zooming

Question

I have an image that has been scaled to fit. From the scaled image a user is selecting a rectangle.

I then re-draw based on this selection:

gc.drawImage(imageDisplayed, minX, minY, width, height, imageDisplayed.getBounds().x,  imageDisplayed.getBounds().y, imageDisplayed.getBounds().width, imageDisplayed.getBounds().height );

So now I want to be able to get the original co-ordinate from the scaled AND zoomed image. Is this correct?:

public Coordinate GetScaledXYCoordinate(int oldX, int oldY, int width, int height, int scaledWidth, int scaledHeight)
{       
    int newX = (int)(oldX * width)/scaledWidth;
    int newY = (int)(oldY * height)/scaledHeight;

    Coordinate retXY = new Coordinate(newX, newY);
    return retXY;
}


public Coordinate GetZoomedXYCoordinate(int oldX, int oldY, int startX, int endX, int startY, int endY,
        int width, int height,int scaledWidth, int scaledHeight)
{       
    // First get x,y after scaling
    Coordinate xy = GetScaledXYCoordinate(oldX, oldY, width, height, scaledWidth, scaledHeight);

    // Now get x.y after zooming
    int minX = Math.min(startX, endX);
    int minY = Math.min(startY, endY);

    int maxX = Math.max(startX, endX);
    int maxY = Math.max(startY, endY);

    int rectWidth = maxX - minX;
    int rectHeight = maxY - minY;
    return GetScaledXYCoordinate(xy.getX(), xy.getY(), width, height, scaledWidth, scaledHeight);
}

Note: I would like an algorithm that would work for many zooms, not just one zoom.

Update:

Ideally, I would like a function that takes a screen Point X,Y, and returns the original image X,Y. The function would still return the correct X,Y after scaling and zooming

Shuping

Still stumped on this one – Shuping

It would help if you'd be able to post an Minimal, Complete, and Verifiable example. – Penrod

Do I understand correctly that you want to translate screen/window coordinates to image coordinates? – Spirogyra

@Spirogyra yes. After scaling and after "zooming". Zooming is just taking a rectangular section and scaling up. – Shuping

Answer 1

Here is a complete working example to zoom into an image using SWT which implements the idea behind Leon's answer. Using affine transformations is the default approach for drawing elements with individual coordinate systems in 2D graphics.

1. Use a Transform for drawing the picture at the right place and scale
2. Use the inverse of that Transform to get the image coordiantes of the selected zoom region.
3. Compute a new Transform to display the zoomed region.

The class below does the following:

• The Transform is stored in paintTransform.
• The screen coordiantes of the zoomed area are stored in zoomStart and zoomEnd
• The image coordinates of the selected area are computed in setVisibleImageAreaInScreenCoordinates from the dragged zoom rectangle.
• The new Transform is computed in setVisibleImageAreaInImageCoordinates
• Most of the rest can be considered boilerplate code.

Please note that the image is never replaced with a scaled version. It is drawn using the paintTransform. That means that the graphics context takes care of painting the image scaled. The actual painting code becomes as simple as

ev.gc.setTransform(paintTransform);
ev.gc.drawImage(img, 0, 0);

All the computation is done during the handling during the state transition triggered by the mouse events, i.e. the zoom() method called in the mouseUp() handler.

import java.io.InputStream;
import java.net.URL;

import org.eclipse.swt.events.MouseEvent;
import org.eclipse.swt.events.MouseListener;
import org.eclipse.swt.events.MouseMoveListener;
import org.eclipse.swt.events.PaintEvent;
import org.eclipse.swt.events.PaintListener;
import org.eclipse.swt.graphics.Color;
import org.eclipse.swt.graphics.GC;
import org.eclipse.swt.graphics.Image;
import org.eclipse.swt.graphics.ImageData;
import org.eclipse.swt.graphics.Point;
import org.eclipse.swt.graphics.RGB;
import org.eclipse.swt.graphics.Transform;
import org.eclipse.swt.widgets.Display;
import org.eclipse.swt.widgets.Shell;

public class Zoom implements PaintListener, MouseMoveListener, MouseListener {
    private static final int MOUSE_DOWN = 1;
    private static final int DRAGGING = 2;
    private static final int NOT_DRAGGING = 3;

    int dragState = NOT_DRAGGING;
    Point zoomStart;
    Point zoomEnd;

    ImageData imgData;
    Image img;
    Transform paintTransform;
    Shell shell;
    Color rectColor;

    public Zoom(ImageData image, Shell shell) {
        imgData = image;
        img = new Image(shell.getDisplay(), image);
        this.shell = shell;
        rectColor = new Color(shell.getDisplay(), new RGB(255, 255, 255));
    }

    void zoom() {
        int x0 = Math.min(zoomStart.x, zoomEnd.x);
        int x1 = Math.max(zoomStart.x, zoomEnd.x);
        int y0 = Math.min(zoomStart.y, zoomEnd.y);
        int y1 = Math.max(zoomStart.y, zoomEnd.y);

        setVisibleImageAreaInScreenCoordinates(x0, y0, x1, y1);
    }

    void setVisibleImageAreaInImageCoordinates(float x0, float y0,
            float x1, float y1) {
        Point sz = shell.getSize();

        double width = x1 - x0;
        double height = y1 - y0;

        double sx = (double) sz.x / (double) width;
        double sy = (double) sz.y / (double) height;

        float scale = (float) Math.min(sx, sy);

        // compute offset to center selected rectangle in available area
        double ox = 0.5 * (sz.x - scale * width);
        double oy = 0.5 * (sz.y - scale * height);

        paintTransform.identity();
        paintTransform.translate((float) ox, (float) oy);
        paintTransform.scale(scale, scale);
        paintTransform.translate(-x0, -y0);
    }

    void setVisibleImageAreaInScreenCoordinates(int x0, int y0,
            int x1, int y1) {
        Transform inv = invertPaintTransform();
        // points in screen coordinates
        // to be transformed to image coordinates
        // (top-left and bottom-right corner of selection)
        float[] points = { x0, y0, x1, y1 };

        // actually get image coordinates
        // (in-place operation on points array)
        inv.transform(points);
        inv.dispose();

        // extract image coordinates from array
        float ix0 = points[0];
        float iy0 = points[1];
        float ix1 = points[2];
        float iy1 = points[3];

        setVisibleImageAreaInImageCoordinates(ix0, iy0, ix1, iy1);
    }

    Transform invertPaintTransform() {
        // clone paintTransform
        float[] elems = new float[6];
        paintTransform.getElements(elems);
        Transform inv = new Transform(shell.getDisplay());
        inv.setElements(elems[0], elems[1], elems[2],
                        elems[3], elems[4], elems[5]);

        // invert clone
        inv.invert();
        return inv;
    }

    void fitImage() {
        Point sz = shell.getSize();

        double sx = (double) sz.x / (double) imgData.width;
        double sy = (double) sz.y / (double) imgData.height;

        float scale = (float) Math.min(sx, sy);

        paintTransform.identity();
        paintTransform.translate(sz.x * 0.5f, sz.y * 0.5f);
        paintTransform.scale(scale, scale);
        paintTransform.translate(-imgData.width*0.5f, -imgData.height*0.5f);
    }

    @Override
    public void paintControl(PaintEvent ev) {
        if (paintTransform == null) {
            paintTransform = new Transform(shell.getDisplay());
            fitImage();
        }

        ev.gc.setTransform(paintTransform);
        ev.gc.drawImage(img, 0, 0);

        if (dragState == DRAGGING) {
            drawZoomRect(ev.gc);
        }
    }

    void drawZoomRect(GC gc) {
        int x0 = Math.min(zoomStart.x, zoomEnd.x);
        int x1 = Math.max(zoomStart.x, zoomEnd.x);
        int y0 = Math.min(zoomStart.y, zoomEnd.y);
        int y1 = Math.max(zoomStart.y, zoomEnd.y);

        gc.setTransform(null);

        gc.setAlpha(0x80);
        gc.setForeground(rectColor);
        gc.fillRectangle(x0, y0, x1 - x0, y1 - y0);
    }

    public static void main(String[] args) throws Exception {
        URL url = new URL(
                "https://upload.wikimedia.org/wikipedia/commons/thumb/"  +
                "6/62/Billy_Zoom.jpg/800px-Billy_Zoom.jpg");
        InputStream input = url.openStream();
        ImageData img;
        try {
            img = new ImageData(input);
        } finally {
            input.close();
        }

        Display display = new Display();
        Shell shell = new Shell(display);
        shell.setSize(800, 600);

        Zoom zoom = new Zoom(img, shell);

        shell.open();
        shell.addPaintListener(zoom);
        shell.addMouseMoveListener(zoom);
        shell.addMouseListener(zoom);

        while (!shell.isDisposed()) {
            if (!display.readAndDispatch())
                display.sleep();
        }
        display.dispose();
    }

    @Override
    public void mouseDoubleClick(MouseEvent e) {
    }

    @Override
    public void mouseDown(MouseEvent e) {
        if (e.button != 1) {
            return;
        }
        zoomStart = new Point(e.x, e.y);
        dragState = MOUSE_DOWN;
    }

    @Override
    public void mouseUp(MouseEvent e) {
        if (e.button != 1) {
            return;
        }
        if (dragState == DRAGGING) {
            zoomEnd = new Point(e.x, e.y);
        }
        dragState = NOT_DRAGGING;
        zoom();
        shell.redraw();
    }

    @Override
    public void mouseMove(MouseEvent e) {
        if (dragState == NOT_DRAGGING) {
            return;
        }
        if (e.x == zoomStart.x && e.y == zoomStart.y) {
            dragState = MOUSE_DOWN;
        } else {
            dragState = DRAGGING;
            zoomEnd = new Point(e.x, e.y);
        }
        shell.redraw();
    }
}

When the window is resized, the transformation is currently not changed. That could be implemented in the same way as zooming: Compute previously visible image coordinates with old window size, compute new transformation with new window size.

Prodrome

Thank you for adding this sample implementation. I added a reference to your answer in mine. – Spirogyra

@Prodrome I don't see how this can be used to go from the screen co-ordinates to the original image co-ordinates – Shuping

@Shuping The screen-to-image transformation is the inverse of paintTransform. The actual calculation is hidden inside inv.transform(points) in setVisibleImageAreaInScreenCoordinates(). – Prodrome

@Shuping The invertPaintTransform() method looks more complex than it is, because Transform.invert() is an inplace-operation and there is no one-liner to clone a Transform. – Prodrome

Answer 2

The method selectionToOriginal should return a Rectangle with the position and dimension of the last zooming selection relative to the original image.

It receives:

• scaledDimensions: Point with the dimension of your scaled image, which is where the zooming selection are performed
• levels: List with the consecutive zooming Rectangle selections; in the first level you put the dimension of the original image

This test program show its use with an original image with dimension 800x600 and scaled dimension of 400x300. Two consecutive zooming selection are applied to it.

import java.util.ArrayList;
import java.util.List;

import org.eclipse.swt.graphics.Point;
import org.eclipse.swt.graphics.Rectangle;

public class ScaleTest {

    public static void main(String[] args) {

        Point scaledDimensions = new Point(400, 300);

        List<Rectangle> levels = new ArrayList<Rectangle>();

        // first level is the original image dimension
        levels.add(new Rectangle(0, 0, 800, 600));

        // other levels are the zooming selection inside the scaled image
        levels.add(new Rectangle(0, 0, 200, 150));
        levels.add(new Rectangle(200, 150, 200, 150));

        Rectangle selectionToOriginal = selectionToOriginal(scaledDimensions,
            levels);

        System.out.println(selectionToOriginal);
    }

    public static Rectangle selectionToOriginal(Point scaledDimensions,
        List<Rectangle> levels) {

        int numberOfLevels = levels.size();
        double scaledX = 0;
        double scaledY = 0;

        // we will work with the size of the last selection
        double scaledWidth = levels.get(numberOfLevels - 1).width;
        double scaledHeight = levels.get(numberOfLevels - 1).height;

        // start from the last selection to the first 
        for (int currentLevel = numberOfLevels - 1; currentLevel > 0; currentLevel--) {

            // get the width of the level N - 1
            double previousSelectionWidth = levels.get(currentLevel - 1).width;

            // convert the width of 1 unit in level N to its width in level N - 1
            double unitaryWidth = previousSelectionWidth / scaledDimensions.x;
            // convert the X position in level N in its X position in level N - 1
            scaledX = unitaryWidth * (levels.get(currentLevel).x + scaledX);
            // convert the width in level N in its width in level N - 1
            scaledWidth *= unitaryWidth;

            // get the height of the level N - 1
            double previousSelectionHeight = levels.get(currentLevel - 1).height;

            // convert the height of 1 unit in level N to its height in level N - 1
            double unitaryHeight = previousSelectionHeight / scaledDimensions.y;
            // convert the Y position in level N in its Y position in level N - 1
            scaledY = unitaryHeight * (levels.get(currentLevel).y + scaledY);
            // convert the height in level N in its height in level N - 1
            scaledHeight *= unitaryHeight;
        }

        return new Rectangle((int) scaledX, (int) scaledY, (int) scaledWidth,
            (int) scaledHeight);
    }

}

The program return a Rectangle with position (200, 150) and size (200, 150), the image show the situation:

Notes:

• in your code you used the class Coordinate which it seem equal to the SWT class Point which I used in my method

• the casts in the return instruction

  return new Rectangle((int) scaledX, (int) scaledY, (int) scaledWidth,
          (int) scaledHeight);
  

  will truncate the value of the doubles, consider using Math.round instead if you prefer to round the values

Scurvy

Thanks for providing the visual. That makes it much easier to see what's happening. – Shavon

I do not understand why this is the top voted answer. It is overly complex. Even looking at the "visuals", I don't understand why the selectionToOriginal takes a point and returns a rectangle. What/where are your actual screen coordinates? The selection is supposed to be a rectangle according to the OP. I expected that changing the scaledDimensions to (800, 600) would make the point (300, 400) appear in the output (bottom right corner of the third rectangle in image coordinates). Instead it returns Rectangle {50, 37, 50, 37}. – Prodrome

IMHO the method is pretty straightforward. scaledDimensions is a Point (2 ints with width and height) because we only need to know the size of where the selections are performed, its x and y coordinates are implied to be 0 (exactly like the OP did, requesting only scaledWidth and scaledHeight in input of his methods). We are not working with screen coordinates but with the image coordinates, moving around the Shell in the screen should not have any effect on the result that OP is asking. If you give the exact input parameters of the test you tried I will explain the result. – Scurvy

Sorry, I think we don't talk about the same thing. For me, your "selection" of the point (400, 300) in the main method means that the upper left quarter of the currently displayed image area is selected for the next zoom and that your selectionToOriginal method computes the according rectangular section of the orginal image. For me that means that the resulting rectangle is (200, 150, 100, 75), not again (200, 150, 200, 150). – Prodrome

The OP question start with: I have an image that has been scaled to fit. From the scaled image a user is selecting a rectangle. scaledDimensions (400, 300) means that the original image (of size 800x600) is displayed as a whole in a 400x300 space. Also the consecutive selections are made inside this space, so the first selection of (0, 0, 200, 150) basically means the upper left quarter of the original image (see the image). The second selection (200, 150, 200, 150) is the bottom right corner of the previous selection. – Scurvy

Ok, I got it. I still think it is easier and more concise to track the zoom state as an affine transformation which makes the tracking of selection rectangles obsolete. Simply, each zoom action alters the affine transformation. This makes it also irrelevant whether the initial zoom is a "scaled to fit" or anything else and simplifies the paint call. Using affine transformations is simply best practice for that kind of task. – Prodrome

I don't understand how I would translate the screen co-ordinates back to the original co-ordinates after scaling and zooming – Shuping

@Shuping The method should do that, have you understood the input parameters? What is the issue you are facing? – Scurvy

@Loris Securo The method returns the original rectangle. But no way of translating between screen co-ordinates and original image co-ordinates – Shuping

@Shuping the method returns the zoomed rectangle (even after multiple zoom) converted in the original image co-ordinates. If that is not what you are looking for please add an example to the question. – Scurvy

Answer 3

SWT has a dedicated class Transform for performing coordinate translations (I'd rather say transformations, since translation in such context is just a special case, the other transformations being scaling, rotation and shearing). AWT has a more convenient AffineTransform class that is not bound to the graphics subsystem.

Using one of these classes simplifies things as follows. Once you construct the transform object that maps coordinates in one direction (e.g. source image coordinates to display coordinates), you can easily obtain the inverse transform (for going from the display coordinates back to the source image coordinates). Use the invert() or createInverse() (the latter, only with AffineTransform) methods to this end.

Perform the actual coordinate conversion with transform() method. In case of SWT.Transform its signature is a little inconvenient if you need to transform a single point, but you can easily wrap it in a helper function.

For your purposes you will need to use only the scale() and translate() methods for defining your coordinate transformation. Most probably you will want to define your transform in terms of the source and target rectangles (similar to your usage of the drawImage() method); this answer shows how that can be done. Then, as you zoom or otherwise manipulate how your image is displayed, you must keep the transform object up-to-date.

UPDATE

@code_onkel has provided an example program using this approach.

Spirogyra

Answer 4

Here is my attempt.

private static Point transformPoint(ArrayList<RectF> rectangleLevels, PointF intPoint)
{
    RectF sourceRec = rectangleLevels.get(rectangleLevels.size()-1);
    Point sourcePoint = new Point((int)intPoint.X, (int)intPoint.Y);
    Point retPoint = sourcePoint;
    for (int i = rectangleLevels.size()-2; i >=0; i--) {
        RectF destRec = rectangleLevels.get(i);
        retPoint = transformPoint(sourceRec, destRec, sourcePoint);

        // Current destination point and rec become source for next round
        sourcePoint = retPoint;
        sourceRec = destRec;
    }
    return retPoint;
}


/*
Rectangle 1 has (x1, y1) origin and (w1, h1) for width and height, and
Rectangle 2 has (x2, y2) origin and (w2, h2) for width and height, then

Given point (x, y) in terms of Rectangle 1 co-ords, to convert it to Rectangle 2 co-ords:
    xNew = ((x-x1)/w1)*w2 + x2;
    yNew = ((y-y1)/h1)*h2 + y2;
 */
private static Point transformPoint(RectF source, RectF destination, Point intPoint)
{
    PointF point = new PointF();
    point.X = intPoint.x;
    point.Y = intPoint.y;
    return transformPoint(source, destination, point);
}

private static Point transformPoint(RectF source, RectF destination, PointF point)
{
    return new Point(
    (int) (((point.X - source.X) / source.Width) * destination.Width + destination.X),
    (int) (((point.Y - source.Y) / source.Height) * destination.Height + destination.Y));
}

So it means I just need to keep track of my scaling and zooming, then pass in the screen X,Y to get my x,y from the original image:

    ArrayList<RectF> rectangleLevels = new ArrayList<RectF>();
    RectF origImage = getRectangle(0,0,320,200);
    RectF scaledImage = getRectangle(0,0,800,800);
    RectF zoomedImage = getRectangle(310,190,10,10);
    RectF scaledZoomedImage = getRectangle(0,0,800,800);
    rectangleLevels.add(origImage);
    rectangleLevels.add(scaledImage);
    rectangleLevels.add(zoomedImage);
    rectangleLevels.add(scaledZoomedImage);
    PointF pointInZoomedImg = getPoint(799, 799);

    Point retPoint = transformPoint(rectangleLevels, pointInZoomedImg);

Shuping

Is this attempt working or do you still need help? Do you consider zoomedImage relative to (800, 800) or to (320, 200)? If it's relative to (800, 800) then my solution would cover the case using scaledDimensions = (800, 800) and levels with these rectangles: (0, 0, 320, 200), (310, 190, 10, 10), (799, 799, 0, 0). It will return (127, 49, 0, 0) and since you want a point you could just consider the x and y part of the rectangle, which is (127, 49). – Scurvy

@LorisSecuro - no there are rounding issues so it does not fully work – Shuping