OpenGL ES 2.0 Object Picking on iOS

Question

What is the best method to select objects that have been drawn in OpenGL ES 2.0 (iOS)?

I am drawing points.

Buhler

lighthouse3d.com/opengl/picking/index.php3?color1 – Constrained

Answer 1

Here is working prototype of color picking, tested on most old ipads and working well. This is actually some part of project called InCube Chess that one may find in app store. The main code you will see is located in a class derived from GLKViewController like this:

@interface IncubeViewController : GLKViewController

This means you have glkview in it: ((GLKView *)self.view).

Here are also some properties:

@property (strong, nonatomic) EAGLContext *context;
@property (strong, nonatomic) GLKBaseEffect *effect;

Don't forget to synthesize them in your *.m file.

@synthesize context = _context;
@synthesize effect = _effect;

The idea is that you have chess pieces on your table (or some objects in your 3d scene) and you need to find a piece in your list of pieces by tapping on a screen. That is, you need to convert your 2d screen tap coords (@point in this case) to chess piece instance.

Each piece has its unique id that I call a "seal". You can allocate the seals from from 1 up to something. Selection function returns piece seal found by tap coords. Then having the seal you can easily find your piece in pieces hash table or array in a way like this:

-(Piece *)findPieceBySeal:(GLuint)seal
{
        /* !!! Black background in off screen buffer produces 0 seals. This allows
           to quickly filter out taps that did not select anything (will be
           mentioned below) !!! */
        if (seal == 0)
                return nil;
        PieceSeal *sealKey = [[PieceSeal alloc] init:s];
        Piece *p = [sealhash objectForKey:sealKey];
        [sealKey release];
        return p;
}

"sealhash" is a NSMutableDictionary.

Now this is the main selection function. Note, that my glkview is antialised and you can't use its buffers for color picking. This mean you need to create your own off screen buffer with antialiasing disabled for picking purposes only.

- (NSUInteger)findSealByPoint:(CGPoint)point
{
        NSInteger height = ((GLKView *)self.view).drawableHeight;
        NSInteger width = ((GLKView *)self.view).drawableWidth;
        Byte pixelColor[4] = {0,};
        GLuint colorRenderbuffer;
        GLuint framebuffer;

        glGenFramebuffers(1, &framebuffer);
        glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
        glGenRenderbuffers(1, &colorRenderbuffer);
        glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);

        glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, width, height);
        glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER, colorRenderbuffer);

        GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
        if (status != GL_FRAMEBUFFER_COMPLETE) {
                NSLog(@"Framebuffer status: %x", (int)status);
                return 0;
        }

        [self render:DM_SELECT];

        CGFloat scale = UIScreen.mainScreen.scale;
        glReadPixels(point.x * scale, (height - (point.y * scale)), 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixelColor);

        glDeleteRenderbuffers(1, &colorRenderbuffer);
        glDeleteFramebuffers(1, &framebuffer);

        return pixelColor[0];
}

Note that function takes into account display scale (retina or new iPads).

Here is render() function used in function above. Note, that for rendering purposes it clear s the buffer with some background color and for selecting case it makes it black so that you can easily check if you tapped on any piece at all or not.

- (void) render:(DrawMode)mode
{
        if (mode == DM_RENDER)
                glClearColor(backgroundColor.r, backgroundColor.g,
                             backgroundColor.b, 1.0f);
        else
                glClearColor(0.0f, 0.0f, 0.0f, 1.0f);

        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        /* Draw all pieces. */
        for (int i = 0; i < [model->pieces count]; i++) {
                Piece *p = [model->pieces objectAtIndex:i];
                [self drawPiece:p mode:mode];
        }
}

Next is how we draw the piece.

- (void) drawPiece:(Piece *)p mode:(DrawMode)mode
{
        PieceType type;

        [self pushMatrix];

        GLKMatrix4 modelViewMatrix = self.effect.transform.modelviewMatrix;

        GLKMatrix4 translateMatrix = GLKMatrix4MakeTranslation(p->drawPos.X,
                                                               p->drawPos.Y,
                                                               p->drawPos.Z);
        modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, translateMatrix);

        GLKMatrix4 rotateMatrix;
        GLKMatrix4 scaleMatrix;

        if (mode == DM_RENDER) {
                scaleMatrix = GLKMatrix4MakeScale(p->scale.X,
                                                  p->scale.Y, p->scale.Z);
        } else {
                /* !!! Make the piece a bit bigger in off screen buffer for selection
                   purposes so that we always sure that we tapped it correctly by
                   finger.*/
                scaleMatrix = GLKMatrix4MakeScale(p->scale.X + 0.2,
                                                  p->scale.Y + 0.2, p->scale.Z + 0.2);
        }

        modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, scaleMatrix);

        self.effect.transform.modelviewMatrix = modelViewMatrix;

        type = p->type;

        if (mode == DM_RENDER) {
                /* !!! Use real pieces color and light on for normal drawing !!! */
                GLKVector4 color[pcLast] = {
                        [pcWhite] = whitesColor,
                        [pcBlack] = blacksColor
                };
                self.effect.constantColor = color[p->color];
                self.effect.light0.enabled = GL_TRUE;
        } else {
                /* !!! Use piece seal for color. Important to turn light off !!! */
                self.effect.light0.enabled = GL_FALSE;
                self.effect.constantColor = GLKVector4Make(p->seal / 255.0f,
                                                           0.0f, 0.0f, 0.0f);
        }

        /* Actually normal render the piece using it geometry buffers. */
        [self renderPiece:type];

        [self popMatrix];
}

This is how to use the functions shown above.

- (IBAction) tapGesture:(id)sender
{
        if ([(UITapGestureRecognizer *)sender state] == UIGestureRecognizerStateEnded) {
                CGPoint tap = [(UITapGestureRecognizer *)sender locationInView:self.view];
                Piece *p = [self findPieceBySeal:[self findSealByPoint:tap]];

                /* !!! Do something with your selected object !!! */
        }
}

This is basically it. You will have very precise picking algorithm that is much better than ray tracing or others.

Here helpers for push/pop matrix things.

- (void)pushMatrix
{
        assert(matrixSP < sizeof(matrixStack) / sizeof(GLKMatrix4));
        matrixStack[matrixSP++] = self.effect.transform.modelviewMatrix;
}

- (void)popMatrix
{
        assert(matrixSP > 0);
        self.effect.transform.modelviewMatrix = matrixStack[--matrixSP];
}

Here also glkview setup/cleanup functions that I used.

- (void)viewDidLoad
{
        [super viewDidLoad];
        self.context = [[[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2] autorelease];
        if (!self.context)
                NSLog(@"Failed to create ES context");

        GLKView *view = (GLKView *)self.view;
        view.context = self.context;
        view.drawableDepthFormat = GLKViewDrawableDepthFormat24;

        [self setupGL];
}

- (void)viewDidUnload
{    
        [super viewDidUnload];

        [self tearDownGL];

        if ([EAGLContext currentContext] == self.context)
                [EAGLContext setCurrentContext:nil];
        self.context = nil;
}

- (void)setupGL
{
        [EAGLContext setCurrentContext:self.context];

        self.effect = [[[GLKBaseEffect alloc] init] autorelease];
        if (self.effect) {
                self.effect.useConstantColor = GL_TRUE;
                self.effect.colorMaterialEnabled = GL_TRUE;
                self.effect.light0.enabled = GL_TRUE;
                self.effect.light0.diffuseColor = GLKVector4Make(1.0f, 1.0f, 1.0f, 1.0f);
        }

        /* !!! Draw antialiased geometry !!! */
        ((GLKView *)self.view).drawableMultisample = GLKViewDrawableMultisample4X;
        self.pauseOnWillResignActive = YES;
        self.resumeOnDidBecomeActive = YES;
        self.preferredFramesPerSecond = 30;

        glDisable(GL_DITHER);
        glEnable(GL_CULL_FACE);
        glEnable(GL_DEPTH_TEST);
        glLineWidth(2.0f);

        /* Load pieces geometry */
        [self loadGeometry];
}

- (void)tearDownGL
{
        drawReady = NO;
        [EAGLContext setCurrentContext:self.context];
        [self unloadGeometry];
}

Hope this helps and may be closes "the picking question" forever :)

Buenabuenaventura

Thank you for such a thorough answer! – Superabundant

@Umka hi can i get any working sample code on this picking that u have explained above?I m new to opengl and hoping a working model will help me to implement this same. – Insalubrious

Sure, just provide me some email and I will send you zip archive. – Buenabuenaventura

@Umka ok.with pleasure – Insalubrious

Hi Umka. Can you please upload your code on GitHub and provide a link here? – Celestyna

Let's do this :) I will send it to anyone who asks, not sure I want to put it to github. – Buenabuenaventura

Good answer. A couple of additional notes: 1) You're probably leaving some performance on the floor by deleting and recreating your offscreen framebuffer on each touch. Create it once at setup time (or lazily the first time you need it), and reuse it by binding to it when you need to pick a pixel. 2) Don't forget to bind back to the onscreen framebuffer with [self.view bindDrawable] or similar when it's time to render the next frame. – Valeryvalerye

I doubt you'll ever settle the picking question forever. :) Reading object IDs from pixels, ray casting, and more exotic approaches all have different performance characteristics, so the best answer will always depend on the scene you're rendering and the algorithms and hardware you use to model and draw it. – Valeryvalerye

Sure, many improvements are possible. This is rather basis for anyone to have it at least working ) However, given it works on iPad 1 without visible glitches, performance question is not most important here. – Buenabuenaventura

@Umka great post, can I get the working sample code that u have explained above? I hope to achieve object picking the same as you. – Marivaux

Unfortunately it is part of bigger project and it would mean to spend long time to extract it. You have all you need right here. – Buenabuenaventura

@iOSDeveloper hi, can I get the working sample code about color picking that u have got from Umka. thank you very much. – Marivaux

@Umka thank you all the same. I'm new to opengl ES 2.0 with GLKit. I have failed many times when using color coding to achieve object picking. so, Can I get the complete IncubeViewController.m file ? Thank you again. – Marivaux

@罗大柚 yes will do.pls give ur mail id – Insalubrious

@iOSDeveloper thanks again, I had achieved object picking with color attribute. – Marivaux

Answer 2

Here is how i do, based on the above solution, with depth buffer for 3D picking and GLKVector3 signature to retrieve the seal :

• all my objects got a seal
• i use a GLKVector3 named color to retrieved the seal from the read pixel. With this solution you can store 255 *255 * 255 = 16,581,375 objects.

In the Shader (vertex or fragment as you want)

add a picking boolean which tell you if you do the pick pass or not

uniform bool picking;

and the GLKVector3

uniform vec3 color;

if the boolean picking is activated so the color will be the GLKVector3

if(picking)
{
    colorVarying = vec4(color, 1.0);
}

In the ViewController

the method which create the GLKVector3 from a seal (when i create a new 3D object) :

-(GLKVector3)pack:(uint)seal
{
    GLKVector3 hash;

    float r = seal % 255;
    float g = (seal / 255) % 255;
    float b = (seal / (255 * 255)) % 255;
    hash = GLKVector3Make(r/255, g/255, b/255);

    return hash;
}

And the view controller code where you get the pixel from the touch position and get the selected seal from the selected color :

-(uint)getSealByColor:(GLKVector3)color
{
    color = GLKVector3DivideScalar(color, 255);
    for (MyObject *o in _objects) {
        if(GLKVector3AllEqualToVector3(o.color, color))
        {
            return o.seal;
        }
    }
    return 0;
}

-(void)tap:(UITapGestureRecognizer*)recognizer
{
    CGPoint p = [recognizer locationInView:self.view];
    GLKVector3 i = [self pickingAt:p];
    _sealSelected = [self getSealByColor:i];
}

-(GLKVector3)pickingAt:(CGPoint)position
{
    CGFloat scale = [UIScreen mainScreen].scale;

    GLsizei w = self.view.bounds.size.width * scale;
    GLsizei h = self.view.bounds.size.height * scale;

    GLuint fb;
    GLuint rb;
    GLuint db;

    Byte pixelColor[4] = {0,};

    glGenFramebuffers(1, &fb);
    glBindFramebuffer(GL_FRAMEBUFFER, fb);
    glGenRenderbuffers(1, &rb);
    glBindRenderbuffer(GL_RENDERBUFFER, rb);

    glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, w, h);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, rb);

    //here we also create a depth buffer for 3D objects picking
    glGenRenderbuffers(1, &db);
    glBindRenderbuffer(GL_RENDERBUFFER, db);

    glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH_COMPONENT16, w, h);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, db);

    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
    if (status != GL_FRAMEBUFFER_COMPLETE) {
        NSLog(@"Framebuffer status: %x", (int)status);
        return GLKVector3Make(0.0, 0.0, 0.0);
    }

    //we render the scene with our picking boolean activated
    [self render:YES];

    glReadPixels(position.x * scale, (h - (position.y * scale)), 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixelColor);

    glDeleteRenderbuffers(1, &db);
    glDeleteRenderbuffers(1, &rb);
    glDeleteFramebuffers(1, &fb);

    return GLKVector3Make(pixelColor[0], pixelColor[1], pixelColor[2]);
}

Hope this help someone, this is how i do to allow color picking for more than 255 objects.

Danford

Took me awhile to investigate why this fail the app due to memory pressure after some monkey testing. You forgot to delete the depth buffer. So you have to add this "glDeleteRenderbuffers(1, & db);" – Banshee

@Danford I'm new to opengl ES 2.0 with GLKit. I have failed many times when using color coding to achieve object picking. can i get any working sample code about color Picking, thank you very much – Marivaux