Eye tracking: finding the pupil (x,y)

Question

I am looking for some suggestions on how to approach the following computer vision problem. Below are 4 samples of an eye tracking dataset that I am working with. I would like to write code takes one such image and calculates the (x,y) position of the center of the pupil. I am currently using MATLAB, but I am open to using other software too.

Can someone recommend an approach I could use for this task? Here are some things I already tried but didn't work TOO well.

• I tried to use circle hough transform, but that requires me to guess the radius of the pupil, which is a bit problematic. Also, due to distortions, the pupil is not always exactly a circle, which may make this approach harder still.
• I tried thresholding the image based on pixel brightness and using regionprops MATLAB function to look for a region of roughly (say) 200 pixel area with very low eccentricity (i.e. as circular as possible). However, this is very sensitive to the threshold value, and some images of the eye are brighter than others based on the lighting conditions. (Note the 4 samples below are mean-normalized already, and still one of the images is brighter than others overall probably because of some very dark random pixel somewhere)

Any comments/suggestions would be appreciated!

EDIT: thanks for the comment Stargazer. The algorithm should ideally be able to determine that the pupil is not in the image, as is the case for the last sample. It's not a big deal if I lose track of it for a while. It's much worse if it gives me wrong answer though.

Davide

Well, at the very least, don't be disappointed if you find an algorithm that only works for the first three. Even I can't really find the pupil in the 4th one. – Bogan

I understand that, of course. The algorithm should in that case determine that, ideally. – Davide

Have you considered using haarcascades (en.wikipedia.org/wiki/Viola-Jones_object_detection_framework) or local binary patterns? (en.wikipedia.org/wiki/Local_binary_patterns). OpenCV provides very easy ways to train these kind of classifiers. – Pheni

In opencv there already haar cascades that were trained to detect eyes. On Unix systems when u install opencv they are saved to: /usr/local/share/opencv/haarcascades Then you center use these cascades with find objects function in opencv Alex – Centenary

This should do the trick for you! https://mcmap.net/q/1164809/-about-finding-pupil-in-a-video – Hapsburg

Answer 1

I'm not sure if this can help you, because you are using a dataset and I don't know your flexibility/needs to change the capture device. Just in case, let's go.

Morimoto et al. use a nice camera trick. They created a camera with two sets of infra-red leds. The first set is put near the camera lenses. The second one is put far from the lenses. Using different frequencies, the two leds sets are turned on in different moments.

Retina will reflect the light from the set near the camera lenses (that is the same thing about the red eye problem in photography), producing a bright pupil. The other set of leds will produce a dark pupil. Compare the results. So, simple difference between the two images give you a near perfect pupil. Take a look in the way that Morimoto et al. explore the glint (nice to approach sight direction).

Pleasing

Thanks, but unfortunately I don't have access to the physical setup, I only got the data and need to run some analysis. The data does come with tracking results, but I thought I could come up with something better/smoother. Turns out the problem is more difficult than I had originally anticipated :) – Davide

Answer 2

Use OpenCV integrated Python . . . It will be very easy for the beginners to work on OpenCV.

Procedure :
* If you are using normal webcam
1. First process the frame with VideoCapture function
2. Convert it into Gray Scale Image.
3. Find Canny Edges using cv2.Canny() function
4. Apply HoughCircles function. It will find the circles in the image as well as center of the image.
5. Use the resulting parameters of HoughCirlces to draw the circle around the pupil. Thats it.

Sidonie

I have applied Canny into my soft and it produces very good results! Thanks. – Aglitter

Answer 3

OpenCV with Python, C, C++, Java and others would be a good tool for doing that. There is a tutorial for Python here: http://docs.opencv.org/trunk/doc/py_tutorials/py_objdetect/py_face_detection/py_face_detection.html, but there are definitely other tutorials out there for the other supported languages. OpenCv has a number of Haar Cascades right out of the box, one for eye detection included. If you actually wanted to implement a solution using HoughCircleTransform, OpenCv has the appropriate function for that, too.

Conduplicate

Answer 4

import java.awt.Robot;%Add package or class to current import listimport java.awt.event.*;robot = Robot();objvideoinput('winvideo',2);%to set the device ID and supported format set(obj, 'FramesPerTrigger', Inf);% trigger infinite set(obj, 'ReturnedColorspace', 'rgb')%video in RGB format obj.FrameGrabInterval = 5;%the object acquires every %5th frame from the video stream start(obj)% to start the vedio time=0;NumberOfFrames=while(true)data=getsnapshot(obj);image(data);filas=size(data,1);columnas=size(data,2);% Centercentro_fila=round(filas/2);centro_columna=round(columnas/2);figure(1);if size(data,3)==3data=rgb2gray(data);% Extract edges.BW = edge(data,'canny')[H,T,R] = hough(BW,'RhoResolution',0.5,'Theta',-90:0.5:89.5);endsubplot(212)piel=~im2bw(data,0.19);piel=bwmorph(piel,'close');piel=bwmorph(piel,'open');piel=bwareaopen(piel,275);piel=imfill(piel,'holes');imagesc(piel);% Tagged objects in BW imageL=bwlabel(piel);% Get areas and tracking rectangleout_a=regionprops(L);% Count the number of objectsN=size(out_a,1);if N < 1 || isempty(out_a) % Returns if no object in the imagesolo_cara=[ ];continue end % Select larger area areas=[out_a.Area];[area_max pam]=max(areas);subplot(211)imagesc(data);colormap grayhold on rectangle('Position',out_a(pam).BoundingBox,'EdgeColor',[1 0 0],...'Curvature', [1,1],'LineWidth',2)centro=round(out_a(pam).Centroid);X=centro(1);Y=centro(2);robot.mouseMove(X,Y);text(X+10,Y,['(',num2str(X),',',num2str(Y),')'],'Color',[1 1 1])if X<centro_columna && Y<centro_fila 
title('Top left')elseif X>centro_columna && Y<centro_fila
title('Top right')elseif X<centro_columna && Y>centro_fila
title('Bottom left')else
title('Bottom right')

Piperine