How does card.io image processing work?

Question

How do services like card.io work behind the scenes? Do they use an OCR library like Tesseract or is it more complex?

Also, in this video, it looks like the app is waiting for you to hold your card in a specific range from the camera inside the green boundaries on the camera, and when you do it takes the photo automatically. If the image recognition is happening on the server, how did they do that?

How can I implement my own mobile 2D object scanner? Where do I start?

Windowshop

Answer 1

Josh from card.io here. I can't tell you the details of how card.io works (hopefully others will speculate here), but I can answer some of your other questions.

card.io does not use an OCR engine; we looked at Tesseract and others and found that they did not work well on many credit cards.

card.io's image recognition happens entirely on the phone. Early versions required some server assistance, but even those did a good chunk of the work on the phone.

To get started, I recommend sitting down and reading Learning OpenCV; it is a good general introduction to computer vision. Then play around and ask more detailed questions. On the machine learning side, Theano and Eigen are very helpful libraries.

Slap

Hei Josh. Why do you recommend Eigen? As far as I know OpenCV can do everything that Eigen Does. It is hard enough to master openCV, and you recommend another 2 libraries :-) – Asgard

OpenCV can do everything that Eigen does. However, for matrix-heavy work (which shows up a lot in machine learning), Eigen is a bit easier to use (readable C++!). And more importantly, Eigen has good NEON implementations of core matrix manipulation, which is key to performance. – Slap

@Josh I am image processing enthusiast, I tried out OpenCV for android that have a java wrapper, it is dead slow only 2 or 3 frames per second where as card.io is able to process more than 10 frames per second, how do I achieve that state, I know card.io directly process the image in native code instead of java wrapper which makes it slow, is there any tutorial for noobs how to use openCV in native code instead of java wrapper? – Farcy