How can HMMs be used for handwriting recognition?

Question

The problem is a bit different than traditional handwriting recognition. I have a dataset that are thousands of the following. For one drawn character, I have several sequential (x, y) coordinates where the pen was pressed down. So, this is a sequential (temporal) problem.

I want to be able to classify handwritten characters based on this data, and would love to implement HMMs for learning purposes. But, is this the right approach? How can they be used to do this?

Abandoned

That's a really interesting idea. Out of curiosity, is the time coordinate of each point recorded as well (so at millisecond 1 coordinate 30 x 45 was pressed) or are they just in sequential order? – Coxswain

Currently it's just sequential. Was thinking of doing some dynamic time warping to account for characters being drawn slower/faster – Abandoned

I think this is the right approach. You could start, as practice, with recognizing the palm graffite alphabet. – Ozzie

Answer 1

I think HMM can be used in both problems mentioned by @jens. I'm working on online handwriting too, and HMM is used in many articles. The simplest approach is like this:

1. Select a feature.
2. If selected feature is continuous convert it to discrete.
3. Choose HMM parameters: Topology and # of states.
4. Train character models using HMM. one model for each class.
5. Test using test set.

for each item:

1. the simplest feature is angle of vector which connects consecutive points. You can use more complicated features like angles of vectors obtained by Douglas & Peucker algorithm.
2. the simplest way for discretization is using Freeman codes, but clustering algorithms like k-means and GMM can be used too.
3. HMM topologies: Ergodic, Left-Right, Bakis and Linear. # of states can be obtained by trial & error. HMM parameters can be variable for each model. # of observations is determined by discretization. observation samples can be have variable length.
4. I recommend Kevin Murphy HMM toolbox.
5. Good luck.

Libyan

Answer 2

This problem is actually a mix of two problems:

1. recognizing one character from your data
2. recognizing a word from a (noisy) sequence of characters

A HMM is used for finding the most likely sequence of a finite number of discrete states out of noisy measurements. This is exactly problem 2, since noisy measurements of discrete states a-z,0-9 follow eachother in a sequence.

For problem 1, a HMM is useless because you aren't interested in the underlying sequence. What you want is to augment your handwritten digit with information on how you wrote it.

Personally, I would start by implementing regular state-of-the-art handwriting recognition which already is very good (with convolutional neural networks or deep learning). After that, you can add information about how it was written, for example clockwise/counterclockwise.

Garzon