In scikit learn, how to deal with the data mixed with numerical and nominal value?

Question

I know that the computation in scikit-learn is based on NumPy so everything is a matrix or array.

How does this package handle mixed data (numerical and nominal values)?

For example, a product could have the attribute 'color' and 'price', where color is nominal and price is numerical. I notice there is a model called 'DictVectorizer' to numerate the nominal data. For example, two products are:

products = [{'color':'black','price':10}, {'color':'green','price':5}]

And the result from 'DictVectorizer' could be:

[[1,0,10],
 [0,1,5]]

If there are lots of different values for the attribute 'color', the matrix would be very sparse. And long features will degrade the performance of some algorithms, such as decision trees.

Is there any way to use the nominal value without the need to create dummy codes?

Fortune

It's worth noting that Weka Instances store nominal values as floating point numbers corresponding to the index of the nominal in the attribute's definition. You could simply follow this same strategy to generate a numeric dataset for use with scikit-learn. – Lauzon

Thanks a lot for enlarging my knowledge. – Fortune

Answer 1

The DecisionTree class in scikit-learn will need some refactoring to deal efficiently with high-cardinality categorical features (and maybe even with naturally sparse data such as text TF-IDF vectors).

Nobody is working on that yet AFAIK.

Breadbasket

thanks a lot. In scikit, is there any smart way to do refactoring compared with manual operation? – Fortune

My answer states that this current state of affair is a limitation of the current implementation of the Decision Tree in scikit-learn. There is no easy fix I know of to remove that limitation. I don't understand what you call "manual operation". – Breadbasket