What's the difference between LibSVM and LibLinear

Question

libsvm and liblinear are both software libraries that implement Support Vector Machines. What's the difference? And how do the differences make liblinear faster than libsvm?

Paederast

Answer 1

In practice the complexity of the SMO algorithm (that works both for kernel and linear SVM) as implemented in libsvm is O(n^2) or O(n^3) whereas liblinear is O(n) but does not support kernel SVMs. n is the number of samples in the training dataset.

Hence for medium to large scale forget about kernels and use liblinear (or maybe have a look at approximate kernel SVM solvers such as LaSVM).

Edit: in practice libsvm becomes painfully slow at 10k samples.

Dematerialize

Aside from the implementation issues, I would also add that the accuracy scores you would receive if you ran both algorithms might be dramatically different. I've found LibLinear to usually (if not always) score a higher accuracy value than LibSVM. – Diaphysis

@Dematerialize so the "L2-regularized logistic regression" classifer in Liblinear has an O(n) complexity? – Burt

Actually I am not sure it probably depends on the solver used by liblinear but I am sure it's significantly better than the SMO solver used by libsvm. – Dematerialize

If of interest, we tried both for datasets with up to 160k samples (after that kernel SVM became prohibitevly slow to train - even on 4 nodes with 16 cores each it took more than a week to finish). Kernel SVM produced sligthly slightly better results, but given the enormous compute hours needed, liblinear should indeed be a better default choice for large datasets. More info available in our (open access) paper: dx.doi.org/10.1186/s13321-016-0151-5 – Smile

Answer 2

SVM is support vector machine, which is basically a linear classifier, but using many kernel transforms to turn a non-linear problem into a linear problem beforehand.

From the link above, it seems like liblinear is very much the same thing, without those kernel transforms. So, as they say, in cases where the kernel transforms are not needed (they mention document classification), it will be faster.

Thrips

Answer 3

From : http://www.csie.ntu.edu.tw/~cjlin/papers/liblinear.pdf

It supports L2-regularized logistic regression (LR), L2-loss and L1-loss linear support vector machines (SVMs) (Boser et al., 1992). It inherits many features of the popular SVM library LIBSVM

And you might also see some useful information here from one of the creators: http://agbs.kyb.tuebingen.mpg.de/km/bb/showthread.php?tid=710

The main idea, I would say, is that liblinear is optimized to deal with linear classification (i.e. no kernels necessary), whereas linear classification is only one of the many capabilities of libsvm, so logically it may not match up to liblinear in terms of classification accuracy. Obviously, I'm making some broad generalizations here, and the exact details on the differences are probably covered in the paper I linked above as well as with the corresponding user's guide to libsvm from the libsvm website.

Faraway