Multi-Class SVM( one versus all)

Question

I know that LIBSVM only allows one-vs-one classification when it comes to multi-class SVM. However, I would like to tweak it a bit to perform one-against-all classification. I have tried to perform one-against-all below. Is this the correct approach?

The code:

TrainLabel;TrainVec;TestVec;TestLaBel;
u=unique(TrainLabel);
N=length(u);
if(N>2)
    itr=1;
    classes=0;
    while((classes~=1)&&(itr<=length(u)))
        c1=(TrainLabel==u(itr));
        newClass=c1;
        model = svmtrain(TrainLabel, TrainVec, '-c 1 -g 0.00154'); 
        [predict_label, accuracy, dec_values] = svmpredict(TestLabel, TestVec, model);
        itr=itr+1;
    end
itr=itr-1;
end

I might have done some mistakes. I would like to hear some feedback. Thanks.

Second Part: As grapeot said : I need to do Sum-pooling (or voting as a simplified solution) to come up with the final answer. I am not sure how to do it. I need some help on it; I saw the python file but still not very sure. I need some help.

Motorman

What's the question exactly? You are asking how to perform one-vs-all classification with LibSVM? Does the program output the result you expected? BTW, the LibSVM parameters should be '-c 1 -g 0.00153' (you lacked the ending single quote). – Erethism

@lakesh: I posted an answer to a similar question, you might find useful: https://mcmap.net/q/540119/-multi-class-classification-in-libsvm-closed – Reuter

Answer 1

%# Fisher Iris dataset
load fisheriris
[~,~,labels] = unique(species);   %# labels: 1/2/3
data = zscore(meas);              %# scale features
numInst = size(data,1);
numLabels = max(labels);

%# split training/testing
idx = randperm(numInst);
numTrain = 100; numTest = numInst - numTrain;
trainData = data(idx(1:numTrain),:);  testData = data(idx(numTrain+1:end),:);
trainLabel = labels(idx(1:numTrain)); testLabel = labels(idx(numTrain+1:end));
%# train one-against-all models
model = cell(numLabels,1);
for k=1:numLabels
    model{k} = svmtrain(double(trainLabel==k), trainData, '-c 1 -g 0.2 -b 1');
end

%# get probability estimates of test instances using each model
prob = zeros(numTest,numLabels);
for k=1:numLabels
    [~,~,p] = svmpredict(double(testLabel==k), testData, model{k}, '-b 1');
    prob(:,k) = p(:,model{k}.Label==1);    %# probability of class==k
end

%# predict the class with the highest probability
[~,pred] = max(prob,[],2);
acc = sum(pred == testLabel) ./ numel(testLabel)    %# accuracy
C = confusionmat(testLabel, pred)                   %# confusion matrix

Motorman

Answer 2

From the code I can see you are trying to first turn the labels into "some class" vs "not this class", and then invoke LibSVM to do training and testing. Some questions and suggestions:

1. Why are you using the original TrainingLabel for training? In my opinion, should it be model = svmtrain(newClass, TrainVec, '-c 1 -g 0.00154');?
2. With modified training mechanism, you also need to tweak the prediction part, such as using sum-pooling to determine the final label. Using -b switch in LibSVM to enable probability output will also improve the accuracy.

Erethism

thanks alot... btw, do u know how to do one vs one using LIBSVM? i am not sure how to do it... – Motorman

Simply putting labels other than 0<=>1 or -1<=>1 as input is fine. LibSVM will recognize it and try to do multi-class classification. – Erethism

btw it is giving me this error when i change it to newClass : Error: label vector and instance matrix must be double model file should be a struct array – Motorman

when i change newClass=c1; to newClass=double(c1);, it gives me 0% classification – Motorman

Maybe you can track in to check the value of c1? Is it having both 1 and 0? – Erethism

An official implementation in python of one-against-all in python based on LibSVM can be found in the website: csie.ntu.edu.tw/~cjlin/libsvmtools/multilabel – Erethism

If you wish to calculate the classification accuracy directly from LibSVM, make sure the ground truth fed to SVMPredict is correct, i.e. they should be like (TestLabel == itr) rather than TestLabel themselves. Or you can write your own implementation to calculate the precision/recall. – Erethism

Once again thanks.. sorry for troubling you.. the pred label are 0 and 1s. Shouldn't it be in numbers that i am using like 1 to 6? – Motorman

Yes, that's by design. Note you are solving the problem by a series of binary classifiers. Therefore the output of the SVMs is binary, but you need to do a sum-pooling (or voting as a simplified solution) to come up with the final answer. You may consult the python file mentioned before. :) – Erethism

@Erethism please can you help me here? thank you a lot #65450434 – Botts

Answer 3

Instead of probability estimates, you can also use the decision values as follows

[~,~,d] = svmpredict(double(testLabel==k), testData, model{k});
prob(:,k) = d * (2 * model{i}.Label(1) - 1);

to achieve the same purpose.

Salter