Multiclass SVM Classification in Encog

Question

Could someone show me how to use multi class SVM classification in Encog 3.1?

I have used their Neural Networks with some success, but can not work out how to set up a multiclass SVM.

The docs have this to say:

"This is a network that is backed by one or more Support Vector Machines (SVM). It is designed to function very similarly to an Encog neural network, and is largely interchangeable with an Encog neural network..... Classification is used when you want the SVM to group the input data into one or more classes. Support Vector Machines typically have a single output. Neural networks can have multiple output neurons. To get around this issue, this class will create multiple SVM's if there is more than one output specified"

Yet i can not see how to specify more than one output, in fact the output property simply returns 1:

 /// <value>For a SVM, the output count is always one.</value>
    public int OutputCount
    {
        get { return 1; }
    }

Answers in Java or c# are greatly appreciated

EDIT still unable to work this out. Really enjoying using Encog, but the support forum is quite with only Jeff Heaton (author of project) himself answering when he gets a chance, so im linking the project code and adding a bounty in the hope that someone can see what im obviously missing.

The project: http://heatonresearch.com/

The SupportVectorMachine class on google code: https://code.google.com/p/encog-cs/source/browse/trunk/encog-core/encog-core-cs/ML/SVM/SupportVectorMachine.cs

Crossbred

Answer 1

Sorry for the slow response. I decided to make this an FAQ for Encog. You can see the FAQ & example here. http://www.heatonresearch.com/faq/5/2

Basically Encog DOES support multi-class SVM. You do not need multiple outputs like you do a neural network. You simply train it with a single output and that output is the class number, i.e. 0.0, 1.0, 2.0, etc.. depending on how many classes you have.

This applies to both the Java and C# versions of Encog. I did the example in C#.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using Encog.ML.SVM;
using Encog.ML.Data;
using Encog.ML.Data.Basic;
using Encog.ML.Train;
using Encog.ML.SVM.Training;

namespace MultiClassSVM
{
    class Program
    {
        /// 
        /// Input for function, normalized to 0 to 1.
        /// 
        public static double[][] ClassificationInput = {
            new[] {0.0, 0.0},
            new[] {0.1, 0.0},
            new[] {0.2, 0.0},
            new[] {0.3, 0.0},
            new[] {0.4, 0.5},
            new[] {0.5, 0.5},
            new[] {0.6, 0.5},
            new[] {0.7, 0.5},
            new[] {0.8, 0.5},
            new[] {0.9, 0.5}
            };

        /// 
        /// Ideal output, these are class numbers, a total of four classes here (0,1,2,3).
        /// DO NOT USE FRACTIONAL CLASSES (i.e. there is no class 1.5)
        /// 
        public static double[][] ClassificationIdeal = {
            new[] {0.0},
            new[] {0.0},
            new[] {0.0},
            new[] {0.0},
            new[] {1.0},
            new[] {1.0},
            new[] {2.0},
            new[] {2.0},
            new[] {3.0},
            new[] {3.0}
        };

        static void Main(string[] args)
        {
            // create a neural network, without using a factory
            var svm = new SupportVectorMachine(2, false); // 2 input, & false for classification

            // create training data
            IMLDataSet trainingSet = new BasicMLDataSet(ClassificationInput, ClassificationIdeal);

            // train the SVM
            IMLTrain train = new SVMSearchTrain(svm, trainingSet);

            int epoch = 1;

            do
            {
                train.Iteration();
                Console.WriteLine(@"Epoch #" + epoch + @" Error:" + train.Error);
                epoch++;
            } while (train.Error > 0.01);

            // test the SVM
            Console.WriteLine(@"SVM Results:");
            foreach (IMLDataPair pair in trainingSet)
            {
                IMLData output = svm.Compute(pair.Input);
                Console.WriteLine(pair.Input[0]
                                  + @", actual=" + output[0] + @",ideal=" + pair.Ideal[0]);
            }

            Console.WriteLine("Done");
        }
    }
}

Intromission

Great, thanks for responding, dont know why i didnt think of trying that. Finding Encog very useful indeed. – Crossbred

Aren't you just using regression for classification? #9161169 – Comptometer

Agreed, it is bad to use regression to do classification, but such is not the case here. In Encog a SVM is designated as either regression or classification in the constructor. These are two very different setups for SVM's. Notice the constructor has a false value, that means it is classification. So therefore values of 1.0, 2.0 etc are class numbers. It always uses a floating point input for consistency. – Intromission

@Intromission do you have any idea how to normalize data in range 0, 1 ? is there any built in functions or something to do so ? ... one last questions how to control kernel type en encog? – Thermobarometer

Ok nice Jeff, but you should precisely highlight - SVM only for NORMALIZED values in range (0,1) – Colloquy

Answer 2

You can't have multiclass SVM. SVMs can only classify into two classes. There are of course methods how to use them for multiclass classification. They are one-vs-one and one-vs-all.

In one-vs-one you train (k * (k-1))/2 SVMs for each pair of classes. Then you let them vote and the class with most votes wins.

In one-vs-all you have only k SVMs and for each class you train one SVM against the rest of the classes and again you let them vote and choose the winner.

I don't know whether there is a support for one-vs-one and one-vs-all in Encog, you could write it yourself in the worst case. However, I am sure that you are looking at wrong part of codebase. It most probably won't be in the implementation of the SVM.

Comptometer

Thanks for your answer, i may end up writing my own implementation or just go back to using Accord NET, but im sure there must be support for this built into Encog. The documentation i highlighted certainly hints at it. – Crossbred