multi layer perceptron - finding the "separating" curve

Question

with single-layer perceptron it's easy to find the equation of the "separating line" (I don't know the professional term), the line that separate between 2 types of points, based on the perceptron's weights, after it was trained. How can I find in a similar way the equation of the curve (not straight line) that separate between 2 types of points, in a multi-layer perceptron?

thanks.

Cacuminal

Answer 1

This is only an attempt to get an approximation to the separating boundary or curve.

Dataset

Below I plotted the separating curve between the two types of the example dataset. The dataset is borrowed from coursera - Andrew Ng's machine learning course. Also the code snippet below borrows the ideas from Ex6 of Andrew's ML course.

Boundary Plot

To plot the separating curve,

• You first train your neural network against your training data;
• Generate a 2d grid of data using the granularity you want, in Matlab, this is something like:

    x1plot = linspace(min(X(:,1)), max(X(:,1)), 100)';
    x2plot = linspace(min(X(:,2)), max(X(:,2)), 100)';
    [X1, X2] = meshgrid(x1plot, x2plot);

• For each data point in the grid, calculate the predicted label using your neural network;
• Drawing the coutour graph of the resulting labels of grid

    vals = zeros(size(X1));
    for i = 1:size(X1, 2)
       this_X = [X1(:, i), X2(:, i)];
       % mlpPredict() is the function to use your trained neural network model
       %    to get a predicted label. 
       vals(:, i) = mlpPredict(model, this_X);
    end

    % Plot the boundary
    hold on
    [C, Lev] = contour(X1, X2, vals, [0 0], 'Color', 'b');
    hold off;

If your goal is only to get the exact mathematical representation of the boundary curve, this method won't work. This method can only give you an approximation of the curve up to the granularity you set up in your grid.

If you do want a precise description of the boundary, SVM might be a good alternative since the whole set of support vectors could serve as the boundary descriptive.

Approximate boundary using contour points

I took a look at octave's documentation about contour. Basically, contour uses the contour matrix C computed by contourc from the same arguments. Here is the signature of contourc:

[C, LEV] = contourc (X, Y, Z, VN)

This function computes contour lines of the matrix Z. Parameters X, Y and VN are optional.

 The return value LEV is a vector of the contour levels.  The
 return value C is a 2 by N matrix containing the contour lines in
 the following format

      C = [lev1, x1, x2, ..., levn, x1, x2, ...
           len1, y1, y2, ..., lenn, y1, y2, ...]

 in which contour line N has a level (height) of LEVN and length of
 LENN.

So if you do want to get an analytical description of the curve, matrix C should contain enough information about it. In my sample plot, after parsing of C, I get 30 levels. The coordinates of the first 6 points in the first level are listed below:

x: 2.3677e-01   2.3764e-01   2.4640e-01   2.4640e-01   2.4640e-01   2.4640e-01 ...
y: 4.0263e-01   4.0855e-01   4.0909e-01   4.1447e-01   4.2039e-01   4.2631e-01 ...

Please notice that they are exactly the points on the contour starting from (0.23677, 0.40263). Using these contour points, it's straightforward to approximate the curve using multiple line segments (because each line segment can be determined by two end points).

Hope it helps.

Anaclinal

Thank you very much. Is there also an analytical method for the approximation of the curve? – Cacuminal

First you might want to try identifying the points on the grid which locate closest to the contour (take a look at the source code of contour function should help.) After that, it is straightforward to approximate the curve using multiple line segments (because each line segment can be determined by two end points). – Anaclinal