Request for example: Recurrent neural network for predicting next value in a sequence

Question

Can anyone give me a practicale example of a recurrent neural network in (pybrain) python in order to predict the next value of a sequence ? (I've read the pybrain documentation and there is no clear example for it I think.) I also found this question. But I fail to see how it works in a more general case. So therefore I'm asking if anyone here could work out a clear example of how to predict the next value of a sequence in pybrain, with a recurrent neural network.

To give an example.

Say for example we have a sequence of numbers in the range [1,7].

First run (So first example): 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6

Second run (So second example): 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6

Third run (So third example): 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7

and so on.

Now given for example the start of a new sequence: 1 3 5 7 2 4 6 7 1 3

what is/are the next value(s)

This question might seem lazy, but I think there lacks a good and decent example of how to do this with pybrain.

---

Additionally: How can this be done if more than 1 feature is present:

Example:

Say for example we have several sequences (each sequence having 2 features) in the range [1,7].

First run (So first example): feature1: 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6
                              feature2: 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7


Second run (So second example): feature1: 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6
                                feature2: 1 2 3 7 2 3 4 6 2 3 5 6 7 2 4 7 1 3 3 5 6    

Third run (So third example): feature1: 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7
                              feature2: 1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6

and so on.

Now given for example the start of a new sequences:

                                            feature 1: 1 3 5 7 2 4 6 7 1 3

                                            feature 2: 1 2 3 7 2 3 4 6 2 4

what is/are the next value(s)

---

Feel free to use your own example as long it is similar to these examples and has some in depth explanation.

Johnsonjohnsonese

Answer 1

Issam Laradji's worked for me to predict sequence of sequences, except my version of pybrain required a tuple for the UnserpervisedDataSet object:

from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet
from pybrain.structure import LinearLayer
ds = SupervisedDataSet(21, 21)
ds.addSample(map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6'.split()),map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()))
ds.addSample(map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()),map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))
net = buildNetwork(21, 20, 21, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)
ts = UnsupervisedDataSet(21,)
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))
[ int(round(i)) for i in net.activateOnDataset(ts)[0]]

gives:

=> [1, 2, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6, 7, 1, 4, 6, 1, 2, 2, 3, 6]

To predict smaller sequences, just train it up as such, either as sub sequences or as overlapping sequences (overlapping shown here):

from pybrain.tools.shortcuts import buildNetwork
from pybrain.supervised.trainers import BackpropTrainer
from pybrain.datasets import SupervisedDataSet,UnsupervisedDataSet
from pybrain.structure import LinearLayer
ds = SupervisedDataSet(10, 11)
z = map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6 1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6 1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split())
obsLen = 10
predLen = 11
for i in xrange(len(z)):
  if i+(obsLen-1)+predLen < len(z):
    ds.addSample([z[d] for d in range(i,i+obsLen)],[z[d] for d in range(i+1,i+1+predLen)])

net = buildNetwork(10, 20, 11, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)
ts = UnsupervisedDataSet(10,)
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3'.split()))
[ int(round(i)) for i in net.activateOnDataset(ts)[0]]

gives:

=> [3, 5, 6, 2, 4, 5, 6, 1, 2, 5, 6]

Not too good...

Annunciate

Your second example seems like a viable answer. – Johnsonjohnsonese

Can you elaborate "viable" answer? How do you judge if the output is "ok"? Based on the original examples? But there so few examples given. – Eatable

Answer 2

These steps are meant to perform what you ask for in the first part of the question.

1) Create a supervised dataset that expects a sample and a target in its arguments,

 ds = SupervisedDataSet(21, 21)
 #add samples (this can be done automatically)
 ds.addSample(map(int,'1 2 4 6 2 3 4 5 1 3 5 6 7 1 4 7 1 2 3 5 6'.split()),map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()))
 ds.addSample(map(int,'1 2 5 6 2 4 4 5 1 2 5 6 7 1 4 6 1 2 3 3 6'.split()),map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))

A succeeding sample is the target or label y of its predecessor x. We put the number 21 because each sample has 21 numbers or features.

Please note that for standard notations in the second half of your question it is better to call feature1 and feature2 as sample1 and sample2 for a sequence, and let features denote the numbers in a sample.

2) Create Network, initialize trainer and run for 100 epochs

net = buildNetwork(21, 20, 21, outclass=LinearLayer,bias=True, recurrent=True)
trainer = BackpropTrainer(net, ds)
trainer.trainEpochs(100)

Make sure to set the recurrent argument as True

3) Create the test data

ts = UnsupervisedDataSet(21, 21)
#add the sample to be predicted
ts.addSample(map(int,'1 3 5 7 2 4 6 7 1 3 5 6 7 1 4 6 1 2 2 3 7'.split()))

We created an unsupervised dataset because of the assumption that we don't have the labels or targets.

4) Predict the test sample using the trained network

net.activateOnDataset(ts)

This should display the values of the expected fourth run.

For the second case when a sequence can have more than sample, instead of creating a supervised dataset, create a sequential one ds = SequentialDataSet(21,21). Then, everytime you get a new sequence, call ds.newSequence() and add the samples -that you call features- in that sequence using ds.addSample().

Hope this is clear-cut :)

If you wish to have the full code to save the trouble of importing the libraries, please let me know.

Greaser

Hmm you don't seem to answer the question... "Now given for example the start of a new sequence: 1 3 5 7 2 4 6 7 1 3 what is/are the next value(s)" So what comes after the last 3 in the sequence ? – Johnsonjohnsonese

Answering your specific question, given just so few examples, has no meaning - anything is possible. But What @Curious has given is the algorithm of doing it, and if you feed it with large number of sequence - which we are not sure HOW you get the sequence - the accuracies of output will be higher. Specific answer is not what we are interested. – Eatable