CNTK C# API - How to set learner module and number of iteration in learner module for training

Question

I am using CNTK C# API (Latest 2.2 release) and have following questions. (using the logisticsRegression example from GitHub and modifying it to set it for my usecase.

Using SGD and also tried SGDMomentim learners.

1. How can I set number of max iteration per sample for training. I can see LR and MR variables in SGDMomentum but nothing for iterations.

2. Is there a way to monitor how each neuron is progressing/stuck as training progresses thru each sample per iteration in the network. Or how the weight parameters were evolving in each iteration?

3. When are other learner modules expected to be released in C#? (Other then SGD and SGD momentum)?

Thanks for your help.

Bacchanalia

Answer 1

1. use TrainingParameterScheduleDouble(VectorPairSizeTDouble schedule, uint epochSize) to specify rate per multiple of epochs. (Yes the API shall be more C# friendly.) Use the input vector to specify a rate for each number of Epochs. For example: {{1, 0.05}, {2, 0.1}, {1, 0.005}} and epochSize = 100.

It means: '0.05' is used for the first 100 samples, then '0.1' is used for the second 200 samples, after which the values is switched to '0.005'

2. To monitor parameter update: see answer to this
3. Most learners are released in 2.2. Do you have a specific learner that is not available from C# API?

Chole

Thanks. I was able to create the vectiorpair... was not intuitive at all :-) – Bacchanalia

PairSizeTDouble p1 = new PairSizeTDouble(1, 0.001); PairSizeTDouble p2 = new PairSizeTDouble(2, 0.1); PairSizeTDouble p3 = new PairSizeTDouble(1, 0.005); CNTK.VectorPairSizeTDouble vp = new VectorPairSizeTDouble() { p1 };//, p2, p3 } ; CNTK.TrainingParameterScheduleDouble epochSchedule = new CNTK.TrainingParameterScheduleDouble(vp, 100); – Bacchanalia

I have an equivalent model in AZure ML studio. It works nicely. The model uses "number of iteration" as a parameter along with other parameters. . I have set the model same as AZure ML in CNTK but this not working in CNTK. The weights are not getting modified at all as training progresses in CNTK. – Bacchanalia

What is meant by rate per multiply. Rate of what? Could you help me understand it better. – Bacchanalia

Regarding rate per multiple (corrected typo in my answer) of epochs: for example {2, 0.1} means for the next 2 * 100 (100 is the epoch size) samples, the learning rate will be 0.1. – Chole

Regarding "number of iteration" Does it mean number of mini-batches? May you increase the learning rate to see whether parameters get updated. If parameters never get updated, it must be that either the model and the criterion is not correct. – Chole

Thanks Liqun, Appreciate your quick help. This helps, I will continue the debug in the model. – Bacchanalia

In Azure ML studio, when initializing the neural network component, there are options for Learning Rate, the momentum, Initial weight and "Number of learning iterations" along with Net# definition of the network. I was trying to figure out what is equivalent parameter to "Number of learning iterations" in CNTK to implement training. – Bacchanalia