1. Neural Network Tools

2. Neural Net Concepts The package provides a “standard” multi-layer perceptron –Composed of layers of neurons –All neurons in a layer are connected to the output of all neurons in the previous layer (or the inputs) Inputs Hidden Layers Output Layer

3. Neurons Combine their inputs and pass the resulting combination through a transfer function (or response) to give a single output Where w i are the weights attached to the u i inputs from the previous layer f may be a tanh, sigmoid or linear response function

4. How the code is structured Top level class NeuralNet –Contains the Neurons in NeuronLayers Three basic types of neuron are available, differing only in their response functions –SigmoidNeuron –TanSigmoidNeuron –LinearNeuron Neurons MUST be created with their builder class – e.g. SigmoidNeuronBuilder This allows for easy addition of your own neuron types with minimal extra effort

5. Create a Neural Net First thing to do is make an “empty” (untrained network) –See the xor.cpp example SigmoidNeuronBuilder *myBuilder = new SigmoidNeuronBuilder; std::vector neuronsPerLayer; neuronsPerLayer.push_back(2); neuronsPerLayer.push_back(1); NeuralNet myNet(2,2,neuronsPerLayer,myBuilder); This code creates a network with 2 inputs, 2 layers of Neurons which are Sigmoid Neurons and has one output neuron. –You can have arbitrary numbers of inputs, layers and neurons per layer (including the output layer).

6. Train the Neural Net An untrained network is completely useless –Weights are set randomly by default There are 4 training algorithms in the package –3 based on back-propagation –1 “Genetic” algorithm based on natural selection ideas Classes are BackPropagationAlgorithm BatchBackPropagationAlgorithm BackPropagationCGAlgorithm GeneticAlgorithm

7. Training Algorithms All training algorithms attempt to minimise the error function where o i and t i are the network outputs and the (user defined) desired outputs respectively They do this by adjusting the weights that each neuron assigns to its inputs

8. Back Propagation Algorithms BackPropagationAlgorithm and BatchBackPropagationAlgorithm use the same method –BatchBackPropagationAlgorithm just runs over more data at one go that the other. Data are held in a NeuralNetDataSet Weight changes are controlled by 2 parameters such that and  are the two parameters called, respectively, the learning rate and momentum constant

9. Using BatchBackPropagationAlgorithm This is probably the most commonly used algorithm –First you need to put your training data into a NeuralNetDataSet –This is just a simple class that holds a set of training data and the output that the neural net should give for each data item NeuralNetDataSet myData; std::vector trainingDataItem; std::vector desiredOutput; Fill training data and desired output for one training point and then myData.addDataItem(trainingDataItem,desiredOutput);

10. Using BatchBackPropagationAlgorithm Now create the algorithm BatchBackPropagationAlgorithm BBPA(myNet,,  ); and  are the two training constants and default to 0.5 if you leave them out. myNet is the network we made earlier. –All the training algorithms take a network as their first parameter. The idea behind this is to allow you to add further training algorithms without having to change the neural net in a way that might not be backwards compatible.

11. Using BatchBackPropagationAlgorithm To train the network use BBPA.train(nEpochs,myData,normaliseData); normaliseData is a bool (true or false) that tells the algorithm to normalise the input data –i.e. calculate mean and sigma of all the training data and then (item by item) subtract the mean and divide by sigma nEpochs is just the number of times to loop through the full dataset. –The weights in the network are only updated after each epoch, so you need to choose your training data set and number of epochs appropriately (by trial and error probably).

12. Using BackPropagationCGAlgorithm BackPropagationCGAlgorithm uses the same ideas as the previous one –It attempts to pick values for and  that maximise the minimisation rate of the error function –Method used is that of conjugate gradients (hence the CG in the name) This is a little more complex as it performs a linear search to optimise the value for Have a look at “Conjugate gradient algorithm for efficient training of artificial neural networks” by C. Charalambous, IEE Proceedings-G Vol. 139, No. 3, June 1992 if you’re interested! Good point about this method is that it seems to converge much (about a factor of 50 or so) faster that the basic back propagation method.

13. GeneticAlgorithm The GeneticAlgorithm treats the weights of the neural net as a genome –It works by having a population of genomes and assessing the “fitness” of each one based on your training data Fitness is just the reciprocal of the error function in this case –The population is then allowed to “breed” – the “best” genomes having a higher probability for “sex” New genomes are created by swapping sections of genes (controlled by CrossOverRate) and mutations (controlled by MutationRate) Convergence is at least as fast, and usually faster, than BatchBackPropagation in the tests I’ve run, but not as fast as BackPropagationCGAlgorithm –Have a look at http://www.ai-junkie.com for an introduction to the ideas behind genetic algorithmshttp://www.ai-junkie.com –I also got the general structure for the neural net from here too

14. Saving the network So you’ve spent days getting a network trained up, now what? –Depending on how you compiled the package, the network will save itself either as an xml file or a plain text file –Use the serialise method of the NeuralNet class to do this You can read the saved network back in by using the constuctor NeuralNet mySavedNet(filename); –Note that if you use xml, the filename could be http://mywebsite/neuralnet.xml so we could have a standard set of trained networks http://mywebsite/neuralnet.xml –This way of building also takes care of creating the right type of neurons, so you could edit the file and change some of the neuron types to have a mixed network. It trains in exactly the same way

15. Real Results…

16. Fortran Bindings There are fortran functions and subroutines to access most (at the moment) of the classes and methods you need to use the package. –There is an example in xor.f –This does exactly the same as the code in xor.cpp, but in fortran… The fortran routines are all listed in the FortranReadMe.txt file They work by calling the C++ code directly behind the scenes and returning integer handles to the C++ objects back to the fortran code –You then use these handles in calls in much the same way as you would pass the objects around in C++.

17. Root Interface In order to keep the basic neural net package as simple as possible, interfaces to root are in a separate library called RootNeuralNet. –The classes in here are just proxy classes that use the same objects as you do from C++ and fortran –They are only designed to allow easy access from interactive root sessions without the need to recompile –I’ve tested them with root 4.00.08 on both RHEL3 and WinXP

18. Have a go! I hope that there are not too many unpleasant “features” in there still –If there are, I’ll try to fix them as fast as possible –If there is some functionality missing that is needed, let me know and I’ll try to add it I’m also available (for a small fee!) to help out with questions –After all, I’ve still got to document and comment all the code properly…