1. Artificial Neural Networks And XML
Presented by : M. Eftekhari
Advisor : Dr. S. Astaneh

2. Outlines Xml-Based format for trained Neural Network definition
Motivations
Neural Network Markup Language (NNML)
Decomposition of Neural Nets Model
The neural model description in NNML
Processing of NNML documents
PMML ….
Introduction
From biological Artificial Neural Nets
Inherent capacities
The Distributed Training Environment (DTE)
Why distributed environment? (Motivation)
Features
JOONE

3. Introduction

4. From biological to Artificial Neuron (Intro.)

5. A simple Artificial Neuron
Out put connections are similar to axons
Sum simulates the
dendrites
w0= f
Activation function
Has the role of events that
Occur in a real neuron of brain w1 w2
Weights are similar to synapse x1 x2
The learning is the process of updating weights

6. Inherent capacities (Intro.)
The neurons are parallel inside each other due to inherent structure of Neural Network.
When a network learns, it works as a autonomous mechanism (speech part of brain).
A central mechanism coordinates, schedules and these self-organize parts. (may be ensemble of parts needed)
So ANNs can be distributed. A learnt ANNs can be shared to use by other applications.

7. The Distributed Training Environment (DTE)

8. Why distributed environment? (Motivation)
use a neural net to resolve complex jobs is not sufficient.
For complex problem, net can fall onto a local minima without finding the best results
what must be developed is a mechanism to train many neural nets in parallel on the same problem, on several machines, governing the whole process from a central point.

9. Java Object Oriented Neural Engine
Joone is a FREE neural net framework to create, train and test neural nets
Distributed Training Environment to train in parallel mode many neural networks to find the fittest one for a given problem.

10. Features Centralized control
The Final results are logged into a XML file to permit to analyze the results from a custom program/script
The training process scale almost linearly adding more machines
No manual configuration needed to add or remove machines
Possibility to add or remove machines dynamically during the training process
The overall process controlled by XML parameters

11. Xml-Based format for trained Neural Network definition

12. Motivations unified way for neural network model definition
Interchanging neural models as well as documentation
store and manipulating them independently from the simulation system that produced it.
The development of the neural-based Web services

13. Neural Network Markup Language (NNML)
XML-based language (Neural Network Markup Language) for the neural network model description.
NNML as an interface between various software systems concerning neural networks (see Fig of next slide)
NNML causes separation of neural networks generators, interpreters, tools for visualization and knowledge extraction (see Fig of next slide)
Applied to the distribution of the neural network models
Integrating the powerful simulation systems like Matlab with Web interface

14. Neural Network Markup Language (NNML)

15. Decomposition of Neural Nets Model and NNML

16. The neural model description in NNML
The problem and model purpose (Task)
Data dictionary
Data preprocessor
Neural network
Postprocessor
Auxiliary information about the model.

17. The neural model description in NNML A simple neuron
Which object
Form Which
layer
layer
Obj

18. Processing of NNML documents
Generation and training by means of the neural network simulator.
Creating hierarchical model by the interface module on the basis of internal representation.
Methods of any XML parser are called, object tree of the model are constructed
NNML file is generated.
For loading of the ready NNML file, actions are performed in the reverse order.

19. Processing of NNML documents

20. PMML (Predictive Model Markup Language )

21. Introduction
The PMML is a set of Document Type Descriptions (DTDs) specified in XML.
The first version (1.0) was provided in July by the Data Mining Group (DMG,
A Markup Language for Predictive modeling, but not only restricted to this field.
Support only Back propagation Nets despite of previous introduced Method.

22. PMML (Contd.)
The PMML 1.1 definition includes DTDs for the following types of models:
1. Naïve bayes
2. Regression Models
3. Decision trees
4. Center and distribution based clusters
5. Sequence and association rules
6. neural nets

23. Advantages of PMML:
Removes the issues of incompatibility between applications and proprietary formats.
DTDs support proprietary extensions to allow for enriched information storage for specialized tools.
Previous solutions to the problem of sharing data models were incorporated into custom-built systems, and thus exchange of models with an application outside of the system was virtually impossible

24. Advantages of PMML For example, it allows users: (sharing the data)
To generate data models using one vendor application.
Use other vendor application to analyze.
Another to evaluate the models.
Another vendor application to visualize the model.

25. PMML (Contd.) The PMML describes the models using eight modules:
Header
Data Dictionary schema
Data Mining schema
Predictive model schema
Definition for predictive models
Definition for ensemble of models
Rules for selecting and combining models and ensembles of models
Rules for exception handling

26. The role of PMML in the Knowledge Discovery process.

27. PMML (Contd.) Using PMML to model Association Rules
<?xml version="1.0" ?>
<PMML version="1.1">
<Header copyright=" description="sample model for association rules"/>
<DataDictionary numberOfFields="1" > <DataField name="item" optype="categorical" /> </DataDictionary>
<AssociationModel>
<AssocInputStats numberOfTransactions="4" numberOfItems="3"
minimumSupport="0.6" minimumConfidence="0.5"
numberOfItemsets="3" numberOfRules="2"/>
<!-- We have three items in our input data -->
<AssocItem id="1" value="Cracker" />
<AssocItem id="2" value="Coke" />
<AssocItem id="3" value="Water" />
<!-- and two frequent itemsets with a single item -->
<AssocItemset id="1" support="1.0" numberOfItems="1">
<AssocItemRef itemRef="1" />
</AssocItemset>
<AssocItemset id="2" support="1.0" numberOfItems="1">
<AssocItemRef itemRef="3" />
<!-- and one frequent itemset with two items. -->
<AssocItemset id="3" support="1.0" numberOfItems="2">
<AssocItemRef itemRef="1" /><AssocItemRef itemRef="3" />
<!-- Two rules satisfy the requirements -->
<AssocRule support="1.0" confidence="1.0" antecedent="1" consequent="2" />
<AssocRule support="1.0" confidence="1.0" antecedent="2" consequent="1" />
</AssociationModel>
</PMML>

28. PMML The General Web Architecture
Application interfaces
Web Warehouse
Materialize and manages useful
Information on web
A software that facilitates the process of
Content extraction

29. PMML and ANNs (DTD)
<!ELEMENT NeuralInput (Extension*, ( NormContinuous | NormDiscrete )) >
<!ATTLIST NeuralInput id %NN-NEURON-ID; #REQUIRED >
NN-NEURON-ID is just a string which identifies a neuron

30. PMML and ANNs (XSD) <xs:element name="NeuralInput">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0“ maxOccurs="unbounded“ ref="Extension" />
<xs:element ref="DerivedField" />
</xssequence>
<xs:attribute name="id" type="NN-NEURON-ID" use="required" />
</xs:complexType>
</xs:element>

31. PMML and ANNs (DTD) <! ELEMENT Neuron (Extension*, Con+) >
<! ATTLIST Neuron
id %NN-NEURON-ID; #REQUIRED
bias %REAL-NUMBER; #IMPLIED
activationFunction %ACTIVATION-FUNCTION; #IMPLIED
threshold %REAL-NUMBER; #IMPLIED
>

32. PMML and ANNs (XSD) <xs:element name="Neuron">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="Extension" />
<xs:element maxOccurs="unbounded" ref="Con" />
</xs:sequence>
<xs:attribute name="id" type="NN-NEURON-ID" use="required" />
<xs:attribute name="bias" type="REAL-NUMBER" />
<xs:attribute name="activationFunction" type="ACTIVATION-FUNCTION" />
<xs:attribute name="threshold" type="REAL-NUMBER" />
<xs:attribute name="width" type="REAL-NUMBER" />
</xs:complexType>
</xs:element>

33. PMML and ANNs (DTD) <!ELEMENT Con (Extension*) >
<!ATTLIST Con
from %NN-NEURON-IDREF; #REQUIRED weight %REAL-NUMBER; #REQUIRED >

34. PMML and ANNs (XSD) <xs:element name="Con">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="Extension" />
</xs:sequence>
<xs:attribute name="from" type="NN-NEURON-IDREF" use="required" />
<xs:attribute name="weight" type="REAL-NUMBER" use="required" />
</xs:complexType>
</xs:element>

35. PMML and ANNs (DTD)
<!ELEMENT NeuralNetwork (Extension*, MiningSchema, ModelStats?, NeuralInputs, ( NeuralLayer+), NeuralOutputs? )>
<!ATTLIST NeuralNetwork
modelName CDATA #IMPLIED
activationFunction %ACTIVATION-FUNCTION; #REQUIRED threshold #REAL-NUMBER; #IMPLIED >
<!ELEMENT NeuralInputs ( NeuralInput+ ) >
<!ELEMENT NeuralLayer ( Neuron+ ) >
<!ELEMENT NeuralOutputs ( NeuralOutput+ ) >

36. PMML and ANNs (XSD) <xs:element name="NeuralNetwork">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="Extension" />
<xs:element ref="MiningSchema" />
<xs:element minOccurs="0" ref="ModelStats" />
<xs:element ref="NeuralInputs" />
<xs:element maxOccurs="unbounded" ref="NeuralLayer" />
<xs:element minOccurs="0" ref="NeuralOutputs" />
</xs:sequence>
<xs:attribute name="modelName" type="xs:string" />
<xs:attribute name="functionName" type="MINING-FUNCTION" use="required" />
<xs:attribute name="algorithmName" type="xs:string" />
<xs:attribute name="activationFunction" type="ACTIVATION-FUNCTION" use="required" />
<xs:attribute name="threshold" type="REAL-NUMBER" />
<xs:attribute name="numberOfLayers" type="xs:nonNegativeInteger" />
</xs:complexType>
</xs:element>

37. PMML and ANNs (XSD) <xs:element name="NeuralInputs">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="Extension" />
<xs:element maxOccurs="unbounded" ref="NeuralInput" />
</xs:sequence>
<xs:attribute name="numberOfInputs" type="xs:nonNegativeInteger" />
</xs:complexType>
</xs:element>

38. PMML and ANNs (XSD) <xs:element name="NeuralLayer">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="Extension" />
<xs:element maxOccurs="unbounded" ref="Neuron" />
</xs:sequence>
<xs:attribute name="numberOfNeurons" type="xs:nonNegativeInteger" />
<xs:attribute name="activationFunction" type="ACTIVATION-FUNCTION" />
<xs:attribute name="normalizationMethod" default="none">
<xs:simpleType> <xs:restriction base="xs:string">
<xs:enumeration value="none" />
<xs:enumeration value="simplemax" />
<xs:enumeration value="softmax" />
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:complexType>
</xs:element>

39. PMML and ANNs (XSD) <xs:element name="NeuralOutputs">
<xs:complexType>
<xs:sequence>
<xs:element minOccurs="0" maxOccurs="unbounded" ref="Extension" />
<xs:element maxOccurs="unbounded" ref="NeuralOutput" />
</xs:sequence>
<xs:attribute name="numberOfOutputs" type="xs:nonNegativeInteger" />
</xs:complexType>
</xs:element>

40. PMML and ANNs <?xml version="1.0" ?> <PMML version="2.1">
<Header copyright="DMG.org"/>

41. PMML and ANNs <DataDictionary numberOfFields="5">
<DataField name="gender" optype="categorical">
<Value value=" female"/> <Value value=" male"/>
</DataField>
<DataField name="no of claims" optype="categorical">
<Value value=" 0"/> <Value value=" 1"/> <Value value=" 3"/> <Value value=" > 3"/> <Value value=" 2"/>
<DataField name="domicile" optype="categorical">
<Value value="suburban"/> <Value value=" urban"/> <Value value=" rural"/>
<DataField name="age of car" optype="continuous"/>
<DataField name="amount of claims" optype="continuous"/>
</DataDictionary>

42. PMML and ANNs <MiningSchema> </MiningSchema>
<MiningField name="gender"/>
<MiningField name="no of claims"/>
<MiningField name="domicile"/>
<MiningField name="age of car"/>
<MiningField name="amount of claims" usageType="predicted"/>
</MiningSchema>

43. PMML and ANNs <NeuralInputs numberOfInputs="10">
<NeuralInput id="0">
<DerivedField>
<NormContinuous field="age of car">
<LinearNorm orig="0.01" norm="0"/>
<LinearNorm orig=" " norm="0.5"/>
<LinearNorm orig="11.44" norm="1"/>
</NormContinuous>
</DerivedField>
</NeuralInput>

44. PMML and ANNs <NeuralInput id="1"> <DerivedField>
<NormDiscrete field="gender" value=" male"/>
</DerivedField>
</NeuralInput>
…. To 9

45. PMML and ANNs <NeuralLayer numberOfNeurons="3">
<Neuron id="10">
<Con from="0" weight=" "/>
<Con from="1" weight=" "/>
<Con from="2" weight=" "/>
<Con from="3" weight=" "/>
<Con from="4" weight=" "/>
<Con from="5" weight=" "/>
<Con from="6" weight=" "/>
<Con from="7" weight=" "/>
<Con from="8" weight=" "/>
<Con from="9" weight=" "/>
</Neuron> ……
</NeuralLayer> I0 . N1
Id=10 . I9

46. PMML and ANNs Output Neuron
<NeuralLayer numberOfNeurons="1">
<Neuron id="13">
<Con from="10" weight=" "/>
<Con from="11" weight=" "/>
<Con from="12" weight=" "/>
</Neuron>
</NeuralLayer> N1
Id=10
output No
Id=13 N2
Id=11 N3
Id=13

47. PMML and ANNs <NeuralOutputs numberOfOutputs="1">
<NeuralOutput outputNeuron="13">
<DerivedField>
<NormContinuous field="amount of claims">
<LinearNorm orig="0" norm="0.1"/>
<LinearNorm orig=" " norm="0.5"/>
<LinearNorm orig=" " norm="0.9"/>
</NormContinuous>
</DerivedField>
</NeuralOutput>
</NeuralOutputs>
</NeuralNetwork>
</PMML>