1. NEURAL NETWORKS FOR DATA MINING
Chapter 8
NEURAL NETWORKS FOR DATA MINING

2. Learning Objectives
Understand the concept and different types of artificial neural networks (ANN)
Learn the advantages and limitations of ANN
Understand how backpropagation neural networks learn
Understand the complete process of using neural networks
Appreciate the wide variety of applications of neural networks

3. Basic Concepts of Neural Networks
Neural networks (NN) or artificial neural network (ANN)
Computer technology that attempts to build computers that will operate like a human brain. The machines possess simultaneous memory storage and works with ambiguous information
A brain metaphor for information processing. These models are biologically inspired rather than an exact replica of how the brain actually functions.
Very promising systems:
Ability to learn from the data
Nonparametric nature(no rigid assumptions)
Ability to generalize

4. Basic Concepts of Neural Networks
Neural computing
An experimental computer design aimed at building intelligent computers that operate in a manner modeled on the functioning of the human brain. See artificial neural networks (ANN)
it is a pattern recognition methodology for machine learning. The resulting model from the neural computing is often called ANN.
Development history (ANN journey)
McCulloch and Pitts, 1943
1950s-1960s popular
1970s-1980s diminished
1990s-, rational developing.
2000s-, data mining,

5. Basic Concepts of Neural Networks
Biological and artificial neural networks
Perceptron （感知器）
Early neural network structure that uses no hidden layer
Neurons (神经元）
Cells (processing elements) of a biological or artificial neural network
Nucleus （神经核）
The central processing portion of a neuron
Dendrite （树突）
The part of a biological neuron that provides inputs to the cell
Axon （轴突）
An outgoing connection (i.e., terminal) from a biological neuron
Synapse （突触）
The connection (where the weights are) between processing elements in a neural network

6. Basic Concepts of Neural Networks

7. Basic Concepts of Neural Networks

8. Basic Concepts of Neural Networks
Biological
Artificial
Soma
Node
Dendrite
Input
Axon
Output
Synapse
Weight
Slow Speed
Fast Speed
Many neurons (10e9)
Few neurons (10-100s)

9. Basic Concepts of Neural Networks
Elements of ANN
Topologies (architectures)
The type neurons are organized in a neural network
Backpropagation (反向传播，多馈前向）
The best-known learning algorithm in neural computing. Learning is done by comparing computed outputs to desired outputs of historical cases
1986年Rumelhart和McCelland小组提出。

10. Basic Concepts of Neural Networks
Processing elements (PEs)
The PE in a neural network are artificial neurons.
each of the neurons receives input, processes them, and deliveries a single output. Show in figure 8.2.
Network structure (three layers)
Input
Intermediate (hidden layer)
Output
A hidden layer is a layer of neurons that takes input from the previous layer and converts those input into output for further processing.
Several hidden layers can be places between input and output layers.
Most of hidden layers process mechanisms are feature extraction.

11. Basic Concepts of Neural Networks

12. Basic Concepts of Neural Networks
Parallel processing
An advanced computer processing technique that allows a computer to perform multiple processes at once—in parallel
Resembles the way the brain works, it differs from the serial processing of conventional computing.

13. Basic Concepts of Neural Networks
Network information processing
Inputs : each input corresponds t a single attribute. E.g., problem is approval or disapproval for a loan.
Numeric value, or representation, of an attribute is the input to the network.
Outputs : contain the solution of a problem. For example, loan can be yes or no. Also, numeric value to the output, such as 0, or 1 for yes or no.
The purpose of ANN is to compute the values of the output. Thus, post-processing of output is required .

14. Basic Concepts of Neural Networks
Network information processing
Connection weights
The weight associated with each link in a neural network model. They are assessed by neural networks learning algorithms
Summation function or transformation (transfer) function
In a neural network, the function that sums and transforms inputs before a neuron fires. The relationship between the internal activation level and the output of a neuron

15. Basic Concepts of Neural Networks

16. Basic Concepts of Neural Networks
Sigmoid (logical activation) function
An S-shaped transfer function in the range of zero to one

17. Basic Concepts of Neural Networks
Threshold value
A hurdle value for the output of a neuron to trigger the next level of neurons. If an output value is smaller than the threshold value, it will not be passed to the next level of neurons
Hidden layer
The middle layer of an artificial neural network that has three or more layers
Theoretically, there can be layers. However, more than three layers are seldom in commercial software.
The more of hidden layers, the more the time will be used for training.

18. Basic Concepts of Neural Networks
Neural network architectures
Common neural network models and algorithms include:
Backpropagation
Feedforward (or associative memory)
Recurrent network

19. Basic Concepts of Neural Networks

20. Basic Concepts of Neural Networks

21. Learning in ANN Learning algorithm ( also, training algorithm)
The training procedure used by an artificial neural network . Learning algorithm specify the process by which a neural network learns the underlying relationship between input and outputs, or just among the inputs.
There are hundreds of them.
It can be classified into to categories: Supervised learning and unsupervised learning.

22. Learning in ANN

23. Learning in ANN Supervised learning
A method of training artificial neural networks in which sample cases (training sets) are shown (teaching ) to the network as input and the weights are adjusted to minimize the error in its outputs. The training set and desired output is iteratively presented to the neural networks. Output of the network in its present form is calculated and compared to the desired output.
Backpropagation learning algorithm: is popular supervised learning.
It is an iterative gradient-descent technique designed to minimize an error function between the actual output of the network and its desired output, as specified in the training data set.
Unsupervised learning
A method of training artificial neural networks in which only input stimuli are shown to the network, it organizes itself internally so that each hidden processing element responds strategically to a different set of input stimuli. No knowledge is supplied about which classification are correct.
So it called self-organizing or clustering its neurons related to the specific desired task.

24. Learning in ANN
Self-organizing A neural network architecture that uses unsupervised learning Adaptive resonance theory (ART) An unsupervised learning method created by Stephen Grossberg. It is a neural network architecture that is aimed at being more brain-like in unsupervised mode Kohonen self-organizing feature maps A type of neural network model for machine learning

25. Learning in ANN The general ANN learning process
The process of learning involves three tasks:
Compute temporary outputs
Compare outputs with desired targets
Adjust the weights and repeat the process

26. Learning in ANN

27. Learning in ANN The general ANN learning process
The process of learning involves three tasks:
Compute temporary outputs
Compare outputs with desired targets
Adjust the weights and repeat the process

28. Learning in ANN Pattern recognition
The technique of matching an external pattern to one stored in a computer’s memory; used in inference engines, image processing, neural computing, and speech recognition (in other words, the process of classifying data into predetermined categories).

29. Learning in ANN How a network learns Learning rate
A parameter for learning in neural networks. It determines the portion of the existing discrepancy that must be offset
Momentum
A learning parameter in feedforward-backpropagation neural networks

30. Learning in ANN How a network learns Backpropagation
The best-known learning algorithm in neural computing. Learning is done by comparing computed outputs to desired outputs of historical cases

31. Learning in ANN How a network learns
Procedure for a learning algorithm
Initialize weights with random values and set other parameters
Read in the input vector and the desired output
Compute the actual output via the calculations, working forward through the layers
Compute the error
Change the weights by working backward from the output layer through the hidden layers

32. Developing Neural Network–Based Systems

33. Developing Neural Network–Based Systems
Data collection and preparation
The data used for training and testing must include all the attributes that are useful for solving the problem
Selection of network structure
Selection of a topology
Topology
The way in which neurons are organized in a neural network

34. Developing Neural Network–Based Systems
Data collection and preparation
The data used for training and testing must include all the attributes that are useful for solving the problem
Selection of network structure
Selection of a topology
Determination of:
Input nodes
Output nodes
Number of hidden layers
Number of hidden nodes

35. Developing Neural Network–Based Systems

36. Developing Neural Network–Based Systems
Learning algorithm selection
Identify a set of connection weights that best cover the training data and have the best predictive accuracy
Network training
An iterative process that starts from a random set of weights and gradually enhances the fitness of the network model and the known data set
The iteration continues until the error sum is converged to below a preset acceptable level

37. Developing Neural Network–Based Systems
Testing
Black-box testing
Comparing test results to actual results
The test plan should include routine cases as well as potentially problematic situations
If the testing reveals large deviations, the training set must be reexamined, and the training process may have to be repeated

38. Developing Neural Network–Based Systems
Implementation of an ANN
Implementation often requires interfaces with other computer-based information systems and user training
Ongoing monitoring and feedback to the developers are recommended for system improvements and long-term success
It is important to gain the confidence of users and management early in the deployment to ensure that the system is accepted and used properly

39. Developing Neural Network–Based Systems

40. A Sample Neural Network Project

41. Other Neural Network Paradigms
Hopfield networks
A single large layer of neurons with total interconnectivity—each neuron is connected to every other neuron
The output of each neuron may depend on its previous values
One use of Hopfield networks: Solving constrained optimization problems, such as the classic traveling salesman problem (TSP)

42. Other Neural Network Paradigms
Self-organizing networks
Kohonen’s self-organizing network learn in an unsupervised mode
Kohonen’s algorithm forms “feature maps,” where neighborhoods of neurons are constructed
These neighborhoods are organized such that topologically close neurons are sensitive to similar inputs into the model
Self-organizing maps, or self organizing feature maps, can sometimes be used to develop some early insight into the data

43. Applications of ANN
ANN are suitable for problems whose inputs are both categorical and numeric, and where the relationships between inputs and outputs are not linear or the input data are not normally distributed