1. Playing Tic-Tac-Toe with Neural Networks
Zachary McNellis CPSC 4820

2. What is a robot?
Sense, think, act

3. Sense, Think, React
Robotics technology consists of mechanisms that can:
Sense – Feedback devices (sensors) allow information about the environment to be recorded
Think – Information is processed in some way (simple or complex)
Act – Most obvious part of a robot. However, it can be anything from outputting a value to making the robot walk
Acting is the most obvious part of robotics technology. The electronic signals that were produced as a result of sensing and thinking then control whatever the robot is designed to do, like lift a sick person, make a facial expression, or control the motors that allow it to navigate around an obstacle.

4. Creating a tic-tac-toe engine
Board representation
What move to make?
Win, Lose, Draw

5. Board Representation 3 0 2 1 1 1 1 1 1 1 1 1 9 board positions4 7 2 5 8 3 6 9
9 board positions
Player 1 Player 2 Empty 0 2 1
Dimension, player 1, player 2
Positions labeled 1-9

6. What Move to Make? What does a tic-tac-toe engine do?
Input: board state
Ex. “ | ./my_engine”
Output: next move
Avoid collisions
Ex. “5”

7. Win, Lose, or Draw “playtictactoe.py” Output Specify number of games
Engine 1
Engine 2
Output
Game progression
Player 1 win ___ times
Draw ___ times
Player 2 win ___ times

8. 1. Random Engine
Implementation details
Results summary

9. Implementation Details
Java
Slow execution
Internal representation of board state
x---ox--o
x: player 1
o: player 2
-: empty position
2 dimensional array
Polymorphism to easily allow different engine implementations
Player player = new RandomPlayer(board, turn);

10. Results Summary random_engine vs random_engine
Player 1 win 49
Draw 13 times
Player 2 win 38
About equal number of wins from player 1 as player 2

11. 2. “Smart” Engine Implementation details Case based reasoning
Results summary

12. Implementation Details
Java
Rules were simple and came from hands-on experience
IF able to get 3 in a row, play winning position
ELSE IF able to block opponent, play blocking position
ELSE IF empty, play edge position
ELSE play random position

13. Case Based Reasoning
Use reverse logic to figure out “rules” governing an unknown engine
Steps
Retrieve
Reuse
Revise
Retain

14. Results Summary random_engine vs smart_engine
Player 1 win 8
Draw 29 times
Player 2 win 63
smart_engine vs smart_engine
Player 1 win 59
Draw 10 times
Player 2 win 31

15. 3. Neural Network Engine Neural network overview
Implementation details
Results summary

16. Neural Network Overview
Provides ability to “learn” how to do tasks based on training data
Requires linear and nonlinear step to produce a set of weights
Weights map training input to training output
Learning rate used to discover a set of weights that result in an error of 0, in which all inputs are precisely mapped to all outputs

17. Implementation Details
Goal: train neural network on data produced by previous “smart engine”
Input: state of the board
Output: next move
Neural network trainer
Python
Allows user to pass in parameters such as learning rate, bias, input, output, and weight files
15 pairs of inputs and outputs used
Difficulty of convergence
Neural network engine
Use set of weights used by trainer to generate “next move”

18. Results Summary neural_engine vs smart_engine
Player 1 win 38
Draw 11 times
Player 2 win 51
neural_engine vs random_engine
Player 1 win 56
Draw 12 times
Player 2 win 32

19. 4. PyBrain Engine PyBrain neural network library
Implementation details
Results summary

20. Implementation Details
Goal: Implement same neural network engine using training weights produced from an external library
PyBrain
Python-Based Reinforcement Learning, Artificial Intelligence and Neural Network Library
Used backpropogation method of training values
Optimization of errors, minimizing loss function
Allows higher chance of convergence for larger data sets
25 pairs of input/output compared to 15

21. Results Summary smart_engine vs neurallib_engine
Player 1 win 51
Draw 6 times
Player 2 win 43
random_engine vs neurallib_engine
Player 1 win 30
Draw times 16
Player 2 win 54

22. (5?) Self Organizing Maps
Another type of neural network
Using weights in different ways
Weights are now nodes instead of connections
Useful for identifying what the inputs should be
Weights are updated based on geography
Useful for pattern completion
Could be used in tic-tac-toe engine to determine whether a given board state is valid or not
You may already be aware of supervised training techniques such as backpropagation where the training data consists of vector pairs - an input vector and a target vector. With this approach an input vector is presented to the network (typically a multilayer feedforward network) and the output is compared with the target vector. If they differ, the weights of the network are altered slightly to reduce the error in the output. This is repeated many times and with many sets of vector pairs until the network gives the desired output. Training a SOM however, requires no target vector. A SOM learns to classify the training data without any external supervision whatsoever.

23. Now I’ll show a demonstration of running the programs I’ve been discussing