1. cs638/838 - Spring 2017 (Shavlik©), Week 11
CS 540 Fall 2015 (Shavlik)
4/18/2018
Today’s Topics
Talk by Sidharth Mudgal Sunil Kumar on text and Deep Nets
Keras (on top of TensorFlow) for Solving Lab 3
I have uploaded to class web page my latest Python
- 10 folds, perturbed examples, Condor, in multiple files, etc
An ImageNet dataset (same six categories as Lab 3)
Some Python/Keras questions I have
April 11: GAN Intro, plus some cs838 progress reports
April 18: Google-Madison talk on TPUs, plus some cs638 progress reports
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

2. cs638/838 - Spring 2017 (Shavlik©), Week 11
Keras – CONV Layer 1
model = Sequential() # A ‘left-to-right’ model with no recurrent links. # The following needs to be a layer, not an argument (‘relu’ can be an arg). leakyReLUtoUse = LeakyReLU(alpha = 0.1) model.add(Conv2D(platesConv1, kernel_size = kernelSizeConv1, # Give two numbers if non-square. input_shape = [imageDimension, imageDimension, numberOfColors], data_format = "channels_last", # Says that the color channels are LAST. strides = strideConv1, padding = "valid", # ???? use_bias = True))
4//17
cs638/838 - Spring 2017 (Shavlik©), Week 11

3. cs638/838 - Spring 2017 (Shavlik©), Week 11
Keras – CONV Layer 1 (pg 2)
model.add(leakyReLUtoUse); # Had been model.add(Activation('relu')) model.add(ZeroPadding2D(padding = zeroPaddingConv1, data_format = "channels_last")) # Not needed. model.add(Dropout(conv1_dropoutProb)) - Seems one CANNOT do dropout on INPUT UNITS (but that’s ok)
4//17
cs638/838 - Spring 2017 (Shavlik©), Week 11

4. cs638/838 - Spring 2017 (Shavlik©), Week 11
Keras – POOL Layer 1
model.add(MaxPooling2D(pool_size = kernelSizePool1, strides = stridePool1, padding = 'valid')) model.add(Dropout(pool1_dropoutProb)) # I set prob to 0. Should use an IF? model.add(ZeroPadding2D(padding = zeroPaddingPool1))
4//17
cs638/838 - Spring 2017 (Shavlik©), Week 11

5. cs638/838 - Spring 2017 (Shavlik©), Week 11
Keras – CONV Layer 2
model.add(Conv2D(platesConv2, // No need for input_shape. kernel_size = kernelSizeConv2, strides = strideConv, padding = "valid", use_bias = True)) model.add(leakyReLUtoUse); model.add(ZeroPadding2D(padding = zeroPaddingConv2)) model.add(Dropout(conv2_dropoutProb))
4//17
cs638/838 - Spring 2017 (Shavlik©), Week 11

6. Keras – POOL Layer 2 (same as 1)
model.add(MaxPooling2D(pool_size = kernelSizePool2, strides = stridePool2, padding = 'valid')) model.add(Dropout(pool2_dropoutProb)) # I set prob to 0. Should use an IF. model.add(ZeroPadding2D(padding = zeroPaddingPool2)) # Later I added a FOR LOOP that allowed N CONV+POOL layers # (all used the parameters of Conv1 and Pool2).
4//17
cs638/838 - Spring 2017 (Shavlik©), Week 11

7. cs638/838 - Spring 2017 (Shavlik©), Week 11
Wrapping Up the Model
model.add(Flatten()) # Flattens the last MAX POOL layer so can fully # connect to the final HU layer.
model.add(Dense(units = numberOfFinalHUs)) # Fully connected ‘flat’ HUs.
model.add(leakyReLUtoUse)
model.add(Dropout(final_dropoutProb))
model.add(Dense(units = numberOfClasses)) # The OUTPUT units.
model.add(Activation("softmax")) # Other choices exist of course.
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

8. Training the Model (setup)
OLD model.compile(loss = 'categorical_crossentropy', optimizer = 'rmsprop', metrics = ['accuracy']) NEW optimizerToUse = Adam() # Had been 'rmsprop' (good for recurrent nets). model.compile(loss = 'categorical_crossentropy', optimizer = optimizerToUse, metrics = ['accuracy
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

9. Training the Model (epochs)
for i in range(epochsToRun): model.fit(X_train, y_onehot_train, epochs = 1, # We manage training ourselves. batch_size = batchSize, verbose = 0, # 0 for no logging to stdout, 1 for progress bar logging, # 2 for one log line per epoc # validation_data=(X_tune, y_onehot_tune), shuffle = True) # Permute examples each epoch! acc_train = accuracy(model, X_train, y_train) acc_tune = accuracy(model, X_tune, y_tune) # Only need this one every epoch. acc_test = accuracy(model, X_test, y_test)
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

10. Creating the Confusion Matrix
If new best TUNE set accuracy:
y_pred_test = model.predict_classes(X_test, verbose = 0)
confusionTestsetAtBestTuneset.fill(0)
# Initialization earlier via: np.zeros((numberOfClasses, numberOfClasses))
# Then whenever new best TUNE result, refill it (no need to keep model)
for y_pred, y in zip(y_pred_test, y_test):
confusionTestsetAtBestTuneset[ y, y_pred ] += 1
# EXAMPLE: zip((a, b, c), (x, y, z)) produces ( (a, x), (b, y), (c, z) ).
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

11. Keras (https://en.wikipedia.org/wiki/Keras)
Lots of possible network topologies and methods for Deep ANNs
Seems well designed, easy to use
Runs fast
Written (mainly) by ONE person!
I’m working on getting Kera on CS Dept computers
I have a Condor-ready version, but working with BMI Dept lab staff on Kera+Python
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

12. Results on New Dataset (from ImageNet, down-sampled to 32x32 images)
               |  airplanes  butterfly   flower    grand   starfish      watch     TRUE SUMs     airplanes  |        111           3                    5          4          |  Row Sum =  125     butterfly  |          9         77                    8         21          |  Row Sum =  125     flower     |         11          8                  3         11          |  Row Sum =  125     grand      |         13          0                  95          6          |  Row Sum =  119     starfish   |         34          1                   13         70          | Row Sum =  125     watch      |         26          3                  24         17         |  Row Sum =  PREDICTED |        204         92                148        129              TOTAL  =  SUMS
This dataset (in 10 folds, zipped) is in the same course directory as the Lab 3 data.
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

13. “I thought this was a starfish”
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

14. “I thought this was a watch”
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

15. “I thought this was a flower”
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

16. Some Python/Keras Questions I Have
Saving models?
Repeatable random-number sequences
Took 27 mins to do early Confusion Matrix (100 epochs, 6000 training examples, 416k weights) – how much would this cost in the cloud?
How to implement waitForEnter?
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

17. In-Class (Ungraded) Lab
Get Lab3 to work in Keras (might want to use my newer code; run main.py)
Add ability to learn an ENSEMBLE
Collect all the REAL-VALUED, TESTSET predictions in a 2D-array (ie, for each ensemble’s ‘best tune epoch”)
testExamples x outputCategories
When done with N models for the ensemble, collect accuracy of choosing the category with the most ‘points’ per training example
Feel free to do a different extension (suggestions?)
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11

18. Version 2 of my Lab3 Keras Code
main.py
you should run this Python script
parameters.py
create an object that holds all the params, collect images, create unique ‘marker string’ per experiment
dribbler.py
prints both to screen and a file
processCondorID.py
convert the single int that condor provides into param settings
utilsJWS.py
read and perturb images, other misc utility/supporting functions
JoeAndJudeCode.py
the Keras code (presented in this lecture) is isolated here
4/3/17
cs638/838 - Spring 2017 (Shavlik©), Week 11