1. BlackBerry Test Validation and Analysis using Deep Learning
Wasif Khan - Software Developer II
BlackBerry Ltd.

2. Outline
Introduction
Context
Challenges
Solution
Demo
Future Work

3. Introduction About BlackBerry About Me Autonomous Vehicles
Enterprise Software
About Me
Enterprise Software -> Quality Assurance -> Tools Development
Tools Developer
Performance Test Validation and Analysis Tool

4. Outline
Introduction
Context
Challenges
Solution
Demo
Future Work

5. Context Performance Test Execution
Run performance test & track various statistics (8 hours / week)
Analyze results (4 hours / week / employee) Instant analysis with Deep Neural Networks
Total time saved = (4 hours / week / employee) * (4 weeks / month) * (3 employees) = ~50 hours / month
Details of analysis
1000 statistics to analyze (CPU usage, throughput, messages sent/received, etc…)
500 data samples
Report Pass/Fail based on correlations on statistics in analysis

6. Outline
Introduction
Context
Challenges
Solution
Demo
Future Work

7. Challenges Main Challenge 1000 Statistics with only 500 data samples
Solution
Reduced statistics from 1000 to 200 by manually discarding useless data
Increase data samples to 5,000 by considering noisy data and simulated data
Attempted PCA to further reduce variables but failed (lack of data)

8. Outline
Introduction
Context
Challenges
Solution
Demo
Future Work

9. Solution 1 Input Layer - 200 Nodes (Input Variables)
1 Hidden Layer - 9 Nodes (ReLU Activation)
1 Output Layer - 2 Nodes (Softmax Activation [P/F])
7 Hidden Nodes % Accuracy 8 Hidden Nodes % Accuracy 9 Hidden Nodes % Accuracy 10 Hidden Node % Accuracy 11 Hidden Nodes % Accuracy
Learning Rate = Error = Cross Entropy Learning = Gradient Descent
Failed Attempts Layers
2 Hidden Layers
Convolutional Layer
Activation
Sigmoid Activation
Tanh Activation
Learning Rate
>= => No convergence
<= => Poor local minima

10. Outline
Introduction
Context
Challenges
Solution
Demo
Future Work

11. Demo

12. Outline
Introduction
Context
Challenges
Solution
Demo
Future Work

13. Future Work Finish Current Network (1 week):
Report which variables are causing failures
Upcoming Networks (4 months):
Neural network to analyze input to test
Neural network to analyze logs (text) on test
Overall Goal Use output from {input, statistics, logs} networks to determine health of BlackBerry software and areas of vulnerability

14. THE END