Bottlenecks Stress Test Demo 20170504

Contents Some Considerations on Stress Testing Highlights of Discussion in Testperf Stress Tests for Danube Test Cases in Discussion Preliminary Works Determine baseline for throughput Life-cycle events for ping Stress Test Demo Testing Contents Video Demo

Some Consideration of Stress Testing Stress testing is a kind of testing determines the robustness of software by testing beyond the limits of normal operation Stress testing does not break the system for fun It will provides level of confidence of the system to users.  It also allows observation of how system react to the failures. Additional purpose behind this madness is to make sure that the system fails and recovers gracefully - a quality known as recoverability.  Load Testing Stress Testing https://wiki.opnfv.org/display/bottlenecks/Sress+Testing+over+OPNFV+Platform

Stress Testing for Danube – Highlights Summary Stress testing principles and test cases discussion in Testperf https://etherpad.opnfv.org/p/DanubeStressTest Test Requirements The stress test should from a user perspective be ... Easy to understand (what the test does and how the system is being stressed) Easy to run (i.e. "out-of-the-box" ... having deployed the OPNFV platform it should be possible to run the test with minimal effort) Where possible use existing proven methods, test-cases and tools Should be possible to work with all OPNFV release scenarios For Danube the stress test result is not part of our release criteria however for future releases a stress test threshold (metric TBD) should be part of the release criteria It should be possible to increase and decrease the load ("stress") and monitor the effect on the system with a simple performance metric The application running on SUT must be highly optimized to avoid being the bottleneck

Test Cases in Discussion - Highlights Summary VM1 VM2 spawn destroy throughput cpu limit ping TC1 TC2 TC3 TC5 TC4 Data-Plane Traffic Determine baseline test cases Life-Cycle Events Perform VM pairs/stacks testing Availability Robustness confidence  Release criteria? 1 Hour Max? General Heavy Load Easy-to-Run Easy-to-Understand

Test Cases in Discussion - Highlights Summary Categories Test Case Description Data-plane Traffic for a virtual or bare metal POD TC1 –Determine baseline for throughput Initiate one v/b POD and generate traffic Increase the package size Throughput, latency and PKT loss up to x% TC2 - Determine baseline for CPU limit Decrease the package size Measure CPU usage up to x% Life-cycle Events for VM pairs/stacks TC3 – Perform life-cycle events for ping Spawn VM pairs, do pings and destroy VM pairs Increase the number of simultaneous live VMs latency and PKT loss up to x% Testing time, count of failures, OOM killer? TC4 – Perform life-cycle events for throughput Spawn VM pairs, generate traffic and destroy VM pairs Serially or paralleled increase the package size Max load and PKT loss vs load Throughput, latency and PKT loss up to x% for either pair TC5 – Perform life-cycle events for CPU limit Serially or paralleled Decrease the package size Measure the CPU usage up to x% Former update: https://wiki.opnfv.org/download/attachments/2926539/Some%20Updates%20on%20Bottlenecks%20Yardstick%20for%20Stress%20Testing%20v2.pptx?version=1&modificationDate=1484121945000&api=v2 Discussion address: https://etherpad.opnfv.org/p/DanubeStressTest

Preliminary Works in Danube TC1 – Determine baseline for throughput Load Manager Load Category “Data-Plane Traffic” Preliminary work from Bottlenecks & Yardstick

Preliminary Works in Danube Type 1 TC3 – Perform life-cycle events for ping Initial stress test Load Manager Run in parallel Time ends or Fail Start End Iterate Increase load Criteria Check Testing Flow create destroy ping run in parallel Bottlenecks Yardstick t1 t4 t5 t0 t2 t3 T6? 5 10 20 50 100 200? Load Manager If VMs successly built If VMs successly built If VMs successly built If VMs successly built Visualization If VMs successly built Quotas Can not determine the exact time for expected number of VMs

Stress Testing Demo for Ping Testing Contents Executing Stress Test and Provide comparison results for different installers (Installer A and Installer B) Up to 100 stacks for Installer A (Completes the test) Up to 40 stacks for Installer B (System fails to complete the test) Testing Steps Enter the Bottlenecks Repo: cd /home/opnfv/bottlenecks Prepare virtual environment: . pre_virt_env.sh Executing ping test case: bottlenecks testcase run posca_factor_ping Clean the virtual environment: . rm_virt_env.sh Video Demo Link will be provided in OPNFV official YouTube channel

Testing Results for the Demo Testing for Installer A Up to 100 stacks in configuration file for Installer B 1 stack SSH error when number of stack raised to 50 When stack number up to 100, most of the errors are heat response time out 100 stacks are established successfully in the end Testing for Installer B Up to 40 stacks in configuration file for Installer B When stack number up to 30, the system fails to create all the stacks 21 stacks are either created failure or keeping in creation To verify the system performance, we choose to do clean-up and run the test again When stack number up to 20, same situation happens as in the last test The system performance degrades Different to the test for Installer A, we do the verification step because the system clearly malfunctions. Which not shown in the demo is that after 3 rounds of the stress test, the system fail even to create 5 stacks

Thank You Jan 2016