1. AT&T M320 NGRE 64bit MIS RR TEST PLAN – 13.2
RBU AT&T CST Team
July 2013

2. Revision History Version Number Owner Date Release Targeted
Version Notes
(Changes Made) V1
Sharon
7/7/2013
13.2R2
Initial Draft Version for 64bit MIS RR V2
7/12/2013
Add Revision History Slide V3
7/18/2013
Update test plan per review feedback from ATT
Hardware updated,both DUTs use 64bit RE
scaling number update
test cases update V4
7/29/2013
Update last two scaling from 15 paths to 20 paths (slide 13) V5
8/14/2013
Update slide 31 to add a few more test cases V6
9/6/2013
Update test case for SNMP polling per doc received from AT&T (add slide 32)

3. Stability Testing for NGRE M320 RR at scaling setup
OBJECTIVE
Measure CPU and memory usage of M320 RR with NGRE 64bit RPD at different scaling setup
64bit Junos, rpd accessible memory increased from 2G to 3G
64bit rpd, rpd can access the full 16G of available memory
Stability Testing for NGRE M320 RR at scaling setup
BGP Convergence Performance Measurement
Find the scaling capacity of NGRE M320 RR with 64bit rpd

4. Overview Test Topology Network Setup Hardware Details
Network Configuration
Scaling Number
Test Methodology
Test Cases
Caveat

5. M320 MIS RR Test Topology …... 64bit M320 MIS-RR Test Topology RR1
15 Virtual Junos as RRs P1
M320
RE-A-1800x2
M10i
MIS-PE
ge-0/0/0
RR1
ge-3/0/0 P4
ge-2/0/0
…...
M10i
MIS-PE P3
ge-0/0/0
M320
RE-A -1800x2
RR15
64bit M320 MIS-RR
Test Topology

6. M320 MIS-RR BGP Sessions M320 MIS-RR BGP Sessions RR1
Simulated PEs
iBGP Peers
15 Virtual Junos as RRs ……
iBGP
M320
RE-A-1800x2
iBGP
iBGP
RR1
M10i
MIS-PE
eBGP
iBGP
eBGP
Internet Feed
Routes
iBGP
MPLS/LDP Core
P Router
Simulated RRs eBGP Peers
Additional
Routes
iBGP
iBGP
eBGP
iBGP
iBGP
eBGP
M10i
MIS-PE
iBGP
RR15
iBGP
M320
RE-A -1800x2
M320 MIS-RR BGP Sessions

7. M320 MIS-RR Network Setup Network Setup
DUT
M320 with RE-A-1800x2 (64bit)
Network Setup
Internet feed with IPv4 routes to P router
IXIA insert addition IPv4/v6 routes to P router
P router establish eBGP peers with RR1 to RR15
RR1 to RR15 iBGP sessions to DUTs reflecting IPv4/IPv6 routes (15 paths per each route)
IXIA ports simulate PEs to establish iBGP sessions (350)
IXIA ports simulate RRs to establish eBGP sessions (30)
Two M10i as MIS PEs
Each M320(RR) is in unique cluster, iBGP Peers between two DUT RRs

8. M320 MIS-RR Network Setup -- Hardware
Core Network
MX240 and MX960 – Simulate core network
RR Service Network
M320 (64bit)
RE: RE-A-1800x2-16G (Dual RE)
FPCs: M320 E2-FPC Type 1 (2)
PICs: PB-4GE-TYPE1-SFP-IQ2 (2)
Software: 13.2
FPCs: M320 E3-FPC Type 1 (2)
PICs: PB-4GE-TYPE1-SFP-IQ2
PB-4GE-TYPE1-SFP-IQ2E
Software:13.2
Virtual Junos and IXIA RT for RRs
VJ Software: 12.1
MIS Network
IXIA RT: Simulate PE routers to advertise IPv4/IPv6 routes to RR for the scaling
Two M10i as Real MIS PEs 8

9. Customer Model Network Design -- Configuration
OSPF
non-zero area
Uplink Metric 15000
SPF timer 50ms
Prefix-export-limit 10
Rib-group OSPF-RIBGP
MPLS: IPv6 tunneling
LDP: Not Enabled
PIM: Not Enabled
BFD: Not Enabled
BGP
ASN 7018
Local address (lo0.0)
keep none
Path-selection external-router-id
Advertise-inactive
mtu-discovery
Precision-timers
Log-updown
Tcp-mss 4096
Uplink to Core
No QoS
MTU: 9192
Filter: None on Interface
Firewall Filter
Applied to lo0.0 (PROTECTHOST)
Management Functions
Services – SSH, telnet, ftp,
Enable path MTU discovery
Enable SNMP
Chassis
GRES enabled Redundancy
NSR not enabled
Router ID
Lo0.0 IPv4 address

10. Customer Model Network Design -- BGP Configuration
BGP Global Properties (DUT)
Local-AS 7018
keep none
Path-selection external-router-id
Advertise-inactive
MTU-discovery
Precision-timers
Log-updown
Tcp-mss 4096
EBGP Session between P router and RR (VJ)
Family inet and inet6
Next-hop changed for the routes to RRs
IBGP Peer Groups (DUT)
IBGP sessions to PE Clients
Divided into 22 groups
IBGP sessions to RR Peers
IBGP sessions to BGPMon
Family inet and inet6 for each group
Export policy to export aggregate routes
EBGP Peer Groups (DUT)
DUT EBGP group to RRs in other regions
Family inet and inet6
Next-hop unchanged

11. Customer Model Network Design -- 64bit RPDBGP Configuration
Config knob to enable 64bit RPD
In order to use 64bit RPD, need to turn on 64bit rpd knob by configuring the following:
system {
processes {
routing {
auto-64-bit } } Or
force-64-bit }
Note:
1. Configure either knob to enable 64bit rpd, auto-64-bit only take effect when RE has 16G memory
2. When commit the change, 64bit RPD takes effect, bgp serssions will bounce.

12. Scaling Number – Baseline Scaling
Baseline Scaling Number (1A)
OSPF routes
BGP
Number of Paths: 10
Number of IPv4 routes: 702,000
Number of IPv6 routes: 22,000
Total IPv4 Paths: 7,020,000
Total IPv6 Paths: 220,000
Total BGP groups: 25
Total BGP peers: 400 (355 PEs, 15 RRs(IBGP), 30 RRs (EBGP))
IPv4 Static Routes: 1000
IPv6 Static Routes: 1000
Multicast Routes: 100
Multicast Paths: 1000
The setup will first be configured with this baseline scaling number. RPD memory usage and RE kernel memory usage will be measured.

13. Scaling number Table (2A) with 64BIT RPD
No. of Paths
Add.
ipv4 routes
Add. ipv6 routes
Total ipv4 BGP routes
Total ipv6 BGP routes
Total ipv4 paths
Total ipv6 paths
BGP peer groups
64bit RPD Converg. time (average)
64bit RE Memory Utilization
64bit RPD Memory Usage
64bit RPD Memory Utilization
(min.) % MB 10
none
702,000
22,000
7,020,000
220,000 25
10,000
802,000
8,020,000
32,000
320,000
36,000
738,000
7,380,000
72,000
20,000
774,000
42,000
7,740,000
420,000
144,000
40,000
846,000
62,000
8,460,000
620,000
216,000
80,000
918,000
102,000
9,180,000
1,020,000
275,000
160,000
977,000
182,000
9,770,000
1,820,000
350,000
170,000
1,052,000
192,000
10,520,000
1,920,000 20
14,040,000
440,000
18,360,000
2,040,000

14. Scaling Capacity – Melt Down Scaling Test
Start point scaling for melt-down scaling testing - 3A
OSPF routes
BGP
Number of Paths: 15
Number of IPv4 routes: 702,000
Number of IPv6 routes: 22,000
Total IPv4 Paths: 7,020,000
Total IPv6 Paths: 220,000
Total BGP groups: 25
Total BGP peers: 400 (355 PEs, 15 RRs(IBGP), 30 RRs (EBGP))
IPv4 Static Routes: 1000
IPv6 Static Routes: 1000
Multicast Routes: 100
Multicast Paths: 1000
Melt-down Scaling Testing
Setup DUT with start point scaling number(3A), increase the number of IPv4 routes, find out the maximum number of IPv4 routes which will break the DUT RR
Setup DUT with start point scaling number(3A), increase the total number of paths, find out the maximum number of IPv4 paths which will break the DUT RR

15. Scaling Number Applied to DUTs
Apply baseline scaling to both DUT with NGRE
Increase the scaling numbers (shown on the table) to measure the resource usage and bgp performance
For operation trigger test and stability testing, use the scaling number 8 on the table(2A) for the testing.
For melt-down scaling testing, start from the baseline scaling (3A) and then increase the number of the routes or the number of paths to find the melt-down point of scaling number

16. Test Methodology
Use customer model network and Focus on RR-DUT (M320s)
Background noise apply to DUT to simulate real network environment
Resource Measurements
RE and PFE resource monitored and measured during the test execution cycle
Test Execution
Only one event/trigger/failure on the setup ( no simultaneous failures/triggers/events)
One iteration of event/trigger/failure executed during all test case execution
Selected event/triggers/failures will be executed ‘n’ iterations . Value of ‘n’ depends on availability of resources and schedule. For 13.2, n is set to 5
DUT resource measurements (CPU usage, memory consumption etc) will be taken
BGP convergence performance will be measured
Melt-down scaling will be tested by increasing the routes or paths

17. Test Summary Steady State Checklist and Resource Measurements
Soak Test (12 hours) and Halt Test (24 to 36 hours)
Background Noise to Simulate Real Network Environment
Operation Triggers Test ( Stability and Robustness Test)
Hardware Operations
Protocol Operations
Process restart
Network Failure operations
BGP Convergence Performance
Memory Leak Test (Valgrind on RPD )
Security Test: MU Dynamics
Scaling limit of DUT

18. Steady State Check List
Check all the configured interfaces, protocols up on each router
All IFLs are up
All OSPF neighbors are at full state on all routers
Expected iBGP peers are established (DUT)
Expect eBGP peers are established (DUT)
Check all routing tables (inet.0, inet.6 routing tables)
Check DUT scaling
Check DUT has the required scaling number
Monitor KRT queu status, syslog message and core dump
Check any anomalies on syslog (syslog message, chassisd log, ksyncd log) and check if any core dumped.
Check KRT queue status
Record RE and PFE resource
RPD process memory usage
Kernel memory usage
RE memory usage
RE CPU usage
Jtree memory usage

19. Background NOISE
From RT port, routes flapping to DUT RR periodically (2% routes from RR every 10 minutes, internet peer flapping every 60 minutes)
From RT port, portion of bgp sessions to DUT RR flapping periodically (10 iBGP Peers randomly flapped at interval of 15 seconds every 10 minutes)
From RT port, send traffic to DUT RR lo0 to stress control plane
SNMP polling
CLI queries
Configure provision for bgp peer add/delete

20. Test Cases Test Case 1 Upgrade and Steady State Check
Test Case 2 Hardware Operation Triggers
Test Case 3 Protocol Operation Triggers
Test Case 4 Process Operation Triggers
Test Case 5 Network Failure Operation
Test Case 6 BGP Convergence Performance
Test Case 7 Customer Network Operation
Test Case 8 HALT Test
Test Case 9 Memory Leak Test
Test Case 10 Security Test
Test Case 11 Operation Trigger Iteration Test
Test Case 12 Melt-down Scaling Number

21. Test Cases1 – Upgrade and Steady State Check
Test Case 1.1 Upgrade from 11.4S3.2 to 13.2R2
Setup Baseline Scaling for DUTs with 11.4S3.2
Load 11.4S3.2 image on both DUTs
Apply baseline scaling 1A to both DUTs
Verify all BGP sessions are up and convergent as expected.
Upgrade DUTs to 13.2R2 image and enable 64bit rpd
Upgrade both REs on each DUT to 13.2R2 image
Configure 64bit rpd knob and commit
Make sure DUT 64bit rpd running

22. Test Cases1 – Upgrade and Steady State Check
Test Case 1.2 Steady State Check and Soak Test on 13.2
Steady State Test Measurements
All BGP peers up and convergent
RE and PFE resources will be measured on DUT RR when the network is at steady state
Duration for monitoring the resources: 60 Minutes
Soak Test Measurements
Soak test will be conducted for a fixed period of 12 hours
Background routes flapping and BGP peer flapping
Resources on RE, PFE, will be monitored and measured on DUT RR during the soak test
Steady state will be checked before and after soak test

23. Test Cases 1 – Upgrade and Steady State Check
Test Case 1.3 Apply test scalings and measure resource on 64bit M320
Setup test scaling
Increase the scaling number to each number from the scaling number table(2A)
Apply the scaling to both DUTs
Steady State Test Measurements
All BGP peers are up with new scaling
RE and PFE resources will be measured on DUT RR at steady state after upgrade is complete
Duration for monitoring the resources: 60 Minutes
Measure BGP Convergence and RE Resources
Measure BGP convergence time for each scaling
Measure rpd memory usage and RE resource usage for each scaling
Measure BGP convergence time for 32bit rpd when scaling number is supported for 32bit rpd for comparision

24. Test Cases 2 – Hardware Operation Triggers
Run One Trigger at a time
RE/PFE resource measurement will be taken
DUT recovers and no cores
Steady state check passes
Test Case 2.1 FPC offline/online
Test Case 2.2 PIC Offline/online
Test Case 2.3 FPC Restart
Test Case 2.4 RE Switchover
Test Case 2.5 SIB Failure/Redundancy
Test Case 2.6 Deactivate/activate interface

25. Test Cases 3 – Protocol Operation Triggers
Run One Trigger at a time
RE/PFE resource measurement will be taken
DUT recovers and no cores
Steady state check passes
Test Case 3.1 Deactivate/Activate OSPF
Test Case 3.2 Deactivate/Activate BGP
Test Case 3.3 Disable/Enable OSPF
Test Case 3.4 Disable/Enable BGP
Test Case 3.5 Clear OSPF neighbor
Test Case 3.6 Clear BGP neighbor
Test Case Clear all BGP neighbor
Test Case Clear BGP neighbor to RR (Virtual Junos RR)
Test Case Clear BGP neighbor to PE
Test Case Clear BGP neighbor to RR in other cluster

26. Test Cases 4 – Process Operation Triggers
Run One Trigger at a time
RE/PFE resource measurement will be taken
DUT will recover after trigger and no cores
Steady State Check passes
Test Case 4.1 rpd process restart
Test Case 4.2 chassisd process restart
Test Case 4.3 dcd process restart

27. Test Cases 5 – Network Failure Operation
Network Failure Testing
Network failure triggers will be tested without doing any CLI configuration/operational command on routers.
One trigger at a time on the network
Network Failure Test Cases
Test Case 5.1 Node Failure
a. One M320 RR failed, make sure no impact on DUT (64bit RR)
b. One P router failed, make sure DUT (64bit RR) will be recovered
c. One VRR restarted, make sure no impact on DUTs
Test case 5.2 Link Failure
One of uplink on DUT (64bit RR) failed, make sure DUT will be able to be convergent
Test Case 5.3 Control Plane/RE Failure
One RE on DUT (64bit RR) failed, make sure RE switchover and all BGP sessions established and convergent

28. Test Cases 5 – Network Failure Operation
Network Failure Test Cases (Cont’d)
Test Case 5.4 Data/Forwarding Plane Failure
Test case FPC Failed
One FPC failed, make sure DUT will update the routes and convergent
Test case PIC Failed
One PIC failed, make sure DUT will update the routes and convergent
Test case 5.5 Routes flapping
Routes flapping from Internet feed and RT port happened on network, make sure DUT routes updated accordingly
Test case 5.6 Upgrade RR peer
Upgrade the other RR peered with DUT, check the impact on DUT RR

29. Test Cases 6 – BGP Convergence Performance
Measure BGP Convergence time with different triggers
Test Case 6.1 BGP Convergence Time after Restart RPD
Restart routing
Measure BGP convergence time
Monitor RE/PFE resource usage
Test Case 6.2 BGP Convergence Time after RE Switchover
RE Switchover

30. Test Cases 6 – BGP Convergence Performance
Test Case 6.3 BGP Convergence Time after Restart FPC
Restart two FPCs
Measure BGP convergence time
Monitor RE/PFE resource usage
Test Case 6.4 BGP Convergence Time after clear BGP neighbor
Clear BGP neighbor
Test Case 6.5 BGP Convergence Time after clear BGP session
Clear BGP session to one of RR

31. Test Cases 7 – Customer Network Operation
Test case 7.1 Customer Network Operation
Run customer network operations SNMP queries, CLI and Provisioning commands on DUT for the duration of 2 hours.
Check for any memory leaks and core-dumps.
RE/PFE resource measurements will be taken during the test execution
Check if any BGP session flap on both RR DUT and RR client
Test case 7.2 BGP policy activate/deactivate provision
Apply BGP import/export policy to filter bgp routes
Check routes updated accordingly and no core seen.
Test case 7.3 Add BGP groups
Split existing bgp peers into more bgp groups to increase the number of bgp groups
Commit the config change and make sure no core seen and bgp peers are up.
Test case 7.4 Merge multiple BGP groups to one
Merge two or three bgp groups to one group, commit the change.
make sure bgp peer re-established and no rpd core seen,

32. Test Cases 7 – Customer Network Operation
Test case 7.5 SNMP Fault Management Trap and MIB
Run SNMP trap and SNMP MIB polling test per document received from AT&T (trimmed from document M320 MIS SNMP MIBS TRAPS SYSLOGS CLI.doc for items applied to MIS-RR).
Test Case 7.6 SNMP Performance MIBs test
Test Case 7.7 Other Pollable MIBs
Test Case 7.8 Cli troubleshooting Commands
SNMP MIBs/CLI commands tested is in this document
M320 MIS-RR SNMP MIBS TRAPS SYSLOGS CLI.doc

33. Test Cases 8 – HALT Test HALT TEST
Periodic network events on DUT for 24 or 36 hours
Monitor for cores and error messages during the test
Verify the steady state check list at the end of the test
Measure RE and PFE resource periodically
Note: HALT Test will run the following triggers randomly
Restart FPC FPC offline/online PIC offline/online
RE switchover deactivate/activate interface
Restart routing Restart chassis Restart dcd
clear ospf neighbor clear bgp neighbor deactivate/activate bgp
deactivate/activate ospf disable/enable ospf disable/enable bgp

34. Test Cases 9 – Memory Leak Test
Valgrind on RPD
Run valgrind on RPD on DUT
Periodic routes flapping and BGP peer flapping
Run test for 8 hours
Check any memory leak on RPD
Note: Need to reduce scaling for valgrind testing
Will try baseline first, if not working, reduce scaling down

35. Test Cases 10 – Security Test
MU Dynamics Test for OSPF, BGP and TCP syn attack
Scaling need to be reduced for this test.
Test with baseline scaling first, if not working, scale down.

36. Test Cases 11 – Operation Trigger Iteration Test
Iterate the triggers 5 times, make sure DUT will be recovered after the trigger.
Test Case 11.1 FPC offline/online
Offline/online FPC 5 times, wait for 5 minutes between the triggers, Make sure DUT will be recovered. No cores happen.
Test Case 11.2 FPC Restart
Restart FPC 5 times, wait for 5 minutes between the triggers, Make sure DUT will be recovered. No cores happen.
Test Case 11.3 RE Switchover
RE Switchover 5 times, wait for 30 minutes between the triggers, make sure DUT operates and no cores.
Test Case 11.4 PIC Offline/Online
PIC offline/online 5 times, wait for 5 minutes between the triggers, make sure DUT recovered and no cores.

37. Test Case 12 - Melt Down Scaling Number
Found maximum number of IPv4 routes
Test Setup and Procedure
Setup 64bit M320 with scaling number 3A
Increase IPv4 routes from IXIA emulated PEs (single path routes)
Wait for DUT to steady state
Monitor memory usage
Repeat Step 2-5 until DUT memory usage reach to 85%

38. Test Case 12 – Melt Down Scaling Number
Found maximum number of IPv4 paths
Test Setup and Procedure
Setup network with scaling number 3A
Increase IPv4 routes reflected from RRs (Increase IPv4 paths)
Wait for DUT to steady state
Monitor memory usage
Repeat Step 2-5 until DUT memory usage to 85%

39. BGP Convergence Comparison between 32bit and 64bit
No. of Paths
Add.
ipv4 routes
Add. ipv6 routes
Total ipv4 BGP routes
Total ipv6 BGP routes
Total ipv4 paths
Total ipv6 paths
BGP peer groups
32bit/64bit RPD Converg. time (average)
32bit/64bit RE Memory Utilization
32bit/64bit RPD Memory Usage
32bit/64bit RPD Memory Utilization
(min.) % MB 10
none
702,000
22,000
7,020,000
220,000 25
10,000
802,000
8,020,000
32,000
320,000
36,000
738,000
7,380,000
72,000
20,000
774,000
42,000
7,740,000
420,000
144,000
40,000
846,000
62,000
8,460,000
620,000
216,000
80,000
918,000
102,000
9,180,000
1,020,000
275,000
160,000
977,000
182,000
9,770,000
1,820,000
350,000
170,000
1,052,000
192,000
10,520,000
1,920,000 20
14,040,000
440,000
18,360,000
2,040,000

40. 40