VXLAN Nexus 9000 Module 6 – MP-BGP EVPN - Design @onecloudinc.com

VXLAN EVPN Design Options

VXLAN Fabric Design – Spine nodes as RR VXLAN Overlay MP-iBGP EVPN RR RR MP-iBGP Sessions Leaf VTEP VTEP VTEP VTEP VTEP VTEP VTEP Functions are on leaf layer Spine nodes are iBGP route reflector Spine nodes don’t need to be VTEP

VXLAN EVPN Fabric with MP-iBGP Design (Cont’d) Spine switches are not capable of running MP-BGP EVPN. Leaf switches are chosen to provide iBGP route-reflector functions to the other iBGP VTEP leaf nodes. All other leaf nodes peer with them through iBGP. Spine Leaf VTEP iBGP RR VXLAN Overlay

VXLAN EVPN Fabric with MP-iBGP Design (Cont’d) Spine switches don’t need to be able to run MP-BGP EVPN. They are purely IP transport devices. Dedicated MP-BGP EVPN route reflectors provide better scalability and control-plane performance. They can be connected to the fabric network in the same way as a leaf node. RR Spine RR Leaf iBGP iBGP iBGP iBGP iBGP VTEP Cisco Nexus 9300 VTEP Cisco Nexus 9300 VTEP Cisco Nexus 9300 VTEP Cisco Nexus 9300 VTEP Cisco Nexus 9300 All leaf VTEPs run iBGP sessions with the dedicated route reflectors.

VXLAN Fabric Design with MP-eBGP EVPN BGP on Spine needs to have the following in address-family l2vpn evpn: BGP next-hop unchanged retain route-target all AS 65000 Spine MP-eBGP Sessions AS 65001 AS 65002 AS 65003 AS 65004 AS 65005 AS 65006 VTEP VTEP VTEP VTEP VTEP VTEP Leaf VTEP Functions are on leaf layer Spine nodes are MP-eBGP Peers Spine nodes don’t need to be VTEP Need to manually configure Route-targets on each VTEP

VXLAN Fabric Design with MP-eBGP EVPN (Cont’d) BGP on Spine needs to have the following in address-family l2vpn evpn: BGP next-hop unchanged retain route-target all AS 65000 Spine MP-eBGP Sessions AS 65100 AS 65100 AS 65100 AS 65100 AS 65100 AS 65100 Leaf VTEP VTEP VTEP VTEP VTEP VTEP VTEP leafs are in the same BGP AS

EVPN VXLAN Fabric Inter Data Center Connectivity (Existing) Spine Leaf VTEP Spine RR Border Leaf VXLAN Overlay EVPN VRF/VRFs Space RR RR DC #1 EVPN iBGP DC #2 EVPN iBGP Border Leaf VTEP VTEP VTEP VTEP VTEP VTEP Leaf VLAN hand-off Flood-&-Learn OVT/VPLS One EVPN Administrative Domain Stretched Across Two Data Centers

EVPN VXLAN Fabric Inter Data Center Connectivity (Option A) Spine Leaf VTEP Spine RR Border Leaf VXLAN Overlay EVPN VRF/VRFs Space RR RR DC #1 EVPN iBGP DC #2 EVPN iBGP Border Leaf VTEP VTEP VTEP VTEP VTEP VTEP Leaf Inter-DC EVPN eBGP One EVPN Administrative Domain Stretched Across Two Data Centers

EVPN VXLAN Fabric Inter Data Center Connectivity (Option A’) Spine Leaf VTEP Spine RR Border Leaf VXLAN Overlay EVPN VRF/VRFs Space DC #1 EVPN iBGP DC #2 EVPN iBGP Border Leaf RR RR VTEP VTEP VTEP VTEP VTEP VTEP Leaf Inter-DC EVPN eBGP One EVPN Administrative Domain Stretched Across Two Data Centers

EVPN VXLAN Fabric Inter Data Center Connectivity (Option B) IP Routing Spine Spine DC #1 EVPN iBGP DC #2 EVPN iBGP VTEP Border Leaf Border Leaf RR RR VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Leaf VTEP VTEP Leaf Inter-DC EVPN eBGP One EVPN Administrative Domain Stretched Across Two Data Centers Red line is the data path, evpn ebgp session between border leafs Global Default VRF Or User Space VRFs Advantage of This Option: No changes on the existing Spine/Aggregation devices Take advantage of existing inter-DC links and routing Only need to add border leaf VTEPs

EVPN VXLAN Fabric Inter Data Center Connectivity (Option B) Spine Leaf VTEP Spine RR Border Leaf VXLAN Overlay EVPN VRF/VRFs Space RR RR DC #1 iBGP DC #2 iBGP Border Leaf VTEP VTEP VTEP VTEP VTEP VTEP Leaf EVPN Administrative Domain #1 EVPN Administrative Domain #2 VLAN Hand-off VTEP VTEP VTEP VTEP Inter-DC eBGP Inter-DC EVPN Administrative Domain

vPC VTEPs in MP-BGP EVPN Underlay IP Network BGP Peer BGP Peer BGP Router ID 1 BGP Peer BGP Router ID 2 Virtual PortChannel Layer 3 Link Layer 2 Link vPC VTEP-1 vPC VTEP-2 BGP Peer When vPC is enabled an ‘anycast’ VTEP address is programmed on both vPC peers Symmetrical forwarding behaviour on both peers provides Multicast topology prevents BUM traffic being sent to the same IP address across the L3 network (prevents duplication of flooded packets) vPC peer-gateway feature must be enabled on both peers VXLAN header is ‘not’ carried on the vPC Peer link (MCT link) vPC VTEP with Anycast VTEP Address

Scalability Limits

Nexus 2000 Series Fabric Extenders (FEX) Verified Scalability Limits Feature 9500 Series Verified Limit 9300 Series Verified Limit Fabric Extenders and Fabric Extender server interfaces Not applicable 16 and 768 VLANs per Fabric Extender 2000 (across all Fabric Extenders) VLANs per Fabric Extender server interface 75 Port channels 500

Interfaces Verified Scalability Limits Feature 9500 Series Verified Limit 9300 Series Verified Limit Generic routing encapsulation (GRE) tunnels 8 Port channel links 32 SVIs 490 250 vPCs 275 100 (280 with Fabric Extenders)

Layer 2 Switching Verified Scalability Limits Feature 9500 Series Verified Limit 9300 Series Verified Limit MST instances 64 MST virtual ports 85,000 48,000 RPVST virtual ports 22,000 12,000 VLANs 4000 3900 VLANs in RPVST mode 500

Multicast Routing Verified Scalability Limits Feature 9500 Series Verified Limit 9300 Series Verified Limit IPv4 multicast routes 32,000 8000 Outgoing interfaces (OIFs) 40 (see CSCum58876)

Layer 2 Switching Verified Scalability Limits Feature 9500 Series Verified Limit 9300 Series Verified Limit MST instances 64 MST virtual ports 85,000 48,000 RPVST virtual ports 22,000 12,000 VLANs 4000 3900 VLANs in RPVST mode 500

Thank You