1 Using VPLS for VM mobility cern.ch cern.ch HEPIX Fall 2015

Agenda CERN data center The objective The idea The solution Conclusions 2

Agenda CERN data center The objective The idea The solution Conclusions 3

Switzerland 1000 racks Hungary 300 racks Wigner Research Centre for Physics Since January x 100GbE links 4 CERN Data Center (I)

Network equipment Brocade routers HP Procurve / HP H3C / Brocade switches 5 CERN Data Center (II)

6 CERN network (I) x100GbE links Distribution CoreAccess x10GbE ToR x10GbE

CERN network (II) Routed network (OSPF) No vlans, no spanning-tree ECMP and LACP IPv4 / IPv6 Dual stack since 2013 x10GbE switch uplinks (LCG) 100GbE p2p router interconnects (LCG) MPLS deployed in the backbone 7

Agenda CERN data center The objective The idea The solution Conclusions 8 8

The objective (I) Migrate virtual machines transparently to the new hardware 9 Virtual Machines

The objective (II) Migrate running virtual machines transparently to the new hardware Conditions : Keep the same ip address on all virtual machines Live migration Restrictions: CERN is a routed network We do not extend VLANs across racks 10

Agenda CERN data center The objective The idea The solution Conclusions 11

The idea MPLS / IP network Switch B Router A Router B Switch A 12 Row A Row B VM

The idea MPLS / IP network Switch B Router A Router B Switch A Multimode fiber 13 Row A Row B VM

The idea MPLS / IP network Switch B Router A Router B Switch A VM 14 Row A Row B

The idea MPLS / IP network Switch B Router A Router B Switch A Connect the switches at router level VM

The idea MPLS / IP network Switch B Router A Router B Switch A Connect the switches logically using VPLS VM

What is VPLS ? Virtual Private LAN Service RFC4761 / RFC4762 Signaling based on BGP or LDP From Wikipedia: “Virtual Private LAN Service (VPLS) is a way to provide Ethernet-based multipoint to multipoint communication over IP or MPLS networks” You need an MPLS enabled backbone 17

What is VPLS ? MPLS / IP network Switch B Router A Router B Switch A VPLS emulates an Ethernet switch VM

Agenda CERN data center The objective The idea The solution Conclusions 19

The solution vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection ! interface ethernet 5/1 no flow-control load-interval 30 enable ! interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area ip ospf cost 1 ip ospf passive ip address /24 ! Layer 1 Layer 2 Layer 3 MPLS 20 Switch A Switch B

The solution vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection ! interface ethernet 5/1 no flow-control load-interval 30 link-fault-signaling ! interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area ip ospf cost 1 ip ospf passive ip address /24 ! Layer 1 Layer 2 Layer 3 MPLS 21 vpls 1 1 auto-discovery vlan 51 untagged ethe 5/1 router-interface ve 51 Switch A Switch B

The solution vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection ! interface ethernet 5/1 no flow-control load-interval 30 link-fault-signaling ! interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area ip ospf cost 1 ip ospf passive ip address /24 ! Layer 1 Layer 2 Layer 3 MPLS 22 vpls 1 1 auto-discovery vlan 51 untagged ethe 5/1 router-interface ve 51 Switch A Switch B (config)# vlan 51 (config-vlan-51)# no router-interface ve 51 (config-vlan-51)# no untagged eth 5/1 (config-vlan-51)# router mpls (config-mpls)# vpls 1 1 (config-mpls-vpls-1)# router-interface ve 51

The loop cable (I) MPLS / IP network Switch B Router A Router B Switch A Loop cable installed in distribution routers VM

The loop cable (II) MPLS / IP network Switch B Router A Router B Switch A Routing still provided by Router A 24 VM Internet

The loop cable (III) MPLS / IP network Switch B Router A Router B Switch A Increase VRRP priority in router B 25 Internet VM

The loop cable (IV) Implemented with 10GigabitEthernet ports Advantage It allows us to create the circuit transparently We can deploy multiple links in parallel if needed It can be used for multiple migrations 26

Operations Workflow 1. Service manager will create a ticket 2. Network team will enable the circuit 3. Migration of virtual machines 4. The circuit will be removed Open questions: Duration of the circuit ? Number of circuits simultaneously ? 27

Agenda CERN data center The objective The idea The solution Conclusions 28

Conclusions It allows us to connect the broadcast domain of different IP services transparently It supports multiple circuits at the same time The solution is based on standard protocols (RFC) It works… although not tested in production yet Our tools are ready to support this configuration It scales as we could connect two IP services from different data centers (Geneva vs Budapest) 29

Thank you 30