1. 1 Using VPLS for VM mobility Carles.Kishimoto @ cern.ch David.Gutierrez @ cern.ch HEPIX Fall 2015

2. Agenda CERN data center The objective The idea The solution Conclusions 2

3. Agenda CERN data center The objective The idea The solution Conclusions 3

4. Switzerland 1000 racks Hungary 300 racks Wigner Research Centre for Physics Since January 2013 2x 100GbE links 4 CERN Data Center (I)

5. Network equipment Brocade routers HP Procurve / HP H3C / Brocade switches 5 CERN Data Center (II)

6. 6 CERN network (I) x100GbE links Distribution CoreAccess x10GbE ToR x10GbE

7. CERN network (II) Routed network (OSPF) No vlans, no spanning-tree ECMP and LACP IPv4 / IPv6 Dual stack since 2013 x10GbE switch uplinks (LCG) 100GbE p2p router interconnects (LCG) MPLS deployed in the backbone 7

8. Agenda CERN data center The objective The idea The solution Conclusions 8 8

9. The objective (I) Migrate virtual machines transparently to the new hardware 9 Virtual Machines

10. The objective (II) Migrate running virtual machines transparently to the new hardware Conditions : Keep the same ip address on all virtual machines Live migration Restrictions: CERN is a routed network We do not extend VLANs across racks 10

11. Agenda CERN data center The objective The idea The solution Conclusions 11

12. The idea MPLS / IP network Switch B Router A Router B Switch A 12 Row A Row B VM

13. The idea MPLS / IP network Switch B Router A Router B Switch A Multimode fiber 13 Row A Row B VM

14. The idea MPLS / IP network Switch B Router A Router B Switch A VM 14 Row A Row B

15. The idea MPLS / IP network Switch B Router A Router B Switch A Connect the switches at router level 15 2 1 1 2 VM

16. The idea MPLS / IP network Switch B Router A Router B Switch A Connect the switches logically using VPLS 16 2 1 1 2 VM

17. What is VPLS ? Virtual Private LAN Service RFC4761 / RFC4762 Signaling based on BGP or LDP From Wikipedia: “Virtual Private LAN Service (VPLS) is a way to provide Ethernet-based multipoint to multipoint communication over IP or MPLS networks” You need an MPLS enabled backbone 17

18. What is VPLS ? MPLS / IP network Switch B Router A Router B Switch A VPLS emulates an Ethernet switch 18 2 1 1 2 VM

19. Agenda CERN data center The objective The idea The solution Conclusions 19

20. The solution vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection ! interface ethernet 5/1 no flow-control load-interval 30 enable ! interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area 128.142.0.0 ip ospf cost 1 ip ospf passive ip address 10.10.10.1/24 ! Layer 1 Layer 2 Layer 3 MPLS 20 Switch A Switch B

21. The solution vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection ! interface ethernet 5/1 no flow-control load-interval 30 link-fault-signaling ! interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area 128.142.0.0 ip ospf cost 1 ip ospf passive ip address 10.10.10.1/24 ! Layer 1 Layer 2 Layer 3 MPLS 21 vpls 1 1 auto-discovery vlan 51 untagged ethe 5/1 router-interface ve 51 Switch A Switch B

22. The solution vlan 51 name S513-C-IP790 untagged ethe 5/1 router-interface ve 51 loop-detection ! interface ethernet 5/1 no flow-control load-interval 30 link-fault-signaling ! interface ve 51 port-name VPLS#S513-C-IP790 ip ospf area 128.142.0.0 ip ospf cost 1 ip ospf passive ip address 10.10.10.1/24 ! Layer 1 Layer 2 Layer 3 MPLS 22 vpls 1 1 auto-discovery vlan 51 untagged ethe 5/1 router-interface ve 51 Switch A Switch B (config)# vlan 51 (config-vlan-51)# no router-interface ve 51 (config-vlan-51)# no untagged eth 5/1 (config-vlan-51)# router mpls (config-mpls)# vpls 1 1 (config-mpls-vpls-1)# router-interface ve 51

23. The loop cable (I) MPLS / IP network Switch B Router A Router B Switch A Loop cable installed in distribution routers 23 1 1 2 2 VM

24. The loop cable (II) MPLS / IP network Switch B Router A Router B Switch A Routing still provided by Router A 24 VM Internet

25. The loop cable (III) MPLS / IP network Switch B Router A Router B Switch A Increase VRRP priority in router B 25 Internet VM

26. The loop cable (IV) Implemented with 10GigabitEthernet ports Advantage It allows us to create the circuit transparently We can deploy multiple links in parallel if needed It can be used for multiple migrations 26

27. Operations Workflow 1. Service manager will create a ticket 2. Network team will enable the circuit 3. Migration of virtual machines 4. The circuit will be removed Open questions: Duration of the circuit ? Number of circuits simultaneously ? 27

28. Agenda CERN data center The objective The idea The solution Conclusions 28

29. Conclusions It allows us to connect the broadcast domain of different IP services transparently It supports multiple circuits at the same time The solution is based on standard protocols (RFC) It works… although not tested in production yet Our tools are ready to support this configuration It scales as we could connect two IP services from different data centers (Geneva vs Budapest) 29

30. Thank you 30