1. Technion - Israel institute of technology department of Electrical Engineering High speed digital systems laboratory Super Computer System Midterm presentation Developers : Anton Vainer and Atila Fuad Mentor : Mr Assad Haick Dec 21, 2011

2. Super Computer System Agenda  Project Overview –Reminder  Final system setup and configuration  Memory map  Startup flow and Driver operation  Gantt Chart

3. Super Computer System Super Computer System Overview –Reminder For Validation of 10G Ethernet products we need a computer that can generate high speed traffic. Standard Computer don’t have the bandwidth needed. Super computers are very large, expensive and loosely coupled systems. In the project we will build a small tightly coupled super computer system with four computer chassis each with 1-4 CPUs. This super computer will be able to deliver the bandwidth needed to stress test 10G products.

4. LADi Haifa INTEL CONFIDENTIAL 10G Link Partners (Option – 1) 4 Agents run above the same TCP stack

5. Super Computer System Super Computer System Final system setup and configuration Slave Master PEX8617 AIC RDK PEX8648 RDK

6. LADi Haifa INTEL CONFIDENTIAL PLX 8617 PCIe Gen II Switch 160 GT/s 16 Lane - 4 Ports Full line rate on all ports Non Blocking SW which can support both modes: Transparent Non-Transparent (NT) Super Computer System Super Computer System Final system setup and configuration

7. LADi Haifa INTEL CONFIDENTIAL PLX 8648 PCIe Gen II Switch 480 GT/s 48 Lane - 12 Ports Full line rate on all ports Non Blocking SW which can support both modes: Transparent Non-Transparent (NT) Super Computer System Super Computer System Final system setup and configuration

8. LADi Haifa INTEL CONFIDENTIAL 8 Super Computer System Super Computer System Final system setup and

9. Super Computer System Super Computer System Final system setup and configuration Each one of the 3 slave computers has a 8617 16-lane PCIe switch configured as a non-transparent bridge. Master computer is connected directly to 8648 48-lane PCIe switch with NT-Bridge. Each slave NT-Bridge is connected to a similar 8617 card on the main board that is configured as clock isolation only (no NT-bridge) because of a bug in spread spectrum clock in Intel Chipsets. Each slave NT bridge is configured via EEPROM to allocate it’s own BAR0 and an additional 1MB for BAR2.

10. Super Computer System Super Computer System Memory map NT-Virtual NT-Link CPU 2-4 CPU 1 BAR0 128k BAR2 1M CPU # BAR2 Address translate allocated 16M For TX data 1M BAR0 Of the NIC port 0 Switch CPU 1 BAR2 Address translate Allocated 16M

11. Super Computer System Startup flow and Driver operation * Slave machine: - Upon boot, the NT-Bridge (configured via EEPROM) will allocate 128k as BAR0 for itself and 1M BAR2 for the NIC. - Upon driver load, the driver will allocate 16M of memory for TX data. * Master machine: - Upon boot, the NIC will allocate 1M BAR0 for itself. NT-bridge will allocate 128k BAR0 for itself and 16M BAR2 for TX data. - Upon driver load, the NT bridges will be configures via internal registers to mirror each slave machine BAR2 to master’s NIC BAR0 (to get NIC register access), and the allocated 16M to master’s NT-bridge BAR2 (to TX every slave machine's data).

12. Super Computer System Gantt Chart TaskDuration Start Date Finish Date Studying the tools301.1130.11 Design the setup and driver abilities151.1215.12 Characterization presentation511.1116.11 Setup bring-up1515.1130.11 Mid. Project Presentation 516.1220.12 Enabling of PCIe switch2020.1210.1 Implementing the driver4020.1231.1 Testing and validation281.228.2 Final project presentation520.325.3