1. Managing SPP Resources: System Resource Manager (SRM) Fred Kuhns fredk@arl.wustl.edu Applied Research Laboratory Washington University in St. Louis

2. 2 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Common Definitions and Types For all commands the message context ID is used to identify the context within which a command is to be executed. –The special value of 0 ( cid = 0 ) indicates a privileged operation performed by the substrate. –Otherwise the context ID is an identifier indicating a user specific context in which the command is to be interpreted and executing. In most cases this is the Global Fastpath ID. See presentation titled Types.ppt

3. 3 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 NATD Interface [egress_map, ingress_map] get_sched_map(ipAddr_t link, hwaddr_t board) –Message context ID = 0. sched_map_t { qmID_t qmID; // 2-bit QM identifier schID_t schID; // 3-bit qm relative scheduler identifier qid_t qid; // 16-bit queue identifier statID_t sindx; // Stats Index vlanID_t vid; // VLAN identifier } Return: egress_map: sched_map_t ingress_map: sched_map_t Arguments: link: Interface IP address (aka external IP) board: Board/Device MAC Address

4. 4 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Reserving Fastpath Resources See Types.ppt for type definition and mapping to substrate API, message context ID = 0. Reserve resources for 0..N fastpaths (on NPEs) and 0..1 GPEs retCode_t make_reservation(plabID, rsvRecord_t) Update an existing reservation, note because all operations need to be idempotent the update argument includes all required resources. retCode_t update_reservation(plabID_t, rsvRecord_t) Cancel any reservation overlapping date_t retCode_t cancel_reservation(plabID_t, date_t) Return any reservation (belonging to slice) that overlaps with date_t rsvRecord_t ? get_reservation(plabID_t, date_t) Return a list of all reservations associated with calling slice rsvRecord_t * get_reservations(plabID_t)

5. 5 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Allocating Fast-path Instance fastpath_t alloc_fastpath(plabID_t, copt, bwspec, rcnts, mem) –Message context ID = 0 plabID_t : GPE/PlanetLab slice identifier uint16_t copt : code option type. { IPv4=1,I3=2 } struct bwspec {bw_t bw-firm, bw-soft;} struct rcnts {cnt_t fltrs, queues, buffers, stats;} struct mem {cnt_t sram, dram;} // Requested size of SRAM and DRAM in Bytes struct fastpath_t { fpid_t fpid; // fastpath id assigned by srm, used as context ID in future messages vlan_t vlan; // VLAN id to associate with fp’s logical interface ipAddr_t fpIP; // IP address of logical fp’s interface on GPE ipAddr_t npeIP; // IP address for fp’s data interface on NPE ipAddr_t fpNM; // Netmask of fp’s logical interface ipAddr_t scdIP; // IP address for fp’s assigned SCD ipPort_t scdPort; // UDP Port number for fp’s assigned SCD hwaddr_t npeMAC;} // NPE MAC address –bw-firm: Expresses the required aggregate processing resources in terms of BW, units are Kbps. –bw-soft: This is just a hint as to the expected maximum processing resources required. Resources are not actually allocated to the fastpath, rather the system uses this number for load balancing best-effort fastpaths across the available NPEs.. Actions required to allocate code option instance and resources: –Select NPE Load balance across available NPEs Must have sufficient resources (SRAM, TCAM space, queues, etc) –Either allocates requested resources or returns error Keeps memory map of SRAM so can perform allocation, though the absolute starting address is not required. If compaction is necessary then must communicate with SCD directly. –Allocate VLAN and configure switch. –Send commands to SCD on NPE set_fastpath(fpid, copt, VLAN, TParams, Mem[]) set_sched_params(sid(gpe) GPE_MIID, GPE_BW_MAX*, GPE_BW_MIN) set_fpmi_bw(fpid, sid(gpe), GPE_MIID, GPE_BW_MIN) set_encap_gpe(fpid, gpeIP(fpid), npeIP(fpid)) Deprecated

6. 6 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Releasing a fast-path instance retCode_t free_fastpath() –Message context ID = Global fastpath ID ( fpid ) –Send command to SCD on NPE set_sched_params(sid(gpe), GPE_MIID, GPE_BW_MAX*, GPE_BW_MIN) retCode_t set_fpmi_bw(fpid, sid(gpe), GPE_MIID, 0) set_encap_gpe(fpid, 0, 0) rem_fastpath(fpid) Deprecated

7. 7 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Instantiate Fastpath and/or Slowpath reservations All resource allocation methods operate on the currently active reservation (i.e. must be used during the interval of time when a reservation is valid). –If free is not called then all resources are automatically freed when reservation expires. allocate any GPE/Slice resources reserved in the current interval retCode_t alloc_plspec(plabID_t) instantiate the “current” fastpath reservation with name fpName fpInfo_t alloc_fpspec(plabID_t, name_t fpName=“”) –if only one fastpath then the fpName parameter can be omitted. free any GPE specific resources currently allocated to slice. retCode_t free_plspec(plabID_t) Destroys fastpath instance, releasing meta-interfaces and NPE resources. retCode_t free_fpspec() –Message context ID set to the Global fastpath ID

8. 8 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Managing Interfaces Reserve Aggregate Interface BW for slice (traffic to/from GPE). retcode resrv_slice_ifbw(plabID, bw, ifn) Release slice interface BW reservation for GPE retcode reles_slice_ifbw(plabID, bw, ifn) Reserve aggregate Interface BW for indicated fastpath retcode resrv_fpath_ifbw(bw, ifn) –message context ID = fpid Release Aggregate Interface BW allocation for indicated fastpath retcode reles_fpath_ifbw(bw, ifn) –message context ID = fpid Deprecated

9. 9 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 System Interfaces, Addresses and Ports Get list of available interfaces if_list get_interfaces(plabID) –Message context ID = 0. –if_list = {ifAttr_t,...} ifAttr_t { ifn_t ifn; // Interface number ifType_t type; // {Multi-Access = 0, Point-to-Point = 1} ipAddr_t ipaddr; // Interface/Link IP address ipAddr_t linkBW; // Raw (Gross) interface (link) BW bw_t availBW; // Available (Net) interface (link) BW } –if type = point-to-point then may call get_ifpeer() Map interface IP address to interface number. ifn_t get_ifn(plabID_t, ipAddr_t) Get updated interface attributes ifAttr_t get_ifattrs(plabID_t, ifn_t) Get the IP address of point-to-point link’s peer node (SPP peer). ipAddr_t get_ifpeer(plabID_t, ifn_t)

10. 10 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Managing Externally Visible Address for GPE Traffic Message context ID set to 0, plabID used to identify Slice making request Managing bandwidth: –SRM: verifies slice meta-interfaces and queues do not exceed interface allocations –SRM: Notifies SCD of per Scheduler slice limits Allocate a local endpoint for slice (linecard directs matching traffic to the corresponding GPE) epInfo_t alloc_endpoint(plabID_t, epInfo_t) –Message context ID = 0. Release allocated endpoint (linecard filters removed and address added bac kto available pool of addresses tuples). retCode_t free_endpoint(plabID_t, epoint_t) –Message context ID = 0

11. 11 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Internal Interfaces for NAT map{bw_tbw; # Kbps ipAddr_tipaddr; # local interface IP address ipPort_tport; # if TCP/UDP and 0 then allocate ipProto_tproto; # TCP/UDP,ICMP dw1_tsmac}; # low order 8 bits of source mac {fid, port} alloc_epmap(map) –Message context ID = 0. –Allocate endpoint and add LC TCAM filter to send pkts to GPE. retCode_t free_epmap(fid) –Message context ID = 0. –Allocate endpoint and add LC TCAM filter to send pkts to GPE. Deprecated

12. 12 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Allocating Meta-Interfaces (tunnel endpoint) Common Types ep {uint32_t bw, uint32_t ipaddr; uint16_t port; uint8_t proto} dw2_t mi : Meta-interface number for an endpoint. Interface numbers start at 0. [mi, ep] alloc_tunnel(bw, ipaddr, tunnel_type, params) Not implemented [mi, ep] alloc_udp_tunnel(bw, ipaddr, port) Message context Id = fpid uint8_t tunnel_type : must be one of { UDP = 0} uint32_t bw : (Minimum guaranteed) BW assigned to endpoint, Kbps uint32_t ipaddr : External IP address of endpoint/tunnel. There is a 1-to-1 relationship between links and ip addresses. uint16_t port : External UDP port number –Actions required to allocate code option instance and resources: verify requested address is available, assign port if necessary (port == 0) Allocate new meta-interface ID allocate and assign filter id for ingress and egress TCAM Lookup existing or allocate qid for ingress and/or egress –Send command to SCD on Line Card Ingress set_sched_params(sid(fabric,NPE), ifn(fabric), BW*, BW min ) set_queue_sched(qid, sid(fabric,NPE)) set_queue_params(qid, threshold, bw) write_fltr(dbid_ingress, fid, key_ingress, mask, result_ingress) write_fltr(dbid_egress, fid, key_egress, mask, result_egress) –Send commands to Line Card Egress set_sched_params(sid(link), ifn(link), BW*, BW min ) set_queue_sched(qid, sid(link)) set_queue_params(qid, threshold, bw) –Send command to SCD on NPE set_sched_params(sid(link, miid), ifn(fabric), BW*, BW min ) set_fpmi_bw(fpid, sid(link, miid), miid, bw*) ep get_endpoint(mi) Message context Id = fpid Deprecated

13. 13 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Releasing Meta-Interfaces Common Types ep {uint32_t bw, uint32_t ipaddr; uint16_t port; uint8_t proto} uint_t mi : Meta-interface number for an endpoint. Interface numbers start at 0. retCode_t free_tunnel(ipaddr, tunnel_type, params) retCode_t free_udp_tunnel(ipaddr, port) –Message context Id = fpid. uint32_t ipaddr : External IP address of endpoint/tunnel uint16_t port : External UDP port number –Actions required to allocate code option instance and resources: –Send command to SCD on Line Card Ingress set_sched_params(sid(fabric,NPE), ifn(fabric), BW*, BW min ) set_queue_params(qid, threshold, bw) rem_fltr_byfid(dbid_ingress, fid) rem_fltr.fid(dbid_egress) –Send commands to Line Card Egress set_sched_params(sid(link), ifn(link), BW*, BW min ) set_queue_params(qid, threshold, bw) –Send command to SCD on NPE set_sched_params(sid(link, miid), ifn(fabric), BW*, BW min ) set_fpmi_bw(fpid, sid(link, miid), miid, bw*) Deprecated

14. 14 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Allocating Meta-Interfaces (tunnel endpoint) --- Simplified interface, see steps from deprecated slides --- Allocate local Address for Slice’s fastpath (MI) and forward to NPE, will also reserve sufficient aggregate BW on corresponding internal and external interfaces. [miID_t, epInfo_t] alloc_udp_tunnel(bw_t, ipAddr_t, ipPort_t) –Message context ID set to the global fastpath ID –It is an error if the sum of requested BW exceeds the reserved fastpath BW. –Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance. –if port = 0 then one is allocated from the free pool –Returns the meta-interface number representing the new endpoint Release BW and Address associated with corresponding meta-interface retCode_t free_udp_tunnel(ipAddr_t, ipPort_t) Map meta-interface ID to corresponding externally visible address tuple epInfo_t get_endpoint(miID_t) –mi must be a valid meta-interface ID returned from a call to alloc_udp_tunnel

15. 15 Washington WASHINGTON UNIVERSITY IN ST LOUIS Fred Kuhns - 10/5/2015 Classes and Tables xsid Entry: { plabID, xsid, gpe, npe, MI } Mappings: plabID to Entry xsid to Enry GPE : {GPE object reference, BW, EPs, ??} NPE : {NPE object reference, vlan bw, sram, fltrs, qs, stats} MI : {EP object reference, ID} Slice Table plabID endpoint (port) maps resvMapavailMapusedMapsxsidMap System tables Interfaces ifn:{type,ipaddr,linkBW,availBW}... NPE Table id:{addr,BW/Port,copts,fltrs,sram,Qs}... VLAN maps range:{start,end} free {...} GPE servMapresvMap endpoint references controlIP BWmaps Slices Per Slice Tables vlan meta-ifaces mi:endpoint... gpe board idBW plab sliceID NPE (allocated) sram {start,size}#flts #Qsboard IDBW#Stats xsidplabID