CC-NIE DANCES Project: Implementing Multi-Site, SDN-Enabled Applications Bryan Learn <blearn@psc.edu> Pittsburgh Supercomputing Center Internet2 Technology Exchange 26 September 2016

DANCES Project Introduction DANCES is an NSF-funded (#1341005), CC-NIE collaborative “Network Integration and Applied Innovation” project DANCES has integrated SDN/OpenFlow into selected supercomputing cyberinfrastructure applications to support network bandwidth reservation and QoS Motivated by the need to support large bulk file transfer flows and efficiently share bandwidth of end site 10G infrastructure Project was started in Jan2014. The two year project will finish in 2016 under a No Cost Extension Concentrating on CI applications that initiate file transfer. These include: User workflow via scheduling/resource management software (e.g., Torque/Moab, Torque/SLURM) Wide area distributed filesystem (SLASH2) Project was motivated by the need to support large bulk wide area file transfers and the desire to provide predictability and stability throughout the transfer. When the project was proposed, end site components typically had 10Gb/s interfaces. Internet2 had recently deployed 100Gb/s uncongested SDN-capable infrastructure in the wide area. Multiple 10G attached servers at end sites could cause local congestion while the WAN path remained uncongested. Large file transfers can take days to complete. Depending on how smaller competing file transfers are initiated, they may either disrupt the larger flow, or the larger flow may block or seriously impede performance of smaller flows. Through the DANCES project we are trying to bring more predictability to file transfer performance by providing dedicated and protected bandwidth for priority bulk data flows.

DANCES Development Team Pittsburgh Supercomputing Center (PSC) National Institute for Computational Sciences (NICS) Pennsylvania State University (Penn State) eXtreme Science and Engineering Discovery Environment (XSEDE) Internet2 Corsa Technologies, Inc. Our core team

DANCES-enhanced Applications Implementation uses SDN/OpenFlow 1.3 metering to implement QoS The enhanced cyberinfrastructure applications are: Supercomputing resource management and scheduling. File transfer via GridFTP scp can then be scheduled and included as part of a workflow Integrated into a distributed wide area file system SLASH2 The metering capability and marking of packets, along with port queueing and weighting functionality, implements QoS SLASH2 was developed by PSC and is an open source wide area network (WAN)-friendly distributed file system offering advanced features such as data multi-residency, system-managed data transfer, and inline checksum verification. We are working to incorporate the network bandwidth scheduling capability with SLASH2 file replication.

SDN/OF Infrastructure Components and Interfaces CONGA bandwidth manager software Receives bandwidth request from application Verifies user authorization with user database Tracks available/allocated bandwidth between sites Initiates OpenFlow path set up via the RYU OpenFlow controller RYU OpenFlow controller software Writes flowmods (routing rules) to switches Reads port statistics to verify operation OpenFlow 1.3-compatible switch hardware Expand as you see fit

The next few slides step through the control and data flow of the system. User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

Workflow Example User submits job (e.g., via qsub) that requires file transfer with reserved bandwidth QoS. Bandwidth request is initiated in Torque Prologue CONGA checks user authorization in XSEDE User Database If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth This slide and the next are text for the previous diagrams so may or may not be useful to include. User submits job (via qsub, for example) that requires file transfer with reserved bandwidth QoS (can also include compute and storage requests) to the Resource Manager / Scheduler. Torque Prologue script initiates RM/S check with CONGA for user auth and available bandwidth between src and dst. CONGA checks user authorization listed in XSEDE User Database for bandwidth scheduling (We worked with the XSEDE allocations/database team and they created a small database with entries for the DANCES project team members. The test db is accessible via REST API) and determines [(based on tracking state and/or polling?)] if bandwidth is available. If not, [best effort and inform user?] If bandwidth is available, CONGA will initiate path and bandwidth provisioning with the RYU OF controller. RYU OF controller provisions local and remote flow and bandwidth

Workflow Example Torque/<scheduler> schedules job when resources are available. System has been tested with Moab local scheduling at NICS and PBS at PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished Torque/<scheduler> schedules job when resources are available. System has been used with Moab@NICS and PBS@PSC; PSC in production will likely use SLURM Execute file transfer Torque Epilogue script or timeout will initiate tear down of provisioned path when transfer is finished

Implementation CONGA RYU OpenFlow controller Custom software created as the “Northbound API” for DANCES project Uses REST API RYU OpenFlow controller Production deployment will require extension to XSEDE allocation system to accommodate a dedicated network bandwidth resource to be managed along with traditional compute and storage resources Accounting by duration of dedicated bandwidth request The implementation of DANCES required significant software development and integration effort. CONGA: Created a “northbound API” for CONGA to enable the applications (Torque and SLASH2) to communicate with the RYU OpenFlow controller framework. With the concept of QoS request scheduling and cross domain operation, CONGA also needed to track bandwidth allocation. REST API chosen to ease extensibility RYU Provided as an OpenFlow controller “framework” with example modules Customize for specific applications Integration with existing XSEDE operational model Using a test database of DANCES development team members in XSEDE user database format

Implementation and Deployment Original plan was full interoperation with Internet2’s AL2S for end-to-end SDN Internet2’s upcoming transition to MPLS removes SDN from WAN production infrastructure to a 10Gb research overlay The DANCES components are still applicable to campuses with congested 10Gb connectivity and DTNs close to the edge Each site would run an instance of CONGA or some other type of resource manager/scheduler and RYU to locally manage devices and congestion Created software module to interface with OESS to provision VLANs on demand. B/W provisioning was not supported by OF1.0 on AL2S, but we tested QoS enforced at end point switches. Will be working with I2 to define interoperation as I2 transitions to MPLS (Someone from I2 may add comments) With the campus to XSEDE SP concept, resource contention is more likely at the campus, not within the WAN path or SP so DANCES functionality would improve throughput without requiring support at both ends or completely end-to-end

Observations Cross-domain control of resources is challenging SDN/OF has been slower to gain WAN deployment support than originally expected (or has gone the opposite direction) Control of multi-vendor SDN environments requires additional custom coding By Bullet2 I’m referring to I2 stepping away from SDN/OF to MPLS

Future Work Additional testing of OpenFlow 1.3 flow metering with various queue sizes and weights Measure network bandwidth utilization achieved by using bandwidth scheduling for large flows along with “best effort” traffic Expand test deployment Prepare and package for production deployment Project is drawing to a close, but in the time remaining, we may...

Questions? https://www.dances-sdn.org dances-dev@dances-sdn.org