Lambda Station: On-demand flow based routing for data intensive GRID applications over multitopology networks GridNets 2006, San Jose,CA, October 1 – 2, 2006 Fermi National Accelerator Laboratory Don Petravick (PI), Phil Demar, Matt Crawford California Institute of Technology. Harvey Newman (co-PI) by Andrey Bobyshev, Fermilab.
Outline of the talk: goals and major directions of the project software architecture, API, SOAP/XML, some details results of using Lambda Station service, SC05 Demo problems and challenges, plans
Basic terms Lambda Station – a host with special software to control traffic path across LAN and WAN on-demand of applications PBR – policy based routing PBR Client – a system or cluster and applications running on it sourcing traffic flows that can be subject for policy based routing Flow - a stream of packets with some attributes in common such as endpoint IP addresses (or range of addresses), protocols, protocol's ports if applicable and differentiated services code point (DSCP).
The main goal of Lambda Station project is to design, develop and deploy a network path selection services to interface production storage and computing facilities with advanced research networks. – selective forwarding on a per flow basis – alternate network paths for high impact data movement – access control in site edge routers for those selected flows – on-demand from applications (authentication & authorization) – current implementation based on policy-based routing & including the support of DSCP marking The goal of the project... deals with the last-mile problem in local area networks
Lambda Station TestBed two HEP sites, Fermilab and Caltech 10G & 1G End-To-End paths via DOE UltraScienceNet and NFS UltraLight components of test and production computing/storage/network infrastructures 10Gb and 1Gb servers
Lambda Station Building Blocks LSInterface LS-Management & Reporting Interface mySQL:requests, history,security LSDIScovery Service LSRESource Scheduler LSController local definitions online updates SOAP/Clarens Vendor specific modules NETWORK CONFIGURATOR CISCO Force10 WAN Storage & application space Management SOAP/Clarens LSInterface Remote lambdastation Data Exchange Control & Management Service-based Architecture: CLARENS is framework for service based architecture, mutual authentication of requests LSController – synchronizes work of all services, has control functions LSI - unified interface for intercommunication between LS and applications, and LS-to-LS LSDIScovery service – detects new lambdastations, and PBR clients at remote and local sites LSRESource Scheduler – estimates bandwidth allocation, monitor real-time usage of resources NETWORK CONFIGURATOR – dynamic reconfiguring of LAN and WAN SOAP
Lambda Station Software Multiple API-calls or primitives (~ 20) divided into several groups: informative (whoiam,sayHello,getClientInfo, getKnownLambdastations....) service or operational (openSvcTicket,cancelTicket,getTicket,getFlowSpec...) internal (getNextDSCP, assignDSCP,updateMyLambdaDefs) single interface for client-to-LS and LS-to-LS communication Scenarios to configure LANs: dynamically configurable local networks with dynamically or statically assigned DSCP without DSCP tagging statically configured networks with predefined matching criterion for flows LS Software v1.0 – a fully functional prototype but no interoperability across different platforms
Java implementation of LS Software Service Oriented Architecture build by using JClarens framework as a grid-aware tool messages are defined by XML schemas core authentication is based on gLite library to support standard Grid proxies and KCA-issued certificates secure document/literal wrapped SOAP messages, Web Services Interoperability Profile (WS-I Basic Profile Version 1.1) Multi threaded LS implementation ( LS Controller, NetConfig and LS/PBR Config as JClarens background processors)
Flows and DSCP tagging Any combination of flow's attributes can be used by Lambda Station software to identify flows on per ticket's basis. Typical steps of alternate paths: generate request for service (application, application's proxy, admin) LS negotiates service and parameters with remote site configure local and wide area network (not yet available) marking traffic (if specified). Current LS software is capable to complete all these steps within 3 – 5 mins. That is why it is desirable to know flow selection parameters before transferring is started: endpoint IP addresses DSCP
DSCP Tagging Complexity of using DSCP tagging: preservation of DSCP is not guaranteed in WAN DSCP tagging needs to be synchronize between sites for dynamically configurable networks ( asymmetry is bad for high- performance transferring ) LS software does support two different modes of DSCP tagging : fixed DSCP values to identify site's traffic. DSCP value is assigned dynamically on per ticket base.
Interaction between applications and LS Multiple scenarios how applications and Lambda Station may communicate: modified applications with LS awareness capability to place and manipulate tickets w/ and w/o DSCP traffic marking unmodified applications, a proxy controls paths for their traffic Operational modes of openSvcTicket request (subject to authentication, authorization and quotas): new ticket - creating a new ticket join - provides ID and required parameters(DSCP) of already opened ticket extend ticket – allows to extend already created ticket.
Lambda Station Aware Applications lsiperf, lsTraceRoute: Full support of Lambda Station awareness Places a ticket for an alternative path, negotiates and synchronizes with remote server's application the parameters (DSCP tagging with IPTables ) of request watches ticket's status SRM/dCache: limited Lambda Station awareness places ticket, join and extend recognizes absence of Lambda Station tries to use it only for specific destinations
1.Data transfer started: – 10GE host; 5 tcp streams – Network path is via ESnet – OC12 bottleneck… – Path MTU is 1500B – LambdaStation service ticket is opened 2.LambdaStation changes network path to USN 3.Host path MTUD check detects a larger path MTU 4.LambdaStation service ticket expires: – Network path changed back to ESnet LSiperf End-to-End Test 1 2 3 4
Project accomplishments Software version 1 (a fully functional prototype supporting whole cycle of LS functionality) positive results of testing between Fermilab and Caltech positive results during SC05 via general Internet and SCinet (10G path) lsiperf, lsTraceroute – wrappers around well known applications to add Lambda Station awareness SRM/dCache integration – testing, added in production 1.7.0 release
Problems and challenges Traffic Asymmetry is bad for high performance applications Network (Lambda Station) awareness is too complex Definition of PBR Clients is a complex issue, auto definition is not yet available References: http://www.lambdastation.org/ Plans release jLS, interoperability across platforms SRM/dCache with production quality LS support add real-time monitoring of resources add WAN control plane module
Miscellaneous Slides Simulation of multiple Lss screen shoots of ticket's queue SC05 Demo
Single web services ServiceTicket request interface for client2LS and LS2LS message flows Multi threaded LS implementation: – LS Controller, NetConfig and LS/PBR Config as JClarens background processors Apache XmlBeans API for XML data binding JClarens utilized to provide User Session management, authorization, authentication and transparent DB connections pooling Interoperability tests with perl, python clients LS Java API
Netconfig Module dynamically modifies the configurations of local network devices. a vendor dependent components. Cisco routers with IOS version supporting sequencing type of named ACLs. Tasks to configure PBR in Cisco devices: interface on which PBR is applied needs to be configured with ip policy route-map statement route map needs to be configured as ordered list of match/action statements match criteria need to be associated with ACLs
openSvcTicket request new ticket - creating a new ticket join - provides ID and required parameters(DSCP) of already opened ticket extend ticket – allows to extend already created ticket. Accepts many input parameters, most of them could also be determined automatically: Dst site's identifiers Src/Dst PBRClient identifier or/and list of Src/Dst IP in (CIDR) IP protocols, protocol's ports localPath, remotePath identifiers BWout,BWin boardTime, startTime, endTime Returns ID of locally assigned ticket Operational modes (subject authentication, authorization and quoting):
Directions of the project. building a wide-are testbed infrastructure designing, developing Lambda Station software, Lambda Station (network) aware applications researching effects of flow based switching on applications
Application's behavior in flow based switching environment T uning of end systems for maximum rates is not a subject of the Lambda Station project, however, we need to see an increase of data movement performance when selectively switch flows DSCP tagging with IPtables switching between two paths with different MTUs
Motivation of the project unprecedented demands for data movement in physics experiments such as CDF, D0, BarBar and coming LHC experiments massive, globally distributed datasets growing to the 100 petabytes by 2010 collaborative data analysis by global communities of thousands of scientists available data communication technology will not be able to satisfy these demands simply by plain increasing bandwidth in LANs and WANs due to technology limitations and high deployment and operation costs. Advance Research optical networks – greater capacity, no production quality of service, are not universally available for all pairs of communicating endpoints
LS multitopology network model PBR-client B Multiple Network Toplogies Red Green Blue Admission Group of network devices RT1 RT2 RT3 RT1 RT2 RT3 PBR-client A H1 H2 H3 NG-A NG-C NG-B NG-ADM H1 H2 H3 PBR-client PBR-clients or regular clients at the remote sites ClientA rules for Red& GREEN topologies: RT1 RT2 RT3 BLUE is Production Path RED-ClientA-IN GREEN-ClientB-IN RED-ClientB-IN GREEN-ClientB-IN RED-B-IN GREEN-OUT RED-OUT Client A, RED & GREEN rules for NGC Cisco IOS, dynamic configuring of PBR, extended sequencing ACLs + access policy ACLs