perfSONAR Architecture: Design, Usage, Extension and Next Steps Presented by Prof. Martin Swany University of Delaware / Internet2 05 August, th APAN Conference Queenstown, New Zealand

What is perfSONAR A collaboration Production network operators focused on designing and building tools that they will deploy and use on their networks to provide monitoring and diagnostic capabilities to themselves and their user communities. An architecture & a set of protocols Web Services Architecture Protocols based on the Open Grid Forum Network Measurement Working Group Schemata Several interoperable software implementations Java, Perl, Python… A Deployed Measurement infrastructure

perfSONAR Architecture Interoperable network measurement middleware (SOA): Modular Web services-based Decentralized Locally controlled Integrates: Network measurement tools and data archives Data manipulation Information Services Discovery Topology Authentication and authorization Based on: Open Grid Forum Network Measurement Working Group schema Currently attempting to formalize specification of perfSONAR protocols in a new OGF WG (NMC) Network topology description being defined in the Network Markup Language WG

Decouple 3 phases of a Measurement Infrastructure

perfSONAR Services Measurement Point (MP) Service Enables the initiation of performance tests Measurement Archive (MA) Service Stores and publishes performance monitoring results Transformation Service Transform the data (aggregation, concatenation, correlation, translation, etc) Resource protector Arbitrate the consumption of limited resources Other services delegate a limited portion of the authorization decision here These services are specifically concerned with the job of network performance measurement and analysis

6 (perfSONAR) Information Services Lookup Service Allows the client to discover the existing services and other LS services. Dynamic: services registration themselves to the LS and mention their capabilities, they can also leave or be removed if a service goes down. Topology Service Make the network topology information available to the framework. Find the closest MP, provide topology information for visualisation tools Authentication Service Based on Existing efforts: Internet2 MAT, GN2-JRA5 Authentication & Authorization functionality for the framework Users can have several roles, the authorization is done based on the user role. Trust relationship between networks These services are the infrastructure concerned with discovering federating the available network services

Where is link utilization for - IPs d,e,f? Example perfSonar client interaction Client Network ANetwork B LS ALS B MA A MA B ab cd e f Where is link utilization for – IPs a,b,c? a,b,c : Network A, MA A Get link utilization d,e,f Here you go Get Link utilization a,b,c Here you go Useful graph gLS Where can I get more about network Doman B/IP d,e,f and Domain A/IP a,b,c? LS A, LS B d,e,f : Network B, MA B

8 perfSONAR works E2E when All Networks Participate FNAL (AS3152) [US] ESnet (AS293) [US] GEANT (AS20965) [Europe] DFN (AS680) [Germany] DESY (AS1754) [Germany] measurement archive m1 m4 m3 measurement archive m1 m4 m3 measurement archive m1 m4 m3 m1 m4 m3 m1 m4 m3 measurement archive performance GUI user Analysis tool Many collaborations are inherently multi-domain, so for an end-to-end monitoring tool to work everyone must participate in the monitoring infrastructure

perfSONAR Architecture Overview perfSONAR schema and protocol Understanding the structure of the schema is informative when discussing the architecture and extensibility mechanisms Information Services The Lookup Service and Topology Service utilize the base model and provide system “glue” that allows for service discovery and contextualization of both data and services Extensibility and ongoing work

perfSONAR Schema Key Goals: Extensibility, Normalization, Readability Break representation of performance measurements down into basic elements Data and Metadata Measurement Data A set of of measurement events that have some value or values at a particular time Measurement Metadata The details about the set of measurement data

Schema Normalization Can simplify the database representation for many types of measurement data While optimizations are possible, many measurement types can be viewed as one value measured over time Assists Combination/Concatenation of metrics Creating derived metrics Normalization helps with inferring relationships between types of metrics

Schema Basic Elements - Metadata Subject (Noun) The measured/tested entity EventType (Verb) What type of measurement, value, or event occurred Characteristic, tool output, or generic event Parameters (Adjectives and Adverbs) How, or under what conditions, did this event occur?

Schema Basic Elements - Data Some sort of value - Datum Existence of an event might point to the case where there no additional value As in “Link up/down” or threshold events Time Is extensible since various representations are appropriate in different cases E.g. UNIX timestamp vs NTP time

A Message Message MetadataData

An Object Store Store MetadataData

A Data is Linked to a Metadata Metadata someId Data someId

A Metadata may be linked to another Metadata someId Metadata someOtherId someId

Schema Namespaces All measurements have some sort of Data and Time All measurements can be described by the Metadata identifying who, what and how The specific structures of the Data and Metadata elements depend on the measurement Approach: Consistently use Data and Metadata elements and vary the namespaces of the specific elements

Schema Namespaces - 2 We encode the measurement/event type in the namespace And as a standalone element Some components of the system can pass Data and Metadata elements through without understanding their specific structure Allows and implementation to decide whether it supports a particular type of data or not Allows validation based on extended (namespace-specific) schemata

Schema Namespaces and Extensibility One key to extensibility is the use of hierarchy with delegation Similar to OIDs in the IETF management world The NM-WG has a hierarchy of network characteristics Good starting point However, not all tools are cleanly mapped onto the Characteristic space Often a matter of some debate

Schema Namespaces and Extensibility - 2 Organization-rooted tools namespace addresses this Some top-level tools ping, traceroute Easy to add new tools in organization- specific namespaces Performance Event Repository Add a schema and get a URI Add Java classes

Linking Metadata Metadata can be linked in series in two ways Merge chaining allows for elements to be reused and a complete metadata can be built Operation chaining requests or describes operations on data sets A A B B ABAB ABAB A A B B B(A)

Operation Metadata Functions applied to the data have URI- based names Common parameters Ideally, a series of these metadata elements can completely describe the provenance of any resultant dataset As well as requesting selection and reduction operations at query time

Topology Schema Topology schema grew from network measurement description Reusable “Subject” elements for common cases Also reduces redundancy Relationships between measurement Subjects Same basic structure at all layers Networks are graphs Define: Node Port (Interface) Link Domain Network Path Service

Topology

Topology - Recursive Links

Topology Schema Structured by layers and the same elements recurring there Varied by namespaces Reuse visualization logic, etc. Validate layer- or technology-specific attributes 4 Layers: Base (both abstract and L1), L2, L3, L4 Also technology-specific layers like Ethernet, SONET/SDH

Relationships between Subjects To completely capture the relationships, we need to do a few more things Recursive definition of links Logical links consist of physical links Ordered lists of links - Path Like above, but we need to introduce an Index attribute Networks Physically consist of links but that is not always the most convenient logical view Special element to which Interfaces or Links belong

Relationships between Subjects Elements at the same layer have relationships A link references two ports/interfaces At Layer2 or Layer3 Elements of the same sort have relationships between themselves at different layers A Layer 1 Interface (physical NIC) can have one or more Layer 2 Interfaces, which can each have one or more Layer 3 Interfaces Node is special Since a Node doesn’t really have any higher-layer characteristic independent of its Interfaces

Schema - Network Element Identifiers A scheme for identifying network elements Each network element gets a unique identifier This identifier will be included with any measurement associated with that element.

Network Element Identifiers Use Cases: A topology service can be used to find the identifier for a network element An LS could then be queried to find all measurements associated with that element Dynamic service path-finding can be integrated with ongoing measurements

Network Element Identifiers Identifiers use URN notation Prefixed with “urn:ogf:network:” Consists of name/value pairs separated by colons Possible field names: domain, node, port, link, path, network Set of rules defined for each field to keep identifiers compact and finite Referred to as NURNs

Network Element Identifiers Examples urn:ogf:network:domain=Internet2.edu urn:ogf:network:domain=internet2.edu:node=packrat urn:ogf:network:domain=internet2.edu:node=rtr.seat:port=so-2%2F1%2F0.16 urn:ogf:network:domain=internet2.edu:node=rtr.seat:port= urn:ogf:network:domain=Internet2.edu:node=packrat:port=eth0:link=1 urn:ogf:network:domain=internet2.edu:link=WASH to ATLA OC192 urn:ogf:network:path=anna

perfSONAR Protocol Utilizes the basic message container With type attribute Contains various data and metadata elements Uses a message-level parameter element to communicate message-level options Messages return a result datum with a type system identical to that of a measurement datum

perfSONAR Information Services The Lookup Service and the Topology Service comprise the perfSONAR IS Topology and Service Lookup information are linked with references and NURNs Not necessarily a single instance of each information service These services are being utilized outside the measurement world

perfSONAR Lookup Service Directory service of perfSONAR deployments Accept service registrations Handles queries for service location and capabilities and location of available data Manage the lifetimes of data and services to keep information up to date Web Service interface to XML Database Sleepycat/eXist XML Databases Service Info/Data kept in native formats Draw away complex query tasks from otherwise 'busy' services

Lookup Service Also use an XML based configuration/protocol Native storage/query mechanisms [Xpath/XQuery] Uses LS-specific messages Targeted at single domain deployment Single instance to manage multiple services Client components and applications use the LS to find services

perfSONAR Topology Service Provides a queryable repository for obtaining topology information about a domain Can obtain the entire network Xquery interface allows the construction of complex queries about the network Topology is specified according to the topology schema Various Uses as a “Subject” repository for measurement information Topology discovery for dynamic circuit pathfinding

Distributed/Global Lookup Service Federation of individual LS instances into a global system “Meta”-lookup phase allows a query to find the specific LS that has relevant information Or perhaps the relevant LSes that have said info The specific query is sent directly to the LS in question

Distributed Lookup Service Service and measurement metadata is “summarized” for propagation to distant domains IP addresses in service and measurement metadata are compressed into network/netmask pairs in the same way that routes are advertised (CIDR-style) These summarized metadata elements are advertised to external “scopes” A “scope” is a set of LSes that are related by e.g. being in the same administrative domain (although multiple scopes within a single domain are possible)

Global Lookup Service Currently deployed solution involves a static set of “root” Lookup Services This works very much like DNS with peer to peer information propagation Summarization is based on the stored eventTypes and serviceTypes

Using perfSONAR How do I share my data using perfSONAR? perfSONAR MAs can publish data gathered with MRTG, Cacti Download one or more distributions MDM Release from GEANT2 Most services in Java Available as RPMs perfSONAR-PS Services in Perl Available as RPMs, via CPAN, Knoppix “live cd” Download various services “a la carte” Register with the Global Lookup Service…

perfSONAR Services Measurement Archive (MA) Services Both RRD and SQL, java and perl FlowSA MA (java) HADES MA (perl) Layer2 Circuit Status MA (java and perl) perfSONAR-BUOY Measurement Point (MP) Services CLI MP SNMP MP Telnet/SSH MP Flow Subscription BWCTL E2EMon Lookup/Topology/Authentication Services

Extending perfSONAR How can I extend perfSONAR? Definition of metric schema If you are publishing a new type of data, schema definition is the first step Reuse or re-implement protocol processing Examples in Java and Perl Register with the Lookup Service Defining a new service type Analysis modules are extended by assigning a URI, defining parameters

Ongoing Work Deployment and refinement of the global LS Recently funded by the US National Science Foundation Deployment of (“live CD”) appliances Initially to support LHC networking Integration with dynamic circuit networks like Internet2’s DCN, GEANT2’s AutoBAHN and ESnet’s SDN Leverage common components for dynamic, visible network

Joining perfSONAR How can I join this effort? Join the mailing lists at perfsonar.net Participate in OGF working groups Participate in Internet2 working groups Let us know that you are interested!

Acknowledgments GRNET HEAnet Indiana University Internet2 ISTF POZNAN RedIRIS Renater RNP SLAC SURFnet SWITCH UNINETT University of Delaware GRNET HEAnet Indiana University Internet2 ISTF POZNAN RedIRIS Renater RNP SLAC SURFnet SWITCH UNINETT University of Delaware ARNES BELNET CARNET CESNET CYNET DANTE DFN ESnet FCCN Fermilab GARR GEANT Georgia Institute of Technology ARNES BELNET CARNET CESNET CYNET DANTE DFN ESnet FCCN Fermilab GARR GEANT Georgia Institute of Technology US National Science Foundation, OCI Collaborators:

end Questions? Visit