CANARIE CA*net 4 Planning Tel:

GigaPOP CA*net 3 National Optical Internet Vancouver Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. Johns Fredericton Charlottetown ORAN BCnet Netera SRnet MRnet ONet RISQ ACORN Chicago STAR TAP CA*net 3 Primary Route Seattle New York CA*net 3 Diverse Route Deployed a 4 channel CWDM Gigabit Ethernet network – 400 km Deploying a 4 channel Gigabit Ethernet transparent optical DWDM– 1500 km Multiple Customer Owned Dark Fiber Networks connecting universities and schools km Consortium Partners: Bell Nexxia Nortel Cisco JDS Uniphase Alcatel Condo Fiber Network linking all universities and hospital

CA*net 3 Status All GigaPOPs and regional networks connected and operational ORANs up and running in Nfld, PEI, Quebec, Alberta Announcements expected shortly for ORANs in BC, Manitoba, Ontario and Nova Scotia, New Brunswick 360network connection to Europe and NY through Dalhousie coming soon Will make Canada & Dalhousie global hub for most international research traffic Several US networks and universities looking to connect directly to CA*net 3 NYSERnet to be completed shortly UoMinnesota in discussions Traffic volumes exceeding 500 Mbps peak on all links Network expected to be reach peak capacity by September with on-going WDD trials and HDTV sessions between UoCalgary and McGill Numerous countries have built or plan to build networks modeled on CA*net 3 and proposed CA*net 4 Chile, Brazil, Poland, SURFnet, Greece Czech Republic, Australia, CENIC, NYSERnet, Quilt Over 200 university & research institutions and 2000 school connected Over 2000 schools connected Over 50 International peers and over 20 International transit networks

Peer Networks USA: 6 networks Abilene (Internet 2), ANL (Argonne), vBNS (NSF), Esnet (Energy), NISN (NASA), NREN (NASA) NEW Nysernet – July 2001 STARTAP in Chicago: 20 national networks CERN, IUCC, APAN/ TRANSPAC (Korea, Malaysia, Australia, Philipines), RENATER2 (France), SINET (Japan), SingAREN (Singapore), SURFnet (Netherlands), NORDUnet (Iceland, Norway, Sweden, Finland, Denmark), TANet2 (Taiwan) New Reuna Chile, RTP Brazil, MIRnet Russia, Renater2 (France), KRnet (Korea) TEN-155 in New York: 26 European networks ACOnet (Austria), ARNES (Slovenia), BELnet (Belgium), CESNET (Czech Republic), DFN (Germany), GARR (Italy), GRNET (Greece), HEAnet (Ireland), HUNGERNET (Hungary), JANET (U.K.), POL34 (Poland), RCCN (Portugal), RedIRIS (Spain), RENATER2 (France), RESTENA (Luxembourg), SWITCH (Switzerland), Nordunet (Iceland, Norway, Sweden, Finland, Denmark) Seattle: 4 networks NTON (DARPA), Supernet (DARPA), PCNW (Washington State) New Australia (AARnet), ESnet

Internet Transit Service Internet 2 and CANARIE offering research transit, e.g. Europe (DANTE) or Asia to US agency networks – NASA, NREN, Esnet Australia to Europe (DANTE) Korea to STAR TAP All European traffic to US research networks (except Internet 2) goes through Canada Will promote international collaboration in advanced research New 360network donation through Dalhousie will significantly enhance Canadas reputation as a global hub for advanced networks If history is any kind commercial networks will soon follow research networks

Applications Distributed caching to remote schools NFB video server multi-channel virtual studio digital mammography (UoT) Pinkas Zukerman interactive violin lessons global brain scan database (McGill) NRC bio-informatics genome sequence national repository strategy for learning objects HDTV over IP – McGill Virtual Vet courses

International Developments Distributed Terabit Facility 40 Gbps network with wavelengths and dark fiber from Qwest Will interconnect over 1000 Linux processors Internet 3 and Quilt Internet 2s next generation network expected to be announced this fall Quilt based on many concepts proposed for CA*net 4 OMNInet in Chicago STAR LIGHT SURFnet (Holland), CERN, NODRUnet, DTF will all bringing wavelengths to Chicago this fall Maybe CA*net 4? PIONIER (Poland), GRnet (Greece), etc DANTE will be issuing RFP for wavelengths across the Atlantic

CA*net 3 Lessons Community Condominium dark fiber networks exploding Non traditional carriers building dark fiber networks Construction companies, university networks, etc LAN architectures, technologies and most importantly LAN economics are invading the WAN Low cost optical technology -CWDM, Gigabit Ethernet Control and management of the optics and wavelengths will increasingly be under the domain of the LAN customer at the edge, as opposed to the traditional carrier in the center Over time the current hierarchical connection oriented telecom environment will look more like the Internet which is made up of autonomous peering networks. These new concepts in customer empowered networking are starting in the same place as the Internet started – the university and research community. The Internet model of networking is moving into traditional telecoms

Customer Empowered Networks School boards and municipalities throughout North America are working with next generation carriers in building condo open access, dark fiber networks and/or purchasing virtual fiber Spokane Chicago CivicNet Manhattan project Illinois Iwire Indiana Gwire California CENIC Polish Pionoier Czech municipal networks Coming soon – BC, Ontario, Nova Scotia, New Brunswick Carrier are selling dim wavelengths or virtual fiber managed by customer to interconnect dark fiber networks Williams, Level 3, Qwest

Research Network Philosophy Research and Education networks must be at forefront of new network architecture and technologies Should be undertaking network technology development that is well ahead of or orthogonal to any commercial interest But any network architecture can only be validated by connecting real users with real applications and must solve real world problems Test networks per se are not sufficient There is a growing trend for many schools, universities and businesses to control and manage their own dark fiber Can we extend this concept so that they can also own and manage their own wavelengths?

The Concept for CA*net 4 Conventional optical networks are built on the paradigm that a central entity has control and management of the wavelengths It therefore must have control of the edge device for the setup and tear down of the wavelengths Customer empowered optical networks are built on the paradigm that customer owns and controls the wavelengths (Virtual Dark Fiber) and dark fiber Customer controls the setup, tear down and routing of the wavelength between itself and other customers Customer may trade and swap wavelengths with other like minded customers ultimately leading to wavelengths as market commodity Network is now an asset, rather than a service Analogy to time sharing computing in the early 1970s versus customer owned mini-computers or client-server computing Will empowering customers to control and manage their own networks result in new applications and services similar to how the mini-computer and PC empowered users to develop new computing applications?

Current View of Optical Internets Big Carrier Optical Cloud using MP S or ASON for management of wavelengths for provisioning, restoral and protection Carrier controls and manages edge devices Optical VLAN Customer ISP AS 1 AS 2 AS 3 AS 1 AS 4 AS 5 UNI NNI

Customer Empowered Network Carrier Neutral IX City A City B City C Carrier Neutral IX Condo Dark Fiber Condo Wavelengths

Future Optical Networks Customer A Customer B Customer C Customer D Customr A elects to cross connect with Customer C rather than D Massive peering at the edge Condo Fiber Condo Wavelength

CA*net 4 Research Areas New optical technologies that support customer empower networking OBGP, CWDM, hybrid optics and HWDM, customer controlled optical switches BGP scaling issues Object Oriented Networking Wavelengths and optical switch treated as an object and method to be incorporate into middleware Or treated as fungible product Distributed Computing Applications and Grids Wavelength Disk Drives (WDD) eScience Grids for weather forecasting, forestry management, education, health, etc

Object Oriented Networking Combines concepts of Active Networks and Grids See DARPA See Globus Customer owns sets of wavelengths and cross connects on an optical switch Network elements can be treated as a set of objects in software applications or grids Complete with inheritances and classes, etc Rather than distributed network objects ( e.g. Java or Corba) distributed object networks In future researchers will purchase networks just like super computers, telescopes or other big science equipment Networks will be an asset – not a service Will be able to trade swap and sell wavelengths and optical cross connects on commodity markets

Example OON Earthquake Visualization Grid Globus Middleware Begin Establish connection to other grid participants Network Object – wavelength to STAR LIGHT – Chicago Network Object – wavelength to Research center Amsterdam Network Object – wavelength to SDSC Visualization Computer Network Object – wavelength to Seismology Center Calgary Link objects and create grid Run Visualization Release Network objects Globus Middleware End Earthquake Visulization End

CA*net 4 & Community Networks CA*net 4 will be a national resource for K-12 networks and supporting community NBTF initiatives eScience grids, learning grids and health grids Researchers & educator may want to use computing resources of schools and homes as part of large distributed computing projects CA*net 4 will interconnect environmental and health grids with students and researchers New grid projects in bio-informatics, pharmaceutical research, particle physics need access to millions of computers

What is eScience? The ultimate goal of e-science is to allow students and eventually members of the general public to be full participants in basic research. Using advanced high speed networks like CA*net 4 and novel new concepts in distributed peer to peer computing, called Grids many research experiments that used to require high end super computers can now use the computer capabilities of thousands of PCs located at our schools and in our homes. High performance computers that are part of C3.ca can be seamlessly integrated with eScience distributed computers using CANARIE Wavelength Disk Drive over CA*net 4 Allows researcher access to the significant computational capabilities of all these distributed computers at our schools and homes With e-science it might be possible that the next big scientific discovery could be by a student at your local school.

Scientists at The Scripps Research Institute (TSRI) are using computational methods to identify drugs that have the right shape and interaction characteristics to fight diseases such as AIDS.The Scripps Research InstituteAIDS Once such candidates are identified, they can be synthesized in a laboratory, tested according to FDA guidelines, and released as prescription drugs to benefit the public. Such computations require a vast number of trial dockings, testing variations in the target protein and the trial drug molecules dockingsproteinmolecules

Philanthropic Peer to Peer The Intel® Philanthropic Peer-to-Peer Program helps to combat life-threatening illnesses by linking millions of PCs to be the largest and fastest computing resource in history. This "virtual supercomputer" uses peer-to-peer technology to make unprecedented amounts of processing power available to medical researchers to accelerate the development of improved treatments and drugs that could potentially cure diseases.peer-to-peer technology

ALTA Cosmic Ray eScience The earth is constantly bombarded by subatomic particles from space, with an energy spectrum that reaches far higher than any terrestrial accelerator could hope to probe. At the highest energies such showers can be detected at the Earths surface over areas on the order of 100 square kilometers. It is believe some of these cosmic rays were created at the creation of the universe Will allow researchers to gainer a deeper understanding of deepest reaches of space and time

ALTA Cosmic Ray eScience The ALTA project is a collaborative scientific research project involving the University of Alberta Center for Subatomic Research and over 50 high schools across Canada in the area of cosmic ray detection.University of Alberta Center for Subatomic Research Teachers and students actively contribute to the physics research while learning about an exciting area of modern science. Distributed computing at schools will be required to analysize data from sensors in near real time Program has now expanded into USA and soon countries around the world CHICOS (California HIgh school Cosmic ray ObServatory), Caltech, UC/Irvine and Cal State/Northridge, California, USA. CHICOS CROP (the Cosmic Ray Observatory Project), University of Nebraska, Lincoln, NE, USA. CROP WALTA (WAshington Large area Time coincidence Array), University of Washington, Seattle, WA, USA. WALTA SALTA Roaring Fork Valley area of Colorado SALTA

Neptune – Undersea Grid

Wavelength Disk Drives CA*net 4 will be nation wide virtual disk drive for grid applications Big challenges with grids or distributed computers is performance of sending data over the Internet TCP performance problems Congestion Rather than networks being used for communications they will be a temporary storage device Ideal for processor stealing transaction intensive applications where you dont know where the next available processor is located CFD Visualization

Wavelength Disk Drives Vancouver Computer data continuously circulates around the WDD Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. Johns Fredericton Charlottetown CA*net 3/4 WDD Node

WDD Architecture Vancouver Calgary Halifax WDD Node WDD Partners: CANARIE, Can-Sol, Viagenie CRC, Carleton U, MACI C3.Ca, Memorial, Dalhousie UdeMontreal, UoToronto, SFU, UoAlberta, BCnet Memorial Dalhousie UdeMontreal UoToronto UoAlberta SFU WDD Node Forest Fire Modeling Raster Engine WDD Node CRC 1. Forest Fire Modeling Raster Engine injects 64K x 64K raster computational tasks into WDD ring 2. Tasks circulate in WDD ring and first available SGI processor removes next task out of the ring and completes computation 3. The SGI writes back the task onto the ring where it is received by Forest Fire Raster Engine and results displayed on X-Window terminal at CRC WDD Ring on CA*net 3

Forest Fire Modeling eScience Emergency officials and civic defense officials need to model forest fires in real time But each forest fire model may take hours to compute By utilizing thousands of distributed computers at our schools and Wavelength Disk Drive on CA*net 4 network forest fire models in near real time First prototype to be demonstrated on CA*net 3 in May using 256 SGI processors across the country on WDD

WDD Process Forest Fire Modeling Raster Engine injects 64K x 64K raster computational tasks into WDD ring at BCnet node in Vancouver Tasks circulate in WDD ring and first available SGI processor removes next task out of the ring and completes computation The SGI writes back the task onto the ring where it is received by Forest Fire Raster Engine and results displayed on X-Window terminal at CRC

CA*net 4 Possible Architecture Vancouver Calgary Regina Winnipeg Ottawa Montreal Toronto Halifax St. Johns Fredericton Charlottetown Chicago Seattle New York Europe Customer controlled optical switches Layer 3 aggregation service Optional Service Available to any GigaPOP Large channel WDM system

STAR LIGHT Interconnection? We see STAR LIGHT, CA*net 4, DTF and Vancouver Transit exchange facing same design issues How do we signal interconnect wavelengths (SDH/SONET subchannels) between STAR LIGHT participants? Like STAR TAP we will probably need a mix of Layer 1-3 solutions Layer 1 cross connect ATM plus Layer 3 router and/or route server Current ATM approaches Full mesh ATM like current STAR TAP Not possible with wavelengths or SDH/SONET channels PVC created on demand E.g Peer maker at MAEs

STAR LIGHT Options Layer 0 - Patch panel or optical switch Needs common wavelength and protocol Not easily subject to change and will not allow multiple peers Layer 1 - SDH/SONET cross connect switch Issues related to how identify and address SDH/SONET channels Layer 2 - GMPLS using IP and SONET/optical switch Main thrust of industry –see Juniper/Nortel, Accelight, Cisco, NTT Requires significant centralized management Layer 2 -Map SDH/SONET channels to GbE channels & use GbE switch Layer 3 - Each network terminates on its own router & routers meshed together N squared meshing Layer 3 - BIG ROUTER Will it scale and needs central management and AS Layer 4 – OBGP with CWDM with optical switch Each CWDM wavelength mapped to SDH/SONET channel Control of switch is by research networks

OBGP Status Report OBGP first draft submitted to IETF Prototype working at Carleton U We want input on next steps for OBGP and see if it will fit within STAR LIGHT plans Key features: SDH/SONET & Optical cross connects controlled by attached networks SDH/SONET & Optical cross connects identified by IP addresses & AS RPSL with OON extensions is database used to query who is connected at switch and at what port BGP OPEN message is used like Peer maker to request optical peering across the switch BGP UPDATE message and community Tags ( and maybe GMPLS) will be used to setup multihop wavelengths

OBGP Proposed new protocol to support control and management of wavelengths and optical switch ports Control of optical routing and switches across an optical cloud is by the customer – not the carrier – true peer to peer optical networking Use establishment of BGP neighbors or peers at network configuration stage for process to establish light path cross connects Customers control of portions of OXC which becomes part of their AS Optical cross connects look like BGP speaking peers – serves as a proxy for link connection, loopback address, etc Traditional BGP gives no indication of route congestion or QoS, but with DWDM wave lengths edge router will have a simple QoS path of guaranteed bandwidth Wavelengths will become new instrument for settlement and exchange eventually leading to futures market in wavelengths May allow smaller ISPs and R&E networks to route around large ISPs that dominate the Internet by massive direct peerings with like minded networks

Wavelength Scenarios Vancouver Calgary Regina Winnipeg Toronto Halifax St. Johns Seattle Montreal Workstation to Workstation Wavelength University to University Wavelength CWDM BCnet RISQ GigaPOP to GigaPOP Wavelength Campus OBGP switch

Wavelength Setup AS 1 AS 2 AS 3 AS 4 AS 5 AS 6 Dark Fiber Wavelength Object owned by primary customer Wavelength Subcontracted by primary customer to a third party AS 1- AS 6 Peer AS 2- AS 5 Peer Regional Network University ISP router

Wavelength Logical Mapping AS 1 AS 2 AS 3 AS 4 AS 5 AS 6 Primary Route Backup Route AS 1- AS 6 Peer AS 2- AS 5 Peer Regional Network University ISP router

Resultant Network Topologies AS 1 AS 5 AS Regional Network University AS ISP router OBGP Potential OBGP Peering BGP Peering on switches at the edge Packet Forwarding in the core

OBGP Variations 1.OBGP Cut Thru OBGP router controls the switch ports in order to establishes an optical cut through path in response to an external request from another router or to carry out local optimization in order to move high traffic flows to the OXC 2.OBGP Optical Peering External router controls one or more switch ports so that it can establish direct light path connections with other devices in support peering etc 3.OBGP Optical Transit or QoS To support end to end setup and tear down of optical wavelengths in support of QoS applications or peer to peer network applications 4.OBGP Large Scale To prototype the technology and management issues of scaling large Internet networks where the network cloud is broken into customer empowered BGP regions and treated as independent customers

OBGP Optical Peering Primary intent is to automate BGP peering process and patch panel process Operator initiates process by click and point to potential peer Original St. Arnaud concept Uses only option field in OPEN messages Requires initial BGP OPEN message for discovery of OBGP neighbors Virtual BGP routers are established for every OXC and new peering relationships are established with new BGP OPEN message Full routing tables are not required for each virtual router No changes to UPDATE messages No optical transit as all wavelengths are owned by peer Uses ARP proxy for routers on different subnets Wade Hong Objects concept Uses an external box (or process) to setup optical cross connects SSH is used to query source router of AS path to destination router Each optical cross connect is treated as an object with names given by AS path Recursive queries are made to objects to discover optical path, reserve and setup NEXT_HOP at source router is modified through SSH End result is a direct peer and intermediate ASs disappear Requires all devices to be on same subnet

Target Market for OBGP University research and community networks who are deploying condominium fiber networks who want to exchange traffic between members of the community but who want to maintain customer control of the network at the edge and avoid recreating the need for aggregating traffic via traditional mechanisms E.g. Ottawa fiber build, Peel County, I-wire, SURAnet, G-Wire, CENIC DCP, SURFnet, etc etc Next generation fiber companies who are building condominium fiber networks for communities and school boards and who want to offer value added fiber services but not traditional telcommunications service E.g. C2C, Universe2u, PF.net, Williams, QuebecTel, Videotron, etc Next generation collocation facilities to offer no-cost peering and wavelength routing Metromedia, Equinix, LINX, PF.net, LayerOne, Westin, PAIX, Above.com, Colo.com, etc etc Over 500 Ixs and carrier hotels worldwide