1. SLAC computing Highlights Les Cottrell, SLAC After dinner talk to the Silicon Valley Tour group from Switzerland at Lakeside Restaurant, Shoreline, Mountain View California May 13 th, 2011 Personal viewpoint of highlights in computing at SLAC Nobel prize winning computing Web at SLAC China’s Internet Record breaking network speeds The future and data centers

2. 1967-1970 Nobel prize experiment Discovery of Quark Use of computers for control, data acquisition, & logging very new Programming led to need for physicist - programmers Acquired a computer 96kBytes, 500kHz, 400KB disk – 100,000 times less than today – Way less than today’s iPhone more like today’s watch – Took a large room

3. Dedicated to experiment no fashionable (at the time) time sharing – “Like wife sharing it tends to work best in special cases” Programmed in FORTRAN IV with real time extensions Experiment would not have been possible without it. – Had a lovely spacewar game ported from MIT with joysticks (from bubble chamber experiment) – Much friendlier than IBM 360/91 supercomputer used for off line analysis “IBM makes good business machines” Kept running for 15 years 1979 – Company SDS long gone, swallowed up by Xerox Nobel prize awarded in 1990

4. Web at SLAC http://www.slac.stanford.edu/history/earlyweb/history.shtml http://www.slac.stanford.edu/history/earlyweb/history.shtml Web at SLAC http://www.slac.stanford.edu/history/earlyweb/history.shtml http://www.slac.stanford.edu/history/earlyweb/history.shtml 1991 – Sep 1991 Paul Kunz (physicist at SLAC) brings word of the World Wide Web's existence to SLAC on his return from a meeting with Tim Berners-Lee at CERN.Tim Berners-LeeCERN – Dec 12 1991 first WWW server at SLAC (and first server outside of Europe) is successfully installed. 1992 – SPIRES-HEP, information system originally designed to provide access to preprints, or scientific papers that have not yet been published in peer-reviewed journals. Timely access to preprints can prevent duplicated effort, confirm or discount the direction of a proposed experiment or provide a vital clue to a struggling researcher, but the publication process can take weeks, months—even years. – Only access via guest accounts, nightmare to manage – First killer app for Web is the SLAC HEP physics preprint data base. No need for guest account at SLAC to access, demoed by Berners-Lee at CHEP in France in February 1992 – Formation of SLAC web wizards, Bebo was a leader/evangelist

5. Explosion Fall 1992 Tony Johnson of SLAC releases the MidasWWW browser. Based on X-windows, allows viewing of PostScript files on the Web from Unix and VMS, and even handles compressed PostScript. February 1993 NCSA release Mosaic for PC November 1993 SLAC puts up official home page on the web with links to web pages developed by others across the laboratory. 1994 IHEP in Beijing China becomes the first Chinese institution to have a fully operational world-wide Internet connection, when a dedicated link with SLAC () is initiated.a dedicated link with SLAC Use of WWW explodes to the world beyond physics...

6. China’s Internet link: Early History http://www.slac.stanford.edu/pubs/slacpubs/6250/slac-pub-6478.pdf http://www.slac.stanford.edu/pubs/slacpubs/6250/slac-pub-6478.pdf China’s Internet link: Early History http://www.slac.stanford.edu/pubs/slacpubs/6250/slac-pub-6478.pdf http://www.slac.stanford.edu/pubs/slacpubs/6250/slac-pub-6478.pdf 1987-1991 BEPC2 (VAX 785) @ IHEP linked to CERN via packet switched data network (PSDN), dial up, email only, expensive

7. Invitation Chinese scientists from IHEP, visit SLAC in April/May 1991 They were interested in computing and networking and as assistant director of computing I was invited to attend a meeting with them They were particularly interested in a network connection to SLAC to support the Beijing Electron Spectrometer collaboration between IHEP, SLAC and other US institutions With a meeting in Tokyo on Computing in High Energy Physics starting 3 weeks later, I suggest extending my trip to a visit to IHEP in Beijing, see the world IHEP were very supportive, invite me

8. Be Careful what you ask for… Not knowing what to expect in Beijing – Just 2 years after the Tianamen Square – What was the technology available in IHEP – I was worried… I met with Pief Panofsky the Emeritus Director of SLAC – He was very encouraging – He had this vision of how excellent networking could make a worldwide physics collaboration really work well But outbound IHEP international calls needed an operator How to make this work with a digital network …

9. Panofsky to the Rescue Panofsky called Nobel laureate T. D. Lee of Columbia for help – Request top priority to installation of 3 phones with unattended international access. – Get me visa 3 weeks later after the CHEP conference in Tokyo I was met at the old Beijing airport – Taken to the Friendship Hotel used to be for Russian experts

10. Working with IHEP staff I was amazed to find the phone lines in place and working They were excited about working with western “experts” – Determined not to let knowledge of English impede things – I was flattered to be considered an “expert”, and by their attention, friendliness and enthusiasm However, soon found that despite nods and smiles I was talking way too fast – So wrote everything down as I talked, this forced me to go slowly and provided a written record

11. Accomplishment while there Hooked up a 9600bps modem between phone & VAX 11/785 Used 2 nd phone line to call Charley Granieri at SLAC (15 hours apart) – Set up asynchronous DECnet dial-up connection SLAC – IHEP – Effective 400 bits/sec, very noisy and hard to use Often unable (no international line available) to make connections Frequent disconnects in mid-session – $3.0/min IHEP VAX SLAC

12. Now it gets really hard US downstation Point Reyes north of San Francisco China downstation Beijing airport From airport microwave to BTA bldg 801 35km away in center of Beijing BTA 801 to 821 exchange bldg 2 blocks from IHEP – Tried infrared, the microwave but error rates to0 high, eventually got a fibre route – Last 2 blocks there was copper, but problems with converting fibre to copper – March 1 st 1993 acceptable error rates, handed over to IHEP – Seconds later monitor program showed the SLAC DECrouter adjacent to IHEP DoE/SLAC paid $50K/yr, similar from China

13. How did it work & for what Ten times better: – 42kps file transfer – Echo time < 1 second – Error rate 1 in 10 million – 1-2 unscheduled outages/month – Twice yearly occulted by sun directly in line with satellite How was it used: – Transfer physics data: 200MB/day (equivalent to that era’s tape cartridge – IBM 3480) – 2500 emails/day 400 sites in 21 countries via SLAC gateway – News groups, collaboration coordination, code management – Copying files, remote login, real-time communication

14. Connecting to the Internet Once 64kbps link established many Chinese institutions wanted to connect to IHEP to get access to Internet Jan 1994 meeting to recommend China domain naming – Node.ihep.ac.cn Proposed replacing DEC routers with Cisco routers – Got export licenses from US DOC – Received in Beijing Feb 1994, installed in March – Worldwide HEPnet connected Dec 1994 Visit by US congressman George Brown to IHEP increased US government interest Agreement for Internet to carry Chinese traffic (DOC, DoD, DoE) – Required Internet wide-area email sent April 18, 1994, saying China connection April 25, 1995 IHEP fully Internetted via US West Coast interconnection point

15. High speed networking: Need Until late 1990s all HEP data was transferred between Labs by tapes. Pack a 747 full of 50 Gbyte tapes, takes 10 hours to Geneva or ~ 2-4 Tbits/s, however – Expensive – At SLAC alone, producing, packaging and shipping the tapes took the full-time effort of two people. – How long did it take for data from SLAC to reach a researcher at CERN? Two weeks so data rate actually ~ < 1Gbits/s. Assuming no errors This century network speed began to achieve and pass 1Gbits/s and so make it possible to use the network to transfer data

16. Breaking the Internet2 Land Speed Record twice 2002/3 Internet2 Land speed record 2002 in Baltimore – http://www-iepm.slac.stanford.edu/lsr/faq.html http://www-iepm.slac.stanford.edu/lsr/faq.html – Big set up for SC002 to demonstrate high performance with loaned equipment from Cisco (router), Level3 (circuits), Intel 10Gbps Ethernet interface, collaboration of SLAC, Caltech, CERN and others for SC2002 – First time Impromptu, hall discussion with student Antony Antony from Amsterdam University and almost a bit of a lark – 900 Gbps equivalent of 2 DVD’s/minute, Sunnyvale to Amsterdam (<6000km) – Repeated Sunnyvale to Geneva in Feb 2003 at 2.38Gbits/s or a Tbyte/hour or heroic tuning of hosts, TCP stacks, team of about 20 people, but COTS hardware & software – Caught attention: reported in English, Spanish, Portuguese, French Dutch and Japanese, on CNN, BBC, Wired, Nature, Times of india, Tech TV and ABC radio – Guinness book of records 2004 – CENIC award, Internet2 record – http://www-iepm.slac.stanford.edu/lsr/ http://www-iepm.slac.stanford.edu/lsr/ – http://www-iepm.slac.stanford.edu/lsr2/ http://www-iepm.slac.stanford.edu/lsr2/

17. Supercomputing Nov 2003 - Phoenix SLAC, Caltech, LANL, CERN, Manchester and Amsterdam 23Gbits/sec multiple computers (6.6 Tbytes in 48 minutes = 2000 DVDs) won SC2003 Bandwidth challenge – Congratulations from US congress woman Anna Eshoo http://www- iepm.slac.stanford.edu/monitoring/bulk/sc2003/ hiperf.html http://www- iepm.slac.stanford.edu/monitoring/bulk/sc2003/ hiperf.html

18. Repeated wins at SC 2004 and 2005 2004 (Pittsburgh): – 101.3Tbit/s 3 full length DVD movies/sec, library of congress in 14 minutes – Used 8 10Gbps waves from multiple providers E.g. ~ 9Gbps from SLAC to Sunnyvale – Limitations in bus speeds 2005 (Seattle): – Sun, Cisco, Ciena, Qlogic, 3Par, Internet2, Esnet, NLR, Surfnet, CANARIE, AMPATH – 22 * 10Gbps fibres (US, UK, Brazil, Korea, Switzerland (CERN) )… – > 150.7Gbps 130 DVDs/min, Lib. of Congress in < 9 mins, serve 10,000 MPEG HDTV movies in real time http://www- iepm.slac.stanford.edu/monitoring/bulk/sc2004/hiperf.html http://www- iepm.slac.stanford.edu/monitoring/bulk/sc2004/hiperf.html

19. Clustered Systems http://www.clusteredsystems.com/ http://www.clusteredsystems.com/ Clustered Systems http://www.clusteredsystems.com/ http://www.clusteredsystems.com/ 50TFlops in a single rack – Using Intel Sandybridge systems, – NextGen => GPUs – 100kW/rack – Build showcase in fish bowl @ SLAC Innovative cooling: – Conductive cold plates water cooled, fanless – Connected to board components by Gel packs – Can use 60 deg F (instead of normal 40 deg F) return cooling water for further cooling 2.5 times less power/ TFlop and less space, more green Disruptive technology Demo later this year

20. Stanford Research Computing Facility (SRCF) Cost of upgrading existing computing centers designed in mainframe days is prohibitive while maintaining services – For each $ spent on computer need another for providing power & cooling – “create world in 6 days but no installed user base” $43M computer center for SU located at SLAC break ground end CY Modular & flexible, high density power & cooling, low operational costs Green: take advantage of California’s temperate climate with ambient air cooling Initial 2.5-3.5MW eventually 6MW