1. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 1 1 © 2010 Cisco and/or its affiliates. All rights reserved. Jarrett Fishpaw, Business Operations Manager Tyler Munger, NMO Lab Intern UC Santa Cruz, ISM 101: Thursday, February 28, 2013

2. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 2 Jarrett’s Personal Background Cisco Systems Overview What is Cisco Services Cisco Services Technology Group Smart Analytics – ‘Actionable Insight’ TM The NMO Lab and Example Data Analytics Projects Day in the life of a Cisco Employee

3. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 3 B.A. Business Management Economics (2009) M.S. Applied Economics and Finance (2010) Business Operations Manager, Cisco Systems Contractor from June 2010 - April 2011 FTE since May 2011 Mayor of Los Altos (2012) Joined City Council in 2010 Job Role: 50% - Incubation and Innovation of new analytical models Automated Router Configuration and Simulation 50% - Specific projects/Team Administration Enhanced NPA

4. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 4 At Cisco (NASDAQ: CSCO) customers come first and an integral part of our DNA is creating long-lasting customer partnerships and working with them to identify their needs and provide solutions that support their success. Husband and wife Len Bosack and Sandy Lerner, both working for Stanford University, wanted to email each other from their respective offices located in different buildings but were unable to due to technological shortcomings. A technology had to be invented to deal with disparate local area protocols; and as a result of solving their challenge - the multi-protocol router was born. Founded: San Fran cisco, California, U.S. (1984) Len BosackLen Bosack, Sandy Lerner, Richard Troiano, Nicholas PhamSandy LernerRichard TroianoNicholas Pham

5. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 5

6. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 6

7. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 7

8. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 8

9. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 9 Q1 FY'13 Revenue: $11.9 billion Q1 FY'13 Employee Count: 72,360

10. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 10 Technical Assistance Center

11. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 11 Employees Worldwide: 11,400+ Certified Professionals: CCIE= 2,000+ Patents ~700 since 1998 Cisco Partners Services is responsible for 80% of the revenue coming from Cisco Partners There are over 280,000 partner employees which provide channel leverage Over 68,000 authorized/registered partners Provide support in over 180 countries worldwide

12. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 12 The Cisco Services Technology Group drives technology innovation and software development to accelerate the evolution of our Advanced and Technical services portfolio. Our global organization consists of five primary groups: Technology Development Support Business Operations Globalization Expertise We are accountable for developing, delivering, and supporting the technologies Cisco Services deploys in our markets.

13. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 13 Who Are These People? Think of him or her as a hybrid of data hacker, analyst, communicator, and trusted adviser. The combination is extremely powerful—and rare. What are their traits? Intense curiosity Associative thinking Credit Card fraud – DNA sequencing Finance - Pharmaceutical Trials - Network Reliability

14. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 14 AnalystsCompetitors IBM opens a Services Innovation Lab (SIL)…the lab will accelerate the expansion of real-time analytics … in both IBM’s technology services offerings and its global services delivery capabilities. (4/2011) IBM has committed $2 billion in service R&D Partners & Customers Smart Analytics represent an excellent delivery of intellectual capital…This type of IC scored in the top 3 of partner demands in the partner focus groups regarding …desirable service elements - Market Intelligence Team, 5/2011 IT complexity will soon outstrip customer’s IT management capabilities – Forrestor Network managers have more information than ever … Making sense of the information is the essential challenge… trends will see the effective transition of development priorities from network technology to IT analytics – IDC, 3/2011 Role of Analytics in Complex Networks

15. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 15 Understand network health Set and measure progress against network improvement plans and change initiatives Compare against peers in the same vertical on certain metrics such as technology maturity and operational maturity

16. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 16 Big Data requires: Hadoop SAS R Denodo Domain Experts Collaborating with Statisticians ‘Big Data’ combined with Analytics

17. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 17 Linear Regression Curve Fitting ARIMA – Autoregressive Integrated Moving Average Data Clustering Data Mining Correlation Analysis Time Series - ARIMA Models Predictive Forecast Multivariate Regressions Cross Sectional Data Historic Trending Process data using advanced tools, and domain expertise Techniques: Multivariate Regressions

18. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 18 Delivery of Intuitive Insight Industry Benchmarking Root Cause Analysis Predictive Forecasting ROI Simulation Visualization

19. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 19 Projects: Short-term undergraduate internships (1-3 quarters) and Masters/PhD level research (3+ quarters) at Cisco Wide range of interesting projects from hardware testing to Supply Chain Management Real world data and problems Potential users (work has a high possibility of actually being used) Time: Generally 20 hours a week. Usually work from home and go into Cisco about once a week Resources Networking labs (UCSC and Cisco) Servers Domain experts

20. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 20 BS in Computer Science from UCSC (2009) Graduate Student in the Technology and Information Management (TIM) program working under Subhas Desa Been part of the NMO lab for 4 years 1 year as an undergraduate student 3 years as a graduate student Research areas Data mining/Information retrieval Software engineering Computer networks Knowledge engineering

21. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 21 When a customer has a problem with a Cisco networking product they open a service request with Cisco’s support teams Cisco support teams receive thousands of service requests every month on a wide variety of Cisco networking products How do we detect emerging (growing) problems before they impact a large number of customers? Software bugs Viruses Hardware failures “We have one ASA with 2 ISP connections and several site to site VPN’s set-up to other sites with ASAs. When we test failing over the VPN’s to use the second ISP connection the tunnels are not coming up. We have confirmed that outbound routing fails over immediately.” Typical Customer Service Request Primary Investigator: Tyler Munger Advisors: Chris Wong (Cisco), Subhas Desa (UCSC)

22. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 22 1. Data Mining: identify the associated product failure mode for customer service requests Support Vector Machines (classification) 2. Prediction: generate an alert when a particular failure mode starts abnormally increasing in frequency (indicating an emerging problem) Double Exponential Smoothing (forecasting) Control Charts (alerts) 3. Software Engineering: develop an easy to use software tool that allows Cisco Engineers to monitor customer service requests for emerging problems Weka (data mining) Ruby on Rails (webserver) MySQL (database) Adobe Flex (visualization and GUI)

23. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 23 Identified 20 key failure modes for a popular Cisco network security product (ASA 5505) Classified a dataset of ~10,000 service requests by failure mode using a Support Vector Machine classifier (SVM) Dominant failure modes were “failover communication between devices”, “VPN tunnels disconnected after switchover”, “configuration not in sync after switchover” Failure ModeFrequency Failover communication between devices 1414 VPN tunnels disconnected after switchover 922 Configurations not in sync after switchover 609 No switchover450 Hardware224 ……

24. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 24 Developed a web-based software tool for Cisco to use for monitoring emerging issues Used the tool to identify emerging problems in historical customer service requests Increase in Virtual Private Networking (VPN) problems caused by a bug in a new software release

25. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 25 Customer: increased network reliability because potential problems are identified and resolved before they occur Cisco: increased customer satisfaction and reduced support costs because customers are opening less service requests

26. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 26 Cisco Internet Operating Systems (IOS) is the core piece of software that runs a wide range of Cisco networking products (routers, switches, firewalls, etc.) There are hundreds of different IOS software releases Older releases are more reliable but have less features Newer releases have more features but are less reliable How do we recommend the most reliable IOS software release that has the features that the customer needs? Primary Investigator: Yali Mu Advisors: Jarrett Fishpaw/Chris Wong (Cisco), Subhas Desa (UCSC)

27. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 27 The reliability of an IOS software release can measured by the number of software bugs that have not been found yet (unfound bugs) How do we estimate the number of unfound bugs for each IOS Software Release? 1.Create a bug forecast using historical bug data 2.Determine the total number of bugs for the software release based on the forecast 3.Calculate the number of unfound bugs Develop an easy to use software tool that allows Cisco Engineers to easily select the most reliable software release for a particular customer network Bugs FoundForecast Total Number of Bugs Unfound Bugs

28. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 28 Created reliability forecasts for over 5,000 IOS software releases across 36 different product families Forecasts were used to create 3 reliability estimates for each IOS software release 1.Total number of bugs 2.How many unfound bugs 3.When all the bugs will be found

29. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 29 Customers: network reliability improves because they can select the most reliable IOS software release for their network Cisco: increased customer satisfaction and reduced support costs because customers have less network problems

30. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 30 Many employees have a unique and flexible work schedule WebEx Meetings Outlook Calendar

31. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 31 Each manager has a different style Responsiveness to Email and IM’s (Phone/text occasionally) Personal discipline to manage multiple projects simultaneously Weekly Breakfast Club on our team

32. © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Confidential 32 Cisco Business questions What does Cisco do? Are Cisco products food or internet related? Do you use Cisco products? What does a Data Scientist do? What year was Cisco Founded? How large is Cisco (people/revenue)? How many authorized partners does Cisco have? Cisco Certification questions Name 2 Cisco networking certifications