1. Information Systems & Computing Leadership Needed: Creating and Implementing an Information Security Vision M. Jost, D.Kassabian, D.Millar University of Pennsylvania EDUCAUSE 2004 Annual Meeting Copyright Trustees of the University of Pennsylvania 2004. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. Information Systems & Computing About Penn The University of Pennsylvania was founded by Ben Franklin in 1740 Penn is part of the Ivy League Located in western Philadelphia Community of more than 35,000 people

3. Information Systems & Computing IT at Penn (1 of 2) Centrally provided administrative applications –Student, Financial and HR systems Distributed academic computing Centrally provided data network, PennNet: –35,000 ports of 10/100 Ethernet –200 campus buildings –1.3 Gbps Internet access

4. Information Systems & Computing IT at Penn (2 of 2) IT funding, management, user support and decision making are very distributed at Penn IT governance and coordination is well established –University CIO –IT Roundtable - Computing Directors –SUG - IT support professionals and developers –NPTF - Financial planners, networking charges –NPC - cross section, developing network policies

5. Information Systems & Computing Situation in Summer’03 Security administration, including patching of operating systems, uneven across campus Anti-virus software on most computers in campus departments, but fewer in residences Student systems becoming compromised at an alarming rate Staff time costs in responding to security compromises was skyrocketing

6. Information Systems & Computing Estimated cost of Blaster/Welchia For August 2003 ITEMEST. COST 1200 compromised machines -Manage detection and notification -Format and rebuild machines -Remove Blaster from machines 15-25% 9,000 Vulnerable machines (patched twice) -20 campus-wide scans, 14 mass notifications of vulnerability -4,500 Patch automatically (twice) -4,500 Patch manually (twice) 2-3% 1-2% 30-40% Total $287,000 Lost productivity of faculty/staff machines disconnected ?

7. Information Systems & Computing Why not rely on perimeter firewalls? Campus firewall not a panacea UniversityDate Netbios ports blocked # Windows machines # infected % infected Penn9/11/200311,0001,20011% Large state university7/28/200312,0001,50013% Ivy League peer1/2/200218,0003,14617%

8. Information Systems & Computing Freedom and Responsibility in personal computing Can we mandate desktop security practices? Do we reduce user freedom in in the name of security? Many in academia are accustomed to great freedom in managing their computers and software Freedom and autonomy are not necessarily at odds with security and legal risks –(with freedom comes responsibility) We needed security vision, complete with user education, balanced policy, technical support, and adequate funding to improve security

9. Information Systems & Computing Security Vision and Plan Vision Get the user community on board Make security the default wherever possible Promote achievable, affordable plans Treat security vision as an ongoing process with a two-year horizon Plan Develop defense-in-depth layered technical approach, including: –Prevention: Anti-virus, patching, secure configs, limited filtering / firewalling –Detection: Vulnerability scanning, IDS, server log reviews –Response: removal of compromised hosts from networks, limited network filtering, strong communications Develop policy, best practices, and end-user communication Communicate benefits to users and their IT support professionals Secure sustainable funding for the key initiatives

10. Information Systems & Computing Building Campus Consensus Risk Assessment to establish priorities –Security discussed informally among security, networking, other central and distributed IT organizations –General consensus on the layered approach Prevention: Patch management, anti-virus, education Detection: intrusion detection, vulnerability scanning Response: locating machines and incident management

11. Information Systems & Computing Building Campus Consensus Network Planning Taskforce –Security discussions over several meetings –Risk discussed in terms of prioritization and specific components of the layered approach –Specific components discussed Patching –Managed vs Unmanaged –Options: Windows, Software Update Server (SUS), Systems management server (SMS), commercial products e.g. HFNetChk Pro, PatchLink, BigFix –Special challenges patching students »Machines we don’t own but need participation from owners »Privacy issues

12. Information Systems & Computing Building Campus Consensus Network Planning Taskforce –Specific components discussed Virus filtering on mail servers –Campus-wide vs individual servers Firewalls/routing, VPNS, personal firewalls –Explained the concepts, terminology and how each work –Discussed pros and cons of different types of implementations Secure out of the box –Default images with strong authentication on truckload sale and Penn machines

13. Information Systems & Computing Building Campus Consensus Network Planning Taskforce –Specific components discussed Vulnerability scanning –Past results –Options to implement at a local level »Develop tools for local support providers –Options to implement on the network at a central level »IDS boxes, router flow logs Better ways to locate compromised and vulnerable machines –General agreement on security direction for Penn reached

14. Information Systems & Computing Building Campus Consensus Network Policy Committee –Worked to establish the policy needed to support the agreed to direction –Earlier had implemented standards for the most critical machines on campus –Recently approved: PennNet Computer Security Policy (Patch Management Policy) Critical updates to all campus connected systems must be applied within three business days or computer may be disconnected from the network Approved in June 2004; Implemented in September 2004

15. Information Systems & Computing Building Campus Consensus Patch Management Policy –Draft out to IT community for review before final approval –Comments and changes strengthened the policy –Discussed with University management to gain appropriate support as policy has broad impact –Communicated to the campus community through several types of communications University publications, newpaper, newsletter Presentations to several IT groups throughout campus

16. Information Systems & Computing Building Campus Consensus Summary –Many people were involved –Discussion at different levels within the organization –Education of community Prioritization using a layered approach What technology was available and how it worked Options and costs to consider –Input welcomed and incorporated into the solution –Participation from community resulted in best solution –Implementation plan developed

17. Information Systems & Computing Funding Funding the Implementation Plan –Costs of implementation were estimated Included costs across all of the central IT organizations, not just networking and security –Funding Source options considered –Start with the most likely Central University Funds NPTF – group that helps set annual user fees for network

18. Information Systems & Computing Funding Funding Options –University funding for central organizations severely constrained –NPTF liked the plan, hated the cost Funding for the schools constrained Looking for network costs to remain flat or go down Wanted a more secure network without additional costs Believed central university funding should pay for security or students should pay for their share of the burden

19. Information Systems & Computing Funding Students –Student behavior and computing support structure was a large cost driver –Bring unpatched, sometimes infected machines back to campus and plug into network –Support for undergraduates in residences provided by student residents not University employees –Limited or no support for Fraternities and Graduate students on campus, off-campus students bringing laptops on campus

20. Information Systems & Computing Funding Funding Proposal –Identified the portion of costs attributable to faculty and staff and separated it from costs attributable to students –Identified key executives financially responsible for student support –Developed presentation to educate execs about the need for security and the cost of delivering it Target audience was Business Administrators, not IT personnel Framed in terms of productivity loss of both end users and IT support personnel due to Blaster

21. Information Systems & Computing Funding Meetings to Look for Funding – Met individually with several key executives that dealt with students Agreed that money was needed and issues were valid No initial agreement on where it should come from Agreed to support a plan for funding to come from student fees via the organizations who collected the fees –Final meeting with all the key players Agreed on student funding for ongoing costs

22. Information Systems & Computing Funding Final Funding Sources Identified –Network Charge would include funding for faculty and staff –Student resident fee (not rent) would increase to cover undergraduates on campus –Fraternities would pay a surcharge for their network connections –Graduates and off campus funds would come from the Provost and/or central University funding –One time costs to implement would be paid for by central IT organization

23. Information Systems & Computing Funding Summary –Funding constraints made it impossible to receive all required funding from existing funding sources –Tension between responsibility for funding students and funding faculty/staff played an important role in final solution –Case for additional funding requests needed to show the benefit/added value the plan would deliver to those paying for it –Educating customers on those benefits is a critical success factor

24. Information Systems & Computing Implementation Challenges: 500+ LSPs, 30,000+ end users Leverage points: –PennConnect CD (Internet Connection Firewall) –Back-to-School Truckload Sale –Prizes and drawings to build awareness –Mass email, banner ads in Daily Pennsylvanian –Vulnerability scanning –Supporting patch management service

25. Information Systems & Computing Implementation Tasks –Evaluate firewalls –Communications plan –Secure Out of the Box – (Dell & IBM images) –PennConnect CD –Security awareness quiz (iPod giveaway) –Implement patch management service & supporting documentation –Contingency plans for router filtering Phased implementation: 8/04 – 9/04 - Communications and awareness 9/04 – 12/04 – Vulnerability scanning and “warning letters” 1/1/05 – Disconnect machines not in compliance

26. Information Systems & Computing Implementation Communications Plan –Identify target audiences (students vs. faculty/staff vs. LSPs) –Identify key messages (“enroll in patch management” vs. establish a patch management service for your users”) –Develop a “media plan” – target vehicles, dates, deadlines, etc.

27. Information Systems & Computing Media Plan

28. Information Systems & Computing Implementation

29. Information Systems & Computing Implementation

30. Information Systems & Computing Implementation

31. Information Systems & Computing Results We now have a program to keep systems secure, rather than dealing with everything as a “one-off” 5300 students/1000 faculty staff took the security quiz 72% fewer machines compromised Fall, 2004 vs. Fall, 2003 Overall sense of campus IT leadership that Fall, 2004 went a lot more smoothly than 2003, though we were also lucky

32. Information Systems & Computing Lessons Learned Present security initiatives as a business case. Measure the cost of poor security whenever you can. ROI’s sell projects. Gain support with appropriate discussions from operational management to executive level. Make strategic planning and budgeting processes transparent to your clients. It was truly amazing to be able to establish a campus consensus for mandatory security standards for all campus machines. Structured and consultative policy development, with a thorough vetting process, yields workable, enforceable policies with a high probability of changing behavior.

33. Information Systems & Computing Lessons Learned Don’t try to go too fast. Allow time to assimilate change. Funding models should drive costs back to their source; administrative units don’t like footing the bill for residential student security problems. Always coordinate end user communications with LSPs. Support large policy changes with a robust, targeted communications plan, supporting services and documentation.

34. Information Systems & Computing Lessons Learned Look for leverage points in developing strategies (patch management) and implementation plans (communications, incentives, mass communications).