1. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Brian Martin Jake Kouns

2. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Overview Inherent Problems Important Issues Major Players Research and Rankings Future

3. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Overview

4. © Open Security Foundation 2005 Vulnerability Database Overview What is a Vulnerability Database (VDB)? Database of information on security vulnerabilities. Simple! What about dictionaries (CVE) or searchable indexes VDB! Key is realizing VDBs will have their focus –Comprehensive Vulnerability Database –Focused Vulnerability Database –Vulnerability Notification Services –Value Added Services

5. © Open Security Foundation 2005 Brief History First VDBs were private, mostly maintained by hackers or budding security geeks (before security professionals were common) First public database? –Unix Known Problem List –Internal Sun Microsystems Bug List –Early CERT database VDBs abandoned (Fyodor), sold to corporations (BID), or home grown (X-Force) Additional VDBs continued to be launched to meet different demands (Secunia, OSVDB)

6. © Open Security Foundation 2005 Basics of a VDB Vulnerability information gathered Identification number/name assigned Adherence to standard format Ability to search and display data Optional: Mail lists (private or public) Exports for integration Other services

7. © Open Security Foundation 2005 Purposes of a VDB Provide accurate information on security vulnerabilities Provide historic reference on software bugs Provide information on solutions Provide innovations to help organizations deal with vulnerabilities But are they?

8. © Open Security Foundation 2005 WIIFM – Whats in it for me? Alerting/Notification –Information provided in timely fashion Detailed Content –Concise description, additional analysis, references Organized Information –Vulnerability statistics –Trending –Historical context

9. © Open Security Foundation 2005 Vulnerabilities Trends CERT Vulnerability Counts ( 1995-2004)

10. © Open Security Foundation 2005 Who uses a VDB? Administrators Auditors Security Testers –Penetration Testing –Vulnerability Assessments –Risk Management Criminals –Hackers, Crackers, Blackhats, Greyhats, OH MY!

11. © Open Security Foundation 2005 Legalities and Liability Issues with disclosure –Bug finder and irresponsible disclosure –Do VDBs have a responsibility to be ethical for bug finders? Liability for providing information –Liability for including exploit code? Copyrights on information –Including unedited original source? –Re-branding or re-writing? Confusing lawsuits –Tegam vs. Guillaume Tena (France) –Sybase vs. NGSS? (US) –HP vs. NGSS? (US)

12. © Open Security Foundation 2005 VDB Sociology VDBs are taken for granted by users Users need them but do not appreciate Users rely on a VDB for 'thoroughness', when they usually are not Users quote VDB information as gospel, as if VDBs confirm and validate every entry Users typically have favorite VDB, and only use that one

13. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Inherent Problems

14. © Open Security Foundation 2005 Inherent Problems with VDBs Dependency –If no entry for Product X, assumption it is secure –Assume information is accurate, becomes gospel –Rely on VDB to alert you? Lack of Updates –Hard to update old entries (why dont new players care about old entries?) –Solutions not there or not fully updated –Workarounds not accurate or helpful Thoroughness –multiple entries –No digging for details –Ignoring obscure products

15. © Open Security Foundation 2005 Lack of standard –Naming conventions –multiples vs. breaking out entries –What deserves an entry at all Accuracy and Integrity –Who updates? What motivation to be accurate? –Myth/Fake –Why is the information inaccurate? Poorly written advisory, Lousy research Poor vendor communication/verification –Why do VDBs trust anything and everything they read? Number of database entries matter Inherent Problems with VDBs

16. © Open Security Foundation 2005 Inherent Problems with VDBs Pros & Cons of adding entries –Fast No external references Incomplete or inaccurate information –Slow Not timely like many people want Statistics & Metrics –Lack of classification (leads to problems) –Lack of severity (debate unto itself) Not only based on remote vs. local … Availability of exploit Impact of exploit Installation base of software

17. © Open Security Foundation 2005 Inherent Problems with VDBs Relying on Bug finders –Double edge sword, good bug hunters provide great information, many do not –Vulns being reported although previously disclosed –Not including versions or vendor site, and not easily Google'd –Vague information, untested –Advisories without dates (big vendors especially guilty.. MS, IBM, Novell, Sun, HP) –People try to use bug finding as a way to advertise their security services

18. © Open Security Foundation 2005 Inherent Problems with VDBs What else? –Many dont make database easily available in full or not portable –Dont support third party utilities and use –VDB snobs, refuse to reference certain other databases or sources –Narrow focus on where to find vulnerability information (life outside Bugtraq) –Often dont give credit where due –[…]

19. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Important Issues

20. © Open Security Foundation 2005 Important Issues for VDBs Most issues are easily overlooked 7 key issues for a VDB to address User Dependency Content Updates Content Depth Standards Accuracy and Integrity Statistics and Metrics Integration Ability

21. © Open Security Foundation 2005 User Dependency Can you rely on a VDB? Do you verify the VDBs statements? Do you read into the information and make assumptions? Rely on VDB to alert you?

22. © Open Security Foundation 2005 Content Updates Turnaround on new entries Older entries need attention –Updated external references –Updated solutions –Updated information on risk ratings Do all VDBs care about older entries? Corrections to entries

23. © Open Security Foundation 2005 Content Depth Number of entries –Catalogue all vulnerabilities or just major issues Vague information on vulnerabilities –Often due to poor research or vendor not providing details (thus, external references are important) Effort to correlate or research –Weeding out duplicate entries Types of products cataloged –Not just about Windows and Unix anymore

24. © Open Security Foundation 2005 Standards Definition of a Vulnerability Naming Conventions Dates Write-ups Risk Ratings Solutions

25. © Open Security Foundation 2005 Accuracy and Integrity Who maintains the data How are updates justified Motivation for entries Motivation for accuracy

26. © Open Security Foundation 2005 Statistics and Metrics How many entries exist? How many entries are missing? How do we know? How many entries have solutions? How many are critical? How many vulns per month/year? How many vulns per vendor/product?

27. © Open Security Foundation 2005 Integration Ability Can users change or ask for updates Is the data easy to obtain Does the VDB support 3 rd parties Does the VDB reference all information Can users dynamically pull information

28. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Major Players

29. © Open Security Foundation 2005 Major Players Comprehensive VDBs –BID - http://www.securityfocus.com/bid –CVE - http://www.cve.mitre.org/ –ISS X-Force - http://xforce.iss.net/ –OSVDB – http://www.osvdb.org/ –Secunia - http://www.secunia.com/ –Security Tracker - http://www.securitytracker.com/ Vulnerability Notification Services –CERT - http://www.cert.org/ –CIAC Advisory - http://www.ciac.org/ciac/index.html Value Added Services –ICAT - http://icat.nist.gov/icat.cfm

30. © Open Security Foundation 2005 BID Started in 1999, acquired by SecurityFocus on 07/17/2002 Full time dedicated resources Free, 72 hour delayed information (SF researched)

31. © Open Security Foundation 2005 BID – Pros/Cons Pros –Brand awareness –Very detailed and technical information provided –Quick posting of new vulnerabilities due to hosting of Bugtraq mail list Cons –Practices changed once acquired by corporation –Little response to feedback provided –Slow to load, banners ads a pain, 39 images per entry –Product information based on erroneous assumptions

32. © Open Security Foundation 2005 CVE/ICAT MITRE and NIST Full time dedicated resources, federal funding CVE started in 1999, ICAT ~2000 Both claim not to be a VDB ICAT adds vulnerability classification and statistics to a predominantly CVE based database Free

33. © Open Security Foundation 2005 CVE/ICAT – Pros/Cons Pros –Detailed statistics and classification scheme –Easy ability to download entire database –Widely adopted, heavily integrated into security products Cons –Heavy use of CVE for vulnerability information –CVE candidate process slow and backlogged –Limited external references (ICAT)

34. © Open Security Foundation 2005 ISS X-Force Run by Internet Security System (ISS) Full time resources dedicated Started around Aug, 1997 VDB is free and public Heavily used and referenced in ISS security products Fast and courteous reply to emails with questions or errors

35. © Open Security Foundation 2005 ISS X-Force – Pros/Cons Pros –Very detailed, very thorough, historical entries –Fairly standard naming conventions –Very thorough external references Cons –Disclosure Issues –Many entries related to IDS events, not classic vulnerabilities –No easy export, cant easily integrate

36. © Open Security Foundation 2005 OSVDB Open Security Foundation, 501(c)3 non-profit organization 3 project leaders, over 200 volunteers since inception First started on 08/30/2002 Free security information Security community driven Vendor dictionary, ethical disclosure service, active integration

37. © Open Security Foundation 2005 OSVDB – Pros/Cons Pros –Vendor Neutral, Un-biased –Integration with open source products –Broad source for data importation (sources, dates) –Very thorough, attention to detail, historical entries Cons –Slow updates on new vulnerabilities –Relies on community for resources –Currently no long term funding

38. © Open Security Foundation 2005 Secunia Corporation located in Denmark Full time staff Launched 03/26/2003 Focus on timely vulnerability alerts Free mailing list of new vulns mailed daily

39. © Open Security Foundation 2005 Secunia – Pros/Cons Pros –Free mailing list –Very strong on monitoring vendor advisories and updates –Attempt to work with open source community Cons –Lack of standards/confusing standards Issues lumped into multiple entries Same vulnerability assigned a dozen entries, one per linux vendor –Only focuses on new vulnerabilities –Some solutions not practical or helpful

40. © Open Security Foundation 2005 Security Tracker Corporation in MD, USA Full time resources dedicated Started in 2002 Free weekly summary of vulnerabilities, fee for instant alerts

41. © Open Security Foundation 2005 Security Tracker – Pros/Cons Pros –Maintain their own standards, uniform entries –Includes data source for vulnerability –Good data importation, monitor broad source of information Cons –No statistics –Limited external references

42. © Open Security Foundation 2005 CERT Carnegie Mellon, funded by US government Full time staff dedicated Started in 1988, after Morris worm Advisories for important issues Maintains CERT-VU/KB Database National Cyber Alert System

43. © Open Security Foundation 2005 CERT – Pros/Cons Pros –US Federally funded and supported –Providing reports to technical and non-technical –Statistics provided Cons –Limited vulnerabilities tracked –Provide early information for exorbitant fee –Not always willing to coordinate with security community –Serious questions about statistics, efficiency of staff/funds –Overlap with CIAC and others

44. © Open Security Foundation 2005 CIAC US funded and supported, DOE Full time dedicated resources Started in 1989 Advisories for major issues Free service

45. © Open Security Foundation 2005 CIAC – Pros/Cons Pros –Stability, around since 1989 –Updated regularly Cons –Limited vulnerabilities covered –Limited external references –Many advisories reprinted, no value added –Overlap with CERT

46. © Open Security Foundation 2005 Additional Resources Vulnerability Sources Not Included: –COOP = https://cirdb.cerias.purdue.edu/coopvdb/public/ –Dragonsoft - http://vdb.dragonsoft.com/ –FrSIRT - http://www.frsirt.com/english/index.php –Securiteam - http://www.securiteam.com/ –Sec Watch- http://www.secwatch.org/ Focused Vulnerability Database –Nikto, Nessus –Sun, HP, IBM, Oracle, Microsoft, etc Vulnerability Sharing Clubs –http://www.idefense.com/ –http://www.immunitysec.com

47. © Open Security Foundation 2005 Government Funded CERT –The CERT/CC is funded primarily by the U.S. Department of Defense and the Department of Homeland Security, along with a number of other federal civil agencies. Other funding comes from the private sector. As part of the Software Engineering Institute, we receive some funds from the primary sponsor of the SEI, the Office of the Under Secretary of Defense for Acquisition and Technology. CIAC –U.S. Department of Energy (DOE) funded CVE –CVE is sponsored by the National Cyber Security Division (NCSD) at the U.S. Department of Homeland Security. US-CERT is the operational arm of the NCSD. ICAT –ICAT is maintained by the National Institute of Standards and Technology. US-CERT –US-CERT is part of the Department of Homeland Security Little overlap? Consolidation? Oversight and audit?

48. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Research and Rankings

49. © Open Security Foundation 2005 Data Harvesting Where is information usually gathered? –Mail lists (Bugtraq, Full-disclosure, Vulnwatch, Ntbugtraq) –Vendors (advisories) Where else should information be gathered? –Mail lists (Freshmeat, Vuln-dev, Dailydave, Pen-test, other specialty security focused lists) –Vendors (Changelogs, Knowledge bases, Vendor forums) –Exploit archives

50. © Open Security Foundation 2005 VDB Incest Who references who? Who refuses? –CVE: ISS, BID, Secunia, SecurityTracker, OSVDB –BID: CVE, Bugtraq, ISS, Secunia, SecurityTracker, OSVDB –ISS: CVE, BID, Secunia, SecurityTracker, OSVDB –Secunia: CVE, OSVDB –SecurityTracker: CVE, OSVDB, Nessus –Nessus: CVE, BID, OSVDB –OSVDB: CVE, BID, Secunia, SecurityTracker, ISS, Nessus, Snort, more Red denotes an apparent refusal to reference, even if the original point of disclosure or only available source.

51. © Open Security Foundation 2005 VDB Ratings Based on important issues identified Score of 1-10 provided for each of the 7 key performance areas 1 = lowest, 10 = highest Ratings given for each issue per VDB Provides baseline for expectations for each service Identifies areas of improvements

52. © Open Security Foundation 2005 VDB Individual Rankings Ratings For Each Category Top 3 VDBs Top 3 Areas for VDB Improvement See research posted at: http://www.opensecurityfoundation.org

53. © Open Security Foundation 2005 Vulnerability Databases: Everything is Vulnerable Future

54. © Open Security Foundation 2005 Future of VDBs Long way to go Hope to improve existing resources –Better search interfaces –Better upkeep of older entries More services available to more people Further integration into products Better statistics and trending

55. © Open Security Foundation 2005 Standardization of Definitions Risk ratings Vulnerability Classifications –Local vs. Remote (Remote Local) –Impact assessment (CIA) –Exploit availability –Access required to exploit (Dependencies) Vulnerability definitions and terminology

56. © Open Security Foundation 2005 VDBs Suck - Expect More 20 years since inception, Limited improvements Same mechanism for updating/verifying info Very few classify or assign risk Still no standardized classification for the few who do Still no standardized risk value for the few who do Still offer limited search ability overall Many don't follow their own standards consistently Most still very weak on external references Barely any new services or ways to use information Many don't seem to care about the vuln disclosure process (why did it take 20 years for a vendor dict to emerge?) Bottom line, VDBs need to drastically improve

57. © Open Security Foundation 2005 Open Security Foundation Vulnerability Databases: Everything is Vulnerable Brian Martin – jericho@attrition.org Jake Kouns – jkouns@opensecurityfoundation.org