2. Vulnerability Management at Scale
Intro: who I am, what I do, where I work.
Vulnerability Management at Scale
Alexandre Fiori
Production Engineer

3. Talk about landscape: different infrastructure, services, and products of each app; and their build/release systems.
Facebook: prod, facilities, corp, EE
WhatsApp, Messenger, Instagram: prod, mobile
Oculus: prod, mobile, embedded

4. Overview
What’s this about
The evolution of the Vulnerability Management program at Facebook, the birth of the Production Engineering team to support the program, and the pragmatic approach to build a highly scalable system for an ever-growing environment.

5. “Facebook was built on an open source stack
“Facebook was built on an open source stack. We support and encourage the use and development of open source software and hardware”
Talk about the pros and cons of open source with regards to security and vulnerabilities.
Pros: public security bulletins and patches
Cons: security information impacting business

6. Vulnerability Management Timeline
…2015: PCI DSS
2016: Internal scanning tools
2017: Broader network scanning
2018: Reboot, build up
2019: VMaaS
Take time to go through each bullet point and their short story.
“Payment Card Industry Data Security Standard”
2015: PCI requires a Vulnerability Management Program to exist
2016: vulnmetrics, aquilles CVE matching
2017: Extending network scanner infra to EE
2018: styx and new pipelines

7. Tell the story of 2018 onwards. How the reboot of the program happened.
Expand the program
Bootstrap new team from 2 ICs and 1 TPM reporting to director
2018

8. New challenge Scan all infrastructure Merge scanning technologies
Improve vulnerability matching
Start and track remediation
Manage remediation lifecycle
Design and build system to support all companies
XFN with teams working in silos, merging tech
Hire proprietary vulnerability database companies
Train security operations team to manage remediation

9. Architecture

10. Mindset Big data Scalability and reliability
On-line vs off-line pipelines
Concept validation and XFN
Fast prototype and launch
Volume: too much data everywhere; consolidation
How to scale the system vs people, add features, maintain SLA
On-line traumas from the detection pipelines: rollout, backfill
Batches are easier to set up, run, test, scale, and maintain (backfills, etc)
Validating concept through XFN, finding early adopters – talk to people

11. Talk about the concept of extract, transform, load and the parallels with the UNIX philosophy.
ETL

12. Collect Process Report
Talk about the concept of running batch-oriented pipelines, e.g. daily.
Collectors are stateless
Processors may be stateless or stateful (e.g. need vulndb, decorators for domain and first seen)
Reporters are stateful – need data from previous runs to proceed (e.g. escalate, cleanup)

13. Collectors
Inventory
applications
operating systems
hardware
Scan asset inventories and print to standard output

14. Processors Aggregate Scan Vulnerabilities Normalize
Pre-process, scan vulnerabilities, post-process

15. Reporters Validate Escalate Notify Track Cleanup Done
Reporters manage escalation lifecycle

16. Inventory Classes Hosted software Installed software Running software
Network scanners
Hardware
Network scanners category already existed but was not integrated
Hosted software and installed software came first

17. Vulnerability Database
Public vs Proprietary
Multi-vendor system
General purpose datasets
Specialized for ecosystem
Standard format for product->vulnerability matching
Proprietary: licensing, restricted use
General purpose databases: NVD
OS-oriented: Apple, RedHat (IBM?), Microsoft
Language / ecosystem: npm, pypi, NuGet
Specialized per ecosystem also includes first-party: our own CVEs

18. Implementation

19. Design Principles Command line tools do one thing
Communicate over a text interface
Core functionality shared as libraries
Composable code, tools, and pipelines
Rely on well established UNIX conventions

20. Industry Standard Technology
MITRE / NIST / NVD
Common Platform Enumeration
Common Vulnerabilities and Exposures
Common Weakness Enumeration
Common Vulnerability Scoring System

21. Infrastructure Tools Services Data warehouse Dashboards Notifications
Tools: our own, internal, open source, generic: grep, sort, jq
Services: Tupperware tailer, vulndb thrift service
Using the data warehouse to leverage orchestration, shared filesystem, persistent storage
Data warehouse provides operational dashboards
Our system’s data exposed to partners/customers via tables and dashboards
Notifications include: cases, tasks, alerts

22. What’s missing in this diagram:
Internal shenanigans for Tupperware and fbpkg
Post-processing and remediation pipelines
$repo2csv: maven, munki, choco, yum/repoquery

23. Internal Pipelines Tupperware Container images and packages
First-party vs third-party codebase
Bad Binary Hunter and Buck
Attribution from package to service
None of this is mentioned in the diagram slide.
Tupperware pipeline uses a tailer + batch job to query all schedulers
Another pipeline scan internal packages for traces of third-party dependencies
Internal packages are shipped to infra containing third-party code into it
BBH scans internal packages using Buck, report third-party dependencies
Internal packages are flagged with vulnerabilities from third-party dependencies

24. Vulnerability Database Tools
nvdsync and $vendor2nvd
vulndb command line tool
Uses NVD CVE JSON 1.0 format
Manages versioned datasets backed by MySQL
Supports vendor snapshots, custom CVEs, and snoozes
vulndb thrift service
CVE lookup and CPE matching
nvdsync is open source, an rsync-like for NVD datasets
$vendor2nvd download vendor databases and convert to NVD CVE JSON 1.0 format
vulndb stores NVD datasets
vulndb can be open sourced
Databases organized by vendor
Import vendor snapshots, export merged datasets with custom CVEs (custom edits/fixes)
Store snoozes used by post-processors (well, grep -vf) to snooze certain CVEs per provider and collector
Thrift service to support UIs and on-demand CVE matching (e.g. vulnquery, “linters”)

25. Decoration CWE and CVSS Domain and sub-domain First seen
Backlog vs influx
Owner (on-call ID or UNIX username)
Threat Intelligence
Decoration is post-processing.

26. Remediation Starts from CVE inventory Depends on decoration data
Supports feedback loop (e.g. snoozes)
Understands release cycles per inventory class
Manages escalation and lifecycle

27. Lessons Learned Normalization, aggregation, and blackholes
Per-customer decision trees are burdensome
Handling delays, XFN work, and fine tuning
Tasks and notification updates are annoying
False positives can compromise credibility

28. 2019: VMaaS

29. Goals Self-service system for vulnerability scan
Custom aggregation defined by collectors
Configurable providers and thresholds
Tier-based service, starting from bronze
Default dashboards and reports per tier
Common CVE inventory for all customers

30. Progress XFN partnership with early adopters
Migrated “hosted software” inventory class
Total of 10+ collectors in operation
Customers fixing bogus CVEs in the database
Snoozes effectively helping fine-tune reports

31. Next up Migrate other inventory classes
Improve data quality and detection speed
Tackle backlog via remediation campaigns
Tackle influx via push-blocking scans
Influence company culture outside security org
Migration: brainstorming graph / CPE attribution
Data quality: better ranking system and risk analysis
Campaigns to tackle backlog *and* emergencies
Influx: linters* or push-blocking tests that do scans
Influence culture: educate through bootcamp, support groups, campaigns

32. Thank you

33. Q&A