Presentation on theme: "Systems Management in an Untrusted Network"— Presentation transcript:
1 Systems Management in an Untrusted Network Dealing with backups, monitoring, administration, and logging in the DMZCory L. Scott, Lead Security ConsultantSecurify, Inc.
2 IntroductionImplementing systems management components in untrusted or semi-trusted networks is difficult……if you are concerned about security!Outline of today’s talk:Example of threatsDMZ Network Architectures in the Real World™Two core designs and advanced design issuesSystem and network configuration for systems management
3 The threat is out there… SNMPMultiple Vendor SNMP World Writeable Community VulnerabilityNAI Sniffer Agent SNMP Buffer Overflow VulnerabilitySniffersMicrosoft Network Monitor Multiple Buffer Overflow VulnerabilitiesSolaris snoop (print_domain_name) Buffer Overflow VulnerabilitySpecific Vulnerability Titles courtesy of SecurityFocus
4 The threat is out there… Remote Control Software (besides its intended functionality)AT&T VNC Weak Authentication VulnerabilityPCAnywhere32 Denial of Service VulnerabilityAdministrative Interfaces (over intended functional protocols)Allaire ColdFusion Server Administrator Login Password DoS VulnerabilityCisco 7xx Series Router DoS VulnerabilityCisco 675 Web Administration Denial of Service VulnerabilitySpecific Vulnerability Titles courtesy of SecurityFocus
5 The threat is out there… System loggingAge-old attacks:log floodlog eraseselective log editLinux syslogd Denial of Service VulnerabilitySolaris syslogd Unresolvable Address Remote Denial of Service VulnerabilityBackupUnauthorized restore/delete, unencrypted backupsVeritas Backup Denial of Service VulnerabilitySpecific Vulnerability Titles courtesy of SecurityFocus
6 The Purpose of the DMZBut… I’m filtering System Management Protocols at the perimeter! Isn’t that enough?No.Why?Two words: Aggravated Penetration.Want two more? Privilege Escalation.More? Insider attack.DMZ hosts are bastion hosts or perimeter service hosts. Why do we spend all this time hardening our DNS servers and then leave a poor password on the ssh service listening on an untrusted interface?
7 The Purpose of the DMZThe DMZ exists to mitigate risk by isolating certain services and functions in a separate segment of the network.Segmentation by isolation is generally not enough. Ingress filtering is usually deployed, but typical designs need work past the “crunchy” outside layer. Defense in depth, along with proper protection of internal hosts from the DMZ, is required.
8 The Purpose of the DMZExample 1. Bastion hosts in a DMZ Segment.
9 The Purpose of the DMZExample 2. Perimeter service hosts in a flat network.
10 The Purpose of the DMZIf it were only that simple…
11 The Purpose of the DMZExample 3. Segmented DMZs.
12 The Purpose of the DMZExample 4. Colocated DMZs.
14 The Purpose of the DMZ Other problems in the DMZ Constant change Too many hands in the potService protocols not designed with security in mindSystems management protocols not designed with security in mindScalability mechanisms create additional separation and obfuscation of a clean network designCollusion of disparate types of traffic going through the DMZ
15 System Management – Composite Sketch Common sighting – the status quo:No centralized logging.SSH inbound from the internal network; often from external network, too.PCAnywhere, VNC, or SMS accessible from some management hosts or worse...Backup system non-existent or backups batch copied to internal hosts.Default administrative protocols and interfaces left accessible within the DMZ and from the internal network:SNMP on routers, web interfaces on serversOpenview or other monitoring system “pinging” from the inside.
16 Dealing with the problem… Securing Systems Management components requires a combination of network architecture and system configuration.
18 DMZ Network Architecture for System Management Example 1. Combination of Management and Production Traffic on the same untrusted segment.Definition of untrusted segment: Where untrusted users and/or processes can place packets on the segment.Advantages:Simple to manage, do not have deal with multiple interfacesEasier firewall rulesets and router ACLs to manage
19 DMZ Network Architecture for System Management Example 1. Combination of Management and Production Traffic on the same untrusted wire.Disadvantages:Bandwidth utilizationFailure to segment different types of traffic introduces security risksMust place loghosts, monitoring consoles, and control components on the internal network to keep isolationHarder to monitor for policy violationsUntrusted segment behind firewall will advertise management servicesFor services that listen for input, must configure host-based inclusion rather than interface/network inclusionCompromised host on segment could spoof management connection
20 DMZ Network Architecture for System Management Example 2. Separate Management LAN.Advantages:Protects bandwidth on untrusted network segmentIntroduces another hurdle for intruders to jump interfaces, which can be locked down more aggressivelyAbility to monitor for violations in both segments improvedCan place loghosts, monitoring hosts, and control components in management LAN with less risk, reducing internal network exposure and relianceAllows for more flexibility with private address space and less border firewall concerns
21 DMZ Network Architecture for System Management Example 2. Separate Management LAN.Disadvantages:Need to make sure that forwarding is disabled; routing must be configured correctly on each host; additional configuration and equipment neededManagement LAN can still be used as a conduit to attack hosts if not properly secured and monitoredAdds complexity to segmented DMZs and potential bypass mechanism between segments
23 DMZ Network Architecture for System Management Example 3. Management Aggregation Points based on natural segregation of the segmented DMZ.Advantages:Works well in segmented DMZsReduces management LAN bandwidthAll of the advantages of segmented DMZsDisadvantages:More equipment and more routesNeed to maintain ACLs and rulesets between Management LANsAdditional points of failureAll of the disadvantages of segmented DMZs
24 DMZ Network Architecture for System Management Example 4. Pushing data versus pulling data.Pushing data from internal network to the DMZ/Admin LAN. Good – but how much do you trust your internal users?DMZ/Admin LAN pulling data from the internal network. Bad.Degrees of push:File / Data one-way with or without validationInteractive transfer with restricted privilegeRemote control administration with full interactivityUse the minimum amount of push whenever possible.When DMZ hosts need to push data for administrative purposes, aggregate in the same trust boundary. Then pull from a more trusted environment.Never have DMZ hosts pull or push from the Internet without appropriate risk analysis and mitigation.
25 The Need for Systems Management BackupDiagnostic information and availability monitoringRemote administrationSystem logging
26 Backup Solutions Risks Bandwidth utilization Unauthorized restore / backupCapture of backup trafficAgent vulnerabilities – authenticationProcedures for restore offsiteLocal backup devices unmanageable or difficult to scaleBackup clients not necessarily designed with security in mind
27 Backup Solutions Securing Backup Solutions Protect the backup server at all costsPlace behind another firewall / filterBackup server should initiate all backup / restore requests to eliminate inbound connectionsConsider the physical security of the server and the mediaImplement tight security controls on server.Encryption – examine the risks / benefitsIs the wire insecure? If so, client has burden of encrypting the data.Store the data encrypted or not? How is key management performed? What happens if the key is lost?Encrypt both on-site and off-site media?
28 Backup Solutions Securing Backup Solutions Administrative LAN segment very beneficial for backup solutionsImplementing a Storage Area Network may provide another means for backup that doesn’t use the LANOne example of a hard-to-secure product:Legato:Server uses default ports /TCP&UDPClient uses default ports /TCP&UDPRuns its own portmapperPorts can be restrictedAuthentication client/server unclearNAT not supportedUnable to determine which interface it listens on
29 Monitoring Solutions SNMP Assume that anything sent over SNMP is readable by all.Community strings should be changed.If possible, limit the hosts that can query SNMP on the queried device itself.Examine the type of information that your device gives via SNMP – it may surprise you.Determine the criticality of the information when deciding whether or not to use SNMP.Never allow reconfiguration of devices via SNMP. Disable write privileges on any SNMP device.Traps should be used sparingly and there should be a dedicated receiver in the DMZ.NT SNMP giveaway.Oftentimes, it is the lesser of another evil.
30 Monitoring SolutionsICMPEcho reply/request is fine on an internal interface.If possible, throttle your ICMP response queue.
31 Remote Administration Solutions Console accessTarget devices include: routers, switches, load balancers, some UPSes, firewalls, and serversAggregate console connections into a terminal serverCan use a hardware terminal server with a serial or network interface to a PC that maintains accessAlternatively, many newer terminal servers support direct network connections via SSH, with RADIUS support and IP filteringCan also connect out-of-band via dial-up modem with callback feature
32 Remote Administration Solutions Console accessAdvantages:Console messages can be logged to a terminal serverCentral point of authentication into console managementProvides the ability to turn off telnet and other administrative clear-text protocols on network equipmentIf ssh or other interactive interface fails to respond, administrator can directly connect to console without physically going to the DMZ
33 Remote Administration Solutions Console accessDisadvantages:The unintentional <BREAK> problemAdditional hardware and cablingAuthentication and logging for console use (once the user has accessed the terminal server) is difficult to implement with a hardware device
34 Remote Administration Solutions SSH bastion gatewayOne (hardened) point of entry via SSH to other hostsCan use ssh-agent to eliminate interactivity on the gateway, while maintaining only a single host that can SSH to the endpointsUse RSA identity filesDisable password authenticationDisable rhosts authentication and root loginBind ssh only to admin LAN interfaceWatch your patch levels – ssh is a popular target
35 Remote Administration Solutions Windows GUI – 2 popular options:PCAnywhereWindows Terminal Services
36 Remote Administration Solutions Windows GUI – PCAnywhereRisksRuns on well-known port – juicy target for attackersPrevious versions have been vulnerable to DoS attacks and weak password encryptionTypical configuration binds to all interfacesShould avoid exposing on an untrusted network segmentTypical configuration bypasses Windows login mechanism
37 Remote Administration Solutions Windows GUI – PCAnywhereSecuring PCAnywhereMake use of the allowed IP addresses feature – limit admin hostsEnable TCPIPHostBindMode to only listen on admin interfaceChange default portMake sure the Windows NT user is logged off after session disconnect (normal and abnormal)Enable event logging and session recording (if disk space permits)Utilize Symmetric encryption / Deny lower-levelIf possible, use X.509 for host authenticationDisable response to PCAnywhere query broadcastsConfigure clients to only use TCP to connect (rather than a UDP query – reduces firewall ruleset)Use separate user account for each admin with strong passwordsLimit login attemptsOnly use PCAnywhere user with PCAnywhere privileges
38 Remote Administration Solutions Windows GUI – Terminal ServicesRisksUtilizes Windows authentication methodRuns on a well-known portShould avoid exposing on an untrusted network segment
39 Remote Administration Solutions Windows GUI – Terminal ServicesSecuring WTS (for administration use)Bind only to the administrative segment interfaceForce all configuration parameters at the server levelUse a separate WTS login from Windows login and give each administrator unique login with strong passwordTake Administrators group out of connection permissionsEnable security auditingRemove TsInternetUser accountUtilize High Encryption for RDPDisconnect idle/broken connections aggressivelyFor those who are paranoid, change the WTS port.
40 The system management need is to centralize logs for analysis. Logging solutionsLog typesThe system management need is to centralize logs for analysis.TypeModeSyslog – UNIX and Network DevicesWrite to local filesystem or send over network (UDP 514)Windows NT/ Event LogsWrite to local filesystem (network support for syslog available from 3rd parties)Application / Service Log FilesSyslog, NT Event Log, flat file, binary file, database entry
41 Logging solutions Network syslog If possible, limit which machines can send log entries to a host.Heartbeat creation and detection is absolutely imperative.Flood detection is also imperative.Syslog servers should sit on administrative LANs if at all possible.Make sure that clients are sending the messages over the administrative LAN interface.Initiatives are out there for secure syslog – not close to implementation yet:Log signingEncrypted transferInsertion / deletion attacksTake a look at syslog-ng.
42 Logging solutions NT/2000 Event Log Need to get those logs off each server posthaste.Two major options:Agent-based forwardingSyslogCommercial solutionsBatch retrievalCan use common resource kit utilities to pull logs out in binary and text formatTo push or to pull?If log is cleared by an intruder, you better know about it! (Use a perl script to check for Event ID 517.)See my SANS NetSec 2000 presentation for many more details!
43 Logging solutions Flat file logs Can always “syslog” them tail -f /var/log/mylog | loggerOk… maybe not! Need to get them off of the originator as soon as possible.If they are too big and/or cumbersome, consider culling them during the push or pull process.How often to push/pull? Determine the criticality of the logs and analyze the worst case scenario: where the attacker blows away the local copy of the log file and your mission is to figure out what happened.When log disappears, you better know about it!
44 General tipsWatch out for administrative interfaces.Follow best practices, especially in regards to:Resource utilizationSegmentationAuthenticationIntegrity