HUIT dns/dhcp redesign and roadmap Improved dns, right size IB, modern design, linux fallback

Redesign / roadmap aims Improved dns durability, reduce risk of recurrence of outage on 12/5 Update design to current best practice Project out development projects related to dns/dhcp

Current design issues Dns and dhcp are on common hardware (many IB boxes serve both dns and dhcp roles) Caching tier did not provide any durability, in fact, was the point of failure in design Box sprawl, too many Infoblox boxes, inefficient use of IB resources No alternative to IB in the event of a grid level failure On premise hosting of external dns view is a liability

Design proposal

New design overview Separate dns and dhcp services Move all internal dns services onto advance dns appliances (new) Replicate internal dns view onto standby linux bind servers Move external dns services from EOL appliances onto existing supported appliances. Move external dns for harvard.edu (but not child zones) to outside provider (eg: Akamai fastdns) Move all dhcp services onto existing appliances Enable dns query logging globally on the grid

New design consideration, recommended Buy advanced dns service boxes from IB Hardware refresh of IB gear due anyway (20 IB boxes are EOL 12/15) Advance dns boxes perform inspection prior to passing data to underlying bind, defending against emerging dns attack vectors (such as NXDOMAIN attack which is suspected as root cause of 12/5 outage) Build manual failover standby dns tier on linux vms Sync zone data via zone transfer Write and test failover plan Manual failover must be entirely programmatic, no datacenter work or on site presence Enable dns query and response logging Has an estimated 20% performance hit to max dns query rate Recommended by IB for forensic purposes – would have helped identify root cause of 12/5 outage Desirable to HUIT security office Significantly simpler to enable inside IB than to build outside of IB with either network taps or bind resolver layer

New design consideration, recommended Consolidate dhcp and dns onto separate dedicated IB boxes Mechanically simpler design, improving diagnostic capabilities Can reduce the overall number of IB boxes in use, resulting in cost savings over like for like replacements using existing design Vendor provided hosting of external dns view Providers like verisign or akamai better capable of handling dns DOS attacks External providers offer dnssec signing service – they manage key rotation Desire for management integration with on premise internal dns Provide IB dns firewall service Subscription to dns RBL service, permits rewriting dns responses for malicious sites Has passive mode, where no dns responses are rewritten, but events are logged

New design by the numbers Total device count of 17 IB, 3 linux (current count is 29 IB) 6x pt 1400 dns recursive resolvers (3 HA pairs), net new 2x 2220 grid manager (1 HA pair), lifecycle replacement 4x 2220 dhcp servers (2 HA pairs), existing hardware 4x 2220/2210 external dns non-recursive resolvers (2 HA pairs), existing hardware 1x 4000 reporting server (non-HA), existing hardware 3x stock linux vms noc vmware for standby internal dns

Improvements – why this is better Advance dns boxes address outage due to malicious queries Logging addresses HUIT security needs and IB support needs Sizing is based off of current utilization, right amount of hardware, cost conscientious Remains a single IB grid, simplest management Failover linux dns tier exists, failover is a manual process, requires no dns client reconfiguration Off site hosting of external view provides defense against external view dns attacks

Budget impact – in budget for capital, new ongoing $47k/yr for support/services Net new equipment: 6x IB pt 1400 (dns resolver), 2x IB 2220 (grid manager), 1x IB 2220 (test grid) IB capital cost $281k (does not include support) 6x Pt $30k = $150k 3x $34k = $102k 2 nd PSU and gbics for 1400s $13k 6x dns firewall license $16k Current FY and next capital budget are $295k, proposal is in budget for capital Support cost for new hardware partially offset by retiring EOL equipment Dns firewall 3 year license cost $52k – net new support costs ~$17k/yr Akamai external view hosting for three zones is $30k/yr, plus $4k one time setup cost this is net new support cost

Future roadmap items Reevaluate manual failover process There is concern over a manual failover process – will failover work when we need it, and will the data be up to date? Consider mixing in linux bind tier to live dns services Evaluate live mix of IB and stock linux bind Possible complications around logging or dns firewall Evaluate anycast as an appropriate technology on the network Could further reduce IB investments (would not need HA paired dns) Evaluate new IB offering for cloud ipam Grid-managed device that does ip address management for AWS (replacing native AWS dhcp)