1. Does Your Time to First Byte Bite?
Collin Woo Enterprise Solutions Engineering
Does Your Time to First Byte Bite?
Data Connectors San Francisco

2. Oracle Confidential – Internal/Restricted/Highly Restricted

3. Two Protocols to Run the Internet
DNS query resolution is the first step in loading a web page with each new page triggering multiple DNS lookups
BGP = Border Gateway Protocol. Very complex protocol that is crucial in advertising where infrastructure clusters live and how to connect with them. Because it is a “border” protocol BGP is concerned with routing between autonomous systems…..not routing within them.
Milliseconds count.
Web pages can have 20, 30 or even 50 DNS lookups
Images, video, audio, ads, social media buttons
100ms difference per object = 2 or more second delay in loading the page
URL:
Host: tapestry.tapad.com
IP:
Error/Status Code: 200
Client Port: 63187
Request Start: s
DNS Lookup: 388 ms
Oracle Confidential – Internal/Restricted/Highly Restricted 3

4. Authoritative and Recursive DNS
Oracle + Dyn is Authoritative DNS not Recursive DNS.
Authoritative DNS serves those that are outside of your firewall
Looking to purchase products/services from your website
Remote employees that connect via VPN
Partners that need to access a portal to process orders with you
Recursive is filtering people inside of your firewall
Authoritative and Recursive DNS
partners.example.com
shop.example.com
vpn.example.com
Oracle Confidential – Internal/Restricted/Highly Restricted 4

5. First in the Chain Initial Connection Content DNS Lookup TTFB Back-end
Front-end
Initial Connection
- Network: global perspective metrics
- BGP: routing changes and reachability
- Providers: market performance analysis
- Prefix: monitoring and alerting BGP performance
Content
-CDN: latency optimization and vendor diversity
- Geo: planning for geographic reach
- Reach: provider reachability alerts
DNS Lookup
- Query: always available answers
- Trace: DNS query hierarchy
- Server: authoritative or caching name servers
- DNSSEC: keychain validation
TTFB
- Geolocation: reduce latency & hops
- Failure routing: only route to live site
- Security: ensure route to server is secure
"First in the Chain" - changed the end item on the right to be a grouping of devices. There were complaints that the original Akamai pull made it seem like DNS was not a big enough part to worry about, and that content was a larger slice
Oracle Confidential – Internal/Restricted/Highly Restricted 5

6. First in the Chain Matters
5 DNS lookups for linkedin.com.
The first 3 items in the waterfall are DNS
LinkedIn uses Dyn for DNS and sees resolution times of 5ms each for the first two lookups
Compare that to netflix.com on AWS Route 53 and the first two lookups take 72ms and 145ms
First in the Chain Matters
DNS query resolution is the first step in loading a web page with each new page triggering multiple DNS lookups
Milliseconds count.
Web pages can have 20, 30 or even 50 DNS lookups
Images, video, audio, ads, social media buttons
100ms difference per object = 2 or more second delay in loading the page
URL:
Host: tapestry.tapad.com
IP:
Error/Status Code: 200
Client Port: 63187
Request Start: s
DNS Lookup: 388 ms
Oracle Confidential – Internal/Restricted/Highly Restricted 6

7. DNS Configurations

8. Primary Cloud DNS Users Recursives Primary 1.1.1.1 1.1.1.1 APM
Example.com?
Example.com?
Oracle Confidential – Internal/Restricted/Highly Restricted 8

9. Primary DNS CON’s PRO’s Still a single point of failure
Faster resolution times
No on prem expense
Use of Dyn’s NOC for DDOS mitigation
APM
Oracle Confidential – Internal/Restricted/Highly Restricted 9

10. Secondary DNS Primary/Master Users Recursives Secondary* 1.1.1.1
Everyone is in Delegation
Primary = Manages the zone, gives updates
Secondary = Only receives updates from primary
Primary/Master
APM
Users
Recursives
Example.com?
Notify via AXFR/IXFR
Example.com?
Example.com?
Secondary*
*This is where that confusing secondary term comes from.
Oracle Confidential – Internal/Restricted/Highly Restricted 10

11. Secondary DNS PRO’s CON’s Multiple Vendors for Resiliency
Fastest Responder Wins
Extremely easy to set up
Use of Dyn’s NOC for DDOS mitigation
APM
CON’s
If primary goes down, no changing records
Not all vendors support AXFR and/or IXFR
Not all vendors support NOTIFY
Advanced intelligent routing schemes cannot be replicated
Oracle Confidential – Internal/Restricted/Highly Restricted 11

12. Hidden Master Hidden Master Data! Users Recursives Authoritative
How it works:
The PRIMARY is on the side of the customer, outside the delegation. The SECONDARY is Dyn which receives updates just like a normal primary - secondary.
Hidden Master
APM
Data!
Users
Recursives
Authoritative
Example.com?
Example.com?
Oracle Confidential – Internal/Restricted/Highly Restricted 12

13. Hidden Master PRO’s CON’s Works great with in-house solutions
Extremely easy to set up
Dyn handles
Zero day attacks
Performance and scale
DDoS protection
CON’s
Not the master server
Responsible for zone management
APM
Oracle Confidential – Internal/Restricted/Highly Restricted 13

14. Security

15. Protection Stack Summary
Upstream Transit Filtration
Network layer attacks (layers 3 & 4)
UDP floods, Syn attacks and ICMP
Bandwidth & Authoritative DNS servers absorb
> 80% of all attacks reported are here
Session layer attacks (layers 5 & 6)
DNS floods and SSL floods
Signature based Filtration methods
Application layer attack (layer 7)
GET floods, SQLi and CSRF
Market Alerts
(BGP alerting on top competitors)
Targeted application attacks (layer 7)
Dyn DDoS Alerts (validate layer 7 DDoS service is advertising routes)
< 20% of all attacks reported are here
Many businesses lease between 1-10Gbps
59% of attacks are over 1Gbps
Neustar: 2015 data on DDoS attacks, found that 73% of global brands and organizations were attacked. More than half reported theft, either of customer data, financial data, or intellectual property. A similar survey showed that DDoS attacks are growing, with frequency 40% higher in the fourth quarter of 2015 over third quarter results.
Ultra: Q Highlights include:
A 23% increase in DDoS attacks and a 26% increase in web application attacks, compared with Q4 2015, setting new records for the number of attacks in the quarter
The rise in repeat DDoS attacks, with an average of 29 attacks per targeted customer – including one customer who was targeted 283 times
The continued rise in multi-vectored attacks (56% of all DDoS attacks mitigated in Q1 2016), making mitigation more difficult
Oracle Confidential – Internal/Restricted/Highly Restricted 15

16. DNS Reflection Attacks
Each bot machine issues one or more DNS queries, but uses the IP address of the target system as its source IP address (i.e spoofing)
The DNS service replies to the target IP address (not the IP address of the querying computer).
The effect of the reflection attack is twofold. First, the target system is overwhelmed by thousands or millions of DNS query responses (one or more for each bot).
Second, the DNS name server is consumed by bogus requests and may lack the Compute/Elastic resources or Bandwidth
This type of attack is the most common at over 55% of all attacks seen
DNS reflection attacks work by flooding a target with bogus DNS responses. In short, a perpetrator implants a “bot” on hundreds or thousands of compromised computers.6 Each bot machine issues one or more DNS queries, but uses the IP address of the target system as its source IP address (a technique known as address spoofing). The DNS service replies to the target IP address (not the IP address of the querying computer). The effect of the reflection attack is twofold. First, the target system is overwhelmed by thousands or millions of DNS query responses (one or more for each bot). Second, the DNS name server is consumed by bogus requests and may lack the computational resources or netwo
Oracle Confidential – Internal/Restricted/Highly Restricted 16

17. DNS Amplification Attacks
DNS query messages < 50 bytes.
Traditional DNS response (such as an DNS messages can contain lots of other information. (For example, anti-spam technologies include cryptographic material.)
These extended response messages can be quite large—1 KB or greater
DNS is designed to send many responses very quickly. If an attacker issues 100,000 short DNS queries of 50 bytes each (5 MB total). If each reply is 1 KB, that’s an aggregate response of 100 MB.
An attacker with 5.6Gbps of bandwidth has generated a 112Gbps attack
Amplification attacks work by issuing requests that generate large responses, potentially flooding the network. DNS infrastructure is a common target for amplification attacks. DNS query messages are very small—often under 50 bytes. But a traditional DNS response (such as an answer containing an IPv4 address) can be ten times larger than the request. And on the internet today, DNS messages can contain lots of other information. (For example, anti-spam technologies include cryptographic material.) These extended response messages can be quite large—1 KB or greater (SEE FIGURE 2). An individual 1 KB response may not seem particularly troublesome, but DNS is designed to send many responses very quickly. Say an attacker issues 100,000 short DNS queries of 50 bytes each (5 MB total). If each reply is 1 KB, that’s an aggregate response of 100 MB.
Oracle Confidential – Internal/Restricted/Highly Restricted 17

18. Result of DDoS Attack
This is the result of a very short lived DDoS attack that our NOC team was able to handle with very little effort.
Can you handle this kind of query volume with your existing bandwidth If you are currently supporting DNS on premise?
How well can your ISP- or registrar-based DNS solution mitigate this type of attack? Do they have the bandwidth on a single provider to absorb these attacks?
DNS reflection attacks work by flooding a target with bogus DNS responses. In short, a perpetrator implants a “bot” on hundreds or thousands of compromised computers.6 Each bot machine issues one or more DNS queries, but uses the IP address of the target system as its source IP address (a technique known as address spoofing). The DNS service replies to the target IP address (not the IP address of the querying computer). The effect of the reflection attack is twofold. First, the target system is overwhelmed by thousands or millions of DNS query responses (one or more for each bot). Second, the DNS name server is consumed by bogus requests and may lack the computational resources or netwo
Oracle Confidential – Internal/Restricted/Highly Restricted 18

19. DDoS Mitigation Monitoring
Sometimes the cure is similar to the poison
Union Bank uses Verisign for DDoS mitigation
Verisign failed to propagate Union Bank routes globally so some of Dyn’s peers still have a route the attacker can use (noted in red on graph and bolded in trace)
Dyn receives full routing tables from over 700 IPv4 and v6 networks.
border5.ae2-bbnet2.phx010.pnap.net Internap Network Services Phoenix United States
unionb-9.edge1.phx010.pnap.net Internap Network Services Phoenix United States Union Bank of California Monterey Park United States
border5.ae2-bbnet2.phx010.pnap.net Internap Network Services Phoenix United States
unionb-9.edge1.phx010.pnap.net Internap Network Services Phoenix United States Union Bank of California Monterey Park United States chns2.unionbank.com Union Bank of California Monterey Park United States
Oracle Confidential – Internal/Restricted/Highly Restricted 19

20. Recent Routing Issues Events
January 20, 2017, TIC announced BGP hijacks for 20 individual IPs associated with Apple’s iTunes service.
April 26, 2017, Rostelecom hijacks 36 prefixes that included HSBC, Visa, Mastercard and smaller European banks. Rostelecom again on July 25, 2017 with Amazon Cloudfront prefixes.
May 2, 2017, Centurylink hijacks address space for Microsoft Livemeeting which results in traffic misdirection
July 14, 2017, Akamai/Prolexic propagates errants routes for over 200 companies including JPMC, Goldman Sachs, Apple, IBM and Oracle
Oracle Confidential – Internal/Restricted/Highly Restricted 20

21. Traceroute Showing Hijack
BGP Hijack Rostelecom
With the passive and active monitoring of BGP announcements and traceroutes we can identify anomalies and prove that traffic is following the hijacked announcement.
Traceroute Showing Hijack
Google Inc. Moscow Russia ms
broadband moscow.rt.ru PJSC Rostelecom Moscow Russia ms
DNS query resolution is the first step in loading a web page with each new page triggering multiple DNS lookups
Milliseconds count.
Web pages can have 20, 30 or even 50 DNS lookups
Images, video, audio, ads, social media buttons
100ms difference per object = 2 or more second delay in loading the page
URL:
Host: tapestry.tapad.com
IP:
Error/Status Code: 200
Client Port: 63187
Request Start: s
DNS Lookup: 388 ms
Oracle Confidential – Internal/Restricted/Highly Restricted 21

22. Traceroute Showing Hijack
BGP Hijack Centurylink/Microsoft
With the passive and active monitoring of BGP announcements and traceroutes we can identify anomalies and prove that traffic is following the hijacked announcement.
Traceroute Showing Hijack
cr1-te sfo.savvis.net Savvis San Francisco United States
er1-te8-0-1.svl.savvis.net Savvis Chesterfield United States hr2-xe sc4.savvis.net Savvis Chesterfield United States Savvis San Francisco United States
DNS query resolution is the first step in loading a web page with each new page triggering multiple DNS lookups
Milliseconds count.
Web pages can have 20, 30 or even 50 DNS lookups
Images, video, audio, ads, social media buttons
100ms difference per object = 2 or more second delay in loading the page
URL:
Host: tapestry.tapad.com
IP:
Error/Status Code: 200
Client Port: 63187
Request Start: s
DNS Lookup: 388 ms
Oracle Confidential – Internal/Restricted/Highly Restricted 22

23. About Dyn

24. DNS Resolution Time for Cloud Providers
APM
Oracle Confidential – Internal/Restricted/Highly Restricted 24

25. DNS Unique Value Oracle + Dyn Unique Value
Consistently High Performance Response Times Worldwide
DNS propagation time < 1 minute
Highly Resilient
Optimized Transit Connections at each POP
Advanced DDoS Attack Processes
Superior Geolocation Accuracy
Extreme Industry Expertise
Dyn NOC successfully mitigates 2 to 3 significant DDoS attacks/week
Dyn’s NOC sees up to 50 DDoS events/month but are absorbed by our network and architecture
Dyn detects & mitigate all attacks to our services at the infrastructure layer
Spanning multiple protocols: DNS, SSDP, NTP, UDP fragments, etc.
Typical mitigation time is less than 10 minutes
> 80+% of all attacks reported are here
Network layer attacks (layers 3 & 4) UDP floods, Syn attacks and ICMP
Session layer attacks (layers 5 & 6) DNS floods and SSL floods
Unique ability to discover and quickly mitigate low volume attacks
Architecture combined with size & expertise of team
Oracle Confidential – Internal/Restricted/Highly Restricted 25

26. Anycast Network “Dyn delivers the best DNS response time worldwide.”
– CloudHarmony
   
Fully redundant anycast network with no outages. Anycast network will be able to provide responses very fast with low latency from every region POPs globally to quickly service your DNS requests. We have analyzed the global internet to strategically place the POPs so they are just a few network hops away.
User’s query resolved and directed to closest available endpoint
Speed: average response times
North America < 15ms
Europe < 30ms
Asia < 45ms
Dbind Servers (dell R430) with dBIND
200,000 Queries on an individual nameserver (one dns4 box)
Anycast A consists of NTT and TATA
Anycast B Consists of Telia, Level3, Cogent, Bharti, Telstra, PCCW and Pacnet
Anycast C Consists of NTT and Tata
Anycast D Consists of Telia, Durand, Telstra, PCCW, Cogent, and Level3
Gig Links (except mumbai and sao paulo, APAC)
Oracle Confidential – Internal/Restricted/Highly Restricted 26

27. Anycast Network
Dyn is connected to multiple tier one transit providers throughout the network.
Four Tier 1 Transit Providers at each POP at 10G each
Transit companies will also shut down paths when under attack
Dyn analytics provide unique insight to select transit companies to manage degradation, market variations, large scale events
During a particularly bad DDoS attack at NS1, Cogent cut off service to NS1 to protect themselves… unfortunately Cogent was the ONLY transit provider used by NS1 and NS1 went down.
Oracle Confidential – Internal/Restricted/Highly Restricted 27

28. Collecting Traceroute and BGP
“It’s good to see this great data being exposed for operational purposes. The internet is so critical for for almost every business today.”
– Gartner (Jonah Kowall, VP)
Active monitoring of BGP.
Real-time global routing table from over 700 sessions
300+ collectors sending traceroutes to over 1.5 million targets daily resulting in over 6B measurements per day
Updates and alerts 30 seconds from real time
Oracle Confidential – Internal/Restricted/Highly Restricted 28

29. Endpoint Agnostic Routing
Route to Anything:
Datacenters
Load balancers
CDNs
Cloud Hosting
Filtration services
VOIP
Pick and Choose
Geography
Round Robin
Weighted
Performance
To cure Internet blindness:
Dyn monitors the whole Internet across multiple datasets
Dyn views Internet organizations from the outside in, just like their customers do
Only correlation across these diverse datasets reveals the high value problem root causes
Only Dyn has non-archived datasets reaching back to 2002 for a unique historic context
Only accept incomplete datasets if you want incomplete Internet performance or security!
Oracle Confidential – Internal/Restricted/Highly Restricted 29

30. Things to Consider DNS of today is not your father’s DNS
DDOS attacks are larger and more complex than ever before
Customer steering to improve experience does not need to be done by a box in your data center
Monitoring and failover can be done while you are sleeping
BGP can be monitored and is now used in ways never seen before
Attackers can use BGP to redirect traffic through an undesirable location
The root cause of a performance issue can be identified so your team does not need to be pulled into emergency troubleshooting
What Internet Service Providers do with routing your traffic can be seen and intelligence decisions can be made around provider choices
Oracle Confidential – Internal/Restricted/Highly Restricted 30