1. SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Protocol Analysis in a Complex Enterprise April 2 nd, 2008 Hansang Bae Senior VP | Citigroup SHARKFEST '08 Foothill College March 31 - April 2, 2008

2. SHARKFEST '08 | Foothill College | March 31 - April 2, 2008 Challenges:  As it turns out, size does matter!  Citi’s branch network spans 5,000+ locations in the US  Citi’s network infrastructure includes 30,000+ devices  300,000 users located in over 100 countries.  Compliance/Security Quagmire  It’s for your own protection, or so I’m told!  Doing a full packet capture is difficult  Wireshark is the only approved protocol analyzer at Citi. It dislodged past market leaders.

3. Challenges (con’t):  Capturing and Analyzing: Two pieces to the same puzzle  Enormous amounts PCAP data are involved.  In most cases, header analysis is adequate.  Wireshark/WinPCAP is not well suited for this much volume  Citi uses a commercial product for packet capturing. Working with the vendor, it took over three years of development before it was deemed “Citi-ready”

4. Example One: Path MTU  Infrastructure size makes it interesting.  Very difficult problem without a proper protocol analyzer

5. Example One: (Con’t)  In depth understanding of routers and protocols were required.  Usenet to the rescue!  ICMP and IP.ADDR filters were key!  So which side am I on in the “religious debate” about whether ICMP messages should be included in the “ip.addr” display filter?..\..\..\Traces\Consumer\CBNA\ICMPRateLimit.pcap  In retrospect, it was an easy problem to solve. Yet the sheer size made it difficult to spot.

6. Example Two: Clock Drift  MarketData driven business complains of extreme delays from UK to US.  At first glance, application logs seem to confirm delays in the 200+ms delays. RTT is 70ms.  Because it’s easy, let’s blame the firewall and the network!  SLA tracking and further investigation of routers/switches gets us nowhere with problem resolution.  Our analysis shows that something is not right!

7. Example Two (Con’t)  Due to mis-matched traffic flow, pcap data itself yield unreliable data.  For example, we would see and an ACK for a packet that was not yet delivered. This was traced to the output buffer of the SPAN on the switch.  The SPAN issue forced us to look a the packets in detail, including the data timestamp

8. Example Two (Con’t)  Charting the pcap timestamp with the data timestamp showed a peculiar pattern.  By spotting the pattern above, we were able to show the vendor that their clock was drifting!

9. Lessons Learned/Feature Request  Picture really is worth a thousand words.  The two pictures above show the same event!  Bounce diagrams can quickly pinpoint issues.

10. Lessons Learned (Con’t)  Allow zoom in feature from the bounce diagram for even easier troubleshooting.  The above shows the slow start in action. It’s immediately obvious what’s going on with one look at the chart!  Increase performance for TCP/IP dissection. Although Wireshark’s support for protocols is impressive, most folks in the enterprise deal with TCP/IP problems.