Troubleshooting

Tools

Troubleshooting Tools Inspecting HTTP Headers In many cases it is useful to view the HTTP headers when debugging various problems, including: Persistence failures when using cookie persistence Performing caching and tuning cache headers Verifying compression is occurring Isolating authentication issues Many tools that are available are very useful for this task and are easy to use: Live HTTP Headers for Mozilla/Firefox IE HTTP Headers for Internet Explorer IEHTTPAnalyzer Fiddler

Troubleshooting Tools Presentation Title Goes Here Insert Version Number Here Troubleshooting Tools Inspecting HTTP Headers: Live HTTP Headers Live HTTP Headers is available at http://livehttpheaders.mozdev.org – free tool Once installed, go to Tools > livehttpheaders to start capturing headers. Once opened, all HTTP request and response headers will be displayed, as shown. If desired, you can also filter what requests will be captured. © 2003 Citrix Systems, Inc.—All rights reserved.

Troubleshooting Tools Inspecting HTTP Headers: IE HTTP Headers IE HTTP Headers can be downloaded from http://www.bluck.info/iehttpheaders - free tool After installation, go to View > Explorer Bar > ieHTTPHeaders to display the view pane.

Presentation Title Goes Here Insert Version Number Here Troubleshooting Inspecting HTTP Headers IEHTTPAnalyzer - must purchase (found on student Desktop) or download from http://www.ieinspector.com/httpanalyzer/ LiveHTTPHeaders for firefox © 2003 Citrix Systems, Inc.—All rights reserved.

Presentation Title Goes Here Insert Version Number Here Troubleshooting Inspecting HTTP Headers Fiddler - Freeware Download from http://www.fiddler2.com/fiddler2/ LiveHTTPHeaders for firefox © 2003 Citrix Systems, Inc.—All rights reserved.

Tracing

Troubleshooting Tools Packet captures can be done on the NetScaler via the “new trace” option on the diagnostics tab, as shown

Troubleshooting Tools The dialog to perform a trace has several options: Packet size: Specify 0 to capture the entire packet Duration of data per file: By default 1 hr. Format: Select “tcpdump” to capture in pcap format Filter expression: TCPDUMP capture filter to only capture certain packets being processed Capturing mode: Additional options for capturing packets

Troubleshooting Tools Packet Traces, continued Once a trace is performed, the trace will have to be downloaded

Troubleshooting Tools Packet Traces, continued Select where you want the trace saved and open with your favorite trace tool

Network Traffic Capture Citrix NetScaler Corporate Intro Presentation March 29th 2006 Network Traffic Capture The nstrace utility: Is used for packet capture on the NetScaler system Has files stored in /var/nstrace Has names nstracexx.cap (Native) or nstracexx.pcap (tcpdump) Has syntax nstrace.sh –sz 0 Prefers the native format for packet capture since it captures more information Can view the native NetScaler and pcap format with Wireshark Nstrace syntax nstrace.sh dumps packets in NS format, can be viewed using NETSTAT utility (release specific) nstrace.sh -sz 0 -tcpdump 1 dumps packet of all length and in tcmpdump format, which can re read using ethereal nstrace.sh -sz 0 -tcpdump 1 -nf 3 -time 5 Dumps packets for 5 seconds and rotates in 3 different files nstrace.sh -sz 0 -tcpdump 1 -m 1 -m with 1 will dump only transmitted packets, with 2 will dump packets buffered for transmission, with 4 will dump only received packets nstrace.sh –stop It will stop any instance of nstrace running in the background © 2006 Citrix Systems, Inc.—All rights reserved. 13

Sample nstrace.sh Output

SNMP

Troubleshooting Tools SNMP Overview SNMP is a standard means of receiving information from a device by either polling variables (OIDs) or receiving alerts (traps). The NetScaler system provides a list of over a thousand variables in a MIB file. In some devices, SNMP can both read data and write configurations, but with the NetScaler system it is read only. Common tools that are used to keep track and plot long-term SNMP data include NetScaler Command Center, HP OpenView and MRTG.

Troubleshooting Tools SNMP Community Configuration The first step in monitoring via SNMP is to configure a community and permission (select ALL). System -> SNMP -> Community

Troubleshooting Tools SNMP Manager Configuration To limit SNMP polling to a set of IPs or networks, configure managers as well.

Troubleshooting Tools Presentation Title Goes Here Insert Version Number Here Troubleshooting Tools Network Monitoring with SNMP Some of the most commonly polled SNMP OIDs HA State: .1.3.6.1.4.1.5951.4.1.1.6.0 Average CPU: .1.3.6.1.4.1.5951.4.1.1.41.1.0 Memory Utilization (%): .1.3.6.1.4.1.5951.4.1.1.41.2.0 Server Connections: .1.3.6.1.4.1.5951.4.1.1.46.1.0 Client Connections: .1.3.6.1.4.1.5951.4.1.1.46.2.0 HTTP Requests: .1.3.6.1.4.1.5951.4.1.1.48.67.0 HTTP Responses: .1.3.6.1.4.1.5951.4.1.1.48.53.0 Interface Stats (table): .1.3.6.1.4.1.5951.4.1.1.54 Since many OIDs are dynamically generated, tools such as the iReasoning MIB browser (http://www.iReasoning.com) are handy for locating them. © 2003 Citrix Systems, Inc.—All rights reserved.

Presentation Title Goes Here Insert Version Number Here Troubleshooting Most Common Issues Do not rule out basic problems. Is there a duplex mismatch? #1 cause of poor performance Is a feature enabled? Check routing, in particular when USIP is enabled. Do not assume coders follow HTTP? Does it work as TCP? When installing a NetScaler system, do not assume the rest of the network is correct—it usually is broken in subtle ways. Divide and conquer. Try to break the problem down into pieces, and verify each. Always perform a trace when you have a problem. A trace tells you what IS happening and not what you think should be happening. © 2003 Citrix Systems, Inc.—All rights reserved.

Logs

Logged Information The following components enable recording and extracting information for troubleshooting the NetScaler: nsconmsg newnslog

Nsconmsg Command Output Citrix NetScaler Corporate Intro Presentation March 29th 2006 Nsconmsg Command Output Nsconmsg is an application that collects all the statistics and system information from the NetScaler system. This information can be used to analyze the traffic information and the NetScaler system information.   When this application starts, it connects to the specified NetScaler system and starts to get the data. It has the facility to log the binary data to the file or view information online. Filters can be defined to filter out only the required information. The Nsconmsg utility, running in FreeBSD user-land, polls the NetScaler kernel to extract performance records. Performance record types include: Symbol Name is every statistics variable has a unique name Device name is every object created on the NetScaler system that has a unique name and number Device Link is the link between two devices Absolute Record is the absolute value of a symbol Differential Record is a change in the value of a symbol Events Record By default, the NetScaler system collects all the performance records and stores in the circular buffer. The application like nsconmsg/GUI, can connect to NetScaler system and starts to collect this performance data real-time. The performance records are dumped into /var/nslog/newnslog.[xxx].gz in binary format. The utility can also be used to generate reports based on the binary data dumped at /var/nslog/newnslog.[xxx].gz At boot time, nsconmsg is run by the script /netscaler/nslog.sh. The Script also manages log rotation. An administrator must access the NetScaler system with SSH to access the logs. Nsconmsg displays debugging information based on the attributes passed to it, including: Display attributes current past oldconmsg (-s along with feature details) consmsg (console messages) event (event details) stats (all the counters) statswt0 (counters which have incremented) auditedcmd Debugging info of other NS nsconmsg –U 10.102.3.171:nsroot:nsroot –d statswt0 Nsconmsg uses the –d option to select what set of information to display Common display sets include current & past (Performance Record Variables) oldconmsg (a textual display, does not support all options) event (event records, usually snmp trap related) stats (an entire list of variables from live system) statswt0 (variables that are non-zero from live system) Options “past”, “stats” and “statswt0” often relate to a live system display, not an input file Nsconmsg utility is used for viewing: Events Console messages CPU/Memory statistics Interface statistics Commands executed by users Feature-specific statistics nsconmsg commands are case-sensitive Arguments are positional, order is important © 2006 Citrix Systems, Inc.—All rights reserved. 23

Accessing Logged Information The log file and time parameters should be replaced with the necessary values to obtain desired log information The log file and time parameters are available to: Uncompress an archived log file Discover the time period covered by the log View load-balancing statistics from the archived log Extract logging information for a shorter duration Start log process for newnslog

Displaying Console Messages

Calendar Dialog

Sample nsconmsg Output Using grep

Displaying Event Information

Displaying CPU Information

Displaying NIC Statistics

Load Balancing Display Output

Statistics

© 2006 Citrix Systems, Inc.—All rights reserved. Citrix NetScaler Corporate Intro Presentation March 29th 2006 Statistics Monitoring statistics of the NetScaler system or a feature can be obtained by running the stat command Stat feature displays summary statistics for the feature Additional stat parameters are: detail: for detailed feature statistics fullValues: for number and strings displayed in full form ntimes: for number of times statistics to be displayed logFile: for reading stats from a log file names internal: for viewing internal counter names © 2006 Citrix Systems, Inc.—All rights reserved. 33

Displaying Statistics in the CLI stat command: Provides statistics on certain components of a NetScaler system E.g.: - stat cpu - stat system - stat lb vserver

NetScaler System Statistical Utility The Statistical Utility displays current statistics: This is not for historical information The dashboard receives a copy of the performance records as they are generated, and displays live data These statistics are updated every seven seconds Each dashboard consumes cpu time An administrator can hover over certain areas of the dashboard and if a hand appears you can click that to get a separate window. Right clicking them brings up plotting. HTTP requests/s shows total requests for all virtual IP addresses/ System Log shows current events such as virtual server up/down messages. If an administrator clicks it, a separate window opens which has unlimited history for the current session. Gauges for CPU/Memory give a percentage of usage and throughput is total in/out for all interfaces. This can be plotted on a per interface basis by clicking or right clicking to get the chart window with a two tabs system and per interface.