1. Troubleshooting

2. Tools

3. Troubleshooting Tools
Inspecting HTTP Headers
In many cases it is useful to view the HTTP headers when debugging various problems, including:
Persistence failures when using cookie persistence
Performing caching and tuning cache headers
Verifying compression is occurring
Isolating authentication issues
Many tools that are available are very useful for this task and are easy to use:
Live HTTP Headers for Mozilla/Firefox
IE HTTP Headers for Internet Explorer
IEHTTPAnalyzer
Fiddler

4. Troubleshooting Tools
Presentation Title Goes Here
Insert Version Number Here
Troubleshooting Tools
Inspecting HTTP Headers: Live HTTP Headers
Live HTTP Headers is available at – free tool
Once installed, go to Tools > livehttpheaders to start capturing headers.
Once opened, all HTTP request and response headers will be displayed, as shown.
If desired, you can also filter what requests will be captured.
© 2003 Citrix Systems, Inc.—All rights reserved.

5. Troubleshooting Tools
Inspecting HTTP Headers: IE HTTP Headers
IE HTTP Headers can be downloaded from - free tool
After installation, go to View > Explorer Bar > ieHTTPHeaders to display the view pane.

6. Presentation Title Goes Here
Insert Version Number Here
Troubleshooting
Inspecting HTTP Headers
IEHTTPAnalyzer - must purchase
(found on student Desktop) or download from
LiveHTTPHeaders for firefox
© 2003 Citrix Systems, Inc.—All rights reserved.

7. Presentation Title Goes Here
Insert Version Number Here
Troubleshooting
Inspecting HTTP Headers
Fiddler - Freeware
Download from
LiveHTTPHeaders for firefox
© 2003 Citrix Systems, Inc.—All rights reserved.

8. Tracing

9. Troubleshooting Tools
Packet captures can be done on the NetScaler via the “new trace” option on the diagnostics tab, as shown

10. Troubleshooting Tools
The dialog to perform a trace has several options:
Packet size: Specify 0 to capture the entire packet
Duration of data per file: By default 1 hr.
Format: Select “tcpdump” to capture in pcap format
Filter expression: TCPDUMP capture filter to only capture certain packets being processed
Capturing mode: Additional options for capturing packets

11. Troubleshooting Tools
Packet Traces, continued
Once a trace is performed, the trace will have to be downloaded

12. Troubleshooting Tools
Packet Traces, continued
Select where you want the trace saved and open with your favorite trace tool

13. Network Traffic Capture
Citrix NetScaler Corporate Intro Presentation
March 29th 2006
Network Traffic Capture
The nstrace utility:
Is used for packet capture on the NetScaler system
Has files stored in /var/nstrace
Has names nstracexx.cap (Native) or nstracexx.pcap (tcpdump)
Has syntax nstrace.sh –sz 0
Prefers the native format for packet capture since it captures more information
Can view the native NetScaler and pcap format with Wireshark
Nstrace syntax
nstrace.sh
dumps packets in NS format, can be viewed using NETSTAT utility (release specific)
nstrace.sh -sz 0 -tcpdump 1
dumps packet of all length and in tcmpdump format, which can re read using ethereal
nstrace.sh -sz 0 -tcpdump 1 -nf 3 -time 5
Dumps packets for 5 seconds and rotates in 3 different files
nstrace.sh -sz 0 -tcpdump 1 -m 1
-m with 1 will dump only transmitted packets, with 2 will dump packets buffered for transmission, with 4 will dump only received packets
nstrace.sh –stop
It will stop any instance of nstrace running in the background
© 2006 Citrix Systems, Inc.—All rights reserved. 13

14. Sample nstrace.sh Output

15. SNMP

16. Troubleshooting Tools
SNMP Overview
SNMP is a standard means of receiving information from a device by either polling variables (OIDs) or receiving alerts (traps).
The NetScaler system provides a list of over a thousand variables in a MIB file.
In some devices, SNMP can both read data and write configurations, but with the NetScaler system it is read only.
Common tools that are used to keep track and plot long-term SNMP data include NetScaler Command Center, HP OpenView and MRTG.

17. Troubleshooting Tools
SNMP Community Configuration
The first step in monitoring via SNMP is to configure a community and permission (select ALL). System -> SNMP -> Community

18. Troubleshooting Tools
SNMP Manager Configuration
To limit SNMP polling to a set of IPs or networks, configure managers as well.

19. Troubleshooting Tools
Presentation Title Goes Here
Insert Version Number Here
Troubleshooting Tools
Network Monitoring with SNMP
Some of the most commonly polled SNMP OIDs
HA State:
Average CPU:
Memory Utilization (%):
Server Connections:
Client Connections:
HTTP Requests:
HTTP Responses:
Interface Stats (table):
Since many OIDs are dynamically generated, tools such as the iReasoning MIB browser ( are handy for locating them.
© 2003 Citrix Systems, Inc.—All rights reserved.

20. Presentation Title Goes Here
Insert Version Number Here
Troubleshooting
Most Common Issues
Do not rule out basic problems.
Is there a duplex mismatch? #1 cause of poor performance
Is a feature enabled?
Check routing, in particular when USIP is enabled.
Do not assume coders follow HTTP? Does it work as TCP?
When installing a NetScaler system, do not assume the rest of the network is correct—it usually is broken in subtle ways.
Divide and conquer.
Try to break the problem down into pieces, and verify each.
Always perform a trace when you have a problem. A trace tells you what IS happening and not what you think should be happening.
© 2003 Citrix Systems, Inc.—All rights reserved.

21. Logs

22. Logged Information
The following components enable recording and extracting information for troubleshooting the NetScaler:
nsconmsg
newnslog

23. Nsconmsg Command Output
Citrix NetScaler Corporate Intro Presentation
March 29th 2006
Nsconmsg Command Output
Nsconmsg is an application that collects all the statistics and system information from the NetScaler system. This information can be used to analyze the traffic information and the NetScaler system information.
When this application starts, it connects to the specified NetScaler system and starts to get the data. It has the facility to log the binary data to the file or view information online. Filters can be defined to filter out only the required information.
The Nsconmsg utility, running in FreeBSD user-land, polls the NetScaler kernel to extract performance records.
Performance record types include:
Symbol Name is every statistics variable has a unique name
Device name is every object created on the NetScaler system that has a unique name and number
Device Link is the link between two devices
Absolute Record is the absolute value of a symbol
Differential Record is a change in the value of a symbol
Events Record
By default, the NetScaler system collects all the performance records and stores in the circular buffer. The application like nsconmsg/GUI, can connect to NetScaler system and starts to collect this performance data real-time.
The performance records are dumped into /var/nslog/newnslog.[xxx].gz in binary format.
The utility can also be used to generate reports based on the binary data dumped at /var/nslog/newnslog.[xxx].gz
At boot time, nsconmsg is run by the script /netscaler/nslog.sh. The Script also manages log rotation.
An administrator must access the NetScaler system with SSH to access the logs.
Nsconmsg displays debugging information based on the attributes passed to it, including:
Display attributes
current
past
oldconmsg (-s along with feature details)
consmsg (console messages)
event (event details)
stats (all the counters)
statswt0 (counters which have incremented)
auditedcmd
Debugging info of other NS
nsconmsg –U :nsroot:nsroot –d statswt0
Nsconmsg uses the –d option to select what set of information to display
Common display sets include
current & past (Performance Record Variables)
oldconmsg (a textual display, does not support all options)
event (event records, usually snmp trap related)
stats (an entire list of variables from live system)
statswt0 (variables that are non-zero from live system)
Options “past”, “stats” and “statswt0” often relate to a live system display, not an input file
Nsconmsg utility is used for viewing:
Events
Console messages
CPU/Memory statistics
Interface statistics
Commands executed by users
Feature-specific statistics
nsconmsg commands are case-sensitive
Arguments are positional, order is important
© 2006 Citrix Systems, Inc.—All rights reserved. 23

24. Accessing Logged Information
The log file and time parameters should be replaced with the necessary values to obtain desired log information
The log file and time parameters are available to:
Uncompress an archived log file
Discover the time period covered by the log
View load-balancing statistics from the archived log
Extract logging information for a shorter duration
Start log process for newnslog

25. Displaying Console Messages

26. Calendar Dialog

27. Sample nsconmsg Output Using grep

28. Displaying Event Information

29. Displaying CPU Information

30. Displaying NIC Statistics

31. Load Balancing Display Output

32. Statistics

33. © 2006 Citrix Systems, Inc.—All rights reserved.
Citrix NetScaler Corporate Intro Presentation
March 29th 2006
Statistics
Monitoring statistics of the NetScaler system or a feature can be obtained by running the stat command
Stat feature displays summary statistics for the feature
Additional stat parameters are:
detail: for detailed feature statistics
fullValues: for number and strings displayed in full form
ntimes: for number of times statistics to be displayed
logFile: for reading stats from a log file
names internal: for viewing internal counter names
© 2006 Citrix Systems, Inc.—All rights reserved. 33

34. Displaying Statistics in the CLI
stat command: Provides statistics on certain components of a NetScaler system
E.g.: - stat cpu - stat system - stat lb vserver

35. NetScaler System Statistical Utility
The Statistical Utility displays current statistics:
This is not for historical information
The dashboard receives a copy of the performance records as they are generated, and displays live data
These statistics are updated every seven seconds
Each dashboard consumes cpu time
An administrator can hover over certain areas of the dashboard and if a hand appears you can click that to get a separate window. Right clicking them brings up plotting.
HTTP requests/s shows total requests for all virtual IP addresses/
System Log shows current events such as virtual server up/down messages. If an administrator clicks it, a separate window opens which has unlimited history for the current session.
Gauges for CPU/Memory give a percentage of usage and throughput is total in/out for all interfaces. This can be plotted on a per interface basis by clicking or right clicking to get the chart window with a two tabs system and per interface.