1. Edgewebhosting.net CEO, Vlad A. Friedman Advanced Application and Server Troubleshooting Techniques © 2005 ACS Edgewebhosting.net

2. Personal Background Vlad A. Friedman CEO, Edgewebhosting.net #1 Rated CF Managed Hosting Provider Hosting since 1998 Operating one of the Most Redundant Datacenters in the world Hundreds of Servers running over 20,000 websites

3. Troubleshooting Overview Different Types of Application Servers Common Problems and Fixes Troubleshooting Tools & Techniques How does Edgeweb locate and eradicate problems and how does that relate to your environment?

4. Types of Application Servers Cold Fusion / Blue Dragon Web application server Dependent on Code IIS SQL Mail Servers IIS/Apache Receives requests from clients Routes requests to application server Returns results back to clients SQL Server/MySQL Server/MS Access Stores and Retrieves Data Mail Servers Sends messages and notifications to users All of the above subsystems must run efficiently for an application to function and fulfill its purpose.

5. Application Dependency Pyramid Cold Fusion Blue Dragon IIS, JRUN SQL Server, Mail Servers Operating Systems Network Infrastructure Hardware Infrastructure It is important to understand The role of each layer in the application pyramid. If any lower layer fails, none of the layers above it can function.

6. Common Problems Completely Unresponsive servers. Crashing Applications (Nothing is working) Errors Hanging Applications (in CF and BD) Web Server unresponsive Slow application response time Bad Code Custom Tags Bad Queries Misconfiguration

7. Completely Unresponsive Servers (Hardware) Check the Hardware 1) Try to ping the server (if ping is allowed in your environment). This is the best way to tell if it is still alive without getting up from your desk. 2) Get physical access on the server. If it is unresponsive or has a BSOD, consider a reboot. 3) Following reboot, check Windows Event Log for errors Physical Locks and BSOD (Blue Screen of Death) Lockups can be caused by a number of physical components BAD CPU BAD Memory BAD Power Supply BAD Hard Disks, especially RAID Ensure hardware is working properly prior to continuing. Hardware Infrastructure

8. Completely Unresponsive Servers (Network) Check the Network Ensure all network connections are in place and working If ping is allowed in your environment Ping the server from your workstation Ping your workstation from the server i.e. ping servername ping workstation If you have access to the server, it appears to be operational, but you cannot ping any machines from it, check your network settings and connections. Run a trace route with the following command from your workstation: tracert servername You may have a network problem outside of your server and workstation environment preventing you from communicating. Trace route shows the path data travels across the internet between two locations. Network Infrastructure

9. Free Internet/Network Troubleshooting Tools A number of free tools are available online to allow you to check your server from multiple worldwide locations to troubleshoot external network problems: http://visualroute.visualware.com/ http://www.alertra.com http://www.checkdns.org

10. http://visualroute.visualware.com/ http://visualroute.visualware.com/ Online visual trace route

11. http://www.alertra.com http://www.alertra.com Free Online site check from 6 worldwide locations

12. Troubleshooting Operating Systems Common Problems Virus Infected Use free online virus scanner http://housecall.trendmicro.com/ http://www.pandasoftware.com/activescan Root Kits Now accounts for over 20% of all server infections SysInternals Root Kit Revealer http://www.sysinternals.com/utilities/rootkitrevealer.html Corrupted Registry Corrupted files in OS Microsoft “Security” Patches Always check the EventLog for problems. Always make sure your server has the latest patches applied Operating Systems

13. Troubleshooting IIS Internet Information Server Receives requests from multiple clients simultaneously. Dispatches requests to application servers Issues data returned by application server back to the correct client. Common problems include: OS Permissions on file system Misconfiguration Corrupt Metabase IIS, JRUN SQL Server, Mail Servers

14. Installing and using IISTracer Available from http://www.iismonitor.nethttp://www.iismonitor.net IISTracer is a real-time internet web site monitoring tool for Microsoft IIS (all versions). The software lets you show what is happening on IIS server right now. You can see or log state of each request, incoming outgoing bytes, http headers and several request running times. IISTracer shows a state of running scripts (.asp,.aspx, asp.net,.cfm,.php,.cgi,...), applications (.dll,.exe,..) and a big downloads (.mp3,.zip,...). Free to try for 60 days, $99 to Buy

15. IISTracer Screenshot

16. Troubleshooting IIS with Netstat Netstat allows you to view connections in real time as well as services that are listening on your server. IIS Runs on ports: TCP 80 for HTTP TCP 443 for HTTPS TCP 21 for FTP TCP 25 for SMTP Run netstat –an Active Connections Proto Local Address Foreign Address State TCP 0.0.0.0:80 0.0.0.0:0 LISTENING TCP 0.0.0.0:135 0.0.0.0:0 LISTENING TCP 0.0.0.0:443 0.0.0.0:0 LISTENING TCP 0.0.0.0:445 0.0.0.0:0 LISTENING TCP 0.0.0.0:1039 0.0.0.0:0 LISTENING TCP 0.0.0.0:1160 0.0.0.0:0 LISTENING TCP 0.0.0.0:2368 0.0.0.0:0 LISTENING TCP 0.0.0.0:2948 0.0.0.0:0 LISTENING TCP 0.0.0.0:3389 0.0.0.0:0 LISTENING TCP 0.0.0.0:4678 0.0.0.0:0 LISTENING TCP 127.0.0.1:1051 0.0.0.0:0 LISTENING TCP 127.0.0.1:4664 0.0.0.0:0 LISTENING TCP 127.0.0.1:5679 0.0.0.0:0 LISTENING TCP 208.39.184.203:139 0.0.0.0:0 LISTENING TCP 208.39.184.203:80 208.39.184.217:1445 ESTABLISHED TCP 208.39.184.203:80 198.65.119.20:3380 ESTABLISHED TCP 208.39.184.203:80 69.63.130.5:4139 ESTABLISHED

17. Troubleshooting with Telnet Telnet allow you to create a low level connection to your server to see if it is evening listening for requests. Run telnet servername 80 If you get a blank screen, the server is listening. Try typing (case is important) GET /index.cfm This will return the actual data that your browser would normally receive. If you do not get a response, there is something wrong with IIS or CF. Try getting an html page from the site. i.e. GET /test.htm If it returns valid results, then there is a problem with CF, but IIS is working properly.

18. Other helpful tools Free tools available from Sysinternals.com FileMon Shows real-time file activity, permissions and file access problems. RegMon Shows real-time registry activity Pskill Allows you to kill running processes TcpView View TCP/IP connections in real time.

19. Troubleshooting SQL Server If you seem to be getting slow response from only your dynamic pages, and all of the static pages are working fine, check your database. Use Microsoft SQL Profiler to connect to your database to measure performance. SQL Server Profiler will show you every query running on the server in real time and amount of resources used on the server. Generally, most normal optimized queries have a CPU of under 20 and READs of under 100. A high number of reads means that you are missing indexes and the server is having to physically scan records in your tables.

20. SQL Profiler Click File->New->Trace Login to the database server Use the default settings

21. Query Analyzer Use query analyzer to see the detailed performance usage breakdown on each query Run query analyzer Connect to your database Click Query-> Show Execution Plan Show Server Trace Show Client Statistics Then cut and paste your query from Profiler into the query window, and hit the run button.

22. Query Analyzer Results

23. Troubleshooting Mail Servers Check your Cold Fusion mail spool folder to ensure messages are being removed from the queue CF5 – c:\cfusion\mail\spool CFMX – c:\CFusionMX\mail\spool If messages are not moving, try moving them to a temporary location, and restarting CF. If mail starts flowing, check the messages that you moved for corruption. Check the undeliverable folder to make sure that messages are not getting rejected. CF5 - C:\CFusion\mail\UnDelivr CFMX - C:\CFusionMX\mail\UnDelivr Enable mail logging in CF administrator and check the CF mail logs to ensure messages are being processed.

24. Troubleshooting Mail Servers Check the mail server Mail sure you are using the correct mail server Review the mail logs and event logs and look for errors. Check that Mail server is able to resolve DNS using nslookup on the mail server itself nslookup Default Server: mail.edgewebhosting.net Address: 69.63.128.71 > set type=mx > ewh.net Server: mail.edgewebhosting.net Address: 69.63.128.71 Non-authoritative answer: ewh.net MX preference = 10, mail exchanger = relay1.edgewebhosting.net ewh.net nameserver = ns2.edgewebhosting.net ewh.net nameserver = ns1.edgewebhosting.net relay1.edgewebhosting.net internet address = 69.63.128.201 ns1.edgewebhosting.net internet address = 69.63.128.230 ns2.edgewebhosting.net internet address = 69.63.128.231 Try telnet to the server on port 25 on your Cold Fusion server telnet mailserver 25 This should return a 220 response: 220 ewhserver18.edgewebhosting.net (IMail 8.20 8743-27) NT-ESMTP Server

25. Cold Fusion and Blue Dragon Top of the Pyramid Cold Fusion Blue Dragon

26. Crashing Applications Programming Errors If your code hasn’t changed, check what has changed Operating System, patches and upgrades Application server patches and upgrades Try restarting Cold Fusion and web services or rebooting the server to see if the error condition clears itself. If your code has changed Make sure you have an error handler on your site. Check the Cold Fusion Log Files for Errors In Cold Fusion 5 C:\CFusion\Log\ In Cold Fusion MX C:\CFusionMX\Logs C:\CFusionMX\runtime\logs Check the Windows EventLogs for Errors Enable Cold Fusion Debugging Programming Errors are typically the easiest type of problem to identify

27. Hanging Applications and Web Server Unresponsiveness When you hit your site or application and the browser just spins Try connecting to port 80 on the server telnet servername 80 GET /pagename.cfm You should receive the results from CF. If not, work your way back down the pyramid. Enable logging of long running requests in CF Administrator Use a 3 rd party application to troubleshoot SeeFusion from SeeFusion.com (2 hour trial) Provides very detailed debugging information. Allows you to see running pages and queries together in debug output. Has a running debug mode to see all debugging information from every page in real time.

28. Debugging CF and BD with SeeFusion.com View Cold Fusion and Db Requests in Real Time

29. Installation Instructions Online Instructions at: http://www.seefusion.com/index.cfm?do=c_guide.install Download the seefusion.jar file from http://www.seefusion.com/seefusion.jar and place the file in c:\cfusionmx\wwwroot\web-inf\libhttp://www.seefusion.com/seefusion.jar download the sample properties file from http://www.seefusion.com/seefusion.properties and place the file in c:\cfusionmx\wwwroot\web-inf\classes http://www.seefusion.com/seefusion.properties edit web.xml, at first servlet definition SeeFusion com.seefusion.SeeFusion SeeFusion /* Restart CFMX Open your browser to http://localhost:8999http://localhost:8999

30. Walkthrough of Actual Installation of SeeFusion

31. The End and Thank you Questions? Shameless Plug: If you have more questions, or would like to know how Edgewebhosting.net’s high capacity managed hosting solutions can benefit your organization, please contact us at 1-866-EDGEWEB.