9 INDISPENSABLE RULES OF TROUBLESHOOTING From Debugging by David J. Agans ISBN #
Rule #1 Understand the System
2013 ECM Training Conference#dbwestECM Hyland Community – Module Reference Guides – Forum Training.OnBase – Web Based Trainings – Education Services Class Schedules Read the Manual
2013 ECM Training Conference#dbwestECM Look up ALL the details on the issue System information Is it supposed to work? DO NOT ASSUME! Trust but verify Know How it’s Supposed to Work
2013 ECM Training Conference#dbwestECM Know what tools to use Know how to use them! Know the limitations too Should you use more than one tool? Maybe… Know Your Tools
Rule #2 Make It Fail
2013 ECM Training Conference#dbwestECM Why reproduce The Issue? – Observe the behavior – Narrow down the cause – Confirm the fix Make It Fail
2013 ECM Training Conference#dbwestECM What are good details of a problem? Detailed steps to reproduce the issue Obtain all facts when troubleshooting Does the problem happen after the same set of actions? Make It Fail
2013 ECM Training Conference#dbwestECM Intermittent Issues – Find the signature of intermittent bugs – Amplify testing – Capture info on every attempt – Never throw away a debugging tool Make It Fail
Rule #3 Quit Thinking & Look
2013 ECM Training Conference#dbwestECM See the actual failure – Do not just assume and fix the wrong thing See the details - get complete info Guessing is okay, but only to focus your search Quit Thinking & Look
Rule #4 Divide & Conquer
2013 ECM Training Conference#dbwestECM Narrow the Search with Successive Approximation Use available info from: – Step 1 – Understand the System – Step 3 – Quit Thinking and Look Break it into smaller bites! Divide & Conquer
2013 ECM Training Conference#dbwestECM Test for an expected result Start at the problem and work backwards It could be more than one thing! – Sometimes they are both the same issue – Sometimes one fix causes another problem Is it noise or is it real? Widen the range if necessary Divide & Conquer
Rule #5 Change One Thing At a Time
2013 ECM Training Conference#dbwestECM Isolate the Key Factor Avoid Quick Fixes Change ONE piece at a time – Test before the next change Determine what you changed since the last time it worked Change One Thing At a Time
Rule #6 Keep an Audit Trail
2013 ECM Training Conference#dbwestECM Write down what you did (DETAILS) – What order? – What happened as a result? Understand that any details could be an important one Correlate Events Keep an Audit Trail
Rule #7 Check the Plug
2013 ECM Training Conference#dbwestECM Do not ignore the obvious Question your assumptions Don’t start at square three Test the tool Check the Plug
Rule #8 Get a Fresh View
2013 ECM Training Conference#dbwestECM Ask for fresh insights – Others are there to help – Explaining the problem may help resolve it! Listen to the voice of experience – Look at past experiences to resolve your problems Get a Fresh View
2013 ECM Training Conference#dbwestECM Report Symptoms – Not Theories! – Reason to ask for help is that your theories are not getting you anywhere Present ALL info even if it may not appear relevant Get a Fresh View
Rule #9 If You Don’t Fix It, It Ain’t Fixed!
2013 ECM Training Conference#dbwestECM Problems don’t occur for no reason and don’t fix themselves! Check that it’s really fixed Know the cause Verify that the fix is what fixed it! If you Don’t Fix It, It Ain’t Fixed!
Database Disk Groups Workstation ODBC NetworkSecurity
HTTP ODBC HTTP Disk Group OnBase Web Server OnBaseApplicationServer NetworkSecurity HTTP (Remoting or SOAP) OnBase Web Client OnBase Core Application Database GatewayCachingServer OnBase Web Client HTTP
Unity Client Unity Client CORE BASED APPLICATIONS
Web Server Application Server
Web Diagnostics – Test.gif – aspnetcheck.aspx – Diagnostics Web Page Diagnostics Console OnBase Web Server Troubleshooting
If a problem occurs in the Web Client, check the Thick Client If yes, probably configuration Notes about the Core Apps
Diagnostics Service Diagnostics Console
2013 ECM Training Conference#dbwestECM A utility that gathers log information related to OnBase Core based modules User interface that allows easy readability of the logs What is the Diagnostics Console?
HYLAND DIAGNOSTICS CONSOLE
2013 ECM Training Conference#dbwestECM A service running on the Application Server and/or Web Server Display messages to the Diagnostics Console or log them to a file Allows for remote monitoring of servers What is the Diagnostics Service?
HYLAND DIAGNOSTICS SERVICE
2013 ECM Training Conference#dbwestECM Two ways to run the Diagnostics Console – Local Mode Logs are gathered from the local workstation Will not work if Diagnostics Service is running – Remote Mode Diagnostics Console connects to the Diagnostics Service Messages are written to the Diagnostics Service Must have rights to start the service Local vs. Remote Mode
2013 ECM Training Conference#dbwestECM Troubleshooting issues with Core based modules – Web Server – Application Server Any module which connects to the Application Server When running the OnBase Thick Client as a Windows Service When to use it?
ENABLING DIAGNOSTICS SETTINGS
Logging Profile option was added in Logging Profile does not require an App Pool restart Reset back to the Web.Config settings (defaults) upon App Pool restart Diagnostics Web Logging Profile
LDAP / NT AUTHENTICATION TAB
0 TIMER / DISTRIBUTION SERVICE
SYSTEM INFORMATION TAB
2013 ECM Training Conference#dbwestECM Registry Settings – 32-bit Workstations HKLM| SOFTWARE | Hyland | Diagnostics – 64-bit Workstations HKLM| SOFTWARE | Wow6432Node | Hyland | Diagnostics Set Log Profile to decimal value of 64 (not HEX) Enabling Thick Client Verbose
Log to file when logging a lot of data Do not run the Diagnostics Console all the time Best Practices
Only enable needed mailslots – Utilize Web Diagnostics Logging Profile Save all tabs when engaging support Best Practices
Enable filtering – If you don’t see what you are looking broaden your filters Enable the User Name column Recommendations
Windows Event Viewer
WINDOWS EVENT VIEWER
2013 ECM Training Conference#dbwestECM Messages from the OnBase Client closing unexpectedly IIS (OnBase Web or Application Server) – Application Pools not starting – Platter Management Errors – Active Directory Authentication OnBase Integration issues related to: – Office Business Application – Outlook Integration What is logged here?
Errors typically correspond to messages in Diagnostics Console Tips on Windows Event Viewer
OnBase Thick Client Verbose
2013 ECM Training Conference#dbwestECM Detailed log of OnBase activity – Logs most SQL queries executed – Used to see what is happening right before an error – Can help pinpoint long running queries What is Verbose?
2013 ECM Training Conference#dbwestECM Command Line Switch – Available with OnBase Thick Client and Config – Application launch issues Tool Bar Option (OnBase Thick Client Only) – Limits the amount of data captured Two ways to enable Verbose
2013 ECM Training Conference#dbwestECM -V SQL,T – Includes SQL with time stamps -V SQL,L – Includes table lock codes -V SQL “itemdata” – Includes only log statements that – include the text of “itemdata” Command Line Verbose Options
2013 ECM Training Conference#dbwestECM It is possible to use multiple command line Verbose options together Example: -V SQL “update” “itemdata”,T,L Command Line Verbose Options
2013 ECM Training Conference#dbwestECM Help | About OnBase Client | Verbose | chatty Enabling Verbose – Tool Bar option
2013 ECM Training Conference#dbwestECM Verbose writes to the verbose.txt file Located in the TempParsePath (by default) What is the output?
Capture the most relevant data – Enable verbose immediately before action – Truncate the log – Disable afterward Verbose files are always appended – Most recent data at the bottom Tips on running Verbose
Verbose files can become very large – Limit the amount of data captured – Don’t leave verbose switch on processes Tips on running Verbose
2013 ECM Training Conference#dbwestECM Typically (but not always) generated when: – OnBase unexpectedly terminates – A debug assert error occurs Located in the same folder as the OnBase executables – Obclnt32.rpt - RPT file for the OnBase Thick Client – Obcfg32.rpt - RPT file for the OnBase Config What are RPT Files?
2013 ECM Training Conference#dbwestECM RPT Files Always click Retry Then click Abort Don’t click Ignore X
If the exception is recreateable, generate a verbose along with the RPT file Check to see if the date/times in the verbose file match the RPT file Tips on RPT Files
Open RPT files in an advanced text editor – Notepad2, Notepad++, WordPad, etc. RPT files are always appended Tips on RPT Files
Where was the OnBase Thick Client opened? – From a Network Share – Locally on the workstation Tips on RPT Files
2013 ECM Training Conference#dbwestECM Let’s take a look… Understanding OnBase Versioning
2013 ECM Training Conference#dbwestECM Understanding OnBase Versioning
2013 ECM Training Conference#dbwestECM The first digit designates the version. It corresponds to the year of release. Understanding OnBase Versioning
2013 ECM Training Conference#dbwestECM The second digit designates release or beta software. Even numbers (0,2,4, etc.) are release software Odd (1,3,5…) are beta software These will only change if there is more than one full release in a calendar year. Understanding OnBase Versioning
2013 ECM Training Conference#dbwestECM The fourth digit designates individual builds within a version. All builds of the Core start with 1 (1 – 499) All builds of the Thick Client start with 500 ( ) The build will increase successively regardless of change in the Service Pack (xx.x.#.xxx) value. Understanding OnBase Versioning
2013 ECM Training Conference#dbwestECM The third digit is for Service Pack versioning. Understanding OnBase Versioning
Check out the Hyland Community !!! Product Information Documentation (MRG’s) Presentations & Films BlogsForums Technical Knowledge Base Troubleshooting Checklist and more…
Check out Training.OnBase.Com!!! Class Schedules Web Based Training Premium Subscription and more…
Engaging your first line of Support
2013 ECM Training Conference#dbwestECM Customer Name or HSI # Level of priority for this issue – Low, Medium, High or Critical OnBase Environment of issue – Production, Test or Disaster Recovery System What information should I have?
2013 ECM Training Conference#dbwestECM OnBase Environmental Information – OnBase Version and Build – Problematic OnBase Modules – Database Platform and Version What questions will I be asked?
2013 ECM Training Conference#dbwestECM Issue Specific Information – Can this issue be reproduced? – When did it start to occur? – Is it specific to a single user or all users? – Is this a new process or existing? – Have there been any changes? – Are there any error messages? What questions will I be asked?
2013 ECM Training Conference#dbwestECM Screenshots – Any errors or error messages Log Files – Verbose – Diagnostics Console (Save all tabs!!) Useful Data
2013 ECM Training Conference#dbwestECM Make sure the time stamps match – Diagnostics Console – Verbose file vs. RPT – Third Party tools Tips on gathering data