EPICS IOC Diagnostics Tools Jeff Hill

Overview 4 EPICS built-in diagnostic tools 4 solutions to specific problems

Background 4 vxWorks shell allows any function to be called 4 vxWorks programs frequently provide special functions for diagnostic purposes 4 syntax is (almost) the same as calling a C function –parenthesis may be omitted 4 strings must always be in quotes

List Names of Function Blocks (Records) Installed 4 listing all records –dbl 4 listing all records of a particular type –dbl (“ ”) 4 list all records with names matching a wildcard pattern –dbgrep (“*portionOfCommiteeDesignedName*”)

Examining and Writing Function Blocks (Records) 4 Examining a function block (record) –dbpr “ ”, 4 Examining a field within a function block (record) –dbgf (“. ”) 4 Writing a Field –dbpf (“. ”, “ ”

Tracing Function Block (Record) Processing 4 set trace processing field, “TPRO”, to true –dbpf “.TPRO”,”1” –generates a message whenever the function block (record) is processed

Setting a Function Block (Record) Processing Break Point 4 set function block (record) processing break point –dbb ( “ ”) 4 delete function block (record) processing break point –dbd ( “ ”)

Single Stepping Function Block (Record) Processing 4 single step lock set after hitting break point –dbs or dbs (“ ”) 4 continue after hitting a break point –dbc or dbc (“ ”) 4 list records that have breakpoints installed –dbstat

Hardware Diagnostics 4 Examine the state of device drivers –dbior “ ”, 4 List all interrupt vectors currently in use –veclist –currently does not work on Intel architecture

Channel Access Communications Status 4 Channel Access server status –casr 4 Channel Access client status –ca_channel_status 4 List Channel Access event subscriptions –dbel 4 Status of database links to other IOCs –dbcar

Diagnosing Specific Problems 4 clients don’t connect 4 unexpected exception

Clients Don’t Connect 4 is the IOC’s CPU saturated? –type “spy” / “spyStop” to see if the server is starved for CPU –CA server will starve first –Common CPU consumption culprits sequencer, driver, record processing load 4 is the network functioning? –from workstation type “ping -s ”

No Response to “ping” 4 response at IOC’s console port? 4 Missing route? –no route to off-subnet clients by default –type “routeShow”

No Response to “ping” 4 creative thin wire (BNC) Ethernet networking? –any network where the connectors fit must be ok - not! 4 check for mBuf starvation with “mbufShow”

No Response at IOC’s Console Port 4 interrupt or DMA related problem are common culprits –exception at interrupt level typically caused by device driver problems –unacknowledged VME interrupts frequently this is a missing jumper on the VME back plane –VME bus master has not released the bus

Clients Don’t Connect 4 client and server not on the same IP subnet –if not, use EPICS_CA_ADDR_LIST environment variable 4 network interface improperly configured? –IP subnet masks of interfaces must match on vxWorks type: “ifShow” on UNIX type: “ifconfig -a” or “netstat -i” on windows type: “ipconfig”

Unexpected Exception 4 the task which generated the exception will be suspended –type “i” to list tasks and find out which one 4 determine if there is stack over flow –checkStack 4 determine if the system is low on memory –memShow strange output indicates corruption of pool

Unexpected Exception 4 before rebooting obtain a stack trace –type: “tt (0x ) frequently this is all that is needed strange output indicates stack corruption 4 send a bug report –certain bugs occur only in rare or intermittent circumstances

Observations 4 in vxWorks any C function can be used as a diagnostic 4 if you experience trouble –try to avoid rebooting without saving a stack trace of suspended tasks