B7: Caring for an Ailing AppServer™ Diego Canziani Senior Technical Support Engineer

Agenda Anatomy of the AppServer Diagnosing Problems Help Treating your AppServer AppServer Broker Agent client Listening Port Cl ients Agents

Anatomy of the AppServer What is the OpenEdge® AppServer?

Anatomy of the AppServer Or What is the Progress® AppServer? Broker JavaTM process Handles communication from clients to agents Manages client and agent pools Writes to <brokername>.broker.log <brokername>.broker.log

Anatomy of the AppServer Or What is the Progress AppServer? Agent “C” process Handles requests and replies to the clients Communicates “STATUS” to broker Writes to <brokername>.server.log <brokername>.server.log

Anatomy of the AppServer 4 Operating Modes State-aware State-reset Stateless State-free One to one - Default Many to few

Anatomy of the AppServer 2 Connection Models Client to Broker gets redirected to Agent State-aware State-reset Client to Agent through the Broker Stateless State-free

Anatomy of the AppServer 2 Connection Models State-aware State-reset 2 Connection Models Stateless State-free Agent Agent Agent Agent client AppServer Broker Agent

Anatomy of the AppServer AppServer Broker Thread Pools Listening Thread L-3090 Client Threads C-0001 C-0002 C-0003 C-0004 C-0005 C-0006 … AppServer Broker Server Threads S-0001 S-0002 S-0003 S-0004 S-0005

Anatomy of the AppServer Broker communication with Agents Broker starts agent and waits for agent to respond, status set to: STARTING Once started the agent reports to broker and broker sets status to: AVAILABLE Client request agent, is redirected to agent port, broker sets status to: CONNECTING Client connects to agent, agent reports to broker, broker sets status to: CONNECTED Agent completes request, notifies broker, broker sets status to:

Anatomy of the AppServer Reading Log Files [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789" [Main Block - authcard.p] [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run createAuthMsg "John Doe 1234567890 Visa " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa " [performCardAuth - authcard.p] [08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p] [08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application Server disconnected. (8360) Application Server connected with connection id: 172.30.1.47: :asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) AS 4GLTRACE Run connectAuthSvc “-H localhost -S 6789” [performCardAuth - authcard.p]

Anatomy of the AppServer Supporting Processes AdminServer - Required JAVA process Manages all OpenEdge Brokers and Servers Writes to admserv.log NameServer - Optional Traffic manager from clients to AppServers Writes to <nameservername>.ns.log

Agenda Anatomy of the AppServer Diagnosing Problems Help Treating your AppServer

Diagnosing Problems Startup Problems $DLC/bin/asbman -name asbroker1 -start OpenEdge Release 10.1C01 as of Fri May 9 21:58:30 EDT 2008 Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Unable to start asbroker1 (8297)

Diagnosing Problems Startup Problems - <brokername>.broker.log ===================================================================== [08/05/18@20:05:25.933-0400] P-000283 T-Main 1 --- --- /usr1/stat/progress/101c/wrk/asbroker1.broker.log opened. [08/05/18@20:05:25.939-0400] P-000283 T-Main 1 --- --- Logging level set to =2 [08/05/18@20:05:25.940-0400] P-000283 T-Main 1 --- --- Log entry types activated: UBroker.Basic,[08/05/18@20:05:26.020-0400] P-000283 T-Main 2 UB Basic ubroker version: v101c (07-May-08) (8038) [08/05/18@20:05:26.314-0400] P-000283 T-L-9090 2 UB Basic Started listenerthread: L-9090 (8043) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- Could not listen on port: 9090 (8044) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- java.net.BindException: Address already in use (errno:226) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.PlainSocketImpl.socketBind(Native Method) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.ServerSocket.bind(ServerSocket.java:325) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.ServerSocket.<init>(ServerSocket.java:186)

Diagnosing Problems Startup Problems $DLC/bin/asbman -name asbroker1 -start Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Starting asbroker1. Check status. (8296) $DLC/bin/asbman -name asbroker1 -query Broker: asbroker1 not running (8313)

Diagnosing Problems Startup Problems - <brokername>.broker.log [08/05/18@20:15:21.807-0400] P-000435 T-S-0004 2 UB Basic Started server: /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logfile /usr1/stat/progress/101c/wrk/asbroker1.server.log -ubpid 435 -Ms 1 -logname asbroker1 -logentrytypes ASPlumbing,DB.Connects -logthreshold 0 -numlogfiles 3 -ASID 4 –ubpropfile /usr1/stat/progress/101c/dlc/properties/ubroker.properties -svrefresh -ipver IPv4 -db mydatabase (8108) [08/05/18@20:15:22.210-0400] P-000435 T-S-0004 1 UB ---------- Message from server : Startup Procedure Failed (8113) [08/05/18@20:15:22.241-0400] P-000435 T-S-0004 1 UB ---------- IOException reading message from server : java.io.EOFException: pipe to server broken (8117) [08/05/18@20:15:22.241-0400] P-000435 T-S-0004 2 UB Basic ERROR: cannot start server. (8100)

Diagnosing Problems Startup Problems - <brokername>.server.log [08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- ** There is no server active for database mydatabase on -H cruz -S 6543. (704) [08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- Error initializing the application server. (5479) [08/05/18@20:29:13.992-0400] P-000667 T-000000 2 AS AS Application Server Shutdown. (5476) [08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- ** There is no server active for database mydatabase on -H cruz -S 6543. (704) [08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- Error initializing the application server. (5479) [08/05/18@20:29:15.002-0400] P-000668 T-000000 2 AS AS Application Server Shutdown. (5476)

Diagnosing Problems Startup Problems - <brokername>.server.log [08/04/03@13:03:33.470-0400] P-008824 T-004332 2 AS AS Starting application server for asbroker1. (5560) [08/04/03@13:03:33.501-0400] P-008824 T-004332 2 AS AS Application Server Startup. (5473) [08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -- C:\tmp\ex08\roy\eg1\asstart.r Database sports2000 not connected. (1006) [08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -- C:\tmp\ex08\roy\eg1\asstart.r startup procedure ended with an ERROR condition. (8025) [08/04/03@13:03:33.814-0400] P-008824 T-004332 2 AS AS Application Server Shutdown. (5476)

Diagnosing Problems Startup Problems All “INITIAL” agents must start or the AppServer will fail to start Set initial agents to start to zero “initialSrvrInstance=0” in ubroker.properties (IPv6 enabled on a machine can cause agents not to communicate with broker) Add “–Djvmstart.debug=1” to the “jvmargs=“ in AdminServerPlugins.properties View $WRKDIR/jvmStart.log file

Diagnosing Problems Phantom Agents Broker Name : asbroker1 Operating Mode : State-reset Broker Status : ACTIVE Broker Port : 3090 Broker PID : 25319 Active Servers : 5 Busy Servers : 0 Locked Servers : 0 Available Servers : 5 Active Clients (now, peak) : (0, 6) Client Queue Depth (cur, max) : (0, 11) Total Requests : 124 Rq Wait (max, avg) : (259 ms, 11 ms) Rq Duration (max, avg) : (5543 ms, 24 ms) PID State Port nRq nRcvd nSent Started Last Change 25320 CONNECTED 02003 000020 000020 000020 May 19, 2008 06:28 May 19, 2008 07:28 25321 AVAILABLE 02018 000031 000031 000031 May 19, 2008 06:28 May 19, 2008 07:28 25323 AVAILABLE 02019 000014 000014 000014 May 19, 2008 06:28 May 19, 2008 07:28 25322 CONNECTED 02006 000026 000026 000026 May 19, 2008 06:28 May 19, 2008 07:28 25324 AVAILABLE 02021 000033 000033 000033 May 19, 2008 06:28 May 19, 2008 07:28

Diagnosing Problems Phantom Agents protrace.25320 protrace.25322 ps -ef | grep _proapsv statqa 25324 25319 0 06:28:47 ? 0:24 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf statqa 25323 25319 0 06:28:46 ? 0:56 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf statqa 25321 25319 0 06:28:45 ? 0:55 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 –log protrace.25320 protrace.25322

Diagnosing Problems Look at log files Increase logging levels Diagnosing Application problems Look at log files <brokername>.broker.log <brokername>.server.log Database log files where agent is connected Increase logging levels srvrLoggingLevel=4 (extended) srvrLogEntryTypes=4GLTrace UBNet - New for 10.1C

Diagnosing Problems Hanging Agents Broker Name : asbroker1 Operating Mode : Stateless Broker Status : ACTIVE Broker Port : 3090 Broker PID : 25319 Active Servers : 1 Busy Servers : 0 Locked Servers : 0 Available Servers : 1 Active Clients (now, peak) : (0, 1) Client Queue Depth (cur, max) : (0, 1) Total Requests : 11 Rq Wait (max, avg) : (11 ms, 11 ms) Rq Duration (max, avg) : (24 ms, 11 ms) PID State Port nRq nRcvd nSent Started Last Change 25320 SENDING 02003 000001 000001 00001 Apr 4, 2008 11:40 Apr 4, 2008 11:41

Diagnosing Problems Diagnosing Agent problems [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789" [Main Block - authcard.p] [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run createAuthMsg "John Doe 1234567890 Visa " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa " [performCardAuth - authcard.p] [08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p] [08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application Server disconnected. (8360)

Agenda Anatomy of the AppServer Diagnosing Problems Help Treating your AppServer

Help Treating your AppServer asbman -query Get status and history of AppServer broker and agents -kill Shutdown an AppServer with busy or hanging agents May need to run twice in a row

Help Treating your AppServer LogRead Available on PSDN ABL source code available Read and Sort multiple log files Unsupported

Help Treating your AppServer Database Connections Don’t connect using the startup parameters Any database down to a single database will cause your AppServer agents to be unresponsive Connect using the “CONNECT” call in the srvrStartupProc= srvrConnectProc= srvrActivateProc= Use “IF CONNECTED” ABL

Help Treating your AppServer Lifespan New in 10.1B Described in ubroker.properties.README Read the CAUTION! Use only when directed by Tech Support Periodically force the shutdown of all agents and allow new agents to start when requested

Help Treating your AppServer OpenEdge Management

Under Development D I S C L A I M E R The next slide includes information about potential future products and/or product enhancements. What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here. D I S C L A I M E R

Help Treating your AppServer List AppServer Connections New in 10.2A Described in ubroker.properties.README List of all clients known to the AppServer, includes: Host/IP Port Connected Agent More!

Help Treating your AppServer Help from Progress Technical Support Knowledge Base (kbase) PSDN

In Summary Understand your AppServer Diagnose your Problem Care for your Ailing AppServer AppServer Broker Agent client Listening Port Cl ients Agents

? Questions

Thank You