1. OPS-6: Caring for an Ailing AppServer ™ Hugo Loera Chavez Principal Tech Support Engineer

2. © 2008 Progress Software Corporation2 Agenda  Anatomy of the AppServer  Diagnosing Problems  Help Treating your AppServer AppServer Broker ClientsClients AgentsAgents Agent clien t Listening Port

3. © 2008 Progress Software Corporation3 Anatomy of the AppServer What is the OpenEdge ® AppServer?

4. © 2008 Progress Software Corporation4 Anatomy of the AppServer  Broker Java TM process Handles communication from clients to agents Manages client and agent pools Writes to.broker.log Or What is the Progress ® AppServer?.broker.log

5. © 2008 Progress Software Corporation5 Anatomy of the AppServer  Agent “C” process Handles requests and replies to the clients Communicates “STATUS” to broker Writes to.server.log Or What is the Progress AppServer?.server.log

6. © 2008 Progress Software Corporation6 Anatomy of the AppServer  State-aware  State-reset  Stateless  State-free 4 Operating Modes One to one Many to few - Default

7. © 2008 Progress Software Corporation7 Anatomy of the AppServer  Client to Broker gets redirected to Agent State-aware State-reset  Client to Agent through the Broker Stateless State-free 2 Connection Models

8. © 2008 Progress Software Corporation8 Anatomy of the AppServer 2 Connection Models State-aware State-reset clien t AppServer Broker Agent 2 Connection Models Stateless State-free

9. © 2008 Progress Software Corporation9 Anatomy of the AppServer AppServer Broker Thread Pools AppServer Broker Listening Thread L-3090 Client Threads C-0001 C-0002 C-0003 C-0004 C-0005 C-0006 … Server Threads S-0001 S-0002 S-0003 S-0004 S-0005

10. © 2008 Progress Software Corporation10 Anatomy of the AppServer Broker communication with Agents Broker starts agent and waits for agent to respond, status set to: STARTING Once started the agent reports to broker and broker sets status to: AVAILABLE Client request agent, is redirected to agent port, broker sets status to: CONNECTING Client connects to agent, agent reports to broker, broker sets status to: CONNECTED Agent completes request, notifies broker, broker sets status to: AVAILABLE

11. © 2008 Progress Software Corporation11 Anatomy of the AppServer [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789" [Main Block - authcard.p] [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run createAuthMsg "John Doe 1234567890 Visa " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa " [performCardAuth - authcard.p] [08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p] [08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application Server disconnected. (8360) Reading Log Files [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47: :asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) AS 4GLTRACE Run connectAuthSvc “-H localhost -S 6789” [performCardAuth - authcard.p]

12. © 2008 Progress Software Corporation12 Anatomy of the AppServer  AdminServer - Required JAVA process Manages all OpenEdge Brokers and Servers Writes to admserv.log  NameServer - Optional JAVA process Traffic manager from clients to AppServers Writes to.ns.log Supporting Processes

13. © 2008 Progress Software Corporation13 Agenda  Anatomy of the AppServer  Diagnosing Problems  Help Treating your AppServer

14. © 2008 Progress Software Corporation14 Diagnosing Problems $DLC/bin/asbman -name asbroker1 -start OpenEdge Release 10.1C01 as of Fri May 9 21:58:30 EDT 2008 Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Unable to start asbroker1 (8297) Startup Problems

15. © 2008 Progress Software Corporation15 Diagnosing Problems ===================================================================== [08/05/18@20:05:25.933-0400] P-000283 T-Main 1 --- --- /usr1/stat/progress/101c/wrk/asbroker1.broker.log opened. [08/05/18@20:05:25.939-0400] P-000283 T-Main 1 --- --- Logging level set to =2 [08/05/18@20:05:25.940-0400] P-000283 T-Main 1 --- --- Log entry types activated: UBroker.Basic,[08/05/18@20:05:26.020-0400] P-000283 T-Main 2 UB Basic ubroker version: v101c (07-May-08) (8038) [08/05/18@20:05:26.314-0400] P-000283 T-L-9090 2 UB Basic Started listenerthread: L- 9090 (8043) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- Could not listen on port: 9090 (8044) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- java.net.BindException: Address already in use (errno:226) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.PlainSocketImpl.socketBind(Native Method) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.ServerSocket.bind(ServerSocket.java:325) [08/05/18@20:05:26.394-0400] P-000283 T-L-9090 1 UB ---------- at java.net.ServerSocket. (ServerSocket.java:186) Startup Problems -.broker.log

16. © 2008 Progress Software Corporation16 Diagnosing Problems $DLC/bin/asbman -name asbroker1 -start Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Starting asbroker1. Check status. (8296) $DLC/bin/asbman -name asbroker1 -query Connecting to Progress AdminServer using rmi://localhost:20931/Chimera (8280) Searching for asbroker1 (8288) Connecting to asbroker1 (8276) Broker: asbroker1 not running (8313) Startup Problems

17. © 2008 Progress Software Corporation17 Diagnosing Problems [08/05/18@20:15:21.807-0400] P-000435 T-S-0004 2 UB Basic Started server: /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logfile /usr1/stat/progress/101c/wrk/asbroker1.server.log -ubpid 435 -Ms 1 -logname asbroker1 -logentrytypes ASPlumbing,DB.Connects - logthreshold 0 -numlogfiles 3 -ASID 4 –ubpropfile /usr1/stat/progress/101c/dlc/properties/ubroker.properties -svrefresh - ipver IPv4 -db mydatabase (8108) [08/05/18@20:15:22.210-0400] P-000435 T-S-0004 1 UB ---------- Message from server : Startup Procedure Failed (8113) [08/05/18@20:15:22.241-0400] P-000435 T-S-0004 1 UB ---------- IOException reading message from server : java.io.EOFException: pipe to server broken (8117) [08/05/18@20:15:22.241-0400] P-000435 T-S-0004 2 UB Basic ERROR: cannot start server. (8100) Startup Problems -.broker.log

18. © 2008 Progress Software Corporation18 Diagnosing Problems [08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- ** There is no server active for database mydatabase on -H cruz -S 6543. (704) [08/05/18@20:29:13.991-0400] P-000667 T-000000 1 AS -- Error initializing the application server. (5479) [08/05/18@20:29:13.992-0400] P-000667 T-000000 2 AS AS Application Server Shutdown. (5476) [08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- ** There is no server active for database mydatabase on -H cruz -S 6543. (704) [08/05/18@20:29:15.001-0400] P-000668 T-000000 1 AS -- Error initializing the application server. (5479) [08/05/18@20:29:15.002-0400] P-000668 T-000000 2 AS AS Application Server Shutdown. (5476) Startup Problems -.server.log

19. © 2008 Progress Software Corporation19 Diagnosing Problems [08/04/03@13:03:33.470-0400] P-008824 T-004332 2 AS AS Starting application server for asbroker1. (5560) [08/04/03@13:03:33.501-0400] P-008824 T-004332 2 AS AS Application Server Startup. (5473) [08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -- C:\tmp\ex08\roy\eg1\asstart.r Database sports2000 not connected. (1006) [08/04/03@13:03:33.814-0400] P-008824 T-004332 1 AS -- C:\tmp\ex08\roy\eg1\asstart.r startup procedure ended with an ERROR condition. (8025) [08/04/03@13:03:33.814-0400] P-008824 T-004332 2 AS AS Application Server Shutdown. (5476) Startup Problems -.server.log

20. © 2008 Progress Software Corporation20 Diagnosing Problems  All “INITIAL” agents must start or the AppServer will fail to start Set initial agents to start to zero –“initialSrvrInstance=0” in ubroker.properties (IPv6 enabled on a machine can cause agents not to communicate with broker)  Add “–Djvmstart.debug=1” to the “jvmargs=“ in AdminServerPlugins.properties View $WRKDIR/jvmStart.log file Startup Problems

21. © 2008 Progress Software Corporation21 Diagnosing Problems C-0001>(Jul 21, 2006 08:46:31:011) Client connected :. (8533) C-0001>(Jul 21, 2006 08:46:31:011) ConnectionID= lockesxe1::appsrv::7120::99729deaf6c56e0b:653108:10c765b99a3:- ecb. (8096) C-0001>(Jul 21, 2006 08:46:31:012) Client disconnected :. (8534) C-0005>(Jul 21, 2006 08:47:22:786) Client connected :. (8533) C-0005>(Jul 21, 2006 08:47:22:786) ConnectionID= lockesxe1::appsrv::7120::99729deaf6c56e0b:653108:10c765b99a3:- eca. (8096) S-2013>(Jul 21, 2006 08:47:25:794) Started server thread: S-2013. (8101) S-2013>Server exec error : Too many open files (8109) S-2013>(Jul 21, 2006 08:47:26:011) ERROR: cannot start server. (8100) Another Startup Problem -.broker.log

22. © 2008 Progress Software Corporation22 Diagnosing Problems Output to apps_ulimit.txt. OS-Command 'date'. OS-Command 'ulimit -n '. Output close.  Simple troubleshoot Appserver program

23. © 2008 Progress Software Corporation23 Diagnosing Problems Broker Name : asbroker1 Operating Mode : State-reset Broker Status : ACTIVE Broker Port : 3090 Broker PID : 25319 Active Servers : 5 Busy Servers : 0 Locked Servers : 0 Available Servers : 5 Active Clients (now, peak) : (0, 6) Client Queue Depth (cur, max) : (0, 11) Total Requests : 124 Rq Wait (max, avg) : (259 ms, 11 ms) Rq Duration (max, avg) : (5543 ms, 24 ms) PID State Port nRq nRcvd nSent Started Last Change 25320 CONNECTED 02003 000020 000020 000020 May 19, 2008 06:28 May 19, 2008 07:28 25321 AVAILABLE 02018 000031 000031 000031 May 19, 2008 06:28 May 19, 2008 07:28 25323 AVAILABLE 02019 000014 000014 000014 May 19, 2008 06:28 May 19, 2008 07:28 25322 CONNECTED 02006 000026 000026 000026 May 19, 2008 06:28 May 19, 2008 07:28 25324 AVAILABLE 02021 000033 000033 000033 May 19, 2008 06:28 May 19, 2008 07:28 Phantom Agents

24. © 2008 Progress Software Corporation24 Diagnosing Problems ps -ef | grep _proapsv statqa 25324 25319 0 06:28:47 ? 0:24 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf statqa 25323 25319 0 06:28:46 ? 0:56 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 -logf statqa 25321 25319 0 06:28:45 ? 0:55 /usr1/stat/progress/101c/dlc/bin/_proapsv -logginglevel 2 –log Phantom Agents protrace.25320 protrace.25322

25. © 2008 Progress Software Corporation25 Diagnosing Problems  Look at log files.broker.log.server.log Database log files where agent is connected  Increase logging levels srvrLoggingLevel=4 (extended) srvrLogEntryTypes=4GLTrace –UBNet - New for 10.1C Diagnosing Application problems

26. © 2008 Progress Software Corporation26 Diagnosing Problems Broker Name : asbroker1 Operating Mode : Stateless Broker Status : ACTIVE Broker Port : 3090 Broker PID : 25319 Active Servers : 1 Busy Servers : 0 Locked Servers : 0 Available Servers : 1 Active Clients (now, peak) : (0, 1) Client Queue Depth (cur, max) : (0, 1) Total Requests : 11 Rq Wait (max, avg) : (11 ms, 11 ms) Rq Duration (max, avg) : (24 ms, 11 ms) PID State Port nRq nRcvd nSent Started Last Change 25320 SENDING 02003 000001 000001 00001 Apr 4, 2008 11:40 Apr 4, 2008 11:41 Hanging Agents

27. © 2008 Progress Software Corporation27 Diagnosing Problems [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS AS Application Server connected with connection id: 172.30.1.47::asbroker1::3090::b0d80cae72::1191a17d1f0:-7ffc. (8358) [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run performCardAuth "John Doe 1234567890 Visa -H localhost -S 6789" [Main Block - authcard.p] [08/04/04@11:41:31.334-0400] P-001268 T-009968 2 AS 4GLTRACE Run connectAuthSvc "-H localhost -S 6789 " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run createAuthMsg "John Doe 1234567890 Visa " [performCardAuth - authcard.p] [08/04/04@11:41:31.350-0400] P-001268 T-009968 2 AS 4GLTRACE Run sendAuthMsg "1000 Name=John Doe;Number=1234567890;Type=Visa " [performCardAuth - authcard.p] [08/04/04@11:42:07.397-0400] P-001268 T-009968 2 AS 4GLTRACE Run disconnectAuthSvc "1000" [performCardAuth - authcard.p] [08/04/04@11:42:07.428-0400] P-001268 T-009968 2 AS AS Application Server disconnected. (8360) Diagnosing Agent problems

28. © 2008 Progress Software Corporation28 Agenda  Anatomy of the AppServer  Diagnosing Problems  Help Treating your AppServer

29. © 2008 Progress Software Corporation29 Help Treating your AppServer  -query Get status and history of AppServer broker and agents  -kill Shutdown an AppServer with busy or hanging agents May need to run twice in a row asbman

30. © 2008 Progress Software Corporation30 Help Treating your AppServer  Available on PSDN  ABL source code available  Read and Sort multiple log files  Unsupported LogRead

31. © 2008 Progress Software Corporation31 Help Treating your AppServer  Don’t connect using the startup parameters Any database down to a single database will cause your AppServer agents to be unresponsive  Connect using the “CONNECT” call in the srvrStartupProc= srvrConnectProc= srvrActivateProc=  Use “IF CONNECTED” ABL Database Connections

32. © 2008 Progress Software Corporation32 Help Treating your AppServer  New in 10.1B  Described in ubroker.properties.README  Read the CAUTION! Use only when directed by Tech Support  Periodically force the shutdown of all agents and allow new agents to start when requested Lifespan

33. © 2008 Progress Software Corporation33 Help Treating your AppServer OpenEdge Management

34. © 2008 Progress Software Corporation34 D I S C L A I M E R Under Development  The next slide includes information about potential future products and/or product enhancements.  What I am going to say reflects our current thinking, but the information contained herein is preliminary and subject to change. Any future products we ultimately deliver may be materially different from what is described here. D I S C L A I M E R

35. © 2008 Progress Software Corporation35 Help Treating your AppServer  New in 10.2A  Described in ubroker.properties.README  List of all clients known to the AppServer, includes: Host/IP Port Connected Agent More! List AppServer Connections

36. © 2008 Progress Software Corporation36 Help Treating your AppServer  Technical Support  Knowledge Base (kbase)  PSDN Help from Progress

37. © 2008 Progress Software Corporation37 AppServer Broker In Summary  Understand your AppServer  Diagnose your Problem  Care for your Ailing AppServer Agent clien t Listening Port ClientsClients AgentsAgents

38. © 2008 Progress Software Corporation38 Relevant PSDN Live Sessions  OPS-7: Building and Deploying a Highly Available Application  DEV-8: AppServer Mode Case Studies  DEV-13 You've got a problem, here’s how to find it

39. © 2008 Progress Software Corporation39 Questions ?

40. © 2008 Progress Software Corporation40 Thank You

41. © 2008 Progress Software Corporation41