Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 PUG Challenge Americas 2013 Click to edit Master title style PUG Challenge Americas 2013 – Westford, MA The Deepest Depths of promon Presented by: Gus.

Similar presentations


Presentation on theme: "1 PUG Challenge Americas 2013 Click to edit Master title style PUG Challenge Americas 2013 – Westford, MA The Deepest Depths of promon Presented by: Gus."— Presentation transcript:

1 1 PUG Challenge Americas 2013 Click to edit Master title style PUG Challenge Americas 2013 – Westford, MA The Deepest Depths of promon Presented by: Gus Bjorklund & Dan Foreman And how it may help in troubleshooting certain DB problems

2 2 PUG Challenge Americas 2013 Gus Bjorklund Progress Wizard

3 3 PUG Challenge Americas 2013 Dan Foreman Progress User since 1984 Author of several Progress related Publications News Flash – A new publication which is a superset of this presentation, titled promon debghb, will be available shortly Author of several cool and useful Progress DBA Tools –ProMonitor & ProCheck & LockMon –Pro Dump&Load –Balanced Benchmark Basketball Fanatic…which sometimes leads to unexpected trips to the ER

4 4 PUG Challenge Americas 2013 Dan Foreman

5 5 PUG Challenge Americas 2013 Brief History of the debghb option in promon Added to promon V6.3 by Gus Purpose: The shared memory architecture introduced in V6.3 was quite a bit different and a way to monitor shared memory activity at a detailed level was needed The debghb option was not a formally endorsed enhancement but written by Gus in his spare time deb = DEBug ghb = Gus’s initials….the middle initial is a Top Secret fanatically guarded by the Finnish government

6 6 PUG Challenge Americas 2013 Warnings The debghb option is not documented by Progress DO NOT call or Gus or PTS for help in using this option Many of the screens and/or metrics have no value to a DBA The view of the data is not transactionally consistent sometimes even on the same screen; example to follow Some of the data is not accurate (data overflow, rounding errors, etc) Some of the screens are broken (don’t display any data) The debghb option can be altered, removed, or hidden by Progress any time they want to

7 7 PUG Challenge Americas 2013 Warnings #2 DB activities in shared memory can be slowed down if certain options are enabled 05/08/13 OpenEdge Release 10 Monitor (R&D) 19:10:44 Adjust Latch Options 1. Spins before timeout: Enable latch activity data collection 3. Enable latch timing data collection 4. Initial latch sleep time: 10 milliseconds 5. Maximum latch sleep time: 5000 milliseconds 6. Record Free Chain Search Depth Factor: 5

8 8 PUG Challenge Americas 2013 How do I access debghb? Start promon Enter: R&D (also works in lower case starting in V10) Enter: debghb You have now entered the debghb “zone” Two main differences in the world of debghb –Extensions to some existing R&D screens –Enables access to “This menu is not here Menu” Enter: “6” even though there is no visible option 6 See next slide

9 9 PUG Challenge Americas 2013 This menu is not here Menu 06/06/13 OpenEdge Release 10 Monitor (R&D) 15:23:01 This menu is not here Menu 1. Cache Entries 2. Hash Chain 3. Page Writer Queue 4. Lru Chains 5. Locked Buffers 6. Buffer Locks 7. Buffer Use Counts 8. Resource Queues 9. TXE Lock Activity 10. Adjust TXE Options 11. Latch Counts 12. Latch Times 13. I/O Wait Time by Type 14. I/O by File 15. Buffer Lock Queue 16. Semaphores 17. Shutdown

10 10 PUG Challenge Americas 2013 Operating Hints Allow at least lines of screen data Allow at least columns of screen width Zero out the stats (“z”) to get a clean starting place –This ‘zeroing’ does not wipe out the actual shared memory counters but only affects the current promon session Update the stats periodically (“u”) to get snapshots All the above can be scripted

11 11 PUG Challenge Americas 2013 Operating Hints - User# -1 Usecnt = # of concurrent processes accessing the block When initially examining the BLQ there were 5 Clients accessing the same DBKEY But before all 5 could be displayed: –One Client dropped off, i.e. released the Buffer Lock, before they could be displayed –Another one of the 5 is partially displayed; i.e. the -1 User# 02/06/13 Status: Buffer Lock Queue 00:37:21 User DBKEY Area T Status Type Usect I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE 3

12 12 PUG Challenge Americas 2013 Useful Screens - Checkpoints Extensions to the ‘normal’ Checkpoint screen Columns of interest –Duration: the amount of time required to complete a Checkpoint; the entire Database is transactionally frozen during this time _CheckPoint._CheckPoint-Duration (V10.2B SP5) –Sync Time: a subset of the ‘Duration’ column; the amount of time required to execute fdatasync() system call _CheckPoint._CheckPoint.Synctime (V10.2B SP5) See for an excellent description of fdatasync (don’t confuse it with fsync).http://www.makelinux.net/alp/060 Sample data on the next slide

13 13 PUG Challenge Americas 2013 Useful Screens - Checkpoints The ‘Duration’ of the Checkpoints (i.e. the total freeze time) is very high for most of the CPs displayed A ‘Duration’ of less than 1 second is a good goal The 10 sec ‘Duration’ is approximately 1/3 of the CP ‘Len’ In other words, a CP is occurring approximately every 30 seconds and for 10 seconds of that period, NO transaction activity can take place. Ckpt Database Writes ---- No. Time Len Freq Dirty CPT Q Scan APW Q Flushes Duration Sync Time :56: :52: :51: :51: :50: :49: :49:

14 14 PUG Challenge Americas 2013 Useful Screens – Resource Queues NHM (Not Here Menu) - Option #8 Do not confuse Resources with Latches Link to Banville’s presentation In general the busiest locks will be: –DB Buf S Lock –DB Buf X Lock –Record Lock Waits that can be problematic: –DB Buf I Lock (I = Intent but these are for Index blocks) –Sample on the next slide

15 15 PUG Challenge Americas 2013 Useful Screens – Resource Queues 01/31/13 Activity: Resource Queues 00:31:57 01/31/13 00:26 to 01/31/13 00:31 (5 min) Queue - Requests Waits Total /Sec Total /Sec Pct Record Lock Trans Commit DB Buf I Lock Record Get DB Buf Read DB Buf Write DB Buf S Lock DB Buf X Lock DB Buf S Lock LRU DB Buf X Lock LRU DB Buf Write LRU BI Buf Read BI Buf Write TXE Share Lock TXE Update Lock TXE Commit Lock

16 16 PUG Challenge Americas 2013 Useful Screens – Latch Counts NHM (Not Here Menu) - Option #11 The R&D Blocked Clients screen doesn’t show Latch contention so debghb is the only place in promon where detailed Latch activity is visible Definition of Naps: When –spin is ‘used up’ by a Progress Client, the process Naps (i.e. does no useful work) for a while and tries again General Principle: Napping is bad Samples on the next few slides

17 17 PUG Challenge Americas 2013 Latch Counts – OM Latch OM (Object Cache) Latch activity can be totally eliminated by setting the -omsize parameter equal to or greater than the number of _StorageObject records. 04/24/13 Activity: Latch Counts 00:59:28 04/24/13 00:54 to 04/24/13 00:59 (5 min 1 sec) Locks Busy --- Naps Spins Nap Max - Owner Total /Sec /Sec Pct /Sec /Sec /Lock /Busy Total HWM MTX USR OM

18 18 PUG Challenge Americas 2013 Latch Counts – USR Latch The small contention on the USR (DB Connection Table) Latch is because Statement Caching is enabled 04/25/13 Activity: Latch Counts 00:33:17 04/25/13 00:28 to 04/25/13 00:33 (5 min 0 sec) Locks Busy --- Naps Spins Nap Max Owner Total /Sec /Sec Pct /Sec /Sec /Lock /Busy Total HWM MTX USR OM BIB SCH LKP GST TXT SEQ AIB TXQ EC LKF BFP BHT PWQ CPQ LRU LRU BUF

19 19 PUG Challenge Americas 2013 Latch Counts – LRU Chain The total number of Locks for LRU is the second highest of all the Resources shown (BHT – Buffer Hash Table – is #1) The # of Naps per Second is the highest of all latches (Zero is the goal) 01/31/13 Activity: Latch Counts 00:05:38 01/31/13 00:00 to 01/31/13 00:05 (5 min 0 sec) Locks Busy Naps Owner Total /Sec /Sec Pct /Sec MTX OM BHT CPQ LRU LRU BUF BUF BUF BUF

20 20 PUG Challenge Americas 2013 Latch Counts – LRU Chain The # of locks on the second LRU (Alternate Buffer Cache) is nil because all the ABC Objects completely fit in the amount of –B2 memory allocated 01/31/13 Activity: Latch Counts 00:05:38 01/31/13 00:00 to 01/31/13 00:05 (5 min 0 sec) Locks Busy Naps Owner Total /Sec /Sec Pct /Sec BHT CPQ LRU LRU BUF BUF

21 21 PUG Challenge Americas 2013 Latch Counts – LRU Chain ‘Owner’ column: if the User# doesn’t change (in value or frequency) that can be a problem indicator because Latches should be held for only a fraction of a second 01/31/13 Activity: Latch Counts 00:05:38 01/31/13 00:00 to 01/31/13 00:05 (5 min 0 sec) Locks Busy Naps Owner Total /Sec /Sec Pct /Sec BHT CPQ LRU LRU BUF BUF

22 22 PUG Challenge Americas 2013 Using Latch Counts to set -spin Short answer – Forget It! If it was that easy Progress would have done it already Past attempts have not been successful Also the optimal value of –spin is not going to be the same for each Latch General guidelines: –Greater than 1000 –Less than –Current Default: 6000 * (# of CPU Cores) Default not advised if you have more than 16 Cores –Dan’s (Patent Pending) Formula: (DBA-Birthday-Year *  ) –Gus’s formula: 5000

23 23 PUG Challenge Americas 2013 Useful Screens – Buffer Lock Queue NHM (Not Here Menu) - Option #15 The ‘normal’ R&D Blocked Clients screen does not show the Area that the DBKEY belongs to The Buffer Lock Queue (BLQ) Screen shows the Area as well as the Block Type Examples on the next two slides

24 24 PUG Challenge Americas 2013 R&D Blocked Clients The R&D Blocked Clients screen doesn’t show enough information to identify the Object involved in this contention storm for DBKEY There were 29 Clients all blocked on the same DBKEY 01/31/13 Status: Blocked Clients 00:26:41 Usr Name Type Wait Wait Info Trans id Login time 730 _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23: _AUTO-B SELF/ABL BKSH /30/13 23:22

25 25 PUG Challenge Americas 2013 Buffer Lock Queue IF there is a matching DBKEY on the the BLQ screen, we can get the Area# and the Block Type (I = Index) There were 29 processes on the Blocked Clients screen with this DBKEY and only 4 on the BLQ screen with the same DBKEY 01/31/13 Status: Buffer Lock Queue 00:26:41 User DBKEY Area T Status Type Usect I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE I LOCKED SHARE 4

26 26 PUG Challenge Americas 2013 Thank You! Questions? Gus: Dan:


Download ppt "1 PUG Challenge Americas 2013 Click to edit Master title style PUG Challenge Americas 2013 – Westford, MA The Deepest Depths of promon Presented by: Gus."

Similar presentations


Ads by Google