P.C. Burkimsher IT-CO-BE July 2004 Scaling Up PVSS Showstopper Tests Paul Burkimsher IT-CO

Aim of the Scaling Up Project WYSIWYAF Investigate functionality and performance of large PVSS systems Reassure ourselves that PVSS scales to support large systems Provide detail rather than bland reassurances

What has been achieved? 18 months PVSS gone through many pre-release versions –“2.13” –3.0Alpha –3.0Pre-Beta –3.0Beta –3.0RC1 –3.0RC1.5 Lots of feedback to ETM. ETM have incorporated –Design fixes & Bug fixes

Progress of the project Has closely followed the different versions. Some going over the same ground, repeating tests as bugs were fixed. Good news: V3.0 Official Release is now here (even 3.0.1) Aim of this talk: –Summarise where we’ve got to today. –Show that the list of potential “showstoppers” has been addressed

What were the potential showstoppers? Basic functionality –Synchronised types in V2 ! Sheer number of systems –Can the implementation cope? Sheer number of displays Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable?

What were the potential showstoppers? Basic functionality –Synchronised types in V2! Sheer number of systems –Can the implementation cope? Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable? } Skip

Sheer number of systems 130 systems simulated on 5 machines 40,000 DPEs ~5 million DPEs Interconnected successfully

What were the potential showstoppers? Basic functionality –Synchronised types in V2! Sheer number of systems –Can the implementation cope? Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable? } Skip

Alert Avalanche Configuration 2 WXP machines Each machine = 1 system Each system has 5 crates declared x 256 channels x 2 alerts in each channel (“voltage” and “current”) 40,000 DPEs total in each system Each system showed alerts from both systems 9491 UI

Traffic & Alert Generation Simple UI script Repeat –Delay D mS –Change N DPEs Traffic rate D \ N –Bursts. –Not changes/sec. Option provoke alerts

Alert Avalanche Test Results - I You can select which system’s alerts you wish to view UI caches ALL alerts from ALL selected systems. Needs sufficient RAM! (5,000 CAME + 5,000 WENT alerts needed 80Mb) Screen update is CPU hungry and an avalanche takes time(!) –30 sec for 10,000 lines.

Alert Avalanche Test Results - II Too many alerts -> progressive degradation 1) Screen update suspended –Message shown 2) Evasive Action. Event Manager eventually cuts the connection to the UI; UI suicides. –EM correctly processed ALL alerts and LOST NO DATA.

Alert Avalanche Test Results - III Alert screen update is CPU intensive Scattered alert screens behave the same as local ones. (TCP) “Went” alerts that are acknowledged on one alert screen disappear from the other alert screens, as expected. –Bugs we reported have now been fixed.

What were the potential showstoppers? Basic functionality –Synchronised types in V2! Sheer number of systems –Can the implementation cope? Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable?

Agreed Realistic Configuration 3 level hierarchy of machines Only ancestral connections, no peer links. Only direct connections allowed. 40,000 DPEs in each system, 1 sys per machine Mixed platform (W=Windows, L=Linux) L LLLLLLLLLLLL W W W

Viewing Alerts coming from leaf systems 1,000 “came” alerts generated on PC94 took 15 sec to be absorbed by PC91. All 4(2) CPUs in PC91 shouldered the load. Additional alerts then fed from PC93 to the top node. –Same graceful degradation and evasive action seen as before. PC91’s EM killed PC91’s Alert Screen Display is again the bottleneck.

Rate supportable from 2 systems Set up a high, but supportable rate of traffic (10,000 \ 1,000) on each of PC93 and PC94, feeding PC91. PC93 itself was almost saturated, but PC91 coped (~200 alerts/sec average, dual CPU)

Surprise Overload (manual) Manually stop PC93 PC91 pops up a message Manually restart PC93 Rush of traffic to PC91 caused PC93 to overload PC93’s EM killed PC93’s DistM PC91 pops up a message

PVSS Self-healing property PVSS self-healing algorithm –Pmon on PC93 restarts PC93’s DistM

Remarks Evasive action taken by EM, cutting connection, is very good. Localises problems, keeping the overall system intact. Self-healing action is very good. Automatic restart of dead managers BUT…

Evasive action and Self-healing Manually stop PC93 PC91 pops up a message Manually restart PC93 Rush of traffic to PC91 causes PC93 to overload PC93’s EM killed PC93’s DistM PC91 pops up a message Pmon restarts PC93’s DistM

Self-healing Improvement To avoid the infinite loop, ETM’s Pmon eventually gives up. Configurable how soon – Still not ideal! ETM are currently considering my suggestion for improvement: –Pmon should issue the restart, but not immediately.

(Old) Alert Screen We fed back many problems with the Alert Screen during the pre- release trials. –E.g. leaves stale information on- screen when systems leave and come back.

New Alert/Event Screen in V Official release now has a completely new Alert/Event Screen which fixes most of the problems. It’s new and still has some bugs, but the ones we have seen are neither design problems nor showstoppers.

More work for ETM: When DistM is killed by EM taking evasive action, the only indication is in the log. But Log viewer, like Alert viewer, is heavy on CPU and shouldn’t be left running when it’s not needed.

Reconnection Behaviour No gaps in the Alert archive of the machine that isolated itself by taking evasive action. No data was lost. It takes about 20 sec for 2 newly restarted Distribution Managers to get back in contact. Existing (new-style!) alert screens are updated with the alerts of new systems that join (or re-join) the cluster.

Is load of many Alerts reasonable? ~200 alerts/sec average would be rather worrying in a production system. So I believe “Yes”. The response to an overload is very good. Though can still be tweaked. Data integrity is preserved throughout.

What were the potential showstoppers? Basic functionality –Synchronised types in V2! Sheer number of systems –Can the implementation cope? Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable? 

Can you see the baby?

What were the potential showstoppers? Basic functionality –Synchronised types in V2! Sheer number of systems –Can the implementation cope? Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable?

Is the load of many Trends reasonable? Same configuration: Trend windows were opened on PC91 displaying data from more and more systems. Mixed platform.

Is Memory Usage Reasonable? RAM (MB) Steady state, no trends open on PC91593 Open plot ctrl panel on On PC91, open a 1 channel trend window from PC03658 On PC91, open a 1 channel trend window from PC04657 On PC91, open a 1 channel trend window from PC05657 On PC91, open a 1 channel trend window from PC06658 On PC91, open a 1 channel trend window from PC07658 Yes

Is Memory Usage Reasonable? RAM Steady state, no trends open on PC91602 On PC91, open 16 single channel trend windows from PC95Crate1Board1604 On PC91, open 16 single channel trend windows from PC03Crate1Board1607 On PC91, open 16 single channel trend windows from PC04Crate1Board1610 Yes

Test 34: Looked at top node plotting data from leaf machines’ archives Performed excellently. Test ceased when we ran out of screen real estate to show even the iconised trends (48 of).

Bland result? No! Did the tests go smoothly? No! –But there was good news at the end

Observed gaps in the trend!! Investigation showed gap was correct –Remote Desktop start-up caused CPU load –Data changes were not generated at this time Zzzzzzz

Proof with a Scattered Generator Steady traffic generation No gaps in the recorded archive –Even when deliberately soak up CPU Gaps were seen in the display –Need a “Trend Refresh” button (ETM) Scattered UI on PC93 Traffic EM Trend UI on PC94 Zzzzzzz

Would sustained overload give trend problems? High traffic (400mS delay\1000 changes) on PC93, as a scattered member of PC94’s system. PC94’s own trend plot could not keep up. PC91’s trend plot could not keep up. “Not keep up” means… Zzzzzzz

“Display can’t keep up” means… Trend screen values updated to here Timenow Zzzzzzz

Evasive action Trend screen values finally updated to here Timen ow EM took evasive action, (disconnected the traffic generator) just here Last 65sec queued in Traffic Generator. Lost when it suicided. Zzzzzzz

Summary of Multiple Trending PVSS can cope PVSS is very resilient to overload Successful tests. Wakey!

Test 31 DP change rates Measured saturation rates on different platform configurations. No surprises. Faster machines with more memory are better. Linux is better than Windows. Numbers on the Web.

Test 32 DP changes with alerts Measured saturation rates; no surprises again. Dual CPU can help in processing when there are a lot of alert screen (user interface) updates.

What were the potential showstoppers? Basic functionality –Synchronised types in V2! Sheer number of systems –Can the implementation cope? Alert Avalanches –How does PVSS degrade? Is load of many Alerts reasonable? Is load of many Trends reasonable ? Conclusions

No showstoppers. We have seen nothing to suggest that PVSS cannot be used to build a very big system.

Further work - I Further “informational” tests will be conducted to assist in making configuration recommendations, eg understanding the configurability of the message queuing and evasive action mechanism. Follow up issues such as “AES needed more CPU when scattered”. Traffic overload from a SIM driver rather than a UI Collaborate with Peter C. to perform network overload tests.

Further work – II Request a Use Case from experiments for a non-stressed configuration: –Realistic sustained alert rates –Realistic peak alert rate + realistic duration i.e. not a sustained avalanche –How many users connected to control room machine? –% viewing alerts; % viewing trends; % viewing numbers (eg CAEN voltages) –Terminal Server UI connections –How many UIs can control room cope with? What recommendations do you want?

In greater detail… The numbers behind these slides will soon be available on the Web at jects/ScalingUpPVSS/welcome.html jects/ScalingUpPVSS/welcome.html Any questions?

Can you see the baby?

Example Numbers CPU PC92Linux2.2 x \1000 PC93W \500 PC94WXP \1000 PC95Linux \1000 PC03Linux \1000 Table showing the Traffic Rates on different machine configurations, that gave rise to 70% CPU usage on those machines. See the Web links for the original table and details on how to interpret the figures.