1. Load Balancing and Intelligent Load Balancing Jesús González Escalation Engineer

2. Objectives Deep understanding of load balancing architecture Troubleshooting techniques Identify the root cause of any LB problem

3. Agenda Architecture Data Collector Updates - Examples Performance Data Helper Citrix and the PDH Where can it go wrong? Black hole effect (Load Throttling - Intelligent Load Balancing) Some servers get all the connections Full load after installing an MUI Troubleshooting

4. Architecture

5. Data Collector DC - Dynamic Store DS LmsSS.dll - IMA Subsystem MFRules.dll - XA counters LMS20Rules.dll - System counters PDH.dll - Performance Data Helper

6. Data Collector Updates Data Collector is updated every 30 s Only when the change in the load evaluator is bigger than 5% Every 5 minutes we send a full update Connection logon or logoff

7. Data Collector Updates Server load - CPU Load example Data Collector XenApp Time 30s60s 15202325235098344556 X20232523509834455640 X 23252350983445564012 384140 Server Load % : 5 minutes

8. Data Collector Updates XA CPU Load != Task Manager CPU Load CPU load XA = 25% Task Manager = 1%

9. Ctxnotif.dll MFSrvSS.dll LMSSS.DLL Data Collector Updates At Logon time - Default load evaluator Data Collector ICA Client Dynamic Store Server A Server B IMA Server B Server A BIAS WI/XML IMA X

10. Performance Data Helper – API The performance data helper interface calls the registry interface to retrieve performance data Uses the PDH.DLL to access the PDH API http://msdn2.microsoft.com/en-us/library/aa373083.aspx

11. Performance Data Helper – C++ Example pdhStatus = PdhAddCounter (hQuery, "\\Processor(0)\\% Processor Time", 0, &hCounter); YOU MUST CALL THE PERFORMANCE COUNTER BY ITS NAME

12. Citrix and the PDH \\Processor(_Total)\\% Processor Time \\System\\Context Switches/sec \\Memory\\% Committed Bytes In Use \\Memory\\Page Faults/sec \\Memory\\Pages/sec \\PhysicalDisk(_Total)\\Disk Bytes/sec \\PhysicalDisk(_Total)\\Disk Reads/sec \\PhysicalDisk(_Total)\\Disk Writes/sec

13. Performance Data Helper - Registry

14. Performance Data Helper – File system English Perfc009.dat Perfh009.dat German Perfc007.dat Perfh007.dat

15. Performance Data Helper - perfmon

16. Performance Data Helper - IMA IMA (At start up) IMA (At start up)

17. Citrix and the PDH We also provide Performance counters

18. Citrix and the PDH HKLM\SYSTEM\CurrentControlSet\Services\IMAService\Performance

19. Where can it go wrong?

20. Black hole effect ProblemSolution At Peak logon times a recently boot up server will get all connections and might become unresponsive Cause: Server is unable to update the DC Load Throttling Intelligent load balancing HKLM\SOFTWARE\Citrix\IMA\LMS\ (DWORD )UseILB = 1 (DWORD ) ILBMultiplier = 2

21. Before Intelligent Load Biasing BIAS Current Load (0) Max Load (10000) 0 sessions Current Load (100) Max Load (10000) 1 session BIAS Max Load (10000) 2 sessions Current Load (200) 1 session comes 2nd session comes Default BIAS = 10000/100 = 100

22. Intelligent Load Biasing BIAS Current Load (0) Max Load (10000) ILBMultiplier (2) 0 sessions Current Load (5000) Max Load (10000) 1 session BIAS Max Load (10000) 2 sessions Current Load (7500) ILBMultiplier (2) 1 session comes 2nd session comes Load = [(Max Load – Current Load) / ILBMultiplier] + Load

23. Load Throttling- Low logon rate Data Collector ICA Client Dynamic Store Server A Server B Server A Server Load BIAS 0 0 0 0 (10000-0)/2+0 = 5000 100 (10000-0)/2+0 = 5000 100

24. Load Throttling- Low logon rate Server load if Low logon rate

25. Load Throttling- Black hole effect Data Collector ICA Client Dynamic Store Server A Server B Server A Server Load BIAS 0 6000 0 0 (10000-0)/2+0 = 5000 (10000-5000)/2+5000 = 7500 (10000-6000)/2+6000 = 8000 (10000-7500)/2+7500 = 8750

26. Load Throttling Server load at Peak logon times

27. Upgrade to W2K3 causes full load ProblemSolution English W2K \\System\\% Total Processor Time English W2K3 \\Processor(_Total)\\% Processor Time In W2K3, use \\Processor(_Total)\\% Processor Time

28. Some servers get all the connections ProblemSolution 1 server all the connections Cause: Fail to read performance counters causes Load = 0 Fail to read performance counters causes Load = 10000

29. Full load after installing an MUI ProblemSolution After installing MUI with advance load evaluators it results in load = 10000 Cause: ->perfc007.dat (German) perfc009.dat (English) HKLM\Software\Citrix\IMA \LMS\ EnableTranslation=1 (dword)

30. Full load after installing an MUI Jesús González in a German restaurant English Menu (perfc009.dat) 22. Vegetable Soup German Menu (perfc007.dat) 22. Gemüsensuppe

31. Troubleshooting

32. Data Collector DC LmsSS.dll LMS20Rules.dll PDH.dll \\Processor(_Total)\\% Processor Time XXXXXXXXXXXXXXXX Full Load

33. Troubleshooting Failing to read performance counters => full load Use procmon (filemon) while restarting IMA Make sure the correct perfcXXX.dat file can be accessed. Consider rebuilding performance counters http://support.microsoft.com/kb/300956/en-us Check that no performance counter are disabled [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\PerfDisk\Performance] "Disable Performance Counters"=dword:00000001 Use CDFControl to gather CDF traces http://support.citrix.com/article/CTX111961

34. LMSRuleDll_Interface::CreatePDHQuery() ERROR!!!!! rc = -1073738823 -1073738823 -> FFFFFFFFC0000BB9 (HEX) http://msdn2.microsoft.com/en-us/library/aa373046.aspx “The specified counter could not be found” Troubleshooting CDF Traces example

35. Summary Deep understanding of load balancing BIAS, PDH… Understanding Intelligent load balancing Load Throttling Typical scenarios Full load versus 0 load, MUI… Troubleshooting techniques So far we are able to resolve all the problems that arrive to EMEA tech support with those tips

36. Before you leave… Session surveys are available online at www.citrixsynergy.com starting Thursday, 7 October Provide your feedback and pick up a complimentary gift card at the registration desk Download presentations starting Friday, 15 October, from your My Organiser Tool located in your My Synergy Microsite event account