3 BizTalk, Testing, No Time! We all know testing is important for every solutionCustomers rarely test well enough... but get away with itSolutions using BizTalk tend to be extremely business critical and tolerate no downtimePoor testing runs the risk of huge technical and political problemsExtensive testing must always be performed, no excuses!
4 Testing approach “Test early, often and comprehensively” Ensure that a demonstrator is created early on to prove concepts, patterns and indicative performanceToo many solutions are found to perform badly in the last few weeks for any number of reasonsHighlights sub-optimal design up-frontIdentifies where the bottlenecks are likely to beIncreases confidence
5 We sometimes hear... “it doesn’t scale” “it doesn’t meet our performance requirements”“we were told it would do XX msgs/sec”BizTalk is notA universal panaceaOptimised for every customer scenario out-of-the-boxAble to make bad code, good codeBizTalk is part of the overall end-to-end solution
6 A great “case study”“ [BizUnit,LoadGen,MSTest] provides the ability to reliably and automatically execute hundreds of test cases that otherwise would have to be performed manually”“improved turn around times by a factor of upwards of 5. For a test pack containing 100 test cases which took up to 3 days to complete manually we can now execute in approx 3 hrs”
7 Laying the foundations Critical to achieving representative performance within your environmentBenchmark your environmentFile Adapter -> File Adapter (3K) message should get ~500msgs/secDon’t run perf tests on workstation hardwareBeware of virtual machinesInvest in SQL Server Hardware and DisksBeware of UNC FILE Adapter locationsMessage TrackingRemember Virus Scanners
8 Foundation ChecklistDatabases and Logs separated onto separate drive letters and therefore separate physical hard drives?Are the drives used for Databases and Logs configured on a fast disk-array and with RAID10/0+1?Next generation SAN technology tends to use huge caches so RAID choice is becoming unimportantDefault Database Autogrowth settings changes or AutoGrowth disabled?SQL Server Agent Running?Have you created separate Receive, Orchestration and Send Hosts and moved the appropriate Adapters and Orchestrations to the appropriate host?Concurrent Web Service Calls configuration file setting changed?<system.net><connectionManagement><add address="*” maxconnection="80"/></connectionManagement></system.net>
9 PerfMon Counters..... ??A huge selection of performance counters are made available by the BizTalk engineIt can be time consuming to identify the counters you need especially when you have multiple machinesCollection of counters is key
11 Visual Studio Profiler New with Visual Studio Team DeveloperIdeal for profiling custom code: custom pipeline components, .NET class librariesProfiler can be installed standalone for servers..vs_profiler.exe located in the \vs\wcu\profiler directory of the VSTS mediaMust be driven via the command line to attach to a service
12 Configuring the VS Profiler VSPerfClrEnv /globalsampleonVSPerfCmd /start:sample /output:“c:\dir\profileroutput"VSPerfCmd /attach:<PID>VSPerfCmd /detach:<PID>VSPerfCmd /shutdownVSPerfClrEnv /globaloff
14 Code CoverageEnables you to understand what code has been executed during your test runIdentifies holes in your testingIdeal for custom pipeline components, helper assemblies, web services, etc.vsinstr –coverage “YOURASSEMBLY”vsperfmon –coverage –output “filename.coverage”vsperfcmd /shutdown
15 How do I test this? MSMQ BizTalk: DarrenJ Travel Services Wiley Flight Services Booking Web ServiceFlight OrchestrationHotel OrchestrationItinerary OrchestrationItinerary Message[ASMX]MSMQHire Car OrchestrationTravel Insurance OrchestrationDirectEventStreamOrchestrationEventStreamBAMPrimaryImport
16 BizUnitTest Cases are constructed from generic and reusable test stepsTest cases described using an XML syntaxDriven from NUNIT or MSTestModel every “use case” within your solutionRun a testIf you get a “green light” your solution functions at a requirements levelIncredibly powerful, not just for the development lifecycle but for changes, new releases.....Not just about BizTalk
17 Test Steps available “in box” BAMDeploymentStepCheckPop3MailStepContextManipulatorStepCrossReferenceData-VerificationStepDatabaseDeleteStepDatabaseRowCountStepDBQueryStepDelayStepDotNetObjectInvokerStepEventLogCheckStepEventLogClearStepExecuteCommandStepFileCreateStepFileMultiValidateStepFilesExistStepFilesMoveStepFileDeleteMultipleStepFileDeleteStepFileMoveStepFileValidateStepHttpPostStepHttpRequestResponseStepMQSeriesClearQueueStepMQSeriesGetStepMQSeriesPutStepMSMQCreateQueueStepMSMQDeleteQueueStepMSMQQueuePurgeStepMSMQReadStepMSMQWriteStepOrchestrationConductorStepReceiveLocationEnabledStepReceivePortConductorStepRenameDirectoryStepRuleEngineStepSMTPReadStepSOAPHTTPRequest-ResponseStepWaitOnFileStepOutlookReadStepYour own.....
18 BizUnit Test Case Format SetupInitialise the platform ready to perform the testExecutionThe execution of the testCleanupRestore the platform to its state prior to test execution
19 Invoke a web service<TestStep assemblyPath="" typeName="SOAPHTTPRequestResponseStep"><WebServiceWSDLURL>http://..</WebServiceWSDLURL><ServiceName>MyWebServiceName</ServiceName><WebMethod>SubmitItinerary</WebMethod><InputMessageTypeName>Itinerary</InputMessageTypeName><MessagePayload>ItineraryMessage.xml</MessagePayload><ContextLoaderStep assemblyPath="" typeName="XmlContextLoader"><XPath contextKey="itineraryReference">//ItineraryReceived/attribute::ItineraryReference</XPath></ContextLoaderStep></TestStep>
20 Validate a MSMQ message <TestStep assemblyPath="" typeName=“MSMQReadStep"><QueuePath>.\Private$\HotelQ</QueuePath><Timeout>10000</Timeout><ValidationStep assemblyPath="“ typeName=“BinaryValidationStep"><ComparisonDataPath>HotelFragment.xml</ComparisonDataPath><CompareAsUTF8>true</CompareAsUTF8></ValidationStep></TestStep>
23 Orchestration Profiler Code Coverage tools don’t really help with OrchestrationsHow do you understand where time is being spend inside Orchestrations?Enabling tracking and using HAT is the only real way right nowNo way to understand which Orchestrations are not being fully testedEdge ScenariosError paths
25 LoadGen BizTalk Load Generation Tool New 2007 release provides WCF supportGenerate messages to test how servers handle normal and extreme workload conditionsData GenerationLoad ThrottlingFind the maximum sustainable rate of your rig, automatically.Multithreaded“Throw away all those custom test harnesses”, please...
26 “in-box” Load Generators FILEOutputs files in a destination folderHTTPSupports both asynchronous and request-response messaging.SOAPSupports both asynchronous and request-response messagingMSMQMQSeriesMessage can be local or remote, in-order or unordered, persistent or non-persistent.WSEWSSWindows® SharePoint® Services.CustomBase your own on the FileTransport sample
27 In-box Message Creators CustomMCCreate new messages based on a templateReplace Items within the message with differing dataSizeMCExplodes a sample message to a certain size (big batch files)
28 In-box Throttle Monitors SQLSpoolEnsure the Spool depth remains within a certain rangeRate100 msgs/sec..CPUKeep the BizTalk servers within 80-90% CPU utilisationPerfMonMonitor any PerfMon counter
30 LoadGen and BizUnit LoadGen BizUnit BizUnit can drive LoadGen Fantastic load generation toolGenerate increasing loadFind the maximum sustainable point of your solutionBizUnitExtensive library of Test StepsCollecting PerfMon and EventViewer LogsStoring Test InformationValidating Test ResultsBizUnit can drive LoadGenAutomate all of your performance testing end-to-end!!
32 Shameless Plug Further Reading In depth chapters covering Testing, Performance & Scalability..
33 Summary Get the environment right before any testing Start testing early, and often.A “demonstrator” is key for complex scenariosUse BizUnit for modelling “use cases”Use LoadGen for generating loadUse Orchestration Profiler to ensure your “touching” all parts of your solution during testingUse BizUnit and LoadGen together to automate your tests