Managing your BizTalk Environment with SCOM Kent Weare

About Me Working with BizTalk since 2004 Implemented solutions with each version since 3 time Microsoft MVP for BizTalk Server Member of the Microsoft Connected Systems Technology Advisors group Currently manage a BizTalk and Microsoft Tools team in Alberta, Canada

Opps….

Agenda – Part 1 Introduction to System Center Operations Manager (SCOM) SCOM Components SCOM Coverage Management Packs What’s New BizTalk 2010 Management Pack SCOM Console Tour Why using SCOM is important Using SCOM in the real world

Agenda – Part 2 Demos SCOM Performance Footprint Challenges Receive Location Shutdown Host Instance Shutdown Pipeline Error Custom Errors Maintenance Mode SCOM Performance Footprint Challenges Lessons Learned

System Center Operations Manager Currently in its 4th release of the product Microsoft Operations Manager 2000 Microsoft Operations Manager (MOM) 2005 System Center Operations Manager (SCOM) 2007 System Center Operations Manager (SCOM) 2007 R2 vNext Used for Proactive and Reactive Monitoring Leverages WMI to query the state of Applications Also can take advantage of Event Viewer(s) SCOM knows what to monitor based upon Management Packs Part of the System Center Family System Center Configuration Manager (SCCM) System Center Service Manager (SCSM) System Centre Virtual Machine Manager (SCVMM)

SCOM Components

SCOM Coverage SCOM Coverage: Applications: BizTalk, Exchange, SQL Server, Custom Application Hosts: Internet Information Services (IIS) Network/OS: Switches, Routers, Firewalls, Windows Server Hardware: IBM, HP, Dell

Management Packs Management Packs provide SCOM with instructions on how to monitor a particular system or application Microsoft provides many Management Packs out of the box at no charge: Exchange SQL Server Windows Operating System Dynamics SharePoint BizTalk AppFabric 3rd party Management Packs available Blackberry Enterprise Server (BES) Oracle Apache

What’s New in BizTalk 2010 Management Pack Improved Performance Fewer round trips to SCOM Database for minor changes to BizTalk artifacts Optimized discovery of artifacts across multiple machines Clustering Issue Improved Diagnostic Support More detailed information includes in Events Application vs Infrastructure Views

Demo SCOM Console Monitoring Authoring Administration My Workspace Active Alerts Windows Computers Distributed Application Authoring Monitors Rules Administration Management Packs Discovery My Workspace Custom Views specific to “your world” Monitoring Active Alerts Windows Computers Distributed Application Authoring Monitors Rules Administration Management Packs Discovery My Workspace Custom Views specific to “your world”

Why using SCOM is important Want visibility into the health and status of your environment Want Operations personnel to be productive Prevent F5 key from wearing out Know when problems exist before your customers/clients notice Resolve small problems before they become large Example: Service Level Agreements – Energize Request Eliminate the guess work associated with root cause identification and triage Example: ERP Outage

Using SCOM in the Real World Just because things shouldn’t fail, doesn’t mean that they won’t

Using SCOM in the real world Utility Distribution Company 24/7 Contact Centre and Dispatch Business Units BizTalk used to exchange data related to: Meter consumption and Billing Customer Energize and De-energize requests Update Customer Information from Retailers Work Order Management Power Outage Trouble Tickets Automated Metering installations and activations Employee and Vehicle timesheets Field Incident Reporting Incidents Locate Screening Ticket Management Also use SCOM for monitoring: Exchange SharePoint SQL Server Windows Operating Systems Custom Alerts

Using SCOM in the real world Middleware team provides day time and after hours call out support Mixture of Developers and Ops employees Warnings/Errors are Sent to Email distribution group Example: Failed Message in Receive Pipeline Critical Errors are sent to On Call Person via SMS Example: Host Instances going offline

Receive Pipeline Failure – Invalid Data Received Alert: ERROR: There was a failure executing a Receive Pipeline Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 14/08/2010 5:00:23 AM Alert description: There was a failure executing the receive pipeline: “Organization.Project.Pipelines.WOMPostInstallRecieve, Organization.Project.Pipelines, Version=2.0.0.0, Culture=neutral, PublicKeyToken=c93ca0362b603989" Source: "Flat file disassembler" Receive Port: "WOM Receive PostInstalls FailedFind" URI: "SFTP://user@sftp.businesspartner.com:22/F:/Outbound/AMI_ORG_PostInstall_FailedFinds_2*.csv" Reason: No Disassemble stage components can recognize the data.

Receive Failure – Subscriber Not Found Alert: ERROR: FTP-Receive-Message-Suspend Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 13/08/2010 12:45:44 PM Alert description: A message received by adapter "nsoftware.FTP v3" on receive location “ORG_ReceiveHeartBeatIn_FTP_SERVER_FOLDER" with URI "FTP://user@server21/server/folder/outbound/work/YYY_BTS_OR81STD*.xml" is suspended. Error details: The published message could not be routed because no subscribers were found. This error occurs if the subscribing orchestration or send port has not been enlisted, or if some of the message properties necessary for subscription evaluation have not been promoted. Please use the Biztalk Administration console to troubleshoot this failure.  MessageId:  {CADF3FBF-FFAA-4EE1-8315-45A35DCA6264} InstanceID: {E3929CC9-6F4E-46A6-A3C6-7BA7E7ED8840}

Send Port Failure – Network Folder Not Found Alert: ERROR: An outbound message is being suspended by the Adapter Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 13/08/2010 4:10:15 PM Alert description: A message sent to adapter "FILE" on send port "Organization.Project.Orchestrations_1.0.0.0_Organization.Project.Orchestrations.processInternalLargeFiles_SendDynLargeFile_b65c4d63c8f76686" with URI "file://\\Server\Folder\xyFiles\ATSXY.dat" is suspended. Error details: The FILE send adapter cannot open file \\Server\Folder\xyFiles\ATSXY.dat for writing.  Details: The network name cannot be found.   MessageId:  {3B4CCA85-8386-408F-8DA2-1041D01E02D3} InstanceID: {600E858C-FDBC-4DC0-8125-041440EDCFC4}

Send Port Failure – Duplicate File Alert: WARNING: An adapter raised an error during message processing Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 13/08/2010 3:30:04 PM Alert description: The adapter "nsoftware.FTP v3" raised an error message. Details "Transmission failed for message "d6d2c088-98c0-4327-a4f3-1553af2e36d9": Error uploading FTP data: Remote file "TEST.txt" exists and overwrite is set to "False".".

Host Instance Failure Alert: BizTalk Server: Availability Monitor raised an Error on Host Instance Source: EnterpriseClusteredSQLReceive.. Path: Server Last modified by: System Last modified time: 13/08/2010 3:26:24 PM Alert description: Host Instance - EnterpriseClusteredSQLReceive : report an error: The Host Instance  is not running.

Throttling From: Operations Manager Sent: May 21, 2010 12:40 AM To: Middleware Team Subject: “Server” one of the BizTalk server processes in the affected computer is being throttled for significant periods because of high database size exceeding the threshold. Alert: one of the BizTalk server processes in the affected computer is being throttled for significant periods because of high database size exceeding the threshold. Source: HostInstance Path: Server Last modified by: System Last modified time: 21/05/2010 12:40:11 AM Alert description: 0.633333333333333

External Connectivity (MQ Series) Alert: WARNING: An adapter raised an error during message processing Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 15/08/2010 9:41:45 PM Alert description: The adapter "MQSC" raised an error message. Details "Failure encountered while attempting to open queue. queue = QueueName queueManager =QueueManager reasonCode = 2009".

Downstream System Failure (SAP) Alert: ERROR: An outbound message is being suspended by the Adapter Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: SERVER Last modified by: System Last modified time: 21/08/2010 9:49:15 PM Alert description: A message sent to adapter "WCF-Custom" on send port "SendLookupRequestToSAP" with URI "sap://CLIENT=000;LANG=EN;@a/SAPSYSTEM/00?RfcSdkTrace=False&AbapDebug=False" is suspended. Error details: Microsoft.ServiceModel.Channels.Common.ConnectionException: Details: ErrorCode=RFC_OK. ErrorGroup=RFC_ERROR_COMMUNICATION. SapErrorMessage=SAP_CMINIT3 : rc=20 > Connect to SAP gateway failed Connect_PM GWHOST=SAPSYSTEM, GWSERV=SAPGATEWAY, SYSNR=00 LOCATION CPIC (TCP/IP) on local host with Unicode ERROR partner ‘00.00.0.00:0000' not reached TIME Sat Aug 21 21:49:14 201 RELEASE 700 COMPONENT NI (network interface) VERSION 38 RC -10 MODULE nixxi.cpp LINE 2823 DETAIL NiPConnect2 SYSTEM CALL connect ERRNO 10061 ERRNO TEXT WSAECONNREFUSED: Connection refused COUNTER 45. AdapterErrorMessage=. ---> Microsoft.Adapters.SAP.RFCException: Details: ErrorCode=RFC_OK. ErrorGroup=RFC_ERROR_COMMUNICATION. SapErrorMessage=SAP_CMINIT3 : rc=20 > Connect to SAP gateway failed

Downstream System Failure (SOAP) Alert: WARNING: An adapter failed to transmit an outbound message Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 14/08/2010 12:32:36 AM Alert description: The adapter failed to transmit message going to send port “Business Partner Metering WebService" with URL "https://partnerserver.com/project/webapi/webservice.asmx". It will be retransmitted after the retry interval specified for this Send Port. Details:"System.Web.Services.Protocols.SoapException: Object reference not set to an instance of an object.    at Partner.Project.WebAPI.Metering.AddMeterToInventory(MeterManufacturerData[] meters)

Downstream System Failure (HTTP) Alert: WARNING: An adapter failed to transmit an outbound message Source: Microsoft.BizTalk.2006.BizTalkServer.ServerRole Path: Server Last modified by: System Last modified time: 15/08/2010 6:26:35 PM Alert description: The adapter failed to transmit message going to send port "GetStockQuoteFromYahoo" with URL "http://download.finance.yahoo.com/d/quotes.csv?s=MSFT&d=t&f=sl1d1t1c1ohgvj1pp2wern". It will be retransmitted after the retry interval specified for this Send Port. Details:"The remote server returned an error: (503) Server Unavailable.".

Custom Alert Alert: Production WorkOrderManagement System Error Detection Rule Source: Organization.WorkOrderManagement Path: Serverl Last modified by: System Last modified time: 13/08/2010 1:51:10 PM Alert description: Description: CreateEvent Response Status was False. Order Number: 000087803314 IDoc Number: 126634934 Orchestration Name: ProcessOrderCreateRequestFromERP Activity ID: {80EB87CB-88E3-4E43-A9F5-C4A97AF2955B} Message Instance ID: 387970ef-7367-4ed0-a83f-e58d2be9adaa Message: <CreateEventX> <msgid>126634934</msgid><status>F</status><errmsg>Error:Failure in I/Call for this request. Could not execute insert for: </errmsg><![CDATA[<input><CreateEvent msgid="126000000" area="01" fieldvalues=“…..”CreateEventX>

Break – Light Humour After the North American Beer Festival, all the brewery presidents decided to go out for a beer. The guy from Corona sits down and says, "Hey Senor, would like the world's best beer, a Corona." The bartender dusts off a bottle from the shelf and gives it to him. The guy from Budweiser says, "I'd like the best beer in the world, give me 'The King Of Beers', a Budweiser." The bartender gives him one. The guy from Coors says, "I'd like the only beer made with Rocky Mountain spring water, give me a Coors." He gets it. The guy from Molson sits down and says, "Give me a Coke." The bartender is a little taken aback, but gives him what he ordered. The other brewery presidents look over at him and ask, "Why aren't you drinking a Molson's?" The Molson president replies, "Well, I figured if you guys aren't drinking beer, neither would I."

Agenda – Part 2 Demos SCOM Performance Footprint Challenges Receive Location Shutdown SCOM as a knowledge base Host Instance Shutdown Web Console Pipeline Error Custom Errors Maintenance Mode SCOM Performance Footprint Challenges Lessons Learned

SCOM Performance Footprint During Installation Processor 29-37% Disk 18-19 additional pages per second Disk Storage 154 mb Network 11mb sent to the system Memory 22mb required during install Post Installation (run time) Processor 3-4% Disk .5 additional pages per second Disk Storage 156 mb data Network .5-.6 mb/minute Memory 11mb required

Challenges BizTalk Developers don’t understand SCOM and SCOM administrators don’t understand BizTalk Email Overload Tune SCOM, don’t create Outlook Rules Issues Clustered Host Instances (BizTalk 2006/2009) Disabled Host Instances Using SCOM Alerts for Business Purposes ESB Exception Management Portal is a better option Planned Maintenance Use Maintenance Mode Operations Team not resolving Alerts Don’t rely strictly on email, use the Console Gaps between BizTalk Admin Console and SCOM Console

Lessons Learned Commitment to the SCOM platform and process is required in order to be successful Tune, tune and re-tune your solutions until you have right level of notifications Leverage the Management console to build your Knowledge base and resolution of alerts. Leverage Maintenance Mode during planned outages to prevent unnecessary alerts during planned outages.

More Info… My Blog: http://kentweare.blogspot.com Twitter: @wearsy TechNet SCOM Virtual Labs http://technet.microsoft.com/en-us/systemcenter/bb539977.aspx BizTalk 2010 Management Pack Documentation http://www.microsoft.com/downloads/details.aspx?FamilyID=868a6c19-d7dc-4682-a068-d67526da0c61&displaylang=en Microsoft BizTalk Dev Center: http://msdn.microsoft.com/en-us/biztalk/default.aspx