Presentation is loading. Please wait.

Presentation is loading. Please wait.

May 2013 TechConnect Genius Webinar Troubleshooting Techniques for Use With FactoryTalk™ AssetCentre Eugene Krymskiy Senior Systems Support Engineer May.

Similar presentations


Presentation on theme: "May 2013 TechConnect Genius Webinar Troubleshooting Techniques for Use With FactoryTalk™ AssetCentre Eugene Krymskiy Senior Systems Support Engineer May."— Presentation transcript:

1 May 2013 TechConnect Genius Webinar Troubleshooting Techniques for Use With FactoryTalk™ AssetCentre
Eugene Krymskiy Senior Systems Support Engineer May 23, 2013

2 Introduction Thank you for your participation today; we recognize that your time is valuable. Today’s discussion will focus on troubleshooting common obstacles with FactoryTalk AssetCentre components in order to sustain a successful system. We hope to reaffirm what you already know, educate you on something new, and confirm best practices on how best to deploy and maintain FactoryTalk AssetCentre systems. In the best interest of our time, we will avoid mentioning any specific customer names and discussions that lead to the dark side.

3 Agenda What is FactoryTalk AssetCentre?
Reading and troubleshooting most common errors/messages FactoryTalk AssetCentre is not the problem; it is a solution Questions

4 FactoryTalk AssetCentre Platform
Set of asset-centric focused tools to securely and centrally manage your automated production environment Centrally archive electronic files/folders Provide backup and compare of operating asset configurations Track users’ actions Secure access Configure process instruments Manage process instruments calibrations Scalable design allows expansion of functionality and device counts Practical application for small-line applications and site-wide installations Low entry cost easily supports testing and proof of concept work FactoryTalk AssetCentre is a tool to manage automation assets

5 FactoryTalk AssetCentre Platform
FactoryTalk AssetCentre Server Governs the system’s operations Microsoft SQL Server 2005 Standard or 2008 R2 Hosts FactoryTalk AssetCentre databases FactoryTalk AssetCentre Client Permits users to configure and use the system, view logged events and perform other associated tasks FactoryTalk AssetCentre Agent Programs that perform certain types of tasks FactoryTalk Directory / Security Common Rockwell Automation security provider Expanded Simplified

6 FactoryTalk AssetCentre Server
Archive Centralized, versioned, secure configuration storage Data repository built on MSSQL Files & Folders (i.e., .acd, .mer, .doc, .xls, .xml, etc..) Binder, collection of files/folders managed as a single entity (i.e. Zip file) Logs Audits: Track authorized user changes (i.e., user check-out & check-in a file) Events: Tracks system activities (i.e., ed report XYZ.pdf to user Built on FactoryTalk Audit & Diagnostics Authentication Security built on FactoryTalk Security Prevent unauthorized changes Restrict viewing of artifacts Reporting Produce reports via Logs (audits/events) Schedule a report or “run now” FactoryTalk AssetCentre Server is a solution unto itself!

7 90% of all material generated from documentation
Introductions Introduction to Troubleshooting Tools Network Verification Commands Windows Event Logs Rockwell Automation Trace Diagnostics FactoryTalk AssetCentre Components Component Review Common Component Problems Troubleshooting FactoryTalk AssetCentre Server Service Troubleshooting FactoryTalk AssetCentre Agent and Scheduled Tasks Troubleshooting Additional Capabilities 90% of all material generated from documentation that already exists

8 Important Documentation
AID 43994: Can't enter design mode in FactoryTalk AssetCentre Explain multiple reasons why design mode is not available. AID 44372: Error initializing FactoryTalk AssetCentre, no connection could be made because the target machine actively refused it. This error usually indicates a failure of the FactoryTalk AssetCentre client to connect to the FactoryTalk AssetCentre server. AID 44965: Debug utilities for FactoryTalk AssetCentre Service called 'Rockwell Automation Trace Diagnostics Service' is installed on all AssetCentre servers, clients and agents. AID 45413: Error initializing FactoryTalk AssetCentre. Cannot initialize application eventing subsystem Communication between FactoryTalk Services Platform Client and Host is broken.

9 Important Documentation (continue)
AID 53693: Waiting for Interface error when trying to run FactoryTalk AssetCentre schedule There are several causes of Waiting for Interface errors when running FactoryTalk AssetCentre schedules. Waiting for Interface errors most commonly means the server is waiting for an agent to respond.. AID 60707: FactoryTalk AssetCentre Server Service Not Starting Step-by-step manual for diagnosing and solving issues related to the FactoryTalk AssetCentre server service not staying started. AID 57483: FactoryTalk AssetCentre System Design Considerations Designed to make system developers aware of the fundamental best-practice guidelines for designing and implementing a FactoryTalk AssetCentre application. AID 56049: FactoryTalk AssetCentre Tips & Best Practices TOC One-stop shop for all important FactoryTalk AssetCentre AIDs, including patch TOCs, troubleshooting, and design.

10 FactoryTalk AssetCentre Troubleshooting:Network Commands
ping <IP Address> Tests ability of host PC to locate remote PC. Can also use PC name in place of IP address. Success will indicate time of reply. Failure will indicate a time out or unreachable host. nslookup <PC name> Tests ability of host PC to resolve the name of the remote PC. Success will indicate first the name server name and IP address, then the remote PC’s name and IP address. Failure will indicate an inability to contact an appropriate name server. netstat –nao Lists the current ports in use on the host PC, and the PID of the process using each. tasklist /svc Lists each task currently running on the host PC and its associated PID

11 FactoryTalk AssetCentre Troubleshooting: Windows Event Log
Application, System Logs Useful for determining PC-level problems Name resolution issues Service start up failures/timing FactoryTalk Diagnostics Often FactoryTalk AssetCentre client may not be in a usable state to allow for Diagnostics viewing Can be found here or through Diagnostics viewer Security Logs Allows for exporting of each log type

12 Rockwell Automation TraceDiagnostics
Installed as part of FactoryTalk AssetCentre Most frequently used troubleshooting tool of Rockwell Commercial Engineering Used to capture step-by-step debug information Real-time or historical information Ability to import/export information Capable of aggregating information from several different PCs

13 Rockwell Automation TraceDiagnostics cont’d
Runs as a service (AID 44965) Enabled via Windows services, can be set to automatic or manual Appears as a magnifying glass in system tray

14 Rockwell Automation TraceDiagnostics cont’d
TraceDiagnostics acts as a central repository for debugging messages from .NET, Windows, and FactoryTalk AssetCentre traces. TraceDiagnostics is a listener service. TraceDiagnostics can be used over TCP/IP to collect messages from a remote machine. TCP/IP

15 General Troubleshooting Considerations: Network Connectivity
Over 10% of customer cases have been narrowed down to faulty network configurations Has the Windows Domain/Workgroup changed recently? PC name and user credentials may be outdated ping, nslookup results Are necessary ports open? Windows Firewall Hardware Firewall Windows Security/Security Software updates Is everything still plugged in? Can you hear me now?!

16 FactoryTalk AssetCentre Client: What Can Go Wrong?
Cannot establish connection to FactoryTalk AssetCentre server Most often a symptom of server service not staying started “No connection made because target machine actively refused it.” or “An existing connection was forcibly closed by the remote host”

17 FactoryTalk AssetCentre Server Service
90% of all customer issues related to server service! Server service fails to stay started on reboot, but will start manually Server services fails to stay started, manually or after reboot

18 FactoryTalk AssetCentre Server Service: Essentials
FactoryTalk AssetCentre Server Service Dependencies Rockwell Directory Multiplexor RPC, Rockwell Event MUX, Rockwell Application Services, Rockwell Redundancy Services FactoryTalk Directory Server Server settings configuration FactoryTalk Activation Server Activation not necessary, but faulty configuration can cause problems Microsoft IIS Server Microsoft SQL Server Internet Connectivity Not required, but has special considerations Who’s running the service?

19 FactoryTalk AssetCentre Server Service: Failing to Start on Reboot
Background Manual startup successful Is often the result of a timing problem Slower PC Lots of startup events Slow network authentication Workarounds: AID 55147: Batch file to start server placed in startup file Reset server service to be dependent on RSVC FactoryTalk AssetCentre won’t have to wait on other components to start Patch: Created to work around architectures with slow startup AID (FactoryTalk AssetCentre (CPR9 SR1))

20 FactoryTalk AssetCentre Server Service: Failing to Stay Started
Background Service not staying started on reboot Service not staying started manually Troubleshooting TraceDiagnostics is usually the quickest debugging solution Any FactoryTalk AssetCentre server service startup errors will occur in TraceDiagnostics log prior to “MasterScheduler, ONLINE!”

21 FactoryTalk Directory Configuration cont’d
Rockwell Software -> FactoryTalk AssetCentre Server -> Server Configuration Reboot suggested Correct name in Server Configuration? Correct ports configured?

22 FactoryTalk Directory Configuration
FactoryTalk Directory configuration for FactoryTalk AssetCentre Before server installation: After server installation:

23 FactoryTalk AssetCentre Server Service: Step by Step

24 FactoryTalk Activation and IIS
FactoryTalk AssetCentre Server service makes an IIS call as part of FactoryTalk Activation checks. Meaning activation errors can appear as IIS errors and vice versa Normally, troubleshooting the activation service first is the recommended course of action. In this case, since the activation server is failing with specific IIS errors, we’ll try the IIS route first.

25 Microsoft IIS Server Configuration
Upon installation, the FactoryTalk AssetCentre server configures a virtual directory in IIS, which links to the installed FactoryTalk AssetCentre server folders. Additionally, an ASP application pool is configured.

26 Microsoft IIS Server: 403 forbidden
Virtual Directory Configuration Frequently misunderstood dialog box: There’s an error at the Rockwell Software layer! Default settings for FactoryTalk AssetCentre server’s virtual directory: Application name: AssetCentre Execute Permissions: Scripts Only Application Pool: AssetCentre Virtual directory configuration problems can lead to HTTP 403 errors

27 Microsoft IIS Server: 401 Unauthorized
IIS Authentication FactoryTalk AssetCentre communications rely on IIS communications being configured correctly. IIS Authentication errors are commonly seen as HTTP 401 errors Anonymous access user missing or password expired Windows file security permissions can propagate up to Virtual Directory AID 49508

28 Microsoft IIS Server Resources
The most commonly seen problem with IIS is when the IIS Service not running. Other service-related problems can be caused by ASP .NET not registering correctly, though this is rarely seen with CPR9 and later versions. AID 48698 A useful IIS debugging tool is made available at no cost from Microsoft IIS Diagnostics Toolkit

29 FactoryTalk Activation Configuration
FactoryTalk AssetCentre Server service DOES NOT require activation to start correctly. Activations are discovered and processed upon server startup. Server Capabilities: Node-Locked Server Capacity: Concurrent FactoryTalk AssetCentre Server service queries FactoryTalk Activation for all available licenses, then requests any applicable FactoryTalk AssetCentre activations. Delays can cause the FactoryTalk AssetCentre Server service to time out

30 Missing FactoryTalk Activation Server
FactoryTalk Activation server configured through FactoryTalk Activation Tool. Local FactoryTalk Activation server listed <servername>’ Remote FactoryTalk Activation servers denoted as If communications cannot be established to the remote FactoryTalk Activation server, a timeout will occur. Often leftover from configuration phase.

31 FactoryTalk Activation Service
FactoryTalk Activation server capable of hosting Activation service. Client not capable of service Service required for concurrent (capacity) activations

32 Cannot Connect to Activation Server
Same message will appear for both FactoryTalk Activation configuration problems. Possible to see activation errors caused by IIS communication problems due to web service call for activation information.

33 Activation Communications
The trace information has previously been failing at FactoryTalk Activation, which has now been resolved. Finding activation communications in the trace logs can rule out the FactoryTalk Activation server connection:

34 FactoryTalk AssetCentre Server Service: Port Utilization
FactoryTalk AssetCentre communications make use of 3 ports User configurable (default: ) If the port range specified is blocked/in use, the FactoryTalk AssetCentre server service will not start correctly Use netstat –nao and tasklist /svc commands to find blocking process

35 FactoryTalk AssetCentre Server Service: Port Utilization cont’d
Note where the exception occurred: And where the success message is now shown, indicating the successful port binding: Exception thrown immediately after GetServerPort call

36 FactoryTalk AssetCentre Server Service:Who’s Running the Services?
By default, FactoryTalk AssetCentre server and AgentController services will run as ‘Local System.’ Best practice (and per Installation Guide), Microsoft SQL server should also be configured to run as ‘Local System.’ Defining a user can lead to authentication issues (e.g., password expiration, user changes, domain changes, domain policies, etc.)

37 Microsoft SQL Server Configuration
After establishing FactoryTalk Activation, the Microsoft SQL Server will be connected. While many architectures co-locate the Microsoft SQL Server with the FactoryTalk AssetCentre server, some database servers can be remote. For these, general PC network connectivity should be considered first. Network connectivity and Microsoft SQL Server service errors will appear the same – an inability to connect to the Microsoft SQL Server.

38 Microsoft SQL Server User Configuration
On non-English (US) operating systems, it is possible for a discrepancy in the default language of the Microsoft SQL users and the OS language to prevent the FactoryTalk AssetCentre server service from starting correctly. AID 60781 FactoryTalk AssetCentre makes use of a SQL-authenticated user (AssetCentreUser by default) to connect to Microsoft SQL Server. User credentials encrypted on FactoryTalk AssetCentre server side Editing of user/credentials must be done on both Microsoft SQL and FactoryTalk AssetCentre servers.

39 FactoryTalk AssetCentre Agent: What Can Go Wrong?
FactoryTalk AssetCentre AgentController Service Dependencies Rockwell Directory Multiplexor RPC, Rockwell Event MUX, Rockwell Application Services, Rockwell Redundancy Services “Waiting for Interface” Asset configuration in Disaster Recovery schedule Remote Computer errors The “Little Black Box” FactoryTalk Diagnostics Error: “AssetCentre Server is unavailable or the AgentController Service version is not compatible with the AssetCentre Server version.” Communication Timeouts with RSLinx Classic

40 FactoryTalk AssetCentre Agent: Waiting for Interface
Waiting for Interface is the first step in the schedule execution of a FactoryTalk AssetCentre agent Executed server side, not on agent Server is looping while waiting for an agent to become available Most frequent cause is no FactoryTalk AssetCentre agent has been installed Verify that the FactoryTalk AssetCentre server has recognized all possible agents (i.e., No 0 count) Verify that FactoryTalk AssetCentre agents are ‘free’

41 FactoryTalk AssetCentre Agent: Multiple NICs
FactoryTalk AssetCentre agents with multiple NICs may require additional configuration. Possible issue where server can ping agent, but agent will not connect. Primary NIC must be attached the route through which the FactoryTalk AssetCentre server is contacted From Network Connections, Advanced -> Advanced Options AID Using FactoryTalk AssetCentre with Multiple Network Interface Cards FactoryTalk AssetCentre Server CLx PLC SLC DH+ Managed Ethernet Switch for Control System Network (CSN) AssetCentre Agent Primary Secondary

42 FactoryTalk AssetCentre Agent: Schedule Execution
Each step is making a call to the FactoryTalk AssetCentre server Waiting for Interface Loop -> no agent available yet Loading Agent Agent assigned, now connecting to agent Loading Data Loading data from server (location data) Loads schedule information Device Path name Filename Running Upload from asset If compare, then asset get will appear in archive view If compare, compare occurs on agent Check-in (if needed)

43 FactoryTalk AssetCentre Agent: Invalid Device Path
All agents must be created equally Includes at least one client to initially create and configure the asset Asset configurations stored server-side, so all agents use the same configuration FactoryTalk AssetCentre components use the path right of the PC name and “!” Many device path problems have been resolved by discovering that the controller is offline or has simply been removed!

44 FactoryTalk AssetCentre Agent: Remote Computer Paths
What is a proper UNC path? AID – Remote Computer asset configuration for FactoryTalk AssetCentre Disaster Recovery

45 FactoryTalk AssetCentre Agent: Remote Computers Paths cont’d
The network name cannot be found. Verify the computer hosting the network share is accessible. Ping, access share from OS Verify that the share on that computer is readable from a remote computer. Is UNC path configured correctly? Copy/paste into Start -> Run Verify that the computer name is resolvable via a DNS nslookup. Replacing a computer name with an IP address is also a good DNS check.

46 FactoryTalk AssetCentre Agent: Remote Computer User Configuration
Logon failure: unknown user name or bad password Shares require read permission Domain security may require the domain name to be added Asset configuration will not automatically update with password resets

47 FactoryTalk AssetCentre Agent: Remote Computers and Locked Files
The process cannot access the file <filename> because it is being used by another process Locked files cannot be backed up by the FactoryTalk AssetCentre agent. For instance: Running FactoryTalk View SE HMI servers Not all files are locked when in use (e.g., Notepad files) End the process accessing the file(s) in question and run the schedule again.

48 Communication Timeouts in RSLinx Classic
FactoryTalk AssetCentre agents make use of the RSBrowse function of RSLinx Classic to establish connections to remote devices. Agent receives route from server, and passes on to RSBrowse. If no success message is received, agent will stand by. After a timeout in the agent, the asset’s schedule will complete with “No Result” Route: ASSETCENTRE-SERVER!AB_ETH-1\ \Backplane\2\A\56, RA.FTAC.AGENTS (100): LegacyVerificationShim.threadLoop, waitresult 258, RSBrowse: Starting cycle: 0, level: 0., RSBrowse: Found bus: ASSETCENTRE-SERVER!AB_ETH-1, RSBrowse: Starting cycle: 0, level: 1., RSBrowse: A device was found: ASSETCENTRE-SERVER!AB_ETH-1\ , RSBrowse: Done Event signaled., RSBrowse: Browsing for level 1 has cycled., RSBrowse: Starting cycle: 0, level: 2., RSBrowse: Found bus: ASSETCENTRE-SERVER!AB_ETH-1\ \Backplane, RSBrowse: Starting cycle: 0, level: 3., RSBrowse: A device was found: ASSETCENTRE-SERVER!AB_ETH-1\ \BACKPLANE\2, RSBrowse: Starting cycle: 0, level: 4., RSBrowse: Found bus: ASSETCENTRE-SERVER!AB_ETH-1\ \Backplane\2\A, RSBrowse: Starting cycle: 0, level: 5.,

49 Comms Timeouts in RSLinx Classic (cont’d)
Many RSLinx communication timeouts are caused by lack of network bandwidth. For this purpose, RSLinx Classic provides the ability to tune its timeout settings. Adjusting the tuning settings will increase or decrease the amount of time RSLinx Classic will wait before timing out a connection.

50 Troubleshooting Communication Timeouts in RSLinx Classic
Adjusting the connection settings in RSLinx Classic will help the browse message complete quickly, thereby returning a valid response to the waiting agent Route: ASSETCENTRE-SERVER!AB_ETHIP-1\ \Backplane\2\A\56, RSBrowse: Starting cycle: 0, level: 0., RSBrowse: Starting cycle: 0, level: 1., RSBrowse: Starting cycle: 0, level: 2., RSBrowse: Starting cycle: 0, level: 3., RSBrowse: Starting cycle: 0, level: 4., RSBrowse: Starting cycle: 0, level: 5., [VERIFICATIONAGENT][Info] CAutoBrowse::DoAutoBrowse() ---> ---> ---> SUCCEEDED browsing for device: ASSETCENTRE-SERVER!AB_ETHIP-1\ \Backplane\2\A\56, [VERIFICATIONAGENT][Info] CAutoBrowse::DoAutoBrowse() ---> ---> ---> Total time to browse is: 00:00:00 (hours:minutes:seconds), Once the connection is made, the agent can continue with the schedule.

51 Questions?

52 Thank you for participating!


Download ppt "May 2013 TechConnect Genius Webinar Troubleshooting Techniques for Use With FactoryTalk™ AssetCentre Eugene Krymskiy Senior Systems Support Engineer May."

Similar presentations


Ads by Google