1. May 2013 TechConnect Genius Webinar Troubleshooting Techniques for Use With FactoryTalk™ AssetCentre
Eugene Krymskiy
Senior Systems Support Engineer
May 23, 2013

2. Introduction
Thank you for your participation today; we recognize that your time is valuable.
Today’s discussion will focus on troubleshooting common obstacles with FactoryTalk AssetCentre components in order to sustain a successful system.
We hope to reaffirm what you already know, educate you on something new, and confirm best practices on how best to deploy and maintain FactoryTalk AssetCentre systems.
In the best interest of our time, we will avoid mentioning any specific customer names and discussions that lead to the dark side.

3. Agenda What is FactoryTalk AssetCentre?
Reading and troubleshooting most common errors/messages
FactoryTalk AssetCentre is not the problem; it is a solution
Questions

4. FactoryTalk AssetCentre Platform
Set of asset-centric focused tools to securely and centrally manage your automated production environment
Centrally archive electronic files/folders
Provide backup and compare of operating asset configurations
Track users’ actions
Secure access
Configure process instruments
Manage process instruments calibrations
Scalable design allows expansion of functionality and device counts
Practical application for small-line applications and site-wide installations
Low entry cost easily supports testing and proof of concept work
FactoryTalk AssetCentre is a tool to manage automation assets

5. FactoryTalk AssetCentre Platform
FactoryTalk AssetCentre Server
Governs the system’s operations
Microsoft SQL Server 2005 Standard or 2008 R2
Hosts FactoryTalk AssetCentre databases
FactoryTalk AssetCentre Client
Permits users to configure and use the system, view logged events and perform other associated tasks
FactoryTalk AssetCentre Agent
Programs that perform certain types of tasks
FactoryTalk Directory / Security
Common Rockwell Automation security provider
Expanded
Simplified

6. FactoryTalk AssetCentre Server
Archive
Centralized, versioned, secure configuration storage
Data repository built on MSSQL
Files & Folders (i.e., .acd, .mer, .doc, .xls, .xml, etc..)
Binder, collection of files/folders managed as a single entity (i.e. Zip file)
Logs
Audits: Track authorized user changes (i.e., user check-out & check-in a file)
Events: Tracks system activities (i.e., ed report XYZ.pdf to user
Built on FactoryTalk Audit & Diagnostics
Authentication
Security built on FactoryTalk Security
Prevent unauthorized changes
Restrict viewing of artifacts
Reporting
Produce reports via Logs (audits/events)
Schedule a report or “run now”
FactoryTalk AssetCentre Server is a solution unto itself!

7. 90% of all material generated from documentation
Introductions
Introduction to Troubleshooting Tools
Network Verification Commands
Windows Event Logs
Rockwell Automation Trace Diagnostics
FactoryTalk AssetCentre Components
Component Review
Common Component Problems
Troubleshooting FactoryTalk AssetCentre Server Service
Troubleshooting FactoryTalk AssetCentre Agent and Scheduled Tasks
Troubleshooting Additional Capabilities
90% of all material generated from documentation
that already exists

8. Important Documentation
AID 43994: Can't enter design mode in FactoryTalk AssetCentre
Explain multiple reasons why design mode is not available.
AID 44372: Error initializing FactoryTalk AssetCentre, no connection could be made because the target machine actively refused it.
This error usually indicates a failure of the FactoryTalk AssetCentre client to connect to the FactoryTalk AssetCentre server.
AID 44965: Debug utilities for FactoryTalk AssetCentre
Service called 'Rockwell Automation Trace Diagnostics Service' is installed on all AssetCentre servers, clients and agents.
AID 45413: Error initializing FactoryTalk AssetCentre. Cannot initialize application eventing subsystem
Communication between FactoryTalk Services Platform Client and Host is broken.

9. Important Documentation (continue)
AID 53693: Waiting for Interface error when trying to run FactoryTalk AssetCentre schedule
There are several causes of Waiting for Interface errors when running FactoryTalk AssetCentre schedules. Waiting for Interface errors most commonly means the server is waiting for an agent to respond..
AID 60707: FactoryTalk AssetCentre Server Service Not Starting
Step-by-step manual for diagnosing and solving issues related to the FactoryTalk AssetCentre server service not staying started.
AID 57483: FactoryTalk AssetCentre System Design Considerations
Designed to make system developers aware of the fundamental best-practice guidelines for designing and implementing a FactoryTalk AssetCentre application.
AID 56049: FactoryTalk AssetCentre Tips & Best Practices TOC
One-stop shop for all important FactoryTalk AssetCentre AIDs, including patch TOCs, troubleshooting, and design.

10. FactoryTalk AssetCentre Troubleshooting:Network Commands
ping <IP Address>
Tests ability of host PC to locate remote PC.
Can also use PC name in place of IP address.
Success will indicate time of reply.
Failure will indicate a time out or unreachable host.
nslookup <PC name>
Tests ability of host PC to resolve the name of the remote PC.
Success will indicate first the name server name and IP address, then the remote PC’s name and IP address.
Failure will indicate an inability to contact an appropriate name server.
netstat –nao
Lists the current ports in use on the host PC, and the PID of the process using each.
tasklist /svc
Lists each task currently running on the host PC and its associated PID

11. FactoryTalk AssetCentre Troubleshooting: Windows Event Log
Application, System Logs
Useful for determining PC-level problems
Name resolution issues
Service start up failures/timing
FactoryTalk Diagnostics
Often FactoryTalk AssetCentre client may not be in a usable state to allow for Diagnostics viewing
Can be found here or through Diagnostics viewer
Security Logs
Allows for exporting of each log type

12. Rockwell Automation TraceDiagnostics
Installed as part of FactoryTalk AssetCentre
Most frequently used troubleshooting tool of Rockwell Commercial Engineering
Used to capture step-by-step debug information
Real-time or historical information
Ability to import/export information
Capable of aggregating information from several different PCs

13. Rockwell Automation TraceDiagnostics cont’d
Runs as a service (AID 44965)
Enabled via Windows services, can be set to automatic or manual
Appears as a magnifying glass in system tray

14. Rockwell Automation TraceDiagnostics cont’d
TraceDiagnostics acts as a central repository for debugging messages from .NET, Windows, and FactoryTalk AssetCentre traces.
TraceDiagnostics is a listener service.
TraceDiagnostics can be used over TCP/IP to collect messages from a remote machine.
TCP/IP

15. General Troubleshooting Considerations: Network Connectivity
Over 10% of customer cases have been narrowed down to faulty network configurations
Has the Windows Domain/Workgroup changed recently?
PC name and user credentials may be outdated
ping, nslookup results
Are necessary ports open?
Windows Firewall
Hardware Firewall
Windows Security/Security Software updates
Is everything still plugged in?
Can you hear me now?!

16. FactoryTalk AssetCentre Client: What Can Go Wrong?
Cannot establish connection to FactoryTalk AssetCentre server
Most often a symptom of server service not staying started
“No connection made because target machine actively refused it.” or
“An existing connection was forcibly closed by the remote host”

17. FactoryTalk AssetCentre Server Service
90% of all customer issues related to server service!
Server service fails to stay started on reboot, but will start manually
Server services fails to stay started, manually or after reboot

18. FactoryTalk AssetCentre Server Service: Essentials
FactoryTalk AssetCentre Server Service Dependencies
Rockwell Directory Multiplexor
RPC, Rockwell Event MUX, Rockwell Application Services, Rockwell Redundancy Services
FactoryTalk Directory Server
Server settings configuration
FactoryTalk Activation Server
Activation not necessary, but faulty configuration can cause problems
Microsoft IIS Server
Microsoft SQL Server
Internet Connectivity
Not required, but has special considerations
Who’s running the service?

19. FactoryTalk AssetCentre Server Service: Failing to Start on Reboot
Background
Manual startup successful
Is often the result of a timing problem
Slower PC
Lots of startup events
Slow network authentication
Workarounds:
AID 55147: Batch file to start server placed in startup file
Reset server service to be dependent on RSVC
FactoryTalk AssetCentre won’t have to wait on other components to start
Patch:
Created to work around architectures with slow startup
AID (FactoryTalk AssetCentre (CPR9 SR1))

20. FactoryTalk AssetCentre Server Service: Failing to Stay Started
Background
Service not staying started on reboot
Service not staying started manually
Troubleshooting
TraceDiagnostics is usually the quickest debugging solution
Any FactoryTalk AssetCentre server service startup errors will occur in TraceDiagnostics log prior to “MasterScheduler, ONLINE!”

21. FactoryTalk Directory Configuration cont’d
Rockwell Software -> FactoryTalk AssetCentre Server -> Server Configuration
Reboot suggested
Correct name in Server Configuration?
Correct ports configured?

22. FactoryTalk Directory Configuration
FactoryTalk Directory configuration for FactoryTalk AssetCentre
Before server installation: After server installation:

23. FactoryTalk AssetCentre Server Service: Step by Step

24. FactoryTalk Activation and IIS
FactoryTalk AssetCentre Server service makes an IIS call as part of FactoryTalk Activation checks.
Meaning activation errors can appear as IIS errors and vice versa
Normally, troubleshooting the activation service first is the recommended course of action.
In this case, since the activation server is failing with specific IIS errors, we’ll try the IIS route first.

25. Microsoft IIS Server Configuration
Upon installation, the FactoryTalk AssetCentre server configures a virtual directory in IIS, which links to the installed FactoryTalk AssetCentre server folders.
Additionally, an ASP application pool is configured.

26. Microsoft IIS Server: 403 forbidden
Virtual Directory Configuration
Frequently misunderstood dialog box:
There’s an error at the Rockwell Software layer!
Default settings for FactoryTalk AssetCentre server’s virtual directory:
Application name: AssetCentre
Execute Permissions: Scripts Only
Application Pool: AssetCentre
Virtual directory configuration problems can lead to HTTP 403 errors

27. Microsoft IIS Server: 401 Unauthorized
IIS Authentication
FactoryTalk AssetCentre communications rely on IIS communications being configured correctly.
IIS Authentication errors are commonly seen as HTTP 401 errors
Anonymous access user missing or password expired
Windows file security permissions can propagate up to Virtual Directory
AID 49508

28. Microsoft IIS Server Resources
The most commonly seen problem with IIS is when the IIS Service not running.
Other service-related problems can be caused by ASP .NET not registering correctly, though this is rarely seen with CPR9 and later versions.
AID 48698
A useful IIS debugging tool is made available at no cost from Microsoft
IIS Diagnostics Toolkit

29. FactoryTalk Activation Configuration
FactoryTalk AssetCentre Server service DOES NOT require activation to start correctly.
Activations are discovered and processed upon server startup.
Server Capabilities: Node-Locked
Server Capacity: Concurrent
FactoryTalk AssetCentre Server service queries FactoryTalk Activation for all available licenses, then requests any applicable FactoryTalk AssetCentre activations.
Delays can cause the FactoryTalk AssetCentre Server service to time out

30. Missing FactoryTalk Activation Server
FactoryTalk Activation server configured through FactoryTalk Activation Tool.
Local FactoryTalk Activation server listed <servername>’
Remote FactoryTalk Activation servers denoted as
If communications cannot be established to the remote FactoryTalk Activation server, a timeout will occur.
Often leftover from configuration phase.

31. FactoryTalk Activation Service
FactoryTalk Activation server capable of hosting Activation service.
Client not capable of service
Service required for concurrent (capacity) activations

32. Cannot Connect to Activation Server
Same message will appear for both FactoryTalk Activation configuration problems.
Possible to see activation errors caused by IIS communication problems due to web service call for activation information.

33. Activation Communications
The trace information has previously been failing at FactoryTalk Activation, which has now been resolved.
Finding activation communications in the trace logs can rule out the FactoryTalk Activation server connection:

34. FactoryTalk AssetCentre Server Service: Port Utilization
FactoryTalk AssetCentre communications make use of 3 ports
User configurable (default: )
If the port range specified is blocked/in use, the FactoryTalk AssetCentre server service will not start correctly
Use netstat –nao and tasklist /svc commands to find blocking process

35. FactoryTalk AssetCentre Server Service: Port Utilization cont’d
Note where the exception occurred:
And where the success message is now shown, indicating the successful port binding:
Exception thrown immediately after GetServerPort call

36. FactoryTalk AssetCentre Server Service:Who’s Running the Services?
By default, FactoryTalk AssetCentre server and AgentController services will run as ‘Local System.’
Best practice (and per Installation Guide), Microsoft SQL server should also be configured to run as ‘Local System.’
Defining a user can lead to authentication issues (e.g., password expiration, user changes, domain changes, domain policies, etc.)

37. Microsoft SQL Server Configuration
After establishing FactoryTalk Activation, the Microsoft SQL Server will be connected.
While many architectures co-locate the Microsoft SQL Server with the FactoryTalk AssetCentre server, some database servers can be remote. For these, general PC network connectivity should be considered first.
Network connectivity and Microsoft SQL Server service errors will appear the same – an inability to connect to the Microsoft SQL Server.

38. Microsoft SQL Server User Configuration
On non-English (US) operating systems, it is possible for a discrepancy in the default language of the Microsoft SQL users and the OS language to prevent the FactoryTalk AssetCentre server service from starting correctly.
AID 60781
FactoryTalk AssetCentre makes use of a SQL-authenticated user (AssetCentreUser by default) to connect to Microsoft SQL Server.
User credentials encrypted on FactoryTalk AssetCentre server side
Editing of user/credentials must be done on both Microsoft SQL and FactoryTalk AssetCentre servers.

39. FactoryTalk AssetCentre Agent: What Can Go Wrong?
FactoryTalk AssetCentre AgentController Service Dependencies
Rockwell Directory Multiplexor
RPC, Rockwell Event MUX, Rockwell Application Services, Rockwell Redundancy Services
“Waiting for Interface”
Asset configuration in Disaster Recovery schedule
Remote Computer errors
The “Little Black Box”
FactoryTalk Diagnostics Error: “AssetCentre Server is unavailable or the AgentController Service version is not compatible with the AssetCentre Server version.”
Communication Timeouts with RSLinx Classic

40. FactoryTalk AssetCentre Agent: Waiting for Interface
Waiting for Interface is the first step in the schedule execution of a FactoryTalk AssetCentre agent
Executed server side, not on agent
Server is looping while waiting for an agent to become available
Most frequent cause is no FactoryTalk AssetCentre agent has been installed
Verify that the FactoryTalk AssetCentre server has recognized all possible agents (i.e., No 0 count)
Verify that FactoryTalk AssetCentre agents are ‘free’

41. FactoryTalk AssetCentre Agent: Multiple NICs
FactoryTalk AssetCentre agents with multiple NICs may require additional configuration.
Possible issue where server can ping agent, but agent will not connect.
Primary NIC must be attached the route through which the FactoryTalk AssetCentre server is contacted
From Network Connections, Advanced -> Advanced Options
AID Using FactoryTalk AssetCentre with Multiple Network Interface Cards
FactoryTalk
AssetCentre Server
CLx
PLC
SLC
DH+
Managed Ethernet Switch for Control System Network (CSN)
AssetCentre Agent
Primary
Secondary

42. FactoryTalk AssetCentre Agent: Schedule Execution
Each step is making a call to the FactoryTalk AssetCentre server
Waiting for Interface
Loop -> no agent available yet
Loading Agent
Agent assigned, now connecting to agent
Loading Data
Loading data from server (location data)
Loads schedule information
Device
Path name
Filename
Running
Upload from asset
If compare, then asset get will appear in archive view
If compare, compare occurs on agent
Check-in (if needed)

43. FactoryTalk AssetCentre Agent: Invalid Device Path
All agents must be created equally
Includes at least one client to initially create and configure the asset
Asset configurations stored server-side, so all agents use the same configuration
FactoryTalk AssetCentre components use the path right of the PC name and “!”
Many device path problems have been resolved by discovering that the controller is offline or has simply been removed!

44. FactoryTalk AssetCentre Agent: Remote Computer Paths
What is a proper UNC path?
AID – Remote Computer asset configuration for FactoryTalk AssetCentre Disaster Recovery

45. FactoryTalk AssetCentre Agent: Remote Computers Paths cont’d
The network name cannot be found.
Verify the computer hosting the network share is accessible.
Ping, access share from OS
Verify that the share on that computer is readable from a remote computer.
Is UNC path configured correctly? Copy/paste into Start -> Run
Verify that the computer name is resolvable via a DNS nslookup.
Replacing a computer name with an IP address is also a good DNS check.

46. FactoryTalk AssetCentre Agent: Remote Computer User Configuration
Logon failure: unknown user name or bad password
Shares require read permission
Domain security may require the domain name to be added
Asset configuration will not automatically update with password resets

47. FactoryTalk AssetCentre Agent: Remote Computers and Locked Files
The process cannot access the file <filename> because it is being used by another process
Locked files cannot be backed up by the FactoryTalk AssetCentre agent.
For instance: Running FactoryTalk View SE HMI servers
Not all files are locked when in use (e.g., Notepad files)
End the process accessing the file(s) in question and run the schedule again.

48. Communication Timeouts in RSLinx Classic
FactoryTalk AssetCentre agents make use of the RSBrowse function of RSLinx Classic to establish connections to remote devices.
Agent receives route from server, and passes on to RSBrowse.
If no success message is received, agent will stand by.
After a timeout in the agent, the asset’s schedule will complete with “No Result”
Route: ASSETCENTRE-SERVER!AB_ETH-1\ \Backplane\2\A\56,
RA.FTAC.AGENTS (100): LegacyVerificationShim.threadLoop, waitresult 258,
RSBrowse: Starting cycle: 0, level: 0.,
RSBrowse: Found bus: ASSETCENTRE-SERVER!AB_ETH-1,
RSBrowse: Starting cycle: 0, level: 1.,
RSBrowse: A device was found: ASSETCENTRE-SERVER!AB_ETH-1\ ,
RSBrowse: Done Event signaled.,
RSBrowse: Browsing for level 1 has cycled.,
RSBrowse: Starting cycle: 0, level: 2.,
RSBrowse: Found bus: ASSETCENTRE-SERVER!AB_ETH-1\ \Backplane,
RSBrowse: Starting cycle: 0, level: 3.,
RSBrowse: A device was found: ASSETCENTRE-SERVER!AB_ETH-1\ \BACKPLANE\2,
RSBrowse: Starting cycle: 0, level: 4.,
RSBrowse: Found bus: ASSETCENTRE-SERVER!AB_ETH-1\ \Backplane\2\A,
RSBrowse: Starting cycle: 0, level: 5.,

49. Comms Timeouts in RSLinx Classic (cont’d)
Many RSLinx communication timeouts are caused by lack of network bandwidth.
For this purpose, RSLinx Classic provides the ability to tune its timeout settings.
Adjusting the tuning settings will increase or decrease the amount of time RSLinx Classic will wait before timing out a connection.

50. Troubleshooting Communication Timeouts in RSLinx Classic
Adjusting the connection settings in RSLinx Classic will help the browse message complete quickly, thereby returning a valid response to the waiting agent
Route: ASSETCENTRE-SERVER!AB_ETHIP-1\ \Backplane\2\A\56,
RSBrowse: Starting cycle: 0, level: 0.,
RSBrowse: Starting cycle: 0, level: 1.,
RSBrowse: Starting cycle: 0, level: 2.,
RSBrowse: Starting cycle: 0, level: 3.,
RSBrowse: Starting cycle: 0, level: 4.,
RSBrowse: Starting cycle: 0, level: 5.,
[VERIFICATIONAGENT][Info] CAutoBrowse::DoAutoBrowse() ---> ---> ---> SUCCEEDED browsing for device: ASSETCENTRE-SERVER!AB_ETHIP-1\ \Backplane\2\A\56,
[VERIFICATIONAGENT][Info] CAutoBrowse::DoAutoBrowse() ---> ---> ---> Total time to browse is: 00:00:00 (hours:minutes:seconds),
Once the connection is made, the agent can continue with the schedule.

51. Questions?

52. Thank you for participating!