2. WW TSS-13 Wonderware MES Support and Troubleshooting Technique
Presenters:
Dan Scott
Jim Stella
JS - This technical session will discuss the various methods and tools that can be used to support and troubleshoot Wonderware MES. Learn about preventative maintenance activities and some best practice tuning and architecture considerations.

3. Session Agenda MES Architecture Overview
Middleware Technology Changes 3.5 to 4.x
Database Tables
Tips and Common Mistakes
Troubleshooting JS

4. MES Architecture OverviewDS

5. Wonderware MES Architecture
MES Client
Operator
MES Client
Supervisor
InTouch w/ .NET Controls
Supervisor
Galaxy
Repository
Reports
Configurator
InTouch w/ MES API
Field I/O
ArchestrA IDE
Supervisory (P)
Shop Floor (P)
MES Admin (P)
AOS Platform (M)
Factory Connector
Server (M)
Operations Capability Obj
Utilization Capability Obj
Middleware
Server (M,S)
Sample Recording Obj
DS – Typical multi-node architecture. Discuss various components. Locations of proxy, server, and MES Service. Poor architecture planning up front can seriously impact the success of your project
Reports
Server (M or None)
Database
Server
App Objects w/ MES API
Key: M = Full Middleware P = Middleware Proxy S = MES Service

6. Technology change in MES Middleware from 3.5 to 4.xJS

7. Wonderware MES Middleware 3.5
JS – COM+ vs WCF. DLLHost memory spikes, now more traceability with new middleware.

8. Wonderware MES Middleware 4.x WCF (Windows Communication Foundation)
JS – 3.5 was a proxy which required Msi and cab file created from server. So, you had proxy or server, not both. Now “proxy” is really a client. You now have either client or client and server locally. More ability to resolve client performance and connection issues.
You can test mw connection, but does not mean 100% connectivity.

9. MES Database Tables JS

10. Common MES Database Tables Used in Troubleshooting
Sessn – User sessions
Who is logged into which session ID, or not in some cases
Records with heartbeat older than 10 minutes can be purged
These should be auto-purged by MES Service if it is running
Only ever one session type 53 – Background User for MES Service
Orphaned type 53 record can stop MES Service from starting
Tech Note 699 covers this table and the client type in more detail
Ent_Logon – User logged on entities
Which user is logged into which entities
User must have privilege for those entities JS

11. Common MES Database Tables Used in Troubleshooting
Item_Prod – Production records
Distinct: Default selection
Can report hourly or shift OEE
Records stored for every production or consumption transaction
Can result in large record count
Shift:
Can report shift OEE only
Most compact record count
Hourly: New for 3.5 p02, MES 4.0 SP2+ (replaces shift)
Compromise between Distinct and Shift option
Should be purged periodically to maintain high performance JS

12. Common MES Database Tables Used in Troubleshooting
Util_Log – Utilization history
Should be purged periodically to maintain high performance
TPM_Stat – OEE raw data
Data only stored when a job is running
Job – Jobs and related status
Contains total production count, allowing item_prod details to be purged
Job_Exec – Active job information
One placeholder record per allow Job Exec per entity
Never delete records JS

13. Common MES Database Tables Used in Troubleshooting
Util_Exec – Current utilization information
Lookup current raw reason
OEE_Exec – Current OEE information
See current Quality and Performance OEE subcomponents
Quality % x Performance % x Availability % = OEE %
Ent – Entities
View current shift ID and times
View capabilities JS

14. Common MES Database Tables Used in Troubleshooting
Shift_to_go – Upcoming shifts
Can be purged, then restart MES Service to rebuild list
Tz_offset – Pre-calculated time zone offsets from UTC
Only time zones of interest are calculated
Purge is required if the database is moved to another time zone
WO – Work Orders
Should be purged at some point. (Archive/Purge has System parameters that can be defined to automatically remove the data)
Item_inv – Inventory JS

15. MES Tips and Common MistakesDS

16. MES Tips and Common Mistakes: Middleware
Middleware proxy must be installed on IDE node if using Entity Model Builder (EMB)
You must open/close a capability object editor window before running EMB
Full middleware should be installed App Object and Manager nodes
License purchase required per full middleware server (multiple inc.)
Middleware is not deployed, must be installed manually
Use Proxy Configuration Editor to test the connection to MW server
Use MES Client or Configurator to test client connectivity since you will get better errors
If middleware is remote from database, MES Client must be installed DS

17. MES Tips and Common Mistakes: Load Management
Server / Platform Loading
What else runs on this node?
Available system resources?
Engine Loading
Engine Scan Rate?
Many capability object functions result in MES database transactions.
Consider separate engines for MES objects versus regular, higher speed, App Objects
Middleware Loading
Capability Object count?
No more than per platform*
No more than 50 SROs per platform*
No more than per engine*
Full middleware on the platform hosting MES App Objects
Reporting?
Database Loading
Dedicated MES Database server?
MES has high transaction count
DS- Consider this when developing an implementation plan.
We see a lot of systems where Dev testing went well, but once record counts and transaction counts get high in Prod, the system runs very slowly or with errors.
Consider how you configure MES related to data storage, examples are rates of recording of production, raw reason code change frequency, etc.
Disk IO capability important in VM environment. Refer to deployment guide.
*Suggestions dependent on transaction volume and scan rate

18. MES Tips and Common Mistakes: Maintain Distinct Production Records
Enabled to report Hourly OEE
Daily/Shift OEE can still be reported with this disabled
This setting can have a significant impact on the MES database size
If enabled (default), a new item_prod record is created every time production is recorded
If disabled, the system will update existing item_prod records
New feature in 3.5 p02, 4.0 SP2 aggregated hourly prod records
JS - Do you really need this?
Pros: hourly reporting, more granular audit data, faster backups
Cons: BIG item_prod table, slower reports, more hardware demands, slower backups, more db maintenance/monitoring reqd
Considerations can be made to turn this on, but use slower rates of production recording

19. MES Tips and Common Mistakes: Multiple Job Execution
How many Jobs can be run concurrently on each entity
More than 10 can cause problems
OCO has a hard cap of 50, but not recommended for entities with large transaction numbers
JS – Handling multiple jobexec requires handling large amount of transactions. Can be handled incorrectly. Operator and job summary will appear differently.

20. MES Tips and Common Mistakes: OCO/UCO Production Counters
Misconfiguration, especially with a rollover counter, can be disastrous
NEVER set Deadband, Max Value, and Update Interval to 0
Production may be recorded every engine scan
Every time production is recorded, you are transacting with the MES Database
If Maintain Distinct Production Records is enabled (default), you are creating a new item_prod records every time production is recorded
UCO rollover counter Max Value MUST be synced with PLC rollover
OCO rollover counter can be reset on demand, or PLC sync (TN 710)
DS - New in MES 4.0, OCO counters are significantly better than in the UCO. Multiple Job Exec, resets, etc.
UCO counters can ONLY be used when there can only be 1 job run at a time run on an entity

21. MES Tips and Common Mistakes: Raw Reason Code Management (UCO)
Every Raw Reason Code change results in multiple MES Database transactions
Changing reasons too fast can cause reason codes to be missed (TN657)
UCO can only handle current change plus queue the next change
Micro stops should be recorded to Historian (events < 30-60s)*
Future product plans in the works to allow for faster changes
DS - Most common bottleneck since v3.5 was released
One of the key understanding of MES is that it is NOT for microstops at this time
Occasional fast changes are OK, but back-to-back-to-back fast changes will result in skipping RRC records
Minimum time before activation field must be used when dealing with a large list of RRCs
Consider the number of RRC being defined. How granular do you really to be here versus historian?
RRC priority is important.
*Suggestion dependent on transaction volume and scan rate

22. MES Tips and Common Mistakes: Continuously Running Work Order
“But we’re only using Performance, we don’t care about Work Orders”
The underlying MES database structure and table relationships makes Work Order management important
Problems with continuously running work orders:
Database performance degrades over time
Updating database records require the system to query large datasets
Archiving capabilities are limited (ex: Archive WO where closed > 30 days)
General recommendation is a new Work Order daily or at shift change
DS - A top 3 common problem.
We are talking about the Work Order, not just the Job

23. MES Tips and Common Mistakes: Work Queue Grid Filtering
The more that you try to display on the Work Queue grid, the slower it will load and refresh
How many Jobs does an operator really need to see?
Use New/Ready status to limit records
Create jobs as New, have a supervisor set them to Ready when they assign the work. Hide New jobs from the operator’s work queue.
Eliminates operator confusion
DS - From implementation diagram, this impacts Supervisory users and Shop Floor Users
The Queue control can become very slow with large record sets
General recommendation is to limit to about 10 jobs depending on the number of entities the user is logged into
Use Deploy Queue Filter and Sequence function to set the filter

24. MES Tips and Common Mistakes: Utilization Grid Filtering
The more that you try to display on the Utilization grid, the slower it will load and refresh
No filter will query the database for ALL utilization records for this entity
What does the operator really need to see vs. what can just show up on a report?
(right-click)
DS - Save Configuration: Saves the filter for yourself on this entity
Save Config for All Logged On Entities: Saves the filter for yourself for all logged in entities
Save as Default Configuration: Saves the filter for all users (without their own filters) on this entity only
Reload configuration: If made changes you have not saved, you can reload your saved fitler
Revert to Default: If made changes you have not saved, you can reload the default filter

25. MES Tips and Common Mistakes: Security
If utilizing OS-based security, the Wonderware Change Network User account must be a domain user
Special steps required to enable OS Group Security - Tech Note 708
Lost Database Connection Editor password
Delete C:\Program Files\Wonderware\ MES\Middleware\Middleware.cfg DS

26. MES Tips and Common Mistakes: OEE Calculation
JS - How OEE is calculated.
Larger record sets queried by these calculations will slow the system

27. MES Tips and Common Mistakes: Data Inconsistencies
Moving database to a new timezone?
Purge shift_to_go and tz_offset tables
Restart MES Service
Errors that Job exec limit exceeded, but Operator work queue does not show that many jobs running
Job state and Job_Exec current job are out of sync
JS - Often overlooked, but can be very important
Maintenance intervals depend on number of records being written
Good tables to monitor if maintenance is needed are item_prod and util_log
Backups are being done, but truncating transactions logs and db/file shrinking is often not being done

28. MES Tips and Common Mistakes: What does MES Service do?
Only one MES Service can run per database
Executes minutely, hourly, and daily housekeeping tasks
Shift changes
Session clean-up
Auto-ready jobs
Run scheduled Supply Chain Connector tasks
Run scheduled archive and purge tasks
Generate time-based Quality Samples
JS -

29. MES Tips and Common Mistakes: System Maintenance
Database maintenance
Truncate transaction logs and shrink files and database (TN 664)
Archive and/or Purge
Large tables:
item_prod (Production details)
util_log (Utilization states)
tpm_stat (OEE stats)
Custom reports can union data from Prod and Archive databases
Available from MES Services, not part of the standard product
Tech Note Using the New MES 4.0 Archive, Purge and Restore Functions
JS - Often overlooked, but can be very important
Maintenance intervals depend on number of records being written
Good tables to monitor if maintenance is needed are item_prod and util_log
Backups are being done, but truncating transactions logs and db/file shrinking is often not being done

30. MES Tips and Common Mistakes: Supply Chain Connector
MES Database
File
Xml or delimited
Supply Chain Connector
(MES Service)
Middleware
Table
3rd party or custom
Supply Chain Connector can have data brought in by either data file or database Table. JS

31. MES Tips and Common Mistakes: Supply Chain Connector
Questions to consider when deciding whether to use files or tables:
How much data will I be importing?
How often will the data be imported?
How much of the importing data will be new and how much will be updating existing records?
What will be creating the new data that will be imported? JS

32. MES Tips and Common Mistakes: Supply Chain Connector
File Base imports:
Pros
Easy to implement
Good for smaller amounts of data
Easy one time configuration scenarios
Cons
Cannot track the processing of individual records
Higher chances of data integrity issues
Slower performance with larger amounts of data JS

33. MES Tips and Common Mistakes: Supply Chain Connector
Table Base imports:
Pros
Able to track and keep information on individual processed records
Reprocessing of individual records is easier
Troubleshooting bad records is easier to find.
Cons
More difficult to implement
Better for larger amounts of data.
Needs more maintenance (cleaning out old records, backing up of tables….) JS

34. MES Troubleshooting Tools and TechniquesDS

35. MES Troubleshooting: Middleware
Licensing
MES Licenses must be installed on Middleware servers (restart MW)
Performance-only or Quality-only license can use OCO for production counts and basic job execution
MES 4.0 license is valid for MES 4.5
Database Connection Editor (restart Middleware if changed)
Verify SQL connectivity
Middleware Configuration Editor
Enable XML Trace of all middleware/SP calls with parameters (restart MW)
Middleware Proxy Configuration Editor
Test connection to Middleware server DS

36. MES Troubleshooting: Error Messages
MES SMC Log Flags
UCO – EquipmentStatusRuntime1
OCO – JobExecutionRuntime1
SRO – SampleCharacteristicRuntime1
Entity Model Builder – EntityModelBuilder
MES Service – FactMES.Server.Services
MES ArchestrA Objects
.ErrorMessage attribute DS

37. MES Troubleshooting: MES Client Debug
Modify C:\Program Files\Wonderware\MES\Client Framework Modules\aaClient.exe.config
Change this section:
<!-- Uncomment to enable the file based trace listener. <add name="LogTraceListener" type="System.Diagnostics.TextWriterTraceListener“ initializeData="Trace.log" /> -->
<!-- Uncomment to enable the file based trace listener. --> <add name="LogTraceListener“ type="System.Diagnostics.TextWriterTraceListener“ initializeData="Trace.log" /> DS

38. Useful Tech Notes
Tech Note Resolving MES Entity Model Builder Errors on a Remote IDE Development Node
Tech Note Using the New MES 4.0 Archive, Purge and Restore Functions
Tech Note Resolving Incorrect Shift Times When MESDB is Moved to Another Time Zone
Tech Note Solving Wonderware MES 4.0 Known Installation Issues
Tech Note 711 – Wonderware MES Operation Capability Object: Configuring Multiple Job Counters

39. Useful Tech Notes
Tech Note Wonderware MES Operations Capability Object: Configuring Rolling Counter Reset
Tech Note Enabling MES 4.0 Operating System (OS) Group Security
Tech Note Wonderware MES Sessn Table and Client Types
Tech Note Truncating Operations & Performance Transaction Logs
Tech Note Wonderware Performance 3.5 Software: Performance Tips

40. Other Useful Documents
MES 4.0 Deployment Guide
MES Database Diagrams
MES 4.0 Development Guide

41. Useful Sessions WW TSS-09: MS SQL Server Performance & Tuning
for Wonderware Database

42. Questions?
THANK YOU