2. Session Description
In this session will discuss issues related to communications between InBatch and both Archestra (IBMX) and PLCs(DAServers) as well as connectivity issues related to InBatch ActiveX Controls within InTouch. We will focus on issues encountered in a production environment to determine cause & resolution, best practices for analyzing and proper usage of diagnostic tools. Duration: 90 Minutes

3. WW TSS-05 InBatch Advanced Troubleshooting
Matt Mummert
Shri Mariyala
October, 2012
Title slide [from Title Slide master]
The title slide is available as a Title Master and the background and graphic elements are fixed. Text is editable for the slide heading and presentation details but the small compliance text exists in editable form only on the Title Master itself. This type area should be used only for the purposes of compliance text and should not be replaced by other content.
Font, type size and colours are fixed for the Title Master style and should not be modified.
Footer details
Footer detail appear in the bottom-left corner of the slide and appear in fixed positions and order. Editing of the Footer details is done by accessing ‘Header and Footer’ under the ‘View’ menu in PowerPoint®.

4. Project Start. Best Practices.
Batch systems are often considered mission critical in the production process; therefore, batch systems need to be designed so that the process integrity is ensured even during unexpected external events.
A batch management solution, facilitating the batch execution on physical process equipment, expands across multiple layers of the system, including (but not limited to):
 Control system infrastructure
 Network infrastructure
 Computer system infrastructure
 Batch Management software
 User interaction
Each one of these layers must be thoroughly analyzed and considered as a critical part of the batch management solution. It is vital that, after the critical variables surrounding the batch execution are identified, the system is designed to address their needs at the right level.
(From InBatch R&D WHITE PAPER - Best Practices and Considerations for Designing Reliable InBatch Solutions)
When you design a topology, consider the following requirements:
• What node types does the InBatch system require? • What station types does the InBatch system require? • Is redundancy required? • Should a tag server be used (to reduce I/O traffic)? • Are terminal services required?
• I/O Servers are shown on a separate node to act as data concentrators and provide a single I/O mapping source and ease PLC request overhead. You can also install I/O Servers on the InBatch server node if network bandwidth is a concern, but doing so is likely to increase the load on the batch server.

5. Project Start. Best Practices.
One of the most important things you can do on a project is to make sure that the system functional specification completely describes the desired operation of the system and no more.
(From InBatch deployment guide)
Concurrent Batch Processing
The phrase concurrent batch processing describes batches running in parallel, at the same time. The following conditions are assumed:
System performance in this context is also considered dynamic because of the many variables involved.
• Batch processing takes place from the batch manager to a programmable logic controller (PLC).
• Network bandwidth is adequate.
• Batch processing is asynchronous.

6. Best Pratices – What Do We Take Away?
InBatch Deployment Guide
InBatch R&D WHITE PAPER - Best Practices and Considerations for Designing Reliable InBatch Solutions

7. Advanced Troubleshooting - Topics
InBatch Architecture
Batch Equipment
Batch Phase Logic
IBCli/DAServer
Batch IBMX
Batch Warm Restart Files
BatchMngr Parameters

8. Advanced Troubleshooting - Topics
BatchGuiConfig Ctrl
Batch Redundancy
Batch History
Reports
System Monitoring
Log Flags

9. InBatch Architecture 1 Each InBatch system includes at least:
Access Reports Remotely
InBatch Runtime Clients/Touch
Batch Server + DAServer
Batch History
Information Server
InBatch Dev Client
Each InBatch system includes at least:
• One InBatch server node
• One InBatch History server node
• One Wonderware Information Server node with InBatch reporting components installed
• Possibly one InBatch client node
PLCs

10. InBatch Integrate with System Platform
Access Reports Remotely
InBatch Runtime Clients/Touch
Batch Server + DAServer GR
Batch History
Information Server
AOS
AOr
InBatch Dev Client
Each InBatch system includes at least:
• One InBatch server node
• One InBatch History server node
• One Wonderware Information Server node with InBatch reporting components installed
• Possibly one InBatch client node
PLCs

11. System Platform Integration – How ?
Extremely flexible and powerful batch management capabilities
InBatch Server
PHASE 1
PHASE 2
PHASE 3
Phases modeled within Control System
MX communication
PHASE 4
PHASE 5
PHASE 6
Phases modeled within System Platform
Device Integration
Device
System Platform
Extremely flexible and powerful platform and plant model 11

12. Batch Equipment Analysis
Batch started but no activity, why?
What is equipment status?
How to track and identify which equipment in use?
What is Unit, Connection & Segment Status?
How to update equipment status (TagView)

13. Equipment Definition - ModelEditor

14. Units/Connections Status - Batch Display

15. Segments Overview

16. Segments Linked to Connections

17. Equipment Status – TagView

18. Where is Tagview.exe?

19. Equipment Status – What Do We Take Away?
Unit Status values must be consistent - Process Model Vs USTAT tags
USTAT tags are linked to the correct addresses in PLC
Blank or incorrect values for USTAT leads to “waiting for available equipment” state
Allocating a connection allocates all segments of that connection
All connections that contain segment allocated by another connection are ‘busy’
Technote 25 – Using Segments to define Model Limitations in InBatch.

20. Monitor Phase Status – Batch Display

21. Monitor Phase Status – Phase Logic Tool

22. Where is InBatch Phase Logic Tool?

23. Phase Logic Tool – Add to Env. Display

24. Phase Logic Tool-Add to Env. Display

25. Monitor Phase Status – TagView

26. Monitor Phase Status – WWClient
Note: Phase status values using WWClient

27. Phase Monitor – What Do We Take Away?
Phase Control/Status bits linked to the control system must be unique and valid addresses in PLC. Duplicate items cause issues
Phase must always be in ready state prior to the beginning of the phase execution
At any point in time, only status is present.
Technotes as follows:

28. Phase Monitor – What Do We Take Away?
TN 555 – Active batch behavior when IBCli looses communications to IOServer
TN Phase control bit behavior
TN 312 – InControl phase logic for InBatch
TN 23 – Developing Allen Bradley Phase logic for InBatch

29. InBatch IBCli Parameters
Response Time - Time after which an IBCli request to the IOServer will timeout. Default 15 seconds
Ping Time - Interval at which IBCli will ping the IOServer to detect a connection loss. Default 2 seconds. To enable debug for IBCli, set the Ping Time to “-999”
Advise All - Performs an Advise All on initialization
Disable Timestamp – Disable the end of phase timestamp feature for this topic.

30. IBCli debug – Log Messages with PingTime = Reading a value from DAServer/IADAS: 2011/07/20 16:20:13.399/-03B4:14A4/IBCli /...(ProcessUpdate) Update for [G2_V9072-V9074-CS-DONE] of [1] 2011/07/20 16:20:13.399/-03B4:14A4/IBCli /...(ProcessUpdate) Time Stamp for [G2_V9072-V9074-CS-DONE] of [2011/07/20 20:20:11] Quality [1c0]

31. IBCli debug – Log Messages with PingTime = InBatch writing value to DAServer/IADAS 2011/07/20 16:20:13.899/-03B4:14A4/IBCli /(tt) wwCliPoke - request: [ ]  tag: <G2_V9072-V9074-CS-RESET> Value:[1] 2011/07/20 16:20:13.899/-03B4:14A4/IBCli  /...(tt) Adding tag to request - request: [ ]  tag: <G2_V9072-V9074-CS-RESET> req type: [1] 2011/07/20 16:20:13.899/-03B4:14A4/IBCli  /...(ProcessUpdate) Update for [G2_V9072-V9074-CS-RESET] of [1] 2011/07/20 16:20:13.899/-

32. Where are IBCli Parameters?

33. InBatch 9.5 – Trace IBCli via Logger

34. InBatch - Environment Editor
Any updates to Environment Editor requires Update Environment to affect changes, which means, stop runtime services, which means, it costs production down time.

35. DAServer/IADAS Parameters

36. DAServer/IADAS – Log Flags

37. InBatch IBMX - Add Application in Env. Editor

38. InBatch IBMX – Configure in TagLinker

39. InBatch IBMX – Launch Galaxy Browser from TagLinker

40. InBatch IBMX – Galaxy Browser

41. IBMX with Verbose Mode-Log Messages

42. InBatch IBMX - ObjectViewer

43. InBatch IBMX – What Do We Take Away?
Platform must be deployed to enable IBMX
IDE must be installed in order to browse Galaxy browser inside of TagLinker
Requires IDE and Runtime license for Platform
Use TagView, Phase Logic, Batch Display and ObjectViewer to track Phase Control/Status tags
InBatch and ArchestrA AppServer Communication Recommendations using IBMX – White Paper (Tech Article)

44. Batch Warm Restart
Warm Restart files - If there is no warm restart folder, batchmngr doesn’t startup
.batchwrdebug – Creating a .batchwrdebug folder from command line enables the debug for warm restart information. This folder created under x:\>program files\wonderware\InBatch\Cfg\Config_a\.batchwrdebug

45. Batch Parameters

46. BatchMngr Parameters
Param Timeout - Time in seconds, of all attempts to read a parameter tag value before timeout failure. Default value set to 30 seconds. If you see any parameter read timeouts in the logger tune this setting
Phase Batch Status – Puts all phases on hold when batch is on hold – Add this setting if all phases needs to be on hold when batch is on hold
Disconnect Clients – Disconnect all deadlock (misbehaving) clients – Add this setting if any clients misbehaving, otherwise it will cause negative impact to the BatchMngr

47. InBatch ActiveX Ctrl - BatchGuiConfig

48. BatchGuiConfig – Properties
Note: MessageBoxes has to be disabled from popup, otherwise it will cause the view to not respond until message is acked.

49. BatchGuiConfig – Events
Terminate/Initialize the control using Startup/Shutdown events of Batch Server accordingly.

50. SFCCtrl and OcxBatchCtrl
Terminate/Initialize these controls inside of view using Shutdown/Startup events of BatchServer

51. InBatch ActiveX Controls – What Do We Take Away?
View locks up due to InBatch ActiveX Controls not terminated gracefully when connection lost to BatchServer (Server shutdown/loss network connection)
Technote 364 – How to configure BatchGuiConfig ActiveX Ctrl for Redundancy Client Switch-Over. This technote is good for Standalone Server also.
Starting with InBatch 9.5, use Technote 797 “Configuring InBatch Runtime Client Communication Timeouts.” These timeouts are configured in registry and technote covers details on configuration.

52. InBatch Redundancy
Configuration Utility

53. InBatch Redundancy (cont.)
RedCfg File
LMHost File
Messages in the logger

54. InBatch Redundancy (cont.)
Contents of RedCfg file
Line 1: Remote machine name.
Line 2: (blank, not used).
Line 3: The path to the remote machine’s config_A directory.
Line 4: (blank, not used).
Line 5: The local hostname for the primary network (must not be a machine name).
Line 6: The local hostname for the secondary/backup network, if one exists.
Line 7: The remote hostname for the primary network (must not be a machine name).
Line 8: The remote hostname for the secondary/backup network, if one exists.
Note: The file content must end with a carriage return. Just press the Enter key after the last entry and then save the file. The contents of the RedCfg file must exactly match the entries in the Hosts or LMHosts files.

55. InBatch Redundancy (cont.)
Contents of LMHost file
BATCHMASTER #PRE BATCHBACKUP #PRE INTOUCH1 #PRE INTOUCH2 #PRE INTOUCH3 #PRE BATCHMAST #PRE BATCHBACK #PRE BACKUP1 #PRE BACKUP2 #PRE

56. InBatch Redundancy (cont.)
Technotes:
139 – Installing and configuring Redundancy Option for InBatch
302 – How to create host file and redcfg file for InBatch redundancy setup which is still valid for later versions also
881 – Configuring InBatch Redundant Servers to Sync in a WorkGroup Environment

57. Batch History & Reports (cont.)
ReportQReader
Custom Reports – Tech Note 767

58. Log Flags for 9.5 XES Diags – all service apps App Load: %
Max Delay: s
Trace – HistQReader
E.g. transactions and admin thread activity
Trace – IBMX / IBCLI
E.g. read, write, data updates, quality, ack/nak, etc
Replaces –d (debug) and –p-999 (PingTime) options

59. Log Flags for 9.5 (cont.)
Trace – OcxBatch ActiveX / Batch Display / Batch Scheduler
Init, term, heartbeat, focus
Trace – Recipe Editor
XML Import and Export

60. Log Flags for 9.5 (cont.)
Security and Active Directory Trace Log Options
[HKLM\SOFTWARE\Wonderware\InBatch\ArchestraSec\Debug]
[HKLM\SOFTWARE\Wonderware\InBatch\NativeSec\Debug]

61. Log Flags for 9.5 (cont.) Batch WR File Log Option
Create folder before startup. Logs to folder. Send to Tech Support.

62. Diagnostic Tools
Operating System Log Files - Application/System/Security Event Log files
ArchestrA Log files
Windows Debugging tool
Phase Logic Tool
WWClient

63. Diagnostic Tools (cont.)
Microsoft WinDbg - used by Tech Support to examine DMP files.

64. Diagnostic Tools (cont.)
Technotes:

65. Questions?
THANK YOU