1. STP Enhancements for z10 EC, z10 BC, z9 EC, z9 BC
Noshir Dhondy (retired)
November 12, 2008

2. Server Time Protocol (STP) Enhancements summary
Enhanced Accuracy to an External Time Source (ETS)
Available on z10 and z9 (EC and BC)
10 microseconds accuracy vs. 100 milliseconds accuracy when using either Dial out from HMC or NTP Client support as the ETS
Utilizes highly stable, accurate Pulse Per Second (PPS) output from NTP servers available from some vendors
Continuous Availability of External Time Source (ETS)
ETS configured for Backup Time Server (BTS) can be used for time adjustments to maintain ETS accuracy, if ETS configured for Preferred Time Server (PTS) not accessible
Applies to NTP server support with or without PPS
Enhanced STP recovery when Internal Battery Feature is in use
Designed to enable BTS to take over as the CTS if
Power outage of PTS/CTS
Power outage of site where PTS/CTS is located

3. Server Time Protocol (STP) Enhancements summary (continued)
System Management enhancement - STP configuration and time information saved across Power on Resets or power outages
Available on z10 and z9 (EC and BC), z990 and z890
Does not require initializing the time or reassigning PTS/CTS roles
“One server” configuration enhancement
Enhanced security - NTP Server on Hardware Management Console
Applicable only for z10 and z9 (EC and BC)
NTP Client on Support Element (SE) can access NTP server using a secure LAN connection
API to automate STP CTN Reconfiguration
Allows automation to reconfigure special roles of servers (PTS, BTS, Arbiter) after a recovery action
Maintains recovery capability in the event of a second failure

4. Enhanced Accuracy to an External Time Source (ETS)

5. Enhanced Accuracy to an External Time Source (ETS)
Simple Network Time Protocol (SNTP) client support added to Support Element (SE) code of z9 and above servers to interface to Network Time Protocol (NTP) servers
NTP Client support GA October 2007
Time reference for NTP server can be GPS, dial-out, DCF-77 etc
NTP servers provide multiple NTP outputs
Time information
Some NTP servers also provide a highly stable, accurate “Pulse per second” (PPS) output
ETR card of z10 EC, z10 BC, z9 EC and z9 BC has additional PPS input
Each System z has 2 ETR cards providing capability of attaching to two NTP Servers with PPS output for redundancy
STP utilizes the PPS signal to provide additional accuracy
Accuracy to PPS – 10 microseconds

6. Enhanced Accuracy to an External Time Source (ETS) (continued)
Only the Current Time Server (CTS) steers the time based on:
Timing information sent by Support Element (SE) code
PPS signal received by its PPS port on ETR card
Recommend configuring NTP server with PPS on PTS/CTS as well as BTS
Provides ETS capability when BTS becomes or is assigned the CTS role
z990, z890 can only dial out from HMC for an ETS
Does not have NTP client capability

7. Enhanced Accuracy to an External Time Source (ETS) - ETS redundancy on same server (PTS/CTS) example

8. Enhanced Accuracy to an External Time Source (ETS) - ETS redundancy on PTS and BTS example
z10 EC
PTS/CTS
z990
Arbiter
z9 BC
(BTS)

9. Continuous Availability of External Time Source (ETS)

10. Continuous Availability of ETS - two site example
BTS transmits adjustment
information to PTS/CTS
based on its NTP/PPS data
When PTS/CTS detects
failures associated with its
NTP/PPS data
PTS/CTS switches to using
NTP/PPS data from BTS
CTS role DOES NOT switch
to BTS

11. Enhanced STP recovery when Internal Battery Feature is in use

12. Power Outage PTS/CTS with Internal Battery Feature
With Internal Battery Feature (IBF) on CEC1
CEC1 power outage, enters IBF state
CEC1 notifies CEC2 it is running on IBF
CEC2 waits for 30 seconds to take action
Could be a power glitch
If notified within 30 seconds that CEC1 back to “normal power”, no further action
If CEC1 in IBF state > 30 seconds,
CEC2 takes over as the CTS
CEC1 becomes S2 until IBF no longer functional and power drops
CEC1 power resumes
Automatic re-takeover as PTS/CTS
HMC
CEC1
PTS/CTS S1 P1
CEC2
BTS S2 P3
Coupling links P2
CEC power outage in same data center
HMC
CEC1
PTS/CTS S1 P1
CEC2
BTS S2 P3
Coupling links P2
Site power outage – 2 data centers

13. Power outage of data center (Site 1) with PTS and Arbiter
HMC
CEC1
PTS/CTS S1 P1
CEC2
(BTS) S2
Coupling links
CEC3
Arbiter P3
CEC4 P4
Site 1
Site 2 P2
With Internal Battery Feature (IBF) on CEC1 and CEC3
Site 1 power outage
CEC1 and CEC3 enter IBF state
CEC1 and CEC3 notify CEC2 it is running on IBF
CEC2 waits for 30 seconds to take action
Could be a power glitch
If notified within 30 seconds that CEC1 back to “normal power”, no further action
If CEC1 and CEC3 in IBF state > 30 seconds,
CEC2 takes over as the CTS
Not dependent on Arbiter Assisted Recovery
CEC1 becomes S2 until IBF no longer functional and power drops
CEC1 power resumes
Automatic re-takeover as PTS/CTS

14. IBF Recommendations Single data center Two data centers
IBF only protects for server power outage
CTN with 2 servers, install IBF on at least the PTS/CTS
Also recommend IBF on BTS to provide recovery protection when BTS is the CTS
CTN with 3 or more servers IBF not required to recover from CTS power outage, if Arbiter configured
Two data centers
IBF protects for both server and site power outage scenarios
CTN with 2 servers (one in each data center) install IBF on at least the PTS/CTS
CTN with 3 or more servers, install IBF on CTS and Arbiter (in same site as CTS)

15. STP configuration and time information saved across Power on Resets or power outages

16. Single server STP-only CTN
Previously
When an IML/POR is done for a single server STP-only CTN, user is required to:
Initialize the Time
Redefine that server as the stratum 1 after IML is complete
System Management enhancement
Customer selects the option to not allow any other servers to join the single server CTN when configuring the CTN
STP CTN configuration (PTS/CTS) will be reinstated after IML/POR,
Initialize Time not required
Fulfilled customer requirement for single server CTN

17. NTP Server on Hardware Management Console

18. Using an NTP server as External Time Source (ETS)
NTP client code on Support Element (SE) of z10 and z9
NTP server must be directly connected to the SE LAN
SE LAN considered to be a private dedicated LAN
Isolated from the corporate and public networks
NTP server on HMC addresses security concerns
NTP client on SE can now use the HMC as an NTP time server.
Provides LAN isolation for SE NTP Client
The HMC has two LAN ports, physically isolated:
HMC/SE LAN used by STP’s NTP client code
Second LAN used by HMC to access an NTP time server
PPS cannot be obtained from ‘HMC as an NTP Server’

19. Internet NTP servers (NTP Project)
Previous capability – Using Internet NTP Servers
Internet NTP servers (NTP Project)
(source for NTP server B)
non-System z time synchronized servers
Ethernet Switch
Corporate network
Internet
HMC
NTP server B
(UNIX, Linux)
Stratum 2
IBM
firewall
NTP server B defined on CTS/PTS
CTN
Remote HMC
(Browser)
z990
Arbiter
z9 EC
PTS/CTS
z9 BC
(BTS)
IBM Support
Center (RSF)
NTP server B defined on BTS

20. Internet NTP servers (NTP Project)
NTP Server on Hardware Management Console
Internet NTP servers (NTP Project)
(source for NTP server B)
LAN isolated from HMC/SE LAN
LAN
non-System z time synchronized servers
Ethernet Switch
Corporate network
Internet
HMC
NTP server B
Stratum 2
IBM
firewall
NTP server B defined on CTS/PTS
CTN
Remote HMC
(Browser)
z990
Arbiter
z9 EC
PTS/CTS
z9 BC
(BTS)
IBM Support
Center (RSF)
NTP server B defined on BTS

21. Redundant NTP server configuration using NTP server on HMC

22. API to automate STP CTN Reconfiguration

23. Automation to reconfigure STP-only CTN
Targeted for GDPS and non-GDPS customers
Fulfilled customer requirement to remove manual reconfiguration required without API.
Provides automation to reconfigure STP-only CTN for the following:
Unplanned outage when BTS takes over as CTS (e.g., Server failure or Power outage)
BTS is single point of failure in case of second failure affecting BTS
To maintain STP recovery capabilities, STP CTN reconfiguration required
If original CTN has 2 servers, no further reconfiguration possible
If original CTN has 3 servers, PTS, BTS reconfiguration possible
If original CTN > 3 servers, PTS, BTS, Arbiter reconfiguration possible

24. Automation to reconfigure STP-only CTN (continued)
Provides automation to reconfigure STP-only CTN for the following:
Planned or Unplanned outage of BTS or Arbiter (e.g., Server failure or Power outage)
PTS is single point of failure in case of second failure affecting PTS
To maintain STP recovery capabilities, STP CTN reconfiguration required
Planned action when disruptive action needs to be performed on CTS (e.g., POR, or activate/deactivate)
CTS role needs to be moved to alternate server (most likely BTS)
Capability to reconfigure CTN to maintain full recovery capability
Capability to move CTS role back to original configuration after planned action completed

25. Summary Enhancements announced October 21, 2008
Enhancements generally available October 28, 2008 for z10 EC and z10 BC
Enhancements generally available January 28, 2009 for z9 EC and z9 BC
STP configuration and time information saved across Power-on-Resets or power outages on z990 and z890 generally available January 28, 2009
Additional Information:
Server Time Protocol Planning Guide, SG
Server Time Protocol Implementation Guide, SG
STP Web Site:
STP education module on Resource Link