Presentation is loading. Please wait.

Presentation is loading. Please wait.

Lead from the front Texas Nodal 1 Texas Nodal Market Implementation Program Infrastructure Update October 22, 2007.

Similar presentations


Presentation on theme: "Lead from the front Texas Nodal 1 Texas Nodal Market Implementation Program Infrastructure Update October 22, 2007."— Presentation transcript:

1 Lead from the front Texas Nodal http://nodal.ercot.com 1 Texas Nodal Market Implementation Program Infrastructure Update October 22, 2007

2 Lead from the front Texas Nodal INF: Project Summary - Initial Charter with Approved Changes Project area: Infrastructure Description: Provision of development, testing, EDS and production environments across the Program Vendor(s): IBM, EMC, OracleProject Manager: David Forfia Key deliverables/short term deliverables: –Hardware specifications –Hardware procurement –Data center capacity resolution –IT Services Catalogue –Service Level Agreements for all Nodal projects –Project development & test (FAT) environments –Integration testing (SAT) environments –EDS environments –Production environments –Market Participant Identity Management (new) –Release Management processes (from INT) –Oracle 10g upgrade for EDW (new) –High availability monitoring environment (new) Key Assumptions: –Infrastructure capacity can be incrementally added as the project progresses using IBM’s capacity upgrade on-demand model –Data center capacity issues will be resolved in the next 90 days Challenges/Risks: –Existing Data Center capacity (power) Comments: –IT Operations will be the first ERCOT function to transition to Nodal operations, starting with setting up development environments http://nodal.ercot.com 2 TPTF Nodal Update 10/22/07

3 Lead from the front Texas Nodal INF: Project Summary - Initial Delivery Plan EDS 3 Release NMMS Computing Infrastructure Q2 2008Q3 2009Q4 2008Q3 2008Q2 2007Q1 2007Q4 2006Q4 2007Q1 2008Q3 2006Q1 2009Q2 2009 EDS 3 Design, Build, Pre-FATFATITEST 12/01/08 Real Time Operations GO LIVE ITESTBuildFAT Requirements, Conceptual Design EDS 4 EDS 4 MP SAT EDS 3 Data Validation Planning UI FATITEST Common model update process Conceptual Design Build 6/30/08 LMP Market Readiness Criteria MET Q3 2007 Operations UI 3/31/08 Single Entry Model GO LIVE 12/08/08 Day Ahead Market/CRR GO LIVE Zonal Shutdown GO/NO GO Storage Software Data Center Capacity Design to Min Requirements Database Licensing Strategy Portal/Integration Licensing Strategy Decommissioning plan Data center virtualization Colocation Existing Data Center Upgrade New Data Center Facility EDS 4 Release Capacity Upgrade Dev Disaster Site #1 Design to Min Requirements Prod #1 Prod #2 Dev DR #2 Capacity needs met? Capacity Planning Capacity Upgrade on Demand Enabled Sufficient Power Recovered ? Capacity Upgrade Initial Storage Upgrade Capacity Planning Capacity Upgrade http://nodal.ercot.com 3 TPTF Nodal Update 10/22/07

4 Lead from the front Texas Nodal INF: Project Summary – Actual Delivery Plan is slightly later than planned EDS 3 Release NMMS Computing Infrastructure Q2 2008Q3 2009Q4 2008Q3 2008Q2 2007Q1 2007Q4 2006Q4 2007Q1 2008Q3 2006Q1 2009Q2 2009 EDS 3 Design, Build, Pre-FATFATITEST 12/01/08 Real Time Operations GO LIVE ITESTBuildFAT Requirements, Conceptual Design EDS 4 EDS 4 MP SAT EDS 3 Data Validation Planning UI FATITEST Common model update process Conceptual Design Build 6/30/08 LMP Market Readiness Criteria MET Q3 2007 Operations UI 3/31/08 Single Entry Model GO LIVE 12/08/08 Day Ahead Market/CRR GO LIVE Zonal Shutdown GO/NO GO Storage Software Data Center Capacity Design to Min Requirements Database Licensing Strategy Portal/Integration Licensing Strategy Decommissioning plan Data center virtualization Colocation Existing Data Center Upgrade New Data Center Facility EDS 4 Release Capacity Upgrade Dev Disaster Site #1 Design to Min Requirements Prod #1 Prod #2 Dev DR #2 Capacity needs met? Capacity Planning Capacity Upgrade on Demand Enabled Sufficient Power Recovered ? Capacity Upgrade Initial Storage Upgrade Capacity Planning Capacity Upgrade http://nodal.ercot.com 4 TPTF Nodal Update 10/22/07

5 Lead from the front Texas Nodal Data Center virtualization was the only viable strategy to make the Nodal timeline Move to a Collocation Site –RFP issued in October 2006 Insufficient capacity available to support ERCOT specialized needs Expand Existing Data Centers –Taylor Lead times for core equipment longer than Nodal program –Austin Existing facilities already expanded to maximum capacity Long term viability of the facility not determined Acquire a new Data Center Facility –Lead times for acquisition and relocation outside Nodal timelines –Currently being explored with the viability of the Austin facility http://nodal.ercot.com 5 TPTF Nodal Update 10/22/07

6 Lead from the front Texas Nodal Expanding existing data center capacity is an integrated process There are 3 components which are balanced to ensure a reliable data center –Standby Generator Capacity –Uninterruptible Power Supply (UPS) capacity –Data Center Air Conditioning (DCAC) capacity The maximum capacity for equipment is determined by the minimum carrying capacity of any one of the components. We have taken all steps possible to maximize the current capacity of the data centers to running at the available capacity of the UPS systems in each site. http://nodal.ercot.com 6 All possible near term facility upgrades have been completed.

7 Lead from the front Texas Nodal The growth in Nodal server deployments and capacity was correctly forecast http://nodal.ercot.com 7 TPTF Nodal Update 10/22/07

8 Lead from the front Texas Nodal Majority of the roadmap is completed, but not all assumptions were right Achievements to date Power Recovery  Executed the Enterprise Architecture Power Recovery Plan Development Storage Retired Development Virtualized Domain Restructured Quality Assurance Moved to ACC Retired unused equipment ½ Server refreshes EMS & Non-EMS Test/Prod Virtualization ¾Database Hosting Refresh  Identified additional compression activities Relocated Development databases servers to Blue Building Austin SAN Refresh Taylor SAN Refresh Database Server Clustering ACC to dedicated UPS Remote access server farm redesign o Application Server Refresh o Self cooled equipment racks (ordered) Key assumptions which were invalid Power consumption server would drop exponentially Power consumption per CPU has declined almost as much as assumed Server memory power consumption has offset power savings in CPU power consumption Nodal redundancy requirements would mirror Zonal The nodal systems require more active/passive and active/active deployments than the current Zonal market. The required level of redundancy and recoverability for the nodal systems was not fully understood when the projections were made in February 2006. Nodal environment requirements would mirror Zonal Integrating a large number of best of breed solution requires more environments to successfully develop the integration points. Market participants required structured and unstructured testing environments to complete their development activities. http://nodal.ercot.com 8 TPTF Nodal Update 10/22/07

9 Lead from the front Texas Nodal Server consolidation timeline was constrained to minimize risks We will do this with the minimum disruption We have a plan to: –Minimize the risk –Maximize the benefit –Lower overall costs –Safeguard Market Operations –Improve service levels –Not affect Texas Set 3.0 However, there are always risks to be aware of in server migration We are working with all the project managers and ERCOT committees to reduce risk and to optimize the timing. In the process of server consolidation, one production migration was deferred and successfully rolled back two. –The net effect was a delay in final migrations and power recovery by 5 weeks. http://nodal.ercot.com 9 TPTF Nodal Update 10/22/07

10 Lead from the front Texas Nodal Migration Metrics Servers Starting TotalDecomRetired% Remaining 7416572% Databases Starting TotalMigrated Retired/ Refresh% Remaining 9158277% Applications Total FilesMigratedRemaining% Remaining 6255792% Scripts ReviewedRemaining% Remaining 229 00% Production Database Storage (in GB) Pre-Migration Bytes Used Post Migration Bytes UsedReclaimed Storage Savings 12,0808,4803,24036% Servers Annual maintenance contracts will be cancelled on the retired database and application servers Servers will be made available on the secondary market to recoup their residual values Databases Unused or underused databases where eliminated resulting in additional licenses for use at Nodal vendor locations Database Storage Properly sizing the databases and compressing the data files for data which has been removed is resulting in a 36% reduction in used storage in production. Server consolidation will result in lower expenses during Nodal and after Nodal implementation http://nodal.ercot.com 10 TPTF Nodal Update 10/22/07

11 Lead from the front Texas Nodal http://nodal.ercot.com 11 System requirements are driven by project detailed design documents Each project team has an assigned architect who is responsible for the architecture for the project’s deliverables that are consolidated into a deployment diagram by IDA that become individual work requests for operations to deploy the systems

12 Lead from the front Texas Nodal http://nodal.ercot.com 12 Market trials and Integration testing will drive changes to the environments The assumption in the infrastructure plan is that the initial deployment specifications will have to be changed. The infrastructure technologies selected as the core for the Nodal system were picked because they adapt well to change.

13 Lead from the front Texas Nodal http://nodal.ercot.com 13 TPTF Nodal Update 10/22/07 Provides the ability to scale up to the maximum capacity of the system to meet usage demands System can recover from multiple component failures (CPU/Memory/Power Supply/ IO) with spare capacity within the system. System availability above the 99.9% threshold. Provides the ability to balance the load across all applications running on the system. Minimum power usage configuration. Critical Path Mitigation Strategies Capacity issues – Scale Up Option Scale up options are implemented on the largest computing systems in the data center. Extra capacity is available to enable when necessary inside the system or can be added without system down time.

14 Lead from the front Texas Nodal http://nodal.ercot.com 14 TPTF Nodal Update 10/22/07 Requires duplicate computing resources on two separate physical servers. Provides the ability to scale up to the maximum capacity of all systems in the cluster to meet usage demands. Classic Windows / Linux capacity expansion strategy typically with a load balancing appliance. Provides the ability to do maintenance on a server without impacting the system. System availability above 99.99% Critical Path Mitigation Strategies Capacity issues – Scale Out Option Scale out options are implemented on the smaller computing systems in the data center. Processing load is split across multiple systems to meet the business requirements. ERCOT will utilize both a scale up and scale out strategy to meet the business requirements of Nodal

15 Lead from the front Texas Nodal Critical Path Mitigation Strategies Procurement and Architectural Delays RiskMitigation Strategy Long procurement lead timesDefine system requirements as soon as possible and place orders. Utilize existing systems or virtual machines where appropriate Late Technical Architecture design documentsPreorder unassembled equipment and have ERCOT staff built to specification Authorize overtime to build systems Pre-build standard server configurations based upon service catalogue and assign to projects as requirements become known Server consolidation decommissions behind schedule Run data center in the safe “buffer” zone of capacity http://nodal.ercot.com 15 TPTF Nodal Update 10/22/07

16 Lead from the front Texas Nodal Questions http://nodal.ercot.com TPTF Nodal Update 10/22/07 16


Download ppt "Lead from the front Texas Nodal 1 Texas Nodal Market Implementation Program Infrastructure Update October 22, 2007."

Similar presentations


Ads by Google