Presentation is loading. Please wait.

Presentation is loading. Please wait.

Best Practices at AT&T Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite Bryon Rickey Ravi Kayarthodi September 20th, 2010.

Similar presentations

Presentation on theme: "Best Practices at AT&T Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite Bryon Rickey Ravi Kayarthodi September 20th, 2010."— Presentation transcript:

1 Best Practices at AT&T Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite Bryon Rickey Ravi Kayarthodi September 20th, 2010

2 Agenda Welcome and Introductions
Supply Chain Management (SCM) environment Overview/highlights Architecture Partnership with Oracle to transform our Business Key project highlights Future initiatives Solution Support Center (SSC) and on-site support Q&A Ravi Kayarthodi: My name is Ravi Kayarthodi. I am a senior architect at AT&T. I am responsible for our Corporate Supply chain platform architecture; platform includes Oracle E-Business Suite on Oracle Real Applications Clusters, several other real applications clusters databases and middleware and monitoring tools. I am an active member of Oracle RAC standards board at AT&T 5 years ago, we migrated our ERP System database (formerly Cingular Wireless’ ERP systems) to RAC to meet the increased availability, fault tolerance and scale-up requirements that arose from acquisition of former AT&T wireless and hence applications system consolidation and optimization initiative we started. After migrating to 3-node RAC, we made this platform as a central nervous system for our Finance and Supply Chain operation, managing all of inventory and assets. Currently it is our platform where all of OneATT’s orders flows though as we show in the next couple of slides. Personally, I have worked on Oracle Products for over 16 years. I worked for Oracle Corporation before I joined AT&T team. I am an engineer by education and I also have a graduate degree in management.

3 About AT&T AT&T is one of the world’s largest Telecommunications companies AT&T Global Headquarters is based in Dallas, TX AT&T has approximately 290,000 employees AT&T has more than 90 million wireless customers AT&T has more than 16 million high speed internet subscribers AT&T has more than 2.5 million uverse subscribers Exclusive U.S. provider of iPhone 3G / 3GS / 4

4 AT&T SCM: Architecture highlights
Oracle E-Business Suite Centric Scale-up with cluster database on very large infrastructure High availability with Oracle Data Guard, Oracle GoldenGate, and site level redundancy Central inventory/financial management; utilizes product hub, planning/forecast with ASCP/Demantra Order orchestration and brokering with in-house developed Oracle based application system Internet deployment for interaction with external suppliers Management reporting/profitability analysis; procurement and spend analytics Industry leading warehouse/distribution management systems

5 AT&T SCM Platform: Financial Integration
Both Network and Retail SCM - financial counterparts are integrated in the same platform: Shared AP, AR, PA, FA, and GL Supply Chain transactions leverage same referential data and accounting rules Financial integration of supply chain transactions are seamless and are recorded real-time in the general ledger No need for reconciliation between sub ledgers Ability to forecast financial implications based on SCM forecasts Online financial budgeting validation on purchasing and requisitions Analytical GL Cube and Inventory Cube

6 Results AT&T has saved more than $500 million and has improved its purchasing power by combining accounts with its vendors. Estimated to save $1B in inventory costs, and $300M per year in reduced operating costs. Warehouses have been able to keep inventories low without missing SLA, resulting reduced overall inventory costs. Ability to identify sold merchandise, resulting in fewer fraudulent returns

7 Industry Recognition 2010 IDG’s CIO Award
Recognizes organizations around the world that exemplify the highest level of operational and strategic excellence in information technology

8 Systems Architecture

9 AT&T SCM: E-Business Suite Centric
This looks very Complex, right? No, it is not so much complex but it does support complex business functions. As you see in this diagram, all our business functions interact with this central system powered by Oracle E-Business Suite. This central system is our system of record It is our operational data store too All the supply chain, asset management and financial function from the order capture to fulfillment, from procure to pay, and from acquire to retire are performed in this system It provides integration with billing applications It provides market making capability utilizing efficient supply chain configuration

10 AT&T SCM: Key Systems Architecture
EDI EDI GL ODP eAuction PO AP OM ASCP RNF PA FA Inventory Planning Purchasing Financials Oracle E-Business Suite 11i Receipts Receipts Audits As we see in this slide, our application system provides links to customers & suppliers Our system supports and integrates business processes It links to supplier on the top and customers on the bottom via distribution center These systems also provide inventory tracking and asset tracking ability As we see here, demand planning for our assets and also for inventory, recording all the financial data in the same system without loosing sight into receipts and shipments. Shipments Moves Repairs Warehouse Mgmt Inventory Tracking Distribution Center Corporate Asset Tracking

11 Oracle 11i E-Business Environment
Forward Orders Store Replenishment Dealers/ iStore National Retailers DSL and uVerse HSIA Direct Fulfilment Upfront EPL Planning Purchasing Order Management Inventory Receivables Financials Oracle E-Business Suite 11i Apps Store XBM/Insurance Dealer/NR Service Depots RU30 Opus Sales Returns Company Owned Retail Reverse In the order management – Forward orders from Store Replenishment, dealers, technician orders and forecasted demand gets into SCM platform for processing. On the reverse order end we have Corporate Stores, returns, Service depots and so on In summary, we have a database cluster having 416 CPU cores and 1.7TB memory in the DB cluster alone On a typical day, this DB cluster processes 25 million DB transactions, over 200,000 concurrent jobs and over 2B SQL executions 416 CPU Cores & 1.7TB of Memory in DB Cluster 14TB of storage for transaction DB 3TB of storage for Archived DB

12 AT&T SCM Platform: Oracle10g RAC Infrastructure: VCS+ CRS
Highlights of this architecture – showing redundancies at network, storage and other infrastructure components level. 416 CPU Cores (104 per node from Sun/Oracle Enterprise 25K frame), and 1.7TB allocated memory 98 CPU cores in 6 Application tier servers In the cluster, we have 8 counts of 1 Gigabit Ethernet for private interconnects per node 4 for UDP/cache fusion traffic while other 4 for shared storage (SFRAC) cluster interconnect 16 storage ports per server Dedicated backup network We have Oracle Cluster ware/ Cluster ready service and also utilize vendor provided cluster – VCS for shared storage and file system We monitor interconnect traffic periodically We have tweaked LMS processes to provide us optimal performance, LMON runs in real time priority

13 AT&T SCM Platform: Transaction Volume
Approximately 3.35 million logons to the production areas per day 2.06 billion SQL executions per day Approximately Billion logical reads and 41TB of physical reads performed per day Approximately 7 TB of writes performed per day Approximately 1.05 billion user calls and 52 million transactions completed per day 200,000 concurrent request per day

14 AT&T SCM Platform: Operational architecture
Decoupled month-end book closing preparation Statistics gathered on a clone Separate reporting environment – refreshed nightly; Inventory and GL Cubes for reporting Revenue share and allocation on a separate system Direct supplier interaction with Internet-deployed application modules Utilizes market making mechanisms (sourcing) Preparing out system for month-end-closing We gather statistics on the cloned environment and then import into production environment. We do not allow scheduled reporting on this cluster database; instead we process of clone reporting from operational database nightly Revenue share/allocation on separate system Very same production environment had sourcing application for reverse auction that is time critical

15 Oracle 11i ERP/SCM– Reporting environment Refresh Summary
Nightly reporting database refresh Nightly Asset Synch and Audit summarization from PA, FA, AP,PO,CATS Network SCM reporting data to external System 251 Stored Discoverer reports with over 21,000 folders Business Objects reports (Ad-hoc and out-of-box) Reporting environment is available for summary and detail reporting. Complete set of data is available as of previous day 11:00pm Asset Tracking Asset Synch ERP PROD Corporate Warehouse Data Guard Data Guard L-TRA Shadow Image RPT External System Archived Transaction History Slide 15: Nightly reporting environment build goes through the shadow image synchronization and slit process. First, we have 2 tiers of recovery environments for business continuity; LTRA (providing database level redundancy) and RTRA (providing site level redundancy) in our deployed architecture Reporting environment build uses LTRA and Hitachi shadow image tool Every night our reporting build job stops this reporting database and starts synchronizing storage blocks with LTRA storage without stopping LTRA; once block level synchronization is completed, it breaks that mirror, opens the database and completes summarization. We utilize transportable tablespace to keep historical summary allowing summarization process only for new records. We use reporting database use for detailed reporting, and we use discoverer, business objects and other ad-hoc tools Reporting environment available for users 2 hours BI Discoverer BOBJ

16 AT&T SCM: Oracle 11i Dev/Test Configurations
Bothell, WA HDS – USP-V HDS – USP-V Alpharetta, GA NetApp GOLD R-TRA L-TRA PROD Data Guard/ Custom Shipping Data Guard Data Guard Data Guard/ Custom Shipping Shadow Image MET2 NetApp GOLD Shadow Image NetApp Master SnapMirror RPT FTP NetApp Master Flex Clone Flex Clone Shadow Image QA DEV DEV QA DEV Flex Clone With several simultaneous development projects running in parallel, we have several copies of production environment to support these projects. We apply strategy to supports these projects. We use NetApp Flex-Clone/Mirror tools to clone Development/test environments quickly. We are able to complete environment clones in 2 hours. We use traditional cloning methods to clone Performance test environments that need to be in SAN/fiber attached storage. MET1 DEV DEV QA Flex Clone FTP DEV DEV DEV QA Storage Efficiency with NetApp de-duplication Tools Performance test environment on tier-1 storage

17 AT&T SCM: Oracle Automatic Storage Management
Local and Remote DR databases and also Reporting databases have been migrated to ASM storage to meet storage and capacity needs Avoided SFRAC costs Performance SCMP DR/BC database converted to RAC database Saved SFRAC license cost on E25K frames Comply with AT&T standard for cluster database (ASM is required) Expected to save cost in the future when we upgrade other Corporate Systems’ databases Since we started with Oracle9i RAC, having stabilized and mature environment performing optimally, we continue to use SFRAC in our production database. However, our standard is to use ASM with RAC. We have migrated business continuity environments on RAC and have avoided costs. With this we are ready for ASM

18 AT&T SCM: Archive and Purge
Archive Stats 16 custom and core modules including (GL, INV, OM, AR, WSH, PO, FA) Executed weekly, monthly, quarterly, and yearly History instance Informatica Data Archive application used Schedule is monitored real-time to ensure adherence Purge Stats 70 jobs via Oracle concurrent programs Executed daily and weekly To keep our production operation database in trimmed to achieve better performance, we archive older transactions. We apply aggressive archive schedule yearly, quarterly, monthly and weekly archive strategy and store that in the database that can be accessed for reference both from production and reporting environments. We also use enterprise archive with compression achiving storage efficiency.

19 AT&T Supply Chain Geographic Distribution – DB Flow to Test and Dev
Primary Site XX XX XX XX Test Site 1 XX Development Site 2 XX Test Site 2 XX XX XX XX As we see in this picture, our application environments spread into six data centers, located various part of the country. We have distributed considering both disaster recovery and proximity to user group groups DR Site Development Site 1 XX XX XX XX Prod XX XX XX XX XX XX XX Dev/Test XX XX XX XX XX XX

20 AT&T SCM: Spend Analytics
Oracle Business Intelligence Enterprise Edition (Oracle OBIEE) Powered Sourcing from Oracle E-Business Suite and Legacy Applications Top 5 Categories Spend Top 5 Suppliers We utilze OBIEE for our Supply chain procurement and spend analytics, extracting data from operation database using data guard. This fulfills our business users to analyze spend metrics and contract leakages and so on. Line Of Business Contract Leakage Off Contract Spend Payables Leakage

21 AT&T SCM: Database Maintenance/Performance Management
Monthly month-end close of books stats validation – database statistics preparation on production clone Table Re-organization – routine and ad-hoc Periodic index rebuild Daily performance call Daily/weekly/monthly/yearly purge/archive schedules – vigilance in adhering to them Extensive performance monitoring on all levels of the system In summary, this architecture provides operational performance we need with our performance management tasks like Decoupled DB statistics management for month book close jobs. Periodic and ad-hoc table reorganization and index rebuilds We have daily performance calls to analyze and discuss performance issues Extensive performance monitoring.

22 Partnering with Oracle Advanced Customer Service
Oracle Solution Service Center (SSC)

23 SSC: Project Spotlight: Patch Recommendations
Project Activities Description Identification of available latest required patches for AT&T environment An initiative to incorporate E-Business Suite recommended patches, such as, Rollup (RUP) in AT&T’s release schedules for 2010 Participation in meetings to review recommendations Value Addition SSC and AT&T worked together to create a new process to efficiently identify required patches with the goal to minimize the impact of rolling them out in our production environment and to current/future projects: Quarterly document for AT&T environments with the latest recommended patches. Participation on AT&T IT meetings to clarify recommendations and questions Provide detailed patch analysis on a ad-hoc basis Successful roll out of AP.O patchset along with other Financials patches in February 2010: stabilized production environment and brought environment to the latest code base. Successful completion of the ASCP Rup 37, Collections Rollup (RUP) 32 and Application Technology Group (ATG) RUP-7 for the April 2010 production rollout. 23

24 SSC: Project Spotlight: Performance Improvement
Project Activities Description Understand AT&T batch window requirements and transaction volumes An initiative to address various performance issues related to Month-End-Close processes and other batch processes after Oracle 10g database upgrade. Identification and root cause analysis of problem areas Implementation of resolutions Value Addition Active participation on Conference bridge calls and OWC sessions. Engage Oracle development groups (Financials and Apps Performance) to provide SQL tuning improvements. Improve month end close activities like AR close, depreciation and asset retirement, optimizer, Invoice Validation processes. Maximize performance during month end, preventing delay; AT&T was able to successfully meet the SLAs for the Month End Close Processes month after month. Provide recommendations for major upgrades (DB, Applications and Tools)

25 SSC: Critical SR Spotlight: Other Key areas of focus
Project Activities Description Workflow background process WF_DEFERRED queue growing exponentially; AT&T engages SSC to evaluate and address the issue. upgrade. RAC – CRS Configuration Collect diagnostic and configuration data Troubleshoot frequent RAC node reboot/performance issue post upgrade Discover unforeseen problems Validate AT&T configuration with RAC Best Practices Device Life Cycle Database Upgrade Provided upgrade expertise and support (pre and post deployment for 11g R2 upgrade. Value Addition Active participation on conference bridge calls and OWC sessions Review current setups and schedules of workflow Background Process Engage on Severity-1 SR and work collaboratively with Oracle development to provide workaround (eg. skip script) Work with at&t and Oracle development organization to provide scripts to clear workflow DEFERRED queue Monitor the implementation of recommended suggestions Engage Oracle Development and RAC COE team Engage on Architectural design discussions Stand-by support for DB/RAC upgrade deployment activities Ensure successful implementation of RAC upgrade Rollout

26 SSC: AT&T Targeted Services: Operations
The Services Yield to the Customer targeted for the next 6-month planning period is comprised of key Support activities, aligned with key customer objectives/activities towards achievement of AT&T’s Account Goals Supply Chain Transformation (Streamline Operations for newly deployed modules) Work with ACS sales to determine appropriate support for upcoming SC rollout. Determine/secure funding for new modules (Demantra and Transportation Management Suite) Develop, Enhance and Secure development sponsors Participate in scheduled project “Deep Dive” calls with PMO and project team Site visit to Memphis to understand warehouse and logistic operations – understand business functions. Supply Chain Business – develop key relationships with Retail and Network business users 11i Financials and Supply Chain Production Supportability (System) Next generation iPhone launch and 2010 Holiday Retail Season readiness “Top Ten” Prioritization Future – implement Tactical vs. Strategic

27 SSC: AT&T Targeted Services: Transportation Manager
OneATT- New Horizon to transform Network and Asset Management: Logistics TMS Key Secondary Activities Participate in scheduled project “Deep Dive” calls with PMO and project team Work with PMO/SSC to determine appropriate support model for New Horizon’s Work closely with OCS /PMO on project milestones – develop relationships Continue to maintain Dev OTM Network Supply Chain Business – develop key relationships Site visit to Warehouses SSC walk-thru’s Key Deliverables Enhance Supply Chain SSC presence and understand usage and architecture for OTM Site visit to understand implementation, document in configuration guide. Performance testing as requested. Amend SSC contract to include OTM products Future – implement Tactical vs. Strategic OneATT initiative to consolidate Supply chain system - New Horizon to transform Network and Asset Management: Project - Logistics TMS

28 SSC: Oracle E-Business Suite Release 12/Database 11g R2 Upgrade
Upgrade existing Oracle 11i.10 SCM and Planning environments to E-Business Suite R12. This will enhances supportability for our complex environment and provide additional needed functionalities Three stage upgrade (2 – 3 year journey) Supply Chain planning to release 12 RDBMS for 10g to 11G Supply chain platform to release 12 Objective: System Availability, Supportability, and Software Currency Key Secondary Activities SSC “Top Ten” Prioritization: Service Request review meetings with Supply Chain, Financials, and DBA support teams Patching/Release Management strategy for Release 12 interoperability; Leverage Tools and Templates Daily SSC Performance Calls Key Deliverables Proactive Diagnostics (Configuration/Performance) Contracted deliverables Semi-Annual Account Review Future – implement Tactical vs. Strategic

29 Solution Support Center
Service Highlights Focused Engineering Team of Oracle Subject Matter Experts Fast track solutions based on intimate Knowledge of customer Environment and business needs Proactive Software Advice and Knowledge Transfers Performance and Patch Assessments Single point of contact for all technical issues - Technical Lead Single point of contact for all escalations – SDM 24/7 coverage Value Mitigate Upgrade Risks substantially All project milestones are met Ensured optimal usage of Oracle features and functionality Reduced overall Maintenance/upgrade/ training costs

30 Solution Support Center
Key Delivered Features Focused Engineering Team Single Point Accountability – SDM Exclusive number 24x7x365 Review of Critical Issues as Needed Patch Analysis Stand-by Support for Critical Go-lives Proactive Alerts / Patch Notifications Project Planning / Migration Assistance Periodic Review Sessions Applied Research

31 Questions

32 Ravi Kayarthodi (
For More Information On this presentation, Bryon Rickey or Ravi Kayarthodi For general information on Advanced Customer Services, go to or

Download ppt "Best Practices at AT&T Oracle Real Application Clusters (Oracle RAC) & Oracle E-Business Suite Bryon Rickey Ravi Kayarthodi September 20th, 2010."

Similar presentations

Ads by Google