Design, performance and capacity - Factors of successful intranet and internet SharePoint sites Sanjay Narang, Anandan Kumaran, Singaram Murigappan

Agenda Real Life Implementation – Real Life Case Study – Capacity Planning Aspects Of Performance Capacity Management Model –Model –Design

SharePoint 2010 Implementation

Publish Content Marketing team needs a robust mechanism to publish content effectively without needing specialized technical skill

Content Deployment Content authored is saved in the authoring server database Content push by marketing admin Content authored is moved to the preproduction environment. It allows the user to verify the content before pushing live Content push by marketing admin Content is moved from preproduction environment to live environment after verification Authoring Pre-Production

Intuitive User Interface

Content Authoring nt Content Author needs a mechanism to add/edit content, upload documents without worrying about the layout or having pre-requisite design skills

Author Content

Edit Page

Edit Page Contd.,

Our Approach – Edit View

Public View

Friendly URLs Need for friendly URLs that are easier for users to remember and for search engines to find

URL Rewrite Powerful rules to transform complex URLs into simple and consistent web addresses Rule-based URL rewriting engine Regular expression pattern matching Downloadable Download:

Configuration - Inbound Inbound rules convert the user friendly URLs provided by the users to actual URLs

Configuration - Outbound Outbound rules convert the actual URLs to the user friendly URLs

Friendly URL - Output

URL Rewrite Support Not officially supported by the Microsoft Product Team

Faster Site Need for the site to equally responsive and quick to all users across the globe. The load of users on the application will vary especially during financial results but I need a cost effective solution

SharePoint Out-Of-Box Output Caching Object Cache This can load but not latency

Content Delivery Network Deliver content faster Scalability Across the globe users has similar experience on accessing the site

Capacity Planning and Management

Case Study – Contoso Intranet Portal 150,000 users spread across globe Existing site –The gateway to all other applications –primarily HTML/ASP/JavaScript New site –Intranet Publishing on SharePoint Server 2010 –Widget based homepage, –Much more content on home page including Flash videos

Capacity Management Capacity Planning –One time sizing –Selection & Configuration Data architecture Topologies (physical and logical) Hardware Capacity Management – Extends Planning – Cyclic Process – Monitored & Optimized – Changing Conditions and Requirements

Analogy

Capacity Management Model

Aspects of Performance Latency Throughput Data Scale Reliability

capacity management Model Design Pilot, Test, Optimize Deploy Monitor & Maintain

Definitions and Conventions Users –Concurrent Users –Active Users RPS

Definitions and Conventions User Types

Model Determine Workload Determine DataSet Set Performance and Reliability Targets Model

RPS Estimation (New Env) NumberRPH/UserRPH Total Users150,000 Active Users 50,000 Light (10%) 5, ,000 Typical (70%) 35, ,000 Heavy (15%)7, ,000 Extreme (5%)2, ,000 Total RPH 2,111,000 RPS (RPH/3600)586 Concurrency50% Effective RPS293 Model

IIS Logs –Take from all servers into a central location –Take it for one of the peak day User Tools –LogParser 2.2 RPS Calculations (Existing Env) Model

Get RPS per second logparser -i:IISW3C -o:CSV " SELECT QUANTIZE(TO_TIMESTAMP(date, time), 3600) AS Hour, QUANTIZE(TO_TIMESTAMP(date, time), 1) AS Second, COUNT(*) AS RPS INTO RPS_No401s.csv FROM *.log WHERE sc-status <> 401 GROUP BY Hour, Second " Model

HourSecondRPS 6/29/ :006/29/ :3010 6/29/ :006/29/ :305 6/29/ :006/29/ :3011 6/29/ :006/29/ : /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/2010 4:006/30/2010 4: /30/ :006/30/ :297 6/30/ :006/30/ :295 RPS Every Second for 24 Hours Model

Get Average maximum RPS [Hidden] logparser -i:CSV -o:CSV "SELECT Hour, AVG(RPS) AS AvgRPS, MAX(RPS) AS MaxRPS INTO RPS_AvgMax_No401s.csv FROM RPS_No401s.csv GROUP BY Hour" Model

HourAvgRPSMaxRPS 6/29/ : /29/ : /29/ : /29/ : /29/ : /29/ : /30/2010 0: /30/2010 1: /30/2010 2: /30/2010 3: /30/2010 4: /30/2010 5: /30/2010 6: /30/2010 7: /30/2010 8: /30/2010 9: /30/ : /30/ : /30/ : /30/ : /30/ : /30/ : /30/ : /30/ : /30/ : Avg and Max RPS Every Hour for 24 Hours Model

Avg RPS vs Max RPS[Hidden] Model

Avoid Image, CSS, JS Get File Extension to_lowercase EXTRACT_EXTENSION(cs-uri-stem)) exclude Where clause fext <> jpg AND fext <> jpeg AND fext <> gif AND fext <> png AND fext <> js AND fext <> css Model

Decision Point – Peak or Average Average RPS on a day Average RPS in a peak hour Maximum RPS in a peak hour

Workload Distribution Model

Dataset [Hidden] Different Calculations for new environment vs Existing Content Size –Number of Documents and Average Size –Versions –Site collection size Service Application –Search, Logging, Web Analytics, UPS, MMS Others – Recycle Bin, Audit Logs Model

File System Analysis[Hidden] The directory C:\intepub\wwwroot NumberSize (in MB)%age Total Files (Content) Pages39, % HTM Pages19, HTML Pages19, aspx pages Images % jpg GIF jpeg bmp Model

Versions[Hidden] Model

Version and Size Calculations[Hidden] Number of Items %age of size Number of versions Size (in MBs) Size with versions Number with versions Pages , Documents ,43951, Images ,62210, Others ,18290, TOTAL162, Size in GB159GB Model

Dataset - Content DB S.NoParameterValueUnitsSource of Value A Existing size of all documents (pages, images etc) without versions40GBActual size from web servers B Existing size of all documents (pages, images etc) with all versions160GBCalculated as in Table 1 C Total Number of documents with versions1,600,000Calculated as in Table 1 D Estimated number of list items per document2 Approximate assumption based on similar projects ETotal number of list items3,200,000C x D F Avg size of a list item or metadata for a document10KB Approximate assumption based on similar projects GTotal size of list items31GB(E x F)/(1024 x 1024) HTotal size of document metadata15GB(C x F)/(1024 x 1024) IContent DB size for existing content206GBB+G+H Model

Content DB Calculations S.NoParameterValueUnitsSource of Value I Content DB size for existing content206GBB+G+H J Growth estimated over next 2-3 years100%Approximate assumption KTotal size with growth412GBI + (I x J)/100 LRecycle Bin30% MTotal size with Recycle bin535GBK + (L x K)/100 Model

All Databases Calculations ParameterValue (in GBs) Content databases for Main Portal site collections535 Custom databases10 Configuration and CA databases5 Database for Usage and Health data collection SA200 Web Analytics Databases200 Managed Metadata SA10 User Profile SA300 TOTAL1,260 Model

Overall SQL Server Storage Storage ForValue (GB)%age of DB Databases1,260 Backup1,890150% System10 Transaction Log25220% temp db25220% Total for Databases3,664 Empty Space (25% of total)916 TOTAL for SQL Server4,580 Model

Performance and Reliability Targets Availability –Numbers of Nines (99%, 99.9%, 99.99%) –Consider Patch Management Code Releases Server Responsiveness –E.g. < 1 sec System Resource Utilization –E.g. Avg CPU < 50%, Available Memory Model

Model - Summary Total Number of Users – 150,000 Total Request in a day – 18,000,000 Throughput (RPS) – 1500, 700, 220 Storage - 5 TB Server Response Time < 1 sec Model

capacity management Design ModelDesign Pilot, Test, Optimize Deploy Monitor & Maintain

Design Identify case study Choose an Architecture Determine Hardware Design

Know SharePoint 2010 Vs SharePoint 2007 –Service Applications & Cross Farm Publishing Oversizing vs. Undersizing Reference Architectures –Single server, small, medium and large farms Design

Service Application Web server CPU Web server RAM Application server CPU Application server RAM SQL Server CPU SQL Server IOPS SQL Server storage SharePoint Foundation ServiceXXX XXXXX Central Admin serviceXX XXX Logging Service *XX XXX SharePoint Search ServiceXXX Word Viewing Service Application *XXXXXXX PowerPoint Service *XX XXXXX Excel Calculation ServiceXXX XXX Visio Service *XXXXX XXX Access Service *XXXXXXXXXX User Profile ServiceXXX XXX XX Managed Metadata Service *XXX XX Web Analytics Service *XXXXX Business Connection Service *XX XXX InfoPath Forms ServiceXX XXX Word Conversion ServiceXXXXXXXXXX PerformancePoint Service Application *XX XXX XXX Project Service *XXXXXXX XX Sandboxed Solutions *XXXXX Workflow capabilities *XXX Timer ServiceXX PowerPivot *XXXXX XX XXX Design

Constraints and Boundaries Site Collection < 100 GB Content DB < 200 GB 1 SQL Instance < 5 TB Content DBs per Web App < 300 Design

Reference Architectures Design

Reference Architectures

Determine Workload[Hidden] Publishing –Read Oriented Collaboration –Read – Write Search Personalized/Social Records Management

Identify Case Study Design

Case Study - Published Intranet Workload CharacteristicsValue Average Requests per Second (RPS)100 Average RPS at peak time226 Total number of unique users per day33,580 Average concurrent users172 Maximum concurrent users376 Total # of requests per day3,800,000 Design

Case Study - Published Intranet

Lab - RPS & Output Cache[Hidden] Design

Lab - RPS & Output Cache [Hidden] Design Output Cache Hit Ratio Measure1x12x14x1 100% Maximum RPS3,4637,33111,032 SQL Server CPU utilization0% 95% Maximum RPS2,1373,9455,937 SQL Server CPU utilization5.93%12.00%21.80% 90% Maximum RPS1,5182,8644,518 SQL Server CPU utilization7.12%14.40%28.00% 50% Maximum RPS ,596 SQL Server CPU utilization9.86%19.50%42.00% 0% Maximum RPS SQL Server CPU utilization9.53%19.00%36.30%

Lab - RPS with Authentication Design

Be Aware of Thumb Rules Number of Servers –1 WFE per 10 K users –1 SQL server instance per 4 WFEs 3-5 WFE cores per SQL core –1 DC per 3 WFEs RAM –2-4 GB memory CPU core Design

Choosing an Architecture High Availability –Hardware vs Software Load Balancer for web servers –Clustering or Mirroring for SQL Server Service Applications –Separate farm or same farm Search Target –Dedicated or Distributed Design

Determining Hardware Server TypeMinimum (as per TechNet) My Recommendation CPUWeb / App64-bit, 4 cores64 bit, 8 cores DB64-bit, 4 – 8 cores64-bit, 16 cores MemoryWeb/App8 GB16 GB DB8-32 GB16-32 GB DiskWeb/App80 GB GB DBAs per requirement IOPSWeb/AppMinimal DBAs per requirement Design

SQL Server RAM Recommendations Design Combined size of content databasesRecommended RAM Minimum for small production deployments 8 GB Minimum for medium production deployments 16 GB Recommendation for up to 2 terabytes32 GB Recommendation for the range of 2 terabytes to a maximum of 5 terabytes 64 GB

Storage Architecture[Hidden] DAS SAN NAS –Only supported for Content DBs with RBS –Ping within 1 ms –First Byte within 20 ms Design

Disk Types[Hidden] Small Computer System Interface (SCSI) Serial Advanced Technology Attachment (SATA) Serial-attached SCSI (SAS) Fibre Channel (FC) Integrated Device Electronics (IDE) Solid State Drive (SSD) or Flash Disk Design

RAID Levels & IOPS RAID 10 Design TypeRAID levelSpindlesOptimization TempDB[RAID 1+0]2 IOPS/ GBWrite optimized Transaction Logs[RAID 1+0]2 IOPS/GBWrite optimized Content databases [RAID 1+0] (Can have RAID 5, if storage is a constraint) 0.75 IOPS / GBRead optimized

Design - Summary Workload – Published Intranet Case Study – WCM Lab Results Architecture –Availability Hardware Load Balancer Clustering for SQL Server –Service Applications in separate farm Number of Servers Hardware Configurations Design

Design - Summary Number of Servers –Lab results: 4 servers = 3200 RPS, 2 servers = 1700 RPS –Scaled UP SQL server for 5 servers Hardware Configurations –RAM oversized because of low cost –Used thumb rule for SQL Memory Design

Possible Variations Internet Website (anonymous) Requirement for local services: Office Web Apps, Excel Services etc. Data Scale more than 5 TB

© 2011 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.