1. 1© Copyright 2012 EMC Corporation. All rights reserved. EMC PERFORMANCE OPTIMIZATION FOR MICROSOFT FAST SEARCH SERVER 2010 FOR SHAREPOINT EMC Symmetrix VMAX, FAST VP, Microsoft Hyper-V

2. 2© Copyright 2012 EMC Corporation. All rights reserved. Agenda  Solution overview: Objectives  Why this solution  Solution architecture (physical and virtualized)  Results and findings (physical and virtualized)  Backup and restore results  Summary

3. 3© Copyright 2012 EMC Corporation. All rights reserved. Solution overview: Objectives  Implement Microsoft FAST Search Server 2010 for SharePoint in a virtualized environment on a Symmetrix VMAX storage system –Validate and show a FAST Search environment on Physical servers for 4 TB SharePoint content and 5 TB file share document –Showcase how to configure and deploy FAST Search servers –Provide guidelines for virtualizing the FAST Search servers –Validate the benefit of FAST VP for the SharePoint farm environment as well as the FAST Search server environment –Validate the SharePoint and FAST Search full farm backup, content database restore and FAST Search full farm backup and restore

4. 4© Copyright 2012 EMC Corporation. All rights reserved. Why this solution  Microsoft FAST Search Server 2010 for SharePoint delivers an exceptional intranet and people search experience and a platform for building custom search-driven applications.  The deployment of the FAST Search Server 2010 for SharePoint needs careful planning and understanding of both SharePoint and FAST Search servers. This solution can provide some guidance to the implementation team.  Virtualization is often called for from many customers, but there is no guideline available. Questions such as “Is it possible?” “How could it be done?” “What kind of performance can I get?” remained unanswered.  This solution addresses the questions above with EMC VMAX as the storage platform and Microsoft Hyper-V as the virtualization platform.

5. 5© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture

6. 6© Copyright 2012 EMC Corporation. All rights reserved. FAST Search for SharePoint topology overview (I)  FAST Search Server 2010 for SharePoint provides enterprise search infrastructure for the SharePoint server farm.  The search solution is hosted on two farm topologies: –SharePoint Server farm hosts the query interfaces and content collection (crawling) interfaces for the search solution. –FAST Search Server 2010 for SharePoint farm provides back-end indexing and query matching capabilities and administration of these features.

7. 7© Copyright 2012 EMC Corporation. All rights reserved. FAST Search for SharePoint topology overview (II) FAST Search for SharePoint farm topology and work flow

8. 8© Copyright 2012 EMC Corporation. All rights reserved. Design considerations  Use minimum servers in a FAST Search farm  Create a redundant Web analyzer to ensure service availability whenever it is possible  Keep storage out of the administrator server for easy recovery  Install FAST on the non-operating system LUN to divert the logging activities and make LUN based replication of the FAST Search farm possible

9. 9© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture: Physical FAST Search farm design

10. 10© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture with physical FAST Search servers (I)

11. 11© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture with physical FAST Search servers (II) Physical FAST Search server farm configuration

12. 12© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture: Virtualized FAST Search farm design

13. 13© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture with virtualized FAST Search servers (I)

14. 14© Copyright 2012 EMC Corporation. All rights reserved. Solution architecture with virtualized FAST Search servers (II) Virtualized FAST Search server farm configuration

15. 15© Copyright 2012 EMC Corporation. All rights reserved. Results and findings

16. 16© Copyright 2012 EMC Corporation. All rights reserved. Testing profiles  User load profile –Heavy (60 requests/hour) –Required response time  Content profile Test typeDetailsRequired response time BrowseUser browseLess than 3 seconds SearchUnique value searchLess than 3 seconds ModifyBrowse and metadata modifyLess than 3 seconds ItemValue Content typedoc, docx, pdf, xls, xlsx, ppt, pptx, txt Average content size1.65 MB SharePoint - Total data4 TB SharePoint – Document size range20 KB – 50 MB SharePoint – Number of documents2,000,000 SharePoint – Total site count220 SharePoint content database size (each)Up to 200 GB SharePoint – Total site collections20 SharePoint – Sites for each site collection11 SharePoint – Total user count22,000 SharePoint -- User concurrency10% File share – Total data5 TB File share – Number of documents3,000,000

17. 17© Copyright 2012 EMC Corporation. All rights reserved. Key results (I) EnvironmentAvailable CPUs Document processors Full crawl rate ( items/minute) SharePoint content File share Physical604215131667 Virtualized2024770978 Server type Average read size (KB) Average write size (KB) Average read/write Total IOPS SQL server2624218:11434 FAST index2756053:1 485 FAST servers13261:66 Environment Profile ( browse/ search/ modify) Average response time (second)Queries per second FAST IOPS total BrowseSearchModify Physical80/10/10 0.940.602.437.1 550 Virtualized80/10/10 0.950.472.838.5

18. 18© Copyright 2012 EMC Corporation. All rights reserved. Key results (II)  The maximum total disk IOPS is around 3,500 on the back end for full crawls and around 1,900 for the query testing.  Crawl rate is 150 GB/hour in the physical environment, and 80 GB/hour in the virtualized environment with less hardware resources.  Search response time is less than 1 second in both physical and virtualized environments supporting 22,000 users with 10% concurrent access.

19. 19© Copyright 2012 EMC Corporation. All rights reserved. Key results (III)  With Replication Manager-engaged TimerFinder snapshot backup: –The whole SharePoint farm, with a total of 9 TB data, can be backed up within 15 minutes. –The FAST Search farm, with a total of 5 TB FAST Search volumes, can be backed up within 8 minutes.  With Replication Manager-engaged TimerFinder snapshot restore: – The SharePoint content database (4 TB) and file share data (5 TB) can be restored and recovered within 1 hour and 30 minutes. –The FAST Search farm, with a total of 5 TB FAST Search volumes, can be restored within 13 minutes. Replication Manager-engaged TimeFinder/Snap backup and restore

20. 20© Copyright 2012 EMC Corporation. All rights reserved. Key results (IV) FAST VP  With FAST VP enabled, the environment can better handle peak I/O request.  The search performance improved about 20% after enabling FAST VP on Symmetrix VMAX storage. QPS result of profile 80/10/10 (percentage of browse/search/modify) FAST VP used? (Yes /No)Queries per second (QPS) No7.1 Yes8.5 Before FAST VP was enabledAfter FAST VP was enabled 100% FAST volumes on FC virtual pool 1.3% on Flash virtual pool 94.6% on FC virtual pool 4.3% on SATA virtual pool

21. 21© Copyright 2012 EMC Corporation. All rights reserved. Test results: Physical FAST Search farm

22. 22© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for SharePoint content in the physical FAST Search farm (I) Full crawl performance of the SharePoint content IOPS and CPU usage of the SharePoint full crawl Duration (hh:mm:ss) Items crawled Crawl rate (items /min) Crawl rate (MB/min) Crawl type 24:15:002,190,9541,513 2,111Full ItemFAST Search serversSharePoint servers Average total IOPS4851,434 Maximum total IOPS1,8094,749 Average CPU usage41%26% Available CPUs in the farm60N/A Document processors42N/A FAST Search index volume I/O characteristics and read/write ratio Server role Average IOPS IOPS (maximum) Read I/O size Write I/O size Read:Write Primary3451,809275 KB605 KB3:1 Secondary4733445 KB664 KB1:50

23. 23© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for SharePoint content in the physical FAST Search farm (II) SharePoint content database full crawl rate The crawl rate of smaller documents was higher with the peak reaching 8,000 items per minute. The crawl rate of larger documents was lower, which was 1,516 items per minute or 2111 MB per minute on average.

24. 24© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for SharePoint content in the physical FAST Search farm (III) Back-end storage Symmetrix VMAX performance Disks and thin pools performed well The disk utilization of SharePoint and FAST Search volumes was less than 50 percent The total IOPS on the back end was 2,500 to 4,500 ItemAverageMaximum IOPS3,3404,730 Throughput (MB) 210275 Disk group utilization Percentage Content DB5%–35% FAST index volumes10%–47%

25. 25© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for file share in the physical FAST Search farm (I) Full crawl performance of the file share content IOPS and CPU usage of the file share full crawl Duration (hh:mm:ss) Items crawled Crawl rate (items/min) Crawl rate (MB/min) Crawl type 35:15:003,124,2231,668 2,602Full ItemFAST Search serversSharePoint servers Average total IOPS208339 Maximum total IOPS1,706930 Average CPU usage45%34% Available CPUs in the farm60N/A Document processors42N/A FAST Search index volume IO characteristics and read/write ratio Server role Average IOPS IOPS (maximum) Read I/O size Write I/O size Read:Write Primary2841,706313 KB617 KB5:2 Secondary4518618 KB756 KB1:25

26. 26© Copyright 2012 EMC Corporation. All rights reserved. The crawl rate was higher than the content database The crawl rate was largely related to the size of the files peak while processing the smaller files Test result: Full crawl for file share in the physical FAST Search farm (II) File share full crawl rate in the physical environment

27. 27© Copyright 2012 EMC Corporation. All rights reserved. Disks and thin pools performed well Disk pools utilization was low Total IOPS on the back end was around 1400-2800 Back-end storage Symmetrix VMAX performance Test result: Full crawl for file share in the physical FAST Search farm (III) ItemAverageMaximum IOPS2,2002,880 Throughput (MB) 234322 Disk group utilization Percentage Content DB5%–25% FAST index volumes 20%–55%

28. 28© Copyright 2012 EMC Corporation. All rights reserved. Test result: Query performance on the physical FAST Search farm (I) VSTS (browse/search/modify)testing Profile (percentage of browse/search/ modify) Requests per second Average response time (second) Queries per second BrowseSearchModify 80/10/1035.7 0.940.602.437.1 70/20/1036.5 0.930.632.149.5 50/30/2038.00.900.832.2616.7

29. 29© Copyright 2012 EMC Corporation. All rights reserved. Test result: Query performance on the physical FAST Search farm (II) Back-end storage performance with VSTS testing Symmetrix VMAX back end IOPS during 80/10/10 (percentages for browse/search/modify) profile Maximum IOPS Total SharePoint and FAST Search servers1,900 FAST Search servers only550

30. 30© Copyright 2012 EMC Corporation. All rights reserved. Test results: Virtualized FAST Search farm

31. 31© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for SharePoint content in the virtualized FAST Search farm (I) Full crawl performance of the SharePoint content IOPS and CPU usage of the SharePoint content full crawl Duration (hh:mm:ss) Items crawled Crawl rate (items/min) Crawl rate (MB/min) Crawl type 47:47:002,190,9547701,395Full ItemFAST Search serversSharePoint servers Average total IOPS675900 Maximum total IOPS6,5924,257 Average CPU usage74%45% Available CPUs in the farm20N/A Document processors24N/A FAST Search index volume I/O characteristics and read/write ratio Server role Average IOPS IOPS (maximum) Read I/O size Write I/O size Read:Write Primary 4836,592 184 KB 345 KB3:1 Secondary 10313,346 42 KB 582 KB1:5

32. 32© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for SharePoint content in the virtualized FAST Search farm (II) Full crawl rate of the SharePoint content The crawl rate of smaller documents was higher with the peak reaching 12,500 items per minute. The crawl rate of larger documents was lower, which was 770 items per minutes or 1395 MB per minutes on average.

33. 33© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for SharePoint content in the virtualized FAST Search farm(III) Disks and thin pools performed well Disk pools utilization low Total IOPS on the back end was around 2,800-4,300 Back-end storage Symmetrix VMAX performance ItemAverageMaximum IOPS3,5004,300 Throughput (MB) 217300 Disk group utilization Percentage Content database20%–30% FAST index volumes30%–50%

34. 34© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for file share in the virtualized FAST Search farm (I) Full crawl performance of the file share content IOPS and CPU usage of the file share full crawl Duration (hh:mm:ss) Items crawled Crawl rate (items/min) Crawl rate (MB/min) Crawl type 60:14:323,124,223978 1383Full ItemFAST Search serversSharePoint servers Average Total IOPS384193 Maximum Total IOPS7,8691,263 Average CPU usage83%15% Available CPUs in Farm20N/A Document Processors24N/A FAST Search index volume I/O characteristics and read/write ratio Server role Average IOPS IOPS (maximum) Read I/O size Write I/O size Read:Write Primary 4637,869 186 KB 373 KB3:1 Secondary 298,704 36 KB 560 KB1:2

35. 35© Copyright 2012 EMC Corporation. All rights reserved. The crawl rate was faster than the content database. The crawl rate was largely related to the size of the files peak while processing the smaller files; same pattern with the physical environment. Full crawl rate of the file share content Test result: Full crawl for file share in the virtualized FAST Search farm (II)

36. 36© Copyright 2012 EMC Corporation. All rights reserved. Test result: Full crawl for file share in the virtualized FAST Search farm(III) The total IOPS on the back end was between 2,500 and 4,300. The disks and thin pools performed well. Disk utilization of SharePoint and FAST Search volumes was less than 50 percent. Back-end storage Symmetrix VMAX performance ItemAverageMaximum IOPS3,5004,300 Throughput (MB) 217300 Disk group utilization Percentage Content database20%–30% FAST index volumes30%–50%

37. 37© Copyright 2012 EMC Corporation. All rights reserved. Test result: Query performance on the virtualized FAST Search farm (I) VSTS testing Profile (percentage of browse/search/ modify) Requests per second Average response time (second) Queries per second BrowseSearchModify 80/10/1038.2 0.950.472.838.5 70/20/1041.60.900.552.5014.4 50/30/2038.50.880.672.0812.9

38. 38© Copyright 2012 EMC Corporation. All rights reserved. Test result: Query performance on the virtualized FAST Search farm (II) Back-end storage performance with VSTS testing Symmetrix VMAX back end IOPS during 80/10/10 (browse/search/modify) profile Maximum IOPS Total SharePoint and FAST Search servers3,400 FAST Search servers only850

39. 39© Copyright 2012 EMC Corporation. All rights reserved. FAST VP test results FAST VP policy: – Flash: 20 percent – Fibre Channel: 30 percent – SATA: 50 percent Before FAST VP was enabled: 100 percent FAST Search volumes were on FC virtual pool. After FAST VP was enabled: About 1.3 percent of data was promoted to the Flash tier; about 4.3 percent of data was demoted to the SATA tier; and 94.6 percent of data resided on the FC tier. In the test above, the search performance improved about 20 percent with FAST VP enabled.

40. 40© Copyright 2012 EMC Corporation. All rights reserved. Test result: Replication Manager- engaged FAST farm backup and restore (I) RM-engaged backup/restore summary Backup: – A snapshot for the whole SharePoint farm took only 11 minutes to complete. – A snapshot for file share completed in less than 2 minutes. – A snapshot for the FAST Search farm took about 8 minutes across three servers. Restore: –Restoring a single SharePoint database took only a few minutes to process. Restoring all 20 SharePoint database across two SQL Server database servers took about 1.5 hours, most of which was spent recovering the databases (log replay). –Restoring file share took less than 2 minutes to finish. –Restoring the FAST Search farm took less than 13 minutes to finish.

41. 41© Copyright 2012 EMC Corporation. All rights reserved. Test result: Replication Manager- engaged FAST farm backup and restore (II) Replication Manager-engaged TimeFinder/Snap backup Replication Manager-engaged TimeFinder/Snap restore *Including database recovery time. ItemTotal size Backup duration (hh:mm:ss) SharePoint farm backup 4 TB 0:11:33 File share backup5 TB 0:01:53 FAST Search farm backup 5 TB 0:07:50 ItemTotal size Restore duration (hh:mm:ss) SharePoint content database restore* 4 TB 1:24:33 File share restore5 TB 0:01:53 FAST Search farm restore5 TB 0:12:49

42. 42© Copyright 2012 EMC Corporation. All rights reserved. Configuration tips The following settings are recommended by Microsoft for better crawl performance. SharePoint crawler registry changes: –HKLM\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager FilterProcessMemoryQuota –Default 100 MB –Changed to 200 MB DedicatedFilterProcessMemoryQuota –Default 100 MB –Changed to 200 MB FolderHighPriority –Default 50 –Changed to 500 Turn off TCP/IP Chimney: http://support.microsoft.com/kb/2570111http://support.microsoft.com/kb/2570111 Apply hotfix KB976462 Document processors could be adjusted in the run time

43. 43© Copyright 2012 EMC Corporation. All rights reserved. Summary (I) SharePoint farm design –The SharePoint farm used three Hyper-V servers supporting 12 virtual machines, which formed the virtualized SharePoint farm. –The SharePoint farm contained 9 TB content source (4 TB content database and 5TB file share). –The SharePoint farm supported 22,000 users to operate with mixed profile (browse, search, and modify) with 10 percent concurrent access. The average search response time was less than 1 second. FAST Search farm design –The physical FAST Search farm used five physical servers. –The virtualized FAST Search farm used two physical hosts with Hyper-V enabled. FAST Search performance in a virtual environment –A full crawl against a 5 TB file share can be completed in 60 hours and 15 minutes in the virtualized FAST Search servers (35 hours and 15 minutes in the physical FAST Search servers). –A full crawl against 4 TB of SharePoint content can be completed in 47 hours and 15 minutes in the virtualized FAST Search servers (24 hours and 15 minutes in the physical FAST Search servers).

44. 44© Copyright 2012 EMC Corporation. All rights reserved. Summary (II) Consolidation ratio in a virtualized environment –The migration of FAST Search servers to a virtualized environment reduced the number of physical host servers from five to two (60 percent reduction). This also leveraged all the management advantages of a virtualized environment while achieving comparable query performance of a physical environment and acceptable crawl performance. I/O characteristics of FAST Search components –The read and write I/O size were around 300 KB and 600 KB on the primary index volume of the FAST Search server. –The read and write I/O size were around 40 KB and 700 KB on the secondary index volume of the FAST Search server. Enabling FAST VP improved the query performance of FAST Search by 20 percent

45. 45© Copyright 2012 EMC Corporation. All rights reserved. Summary (III) Crawl rate –A full crawl against a 5 TB file share completed in 35 hours and 15 minutes in the physical FAST Search servers, or 60 hours and 15 minutes in the virtualized FAST Search servers. –A full crawl against 4 TB of SharePoint content completed in 24 hours and 15 minutes in the physical FAST Search servers or in 47 hours and 15 minutes in the virtualized FAST Search servers. Search Rate –Mixed workload with 30 percent query achieved 16.7 queries per second (QPS) in the physical FAST Search farm. –14.4 QPS were processed in the virtualized FAST Search farm while supporting 22,000 users with 10 percent concurrent access. –Pure search test achieved 205 QPS while keeping search the mean query latency response time less than 0.072 seconds. Backup and Restore – Replication Manager ensures that the environment is protected. The SharePoint content and FAST Search farm were replicated and recovered with minimal impact on the users.

46. 46© Copyright 2012 EMC Corporation. All rights reserved. Demonstration

47. 47© Copyright 2012 EMC Corporation. All rights reserved. Q&AQ&A