Jeff Fried Senior Product Manager Microsoft OFS210

Scale – content and query scale becomes a non-issue User experience – guide users with rich, visual navigation Social search – search unlocks the value of the people and expertise Relevance –improving matching and ranking and language support Platform –tools to customize, extend, or build new applications Management & reliability – reliable crawling, scriptable admin, improved monitoring Rich Content Processing– extract metadata from my enterprise content Big Improvements in Search In areas with common customer requests

Microsoft Enterprise Search- Productivity Products for Every Customer Need Complete intranet search High end search delivered through SharePoint Common UI Framework Social search features and integration SharePoint platform integration End user and site administrator enablement Common across the product line Common Connector Framework (BDC) APIs and developer Experience Admin & deployment capabilities Operations advantages (SCOM, scripting)

Query Object Model Content OpenSearch Source Crawling - Traverse URL space to record items in search catalog Indexing - Extract information from items to enable efficient matching Query Servers - Accept query requests from users and return results Search Center - UI for users to issue queries and interact with results Query Federation - Return results from non-SharePoint Indexes Crawler Indexer Query Servers Index Partition Connectors - Know how to process different content sources Index Partition - Subset of the overall index

Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability

End-User UI Out-of-box refinement Refine over key results properties Easy to extend over custom properties One-stop Search Center Scopes, web parts, best bets, top answers, advanced search Query federation brings together results from all over - native support for OpenSearch Core search experience Metadata, taxonomy and social tags based results refinement Improved did you mean suggestions New pre-query and post related query suggestions “View in browser” link (for most office docs) Improved query syntax

End-User UI (continued) Improved relevance ranking New ingredients: URL fuzzy matching, social tags, results click through, implicit phrase matching, extracted metadata, etc. Improved low-noise snippets in summaries Enhanced multi-lingual support Automatic detection of language of many document types and part of documents Compound word handling - e.g., Innovationszyklen” and ”innovation“, “zyklen” Improved ranking of documents in multilingual collections New form factors Mobile search from Smartphone browsers Desktop search integration in Windows 7

New Query Syntax Support for Boolean operators for FreeText queries and property queries (“SharePoint Search” OR “Bing Search”) AND (title:“keyword syntax” OR title:”query syntax”) Prefix matching support for keywords and properties Micro* author:bill* Improved operator support for property restrictions =, >, = Can create range refinements

End-user UX Jeff Fried Enterprise Search Group

Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability

Search is Social People finding experience Better expertise & interest search mining to bootstrap profiles with interests and colleagues “Address book style” search Phonetic name matching Nickname matching Relevance tuned for people search Metadata refinement, better hit highlighting, recently authored content

Search is Social (continued) Social behavior drives search quality Search click through behavior drives relevance ranking Query suggestions mined from search logs Social tagging influences relevance ranking Self search - to drive people to participate content Social definitions extracted from indexed content

People Search Jeff Fried Enterprise Search Group

Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability

Scale-out Architecture 2010 core engine tenets Sub-second query latencies at large scale Fresher indexes Better resiliency/higher availability Basic philosophy Componentize the system Remove system bottlenecks through scale-out

MOSS 2007 Search Scale-out “Single point of failure” “Bottleneck” “The whole index” “Bottleneck” + Query

SharePoint Server 2010 Search Scale-out “Single point of failure” “Bottleneck” “The whole index” “Bottleneck” Multiple Indexers Crawl Distribution Query Mirroring Query Partitioning Stateless Crawlers Multiple Property DBs Admin Database + Admin Component

Content Distribution Crawl Distribution In-built load balancer distributes hosts to crawl databases Crawlers crawl content that is covered by crawl database Default configuration can be overwritten using host distribution rules E.g. purchasing a new connector Query Distribution Low query latency if all index partitions equal in size Distribution by hash of documentId Crawlers partition indexed data and propagate to query servers

Industrial Strength Resiliency 2007 style mirroring for index partitions Redundant components provide failover Ability to add multiple crawl components to minimize crawl downtime Machine down doesn’t result in crawl downtime Native support for SQL mirroring

Scale-out from an art to a recipe To improve this…Take these actions Index freshness/crawl times If Indexer machine is saturated, add more Indexer machines and/or crawl components If Crawl Database is I/O bound on SQL Server, add additional crawl database on same SQL Server If SQL server is Memory/CPU bound, add additional SQL Server with additional crawl databases Query Latency/Throughput If time spent in resolving query in full-text index is high, partition the index into smaller index partitions. Each index partition can contain ~10M items. If query throughput is low, add query components with mirror index partitions. If Property Database is I/O bound on SQL Server, add additional crawl database on same SQL Server If SQL server is Memory/CPU bound, add additional SQL Server with additional crawl databases Query Availability Deploy redundant query servers, redundant index partitions and components, and use clustered or mirrored database servers to host property databases. Crawl/Indexing Availability Use multiple crawler components on redundant index servers, and add crawl databases. Crawl components associated with a given crawl database can be distributed across index servers for availability and load distribution.

Other Engine Enhancements Support for regular expressions in Crawl Rules Native support for crawling case sensitive repositories Ability to prioritize Content Sources so as to distribute crawler resources New ‘Crawl Policy’ to define how crawler treats error conditions Low indexing downtime Search Backups

Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability

Do More With Search Across a Spectrum of Customization and Development ConfigureExtendCreate

SharePoint Search Platform Architecture

Search Extensibility: End-User UI Change web part properties – no code Modify XSLT Modify config XML Refinement panel – control metadata available for refinement Advanced search – control metadata available for advanced search queries Extend OOB web parts programmatically All OOB web parts are public Extend web parts to change default behavior E.g. Extend core results web part to show result previews Extend Search Center with vertical experiences All web parts communicate through public interface (SharedQueryManager object)

Search Extensibility - Federation Query Federation Bring external search results into the SharePoint experience Based on Open Search standard Exposed through UI + public federation object model

Connector Framework New features Support for attachments Item level security Crawl through entity associations Inline caching + batching for better citizenship Richer crawl options Regular full crawl (like 2007) Time stamp based incremental crawl Change log crawl + deleted count Change log + delete log crawl * Protocol handler API still supported

Connector Architecture Search Engine/Index Search BDC Runtime WCF Databases.NET Shim Execution call routing through BDC Shim using model file Execution call routing through BDC Shim using model file Custom shim Search user queries BDC Profile Pages or Virtual Lists Search results page – results based on metadata Search page points to Profile Page or custom page to show search result Fetch data from repository Protocol Handlers

How to write connectors Use OOB shims (Database/WCF/.NET) Create/Deploy model file using SPD and use search UI to configure crawls Create/Deploy.NET classes using Visual Studio and use search UI to configure crawls Recommended if backend structure is static Write Custom Shim + Model file Useful for dynamic back-end structures. E.g. exchange public folders

Topics End-user Enhancements Search is Social Scale-out Architecture Extensibility Enhancements Deployment and Manageability

Deployment simplicity

Search Installation and Topology extension Jeff Fried Enterprise Search Group

Richer Manageability Consolidated administration UI dashboard Automated service password management through “managed accounts” PowerShell support for scripted administration Built-in system health monitoring, support for SCOM monitoring and alerting Built-in and extensible search analytics reporting

Search Administration Jeff Fried Enterprise Search Group

SharePoint Server 2010 Search: Best Infrastructure Search Major improvements over MOSS 2007 AreaNew features and benefits RelevanceProvide great search results New ingredients, more experience -> better OOB search Better wordbreakers in more languages Social Distance, tagging, click-through influence relevance RefinementFind information faster Metadata-based refiners provide drill-down Related query suggestions show popular queries SocialConnect with people and expertise Address-book style name lookup with phonetic matching Improved expertise matching - mined inbox data Social behavior improves search; search improves mySites PlatformIndustrial Strength Search at High Scale Index partitioning -> sub-second latency at 100M items Full failover support Connector and BDC enhancements and tooling support ExtensibilityCreate powerful applications based on search Web parts, query OMs are public -> easy to modify Enhanced query syntax TCODeploy and Operate with less effort Virtualization and native 64-bit simplify deployment Powershell, Admin web parts, SCOM provide easier admin

Microsoft SharePoint 2010

Sessions On-Demand & Community Resources for IT Professionals Resources for Developers Microsoft Certification & Training Resources Resources Required Slide Speakers, TechEd 2009 is not producing a DVD. Please announce that attendees can access session recordings at TechEd Online. Required Slide Speakers, TechEd 2009 is not producing a DVD. Please announce that attendees can access session recordings at TechEd Online.

Related Content CLI317 Windows 7 and its role in Enterprise Search 11/10/ :45-17:00 London 3 - Hall 7-1b CLI07-IS Search, Organize, and Browse Discussion 11/11/ :30-18:45 Interactive Theatre 3 – Blue Required Slide Speakers, please list the Breakout Sessions, TLC Interactive Theaters and Labs that are related to your session. Required Slide Speakers, please list the Breakout Sessions, TLC Interactive Theaters and Labs that are related to your session. OFS05-IS Microsoft FAST Search Server 2010 for SharePoint 11/13/ :00-10:15 Interactive Theatre 1 - Red

Complete an evaluation on CommNet and enter to win an Xbox 360 Elite!

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION. Required Slide