1. Making Serious Sites SharePoint Search Mirjam van Olst Information Worker Solutions Macaw

2. Agenda Enterprise Search Challenges Microsoft Search Vision SharePoint End-user improvements Knowledge Network Administration Improvements Performance and Capacity Planning Customizing the Search UI Develop Search Solutions

3. Making Serious Sites Enterprise Search Challenges

4. Information is stored in many places and in many formats People often know more than systems Information access is controlled Sales Executive Responding to an RFP Business Analyst Preparing a Briefing Finance Manager Reviewing a Budget

5. Making Serious Sites Microsoft Search Vision

6. Enterprise Search from Microsoft Empower people to find information and expertise Secure & manage your search deployment Find people & expertise Unlock unstructured and structured information Deploy search as an infrastructure service Search as an integral part of your business productivity infrastructure

7. Use Share Find Find information across the desktop, intranet and Internet Use information via intuitive, familiar interfaces Share information among teams and connect people in real-time Microsoft Search Solutions Enable professionals to find, use, and share relevant business data quickly, easily and securely

8. Search As an Infrastructure Service Integrate search into important business applications –Search your collaboration platform for information in blogs, wikis, & workspaces –Unlock the value of content stored in your ECM solutions –Deliver comprehensive search for your web sites and portal sites Find files, programs, and communications on the desktop –Search for email, appointments, contacts, tasks and notes in Outlook –View, organize and find your files using Windows Vista saved searches –Deploy a single desktop indexer across your clients Search with rich, actionable interfaces –Find, use and share information on the desktop and in business applications –Integrate search anywhere using web parts, the OM, or the query web service –Design customized search experiences using tabs and query/results interfaces

9. Search As an Infrastructure Service Scenario spectrum –Within-site search– WSS or MOSS –Search-driven business applications - MOSS –Intranet Search – MOSS –Internet point of presence search - MOSS

10. Enterprise Search Architecture

11. SharePoint Server 2007 SKUs for Internet Sites Standard Edition Enterprise Edition

12. Microsoft® Office SharePoint® Server 2007 for Search (Standard Edition) Microsoft® Office SharePoint® Server 2007 for Search (Enterprise Edition) Microsoft® Office SharePoint® Server 2007 (Standard CAL) Microsoft® Office SharePoint® Server 2007 (Enterprise CAL) Search File shares, Web Sites, SharePoint Sites, Exchange PFs, Notes DBs OOB Search 3 rd Party Document Repositories Indexes 200 File Types OOB (extensible) Secure Content Access Control Search for People and Expertise Enhanced Search Center UI Search Structured Data Sources Document Limit 500,000No Limit Search Feature Comparison

13. WSSMOSS Can Index Local SharePoint content SharePoint Web, Exchange, File shares, Notes, LOB… Rich, relevant results XX Alerts, RSS, DYM, Dup collapsing XX Scopes, Managed Properties X Best Bets, Result Removal, Query Reports X Tabs X People Search, KN X BDC Search X APIs Provided QueryQuery + Admin

14. Secure & Manage Your Search Deployment Control access to information with permissions –Perform standard query time security trimming –Implement custom security trimming Manage your search solutions –Analyze usage patterns using powerful out-of-the-box reporting –Tune relevance parameters for better results –Control deployment of desktop search with group policies Scale your deployment –Cluster web front ends, query servers, and index databases –Improve indexing performance with a dedicated index server –Index up to 50 million items per index

15. Unlock Unstructured & Structured Data Find information in unstructured repositories –Find content in web sites, file shares, SharePoint sites, public folders & Notes DBs –Use Protocol Handlers to index content in proprietary repositories –Support any file type with iFilters (over 200 supported formats out-of-the-box) Use search as an interface for line-of-business systems –Index structured data sources using the Business Data Catalog (BDC) –Customize search results using out-of-the-box Business Data Web Parts –Make results actionable with Business Data Actions Refine results and save important queries –Refine results using the keyword query syntax –Filter and sort result lists using relevant attributes –Save important queries and subscribe to RSS feeds or email alerts

16. IFilters iFilters available for lots of different file types http://www.citeknet.com/http://www.citeknet.com/ for: CAB CHM EXE HLP RAR ZIP https://support.cadcompany.nl/ifilter/default.aspxhttps://support.cadcompany.nl/ifilter/default.aspx for: DWG (Looking for Beta users) http://www.adobe.com/support/downloads/detail.jsp?ftpID=2611 PDF

17. Making Serious Sites SharePoint End-user Improvements

18. Relevance Improvements Best results on first page, every time Enterprise Search is different from Internet Search –Far less cross-linking of content –Lots of non-Web content: Office docs, line-of-business data, etc. –Security is paramount Revamped relevance & ranking engine –Extensive collaboration with Microsoft Research, MSN, Windows –Algorithms tuned for web and enterprise content –New search ingredients

19. Relevance Improvements New ingredients added, including –Click Distance –Anchor Text –URL Depth –Missing Metadata Creation –Automatic Language Detection Result is noticeably more relevant search –100% better on all queries –500% better on common queries

20. Influencing Ranking Best Bets Additional Metadata Authoritative Sites

21. Enhanced User Experience Simple yet powerful user interface –Query correction – “Did you mean?” –Hit highlighting –Duplicate Collapsing –Alerts –RSS Feeds –Security-trimmed search results –Actionable results through integration with Microsoft Office applications, Windows Desktop Search, Live Communication Server

22. Enhanced Search Center Search Center –Web part-based query and results pages –Many options for customization Search tabs OOB Search Web Parts –Search Box –Search Core Results –Search Best Bets –Search High Confidence Results –Search Statistics –Search Paging –Search Action Links

23. Anatomy Of Results Page UI SEARCH BOX CORE SEARCH RESULTS PAGINATIONSTATISTICSACTION LINKS MATCHING KEYWORDS AND BEST BETS HIGH CONFIDENCE MATCH HIDDEN SHARED OBJECT

24. Query & Results –Scope –Duplicate removal –Sort order –Stemming –Noise words –# of results –Summary length –SELECT columns Search box –Scopes shown –Target pages –Appearance: Images, width Static query Best Bets / High Confidence –Pick properties to display Search UI Web Part Properties

25. Use of Search functionality in MOSS Content Query Web Part Login Name and Display Name My Team Sites Web Part My Links Colleague Tracker Web Part Alerts

26. Search Center The Search Center, the home for Enterprise Search

27. Business Data Search Search data, not just documents Information in LOB systems is often hard to access MOSS 2007 can bring that data to your users –Data is accessed through the Business Data Catalog –Exposed to many features in SharePoint Search can easily index the data –No need to write code –Highly customizable results –Integrated with scopes and search center

28. LOB Search

29. Find People and Expertise Getting your job done means working with the right people Build rich personal profiles –Customize personal profile attributes –Manually update profile information –Automatically populate profiles using AD, LDAP directories, or LoB systems Publish personal profiles using SharePoint My Sites –Create personal portals –Share files, links, and other useful information with others –Control access to information using security and privacy controls Identify people who can help => Knowledge Network –Find people based on keyword matches with SharePoint personal profiles –Find “in-common” connections—managers, memberships, and colleagues –Group results by social distance and filter by common attributes

30. People And Expertise Bring people into the search experience –Getting your job done means working with the right people –Find subject matter experts based on their knowledge and contacts Numerous improvements over SPS 2003 –Index any LDAP V3 directory –Dedicated tab for people finding –Results grouped by “social distance” to you Knowledge Network puts People Search on steriods

31. Making Serious Sites Vragen?

32. Making Serious Sites Knowledge Network

33. Knowledge Network Overview Client Profile Creation SharePoint Server Integration Privacy and Anonymous Brokering Deployment and Administration Extensibility

34. Knowledge Network is software for enterprise social networking that helps users collaborate more effectively by automating the discovery and sharing of undocumented knowledge and relationships Who knows what? Who knows whom? Better Decisions More Quickly What is Knowledge Network?

35. Why Is Microsoft Creating KN? Three Core Beliefs –Most information is undocumented –It’s difficult to connect to the right person –“Weak ties” deliver significant value

36. Employees are more likely to turn to colleagues for information* Not stored in documents Not stored in databases It’s Stored in peoples’ heads Source: Delphi Consulting Group Documented Knowledge 20% Individual Knowledge 80% Most knowledge is undocumented Core Belief #1

37. Finding the right person often involves a referral by an intermediary It’s difficult to connect to the right person Core Belief #2

38. People in our inner circle know basically the same people and the same things that we know People that they know have connections that we don’t have By leveraging “weak ties” we can learn about new opportunities, share new ideas or meet a new person or group of people 3+ Degrees Their Extended Circle 2 Degrees Their Extended Circle 1 Degree Your Inner Circle You “Weak ties” deliver significant value Core Belief #3

39. The KN Solution 1.KN Client Analyzes email to create profile of Keywords Colleagues External Contacts 1.KN Client Analyzes email to create profile of Keywords Colleagues External Contacts 2.Publish Profile Incremental updates 3.KN Server Aggregate profiles Expertise information Social network 4.Seeker Searches for people Who knows what Who knows whom 5.People Results Ranked by Social distance to seeker Expertise and relationship relevance

40. The goal is to strike the right balance between Utility, Simplicity and Privacy –Utility is how useful will this software be to me? –Simplicity is how easy will this software be to install, upload, maintain and use? –Privacy is how much personal information will this software reveal and how much control do I have? Striking the Right Balance Privacy Concerns

41. The Simple Privacy Model Notification –Communicate steps of the profile creation and publishing process –Customers can expose privacy policy in the client profile wizard Control –User can choose which items to include/exclude –User can choose from 5 levels of privacy to apply to each profile item to control who is allowed to view that information on the server –Administrators can configure the default operation of the client, including opt-in/opt-out and the default privacy visibilities for profile items –Administrators can determine which aspects of the product functionality to leverage including external contacts, anonymous results, and DL keywords Consent –KN sends no data to the server before the user has approved it Caveat is keywords associated with “public” Distribution Lists – aggregated into DL profiles on the server

42. Top Customer Questions… Why client-side mining? –Privacy – Nothing leaves user’s machine until they “publish” –Access to information – PSTs, future data sources –Distribute the processing Why not mine sources other than email? –E-mail is by far the richest and most pervasive source today –Calculating strength across different data sources adds complexity How long does analysis take? –Depends on a number of factors On the amount of email and unique contacts On disk performance, RAM, and CPU User activity – since analysis process runs at low priority

43. Client Profile Creation Set Data Sources Outlook folders to analyze Import Outlook contacts Import Instant Messaging contacts 1. Sync Read each email Capture key “interaction data” in MDB Read in contacts from Outlook and IM clients Sync Colleagues from SPS profile Select Information Run Analysis Review Profile Publish Profile 2. Contact Resolution Lookup contacts against GAL (MAPI) Internal or external? Is it a DL? Discard Capture key properties 3. Update Aggregate counts Check thresholds Calculate strength 4. Recommendation “Exclusion lists” Special rules Limits applied Organization name mapping for External Contacts KNClient.log file and MDB files C:\Documents and Settings\ \Local Settings\Application Data\Microsoft\Knowledge Network

44. SharePoint Integration – Overview Active Directory Index AD properties DL memberships People ProfilesIndex and Search KN Client MDB Search Center - Find People - Extract and Publish Keywords Colleagues External Contacts Manually enter Interests, Skills, Responsibilities, etc

45. SharePoint Integration – Data SPS User Profiles KN Profile Management Web Service KN Store Publish KN Data to Server Client Server Client Server KN Data Crawl from SPS KN Data KN Client Data Keywords (public) Colleagues AD property changes SPS property changes Keywords (non- public) External Contacts Keywords Colleagues External Contacts

46. SharePoint Integration – My Site Keywords In Common Profile Detail Tabs KN Member Indicator Keywords Property  Colleagues  External Contacts  Keywords  Summary of public keywords

47. Desktop Search Integration

48. Anonymous Brokering  Seeker Initiates Search  Anonymous Results  Brokered E-mail  Reply to Help or Decline  Request Deleted

49. Knowledge Network

50. Client Deployment System Requirements Windows XP SP2 or Vista Office 2007 or 2003 –Strongly recommend “Cached Exchange Mode” Exchange 2007 or 2003 –With Active Directory.NET Framework 2.0 Configuration Options Client policies (GPO) –Update interval –Recommendation limits –Default “Show To” privacy –Opt-in/out first experience –Anonymous brokering –Retention “clean up” Custom privacy policy KN Server settings –Organization mapping –Exclusion lists

51. Server Deployment And Admin Deployment Leverages Office Server 2007 administration, provisioning, and deployment infrastructure Administration Options View and manage jobs User management –Manage KN “members” –Manage External Contacts Client configuration –Organization mapping –Exclusion lists Server statistics E-mail templates

52. KN Extensibility Client-side –Managed API exposing data in KN DB –Retrieve user Keywords, Colleagues, and External Contacts Server-side –Web service exposing a full-fledged query language for expertise/social network people search Can go beyond UI scenarios – e.g., multi-target queries –Web service to retrieve (read) and augment (write) profile data Bootstrap the profile or add custom keywords and contacts

53. Making Serious Sites Vragen?

54. Making Serious Sites Search Administration Improvements

55. Search Administration Design goals –Address SPS 2003 admin UI pain points –Unify WSS and MOSS search –Enable full programmability via the OM –Even better scalability and performance

56. Improved Admin & Management Consistent search system across WSS and SPS Revamped indexing management –Simpler UI –Better indexing logs Flexible scoping –Scopes decoupled from content sources –Simple and compound scopes Schema –Map foreign schema into search system –Streamlined UI Query and Result Reporting –Get visibility into user search behavior –Logging for more detailed custom reports Best Bets –Editorial results for specific queries –Streamlined UI Continuous propagation –Indexed content is immediately searchable Updated object model and APIs –Custom tools and services

57. Indexing Management Streamlined experience and more control One index per shared service  no need to worry about managing discrete indexes Multiple start addresses per content source MOSS indexes can drive the WSS search experience –Allow upgrade from WSS to MOSS

58. Security Query-time security trimming in SPS 2003 –File shares, WSS/SPS 2003, Exchange, Lotus Notes (via mapping) Now support pluggable authentication for content in WSS/MOSS sites –Based on ASP.NET 2.0 model Minimum required crawler permission is now just Full Read, not administrator –Still provides the same security trimming functionality Ability to remove Single Items

59. Search Administration

60. Customization Search in your company is different –Different metadata might matter Documents: Title, Author, File location, size Records: Patient, Doctor, Healthcare provider, SSN… –How users meaningfully scope searches differs “All finance documents” “All patient records” “All published documents” Customize results to “pop” metadata that matters Customization offered at many levels –Webparts, XSLT/CSS, full object model… –SDK available

61. Search Administration Adding a tab Customizing the Search results

62. Query Reporting Best way to improve search is to understand current usage New out of the box usage reporting –Query volume trends, top queries, click through rates, queries with zero results, etc. –At both site and service provider levels –Export data for extended reporting in Excel –Respond to feedback with configuration changes or editorial results

63. Query Reporting

65. Making Serious Sites Performance and Capacity Planning

66. Performance Improvements Key new features make the crawls faster so the content is fresher –More efficient SharePoint Crawling (Change Log Crawl) –Continuous Propagation –Unified WSS and MOSS search –Security Change Only Crawl Maximum scale is 10s of Millions of documents per indexer

67. Search Performance Planning Key learnings –Think about freshness goals, change rate –Evaluate remote server responsiveness –Upgrade WSS 2003 sites to WSS 2007 sites to index them faster –Balance results freshness with load on servers –In most cases, your search index is on one server Scale strategies might include –Dedicated WFE for crawling –Separate indexer machine –Separate SQL DB for Search –Fast disk for SQL, fast CPU for Indexer, more memory

68. Making Serious Sites Vragen?

69. Making Serious Sites Customizing the Search UI

70. Search UI Programmability Easy and powerful: –Customize using web part properties –Customize using XSLT, XML, and script Manipulate URL parameters –Build web parts and apps using the query OM keyword and SQL syntax –Remote apps using the query web service

71. POST or GET search results page Query.Execute() Get Doc IDs from index, Properties from search DB Return results in set of IDataReader objects Wrap results in XML CSS applied Client-side code runs Get query, scope from request parameters Get query, scope from request parameters Get SELECT list from tool pane Build links with URL parameters Transform Render Transform Render Change the transform Edit style sheets Write custom Web Parts to OM Results Page Information Flow Browser Server Search OM Service

72. Anatomy Of Results Page SEARCH BOX CORE SEARCH RESULTS PAGINATIONSTATISTICSACTION LINKS MATCHING KEYWORDS AND BEST BETS HIGH CONFIDENCE MATCH HIDDEN SHARED OBJECT

73. Customize UI With XSLT Web part property specifies query columns Search web parts get results XML Web parts apply transform to XML nodes Search page URL parameters enable query manipulations Appropriate for scenarios requiring –Change results layout, look and feel –Keyword syntax queries

74. Properties must have been added as Managed Properties XSLT uses them for display or logic Use the core search results web part 'Columns' property in the tool pane: <root><SelectColumns> <...> </root> Specify Properties To SELECT

75. Shirt Size: Shirt Size:</xsl:with-param><xsl:call-template>..(Later…).. </xsl:if> Displaying Custom Properties

76. Pivot or drill-down on a property to offer different queries Construct URL, re-post to page with new query - - </xsl:template> Property Filtering

77. FileExtension FileExtension File File Property Filtering

78. Established search UI element –Properties monty -actor:"John Cleese" monty -actor:John same-property clauses are ORed Equality only, prefix or exact phrase match –And / Or / Not / Phrase "chocolate covered" -frog No elaborate groupings –Scopes: scope:movies Keyword Syntax

79. URL Request Parameters Content –results.aspx?k=fish Scopes –results.aspx?k=fish&s=BBC Sort –results.aspx?v=date –results.aspx?v=relevance Page –results.aspx?start=21

80. Making Serious Sites Develop Search Solutions

81. Use query OM: –To build custom search UI web parts –To have direct access to query and results properties –When you need to manipulate results on the server Same flow, keyword or SQL syntax –Set up query text and options –Execute() –Retrieve a set of IDataReader objects Using The Query OM

82. 'ResultsTableCollection': IDataReaders Custom properties provide information on query and query execution One table for each requested result type –RelevantResults –SpecialTermResults –HighConfidenceResults –DefinitionResults Process Results

83. ResultTable relevantResults = null; DataTable relTable = New DataTable(); SQLFullTextQuery q = new SQLFullTextQuery(app); // Set up the query and ask for standard result set q.QueryText = queryBox.Text; q.KeywordInclusion = KeywordInclusion.AllKeywords; q.ResultTypes |= ResultType.RelevantResults; q.TrimDuplicates = true; // Execute! ResultTableCollection myResult = q.Execute(); // Load the results into your DataTable relevantResults = myResult[ResultType.RelevantResults]; relTable.Load(relevantResults); // Display the results in a Grid resultsGrid.DataSource = relTable; Submit Query, Process Results

84. http:// /_vti_bin/search.asmx Two query methods: –Query Research and Reference Service- compatible XML Use in the Office Research pane –QueryEx Returns.NET DataSet Full set of result types, features Using The Query Web Service

85. Select from scope() where freetext(defaultproperties,' ') order by rank desc freetext()

86. Select from scope() where freetext(defaultproperties,' ') AND (SITE = OR SITE = ) order by SITE asc Combined Clauses

87. Select from scope() where contains(' ') order by SiteName 'Contains' Predicate

88. Customizing With XSLT Putting it all together

89. Making Serious Sites Vragen?

90. Macaw Beech Avenue 140 1119 PR Schiphol-Rijk Tel +31 (0)20 8 510 510 Fax +31 (0)20 8 510 511