1. MEC 2014
4/11/2017 9:45 AM
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

2. Search in Exchange Kumar Venkateswar, Sr. Program Manager
Kutlay Topatan, Sr. Program Manager
Microsoft
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Search in Exchange 2013 Infrastructure What’s new in O365?
How does information get into the index and how is it maintained?
What types of information can be indexed?
What processes are essential and what resources are used?
How can search be monitored and managed?
How does search provide high availability?
How does querying work, and what problems do end users encounter?
What’s new in O365?

4. Big picture
Indexing
Query
Processing
Index

5. Indexing flow (mailbox)
Good: Works well
Less good: Can’t write back to database
Per-copy processing
Per-mailbox processing (DLs)
How do we mitigate these?
Can we pre-process what goes into Store? Yes!
Search service
Filter
Word Break
Content XForm
DocIDs
Content flow operators (simplified)
Notifications
Index Writer
Store
Content Engine
Index Node
Tokens
Fetch
Content DB
Index
Document Parser(s)

6. Indexing flow (transport)
Good:
Writes metadata to message before delivery
Writes natural language processing data
Less good: Best effort processing
Both flows together let us index reliably!
Filter
Word Break
Content XForm
Content flow operators (simplified)
Index Writer
Store
Transport
Content Engine DB
Document Parser(s)

7. Indexing Metadata (aka annotation stream) Words to index only
Filter
Word Break
Content XForm
Index Writer
Transport
Annotation stream writer
Annotation stream reader
Content Engine
Index Node
Store
Index
Document Parser(s) DB
Search service
Metadata (aka annotation stream)
Words to index only
Word break boundaries
Stored on message
Usually much smaller than body

8. Mailbox Flow: More Operator Detail
Amount of data read from the mailbox item will be reduced if an “annotation stream” is already present
Mailbox Database
Document
Index System
Tokens (or an error record) are written out to the index at the end of the flow…
Tokens
Document ID enters here…

9. Index – parts and merging
Master L5 L4 L3 L2 L1
Two update groups - %default and folder update group
On-disk index parts have five levels plus master index (L0 is in memory)
Each merge occurs when there are three lower parts “full”
Throttled, <=4 simultaneous merges, <=1 master
Master merge occurs with 20% of content outside the master index
This is subject to future tuning

10. Message and Attachment Processing
Message structure can be complex
Even if format is supported, not all attachments may be processed because…
There are too many attachments, or…
Nesting is too deep, or…
Processing is disabled
Administrators can change the defaults:
HKLM\Software\Microsoft\ExchangeServer\v15\Search\SystemParameters
MaxAttachmentDepth (default value: 2)
MaxAttachmentCount (default value: 10)
ProcessImages (default value: 0)
MarkSkippedImagesAsPartiallyProcessed (default value: 0)

11. Formats Supported “Out of the Box”
Group
Format
Handled by Parsers
Microsoft Office
Excel
OneNote
Outlook
PowerPoint
Publisher
Visio
Word
.xls, etc.
.one
.msg
.ppt, etc.
.pub
.vsd, .vsdx, etc.
.doc, .rtf, etc.
Other Microsoft
XML Paper Specification
.eml, .mhtml, .rss, etc.
.xps
Adobe
Acrobat
.pdf
OpenOffice.org
OpenOffice
.odp, etc.
Image
GIF
JPEG
TIFF
.gif, etc.
.jpg, etc.
.tif, etc.
Other
HTML
Plain Text
XML
ZIP
.htm, etc.
.txt, .csv, etc.
.xml, etc.
.zip
FAST engine has expanded the list of supported formats
Third party IFilters will be picked up and used
Example:
New- SearchDocumentFormat - Name "Proprietary SCT Formats" -MimeType text/scriptlet - Extension .sct - Identity ProprietarySCT1

12. New Search Processes HostControllerService.exe (Windows service)
XML config
Host Controller Service is started by the Exchange Search service. Host Controller starts 4 worker processes, each named NodeRunner. These communicate with each other and with Exchange with Windows Communication Foundation.
NodeRunner.exe
(“Admin Node”)
XML config
XML config
XML config
NodeRunner.exe
(“Content Engine Node”)
NodeRunner.exe
(“Interaction Engine Node”)
NodeRunner.exe
(“Index Node”)
XML config
ParserServer.exe
(Ifilter sandbox)

13. Resource consumption Disk Memory CPU
Per-item index size is approximately 10% of the per-item database size
Merges cause the index to need up to 20% of database size temporarily
IO is relatively sequential, since items are appended to parts and then merged
Memory
Rule of thumb: around 15% of RAM for search
More precise: constant cache cost + constant per-index system cost + constant per-item cost
Capacity planning spreadsheet gives the best estimate
CPU
Variable, based on rate, size, and content type of incoming items
Merges consume CPU as well

14. Search Management Same cmdlets as in Exchange 2010
Some new properties exposed by Get-MailboxDatabaseCopyStatus
Check search health per Server or per MDB with Test-ExchangeSearch
List unindexed documents per Server or per MDB or per MBX with Get-FailedContentIndexDocuments

15. Search Monitoring: Perf Counters
MS Exchange Search Indexes object (one instance per database):
Crawler: Items Processed
Notifications: Delayed Items
Notifications: Processed/sec
Crawler: Items Sent for Processing
Notifications: Deletes Processed
Retry: Deleted Mailboxes Remaining
Crawler: Mailboxes Remaining
Notifications: Deletes Processed/sec
Retry: Items Deleted
Crawler: Submission Delay Time
Notifications: Items Processed
Retry: Items Processed
Crawler: Submission Delays
Notifications: Items Sent for Processing
Retry: Items Sent for Deletion
Failed Items
Notifications: Last Successful Poll Timestamp
Retry: Items Sent for Processing
Feeding Sessions
Notifications: Moves Processed
Retry: Retriable Items
Items Processed
Notifications: Moves Processed/sec
Retry: Submission Delay Time
Items Processed/sec
Notifications: Processed
Retry: Submission Delays
Notifications: Age of Last Notification Processed
Notifications: Awaiting Processing
Notifications: Queue Length
Notifications: Creates Processed
Notifications: Stall Time
Notifications: Creates Processed/sec
Notifications: Updates Processed
Rate at which items are being processed into the index(es)
Number of items that are scheduled for reprocessing (because of a previous timeout or failure)

16. High Availability for Search
Based around Database High Availability architecture
Indexing always reads from the active database copy
Look for reductions in network usage in the future
Transport
Role
Transport Content Node Flow
Before DL Expansion
Local delivery
MBX1
MBX2
MBX3 DB
Index DB
Index DB
Index
Retrieval/indexing
Log shipping
Log shipping
Retrieval/indexing
Retrieval/indexing

17. Copy selection, failovers, and seeding
Index health is an important factor in best copy selection, second only to DB health
Healthy > Crawling > other status
This is health, not queue length.
Search status can trigger failovers
Disabled or failedandsuspended index on mounted
Index is suspended but database is not
Stalled seed, not able to reseed, or failed for too long on passive, plus restarting services doesn’t help
No results from query
Seeding
Since index is smaller, takes much less time than seeding database
Used to remedy a variety of index issues, on passives and actives

18. Big picture revisited: Query
Indexing
Query
Processing
Index

19. Middle Tier (depends on protocol)
Query
Word break
Parse
Query Plan
TWIR Filter
Word break
Parse
Content XForm
Middle Tier (depends on protocol)
Store
Interaction Engine
Index Node DB
Index
Queries are composed of AQS and query restrictions
Queries are wordbroken and parsed to FQL plus “that which is residual” (TWIR)
Index returns document IDs from FQL, and store processes TWIR and links in message data before returning

20. Language detection Transport flow Mailbox flow Query
Body is run through language detection
If the body is <12 characters or language can’t be detected, this is left blank
Wordbreaking uses the detected language, or English if not detected
Mailbox flow
Body + subject + contacts are run through language detection
Since this is a greater number of characters it is more likely to succeed
Fall back to English
Query
Mailbox session culture is used for language identification

21. Query troubleshooting – common problems
Queries will return 250 maximum results
Capping results improves query latency
Most users don’t want to search through many results, so our focus is on tools to improve querying
Indexing issues (transient or permanent)
Attachments that are excessively complex
Attachments without filters installed
IRM server unavailable

22. Demo: Query troubleshooting
Kumar Venkateswar

23. What is new in O365 mail search?
4/11/2017 9:45 AM
What is new in O365 mail search?
Kutlay Topatan
Program Manager
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. Let’s talk numbers 93 19% 4.5 BILLION YEARS
AVERAGE IW SPENDS SEARCHING FOR AND GATHERING INFORMATION
BEFORE AN O365 USER NEEDS TO DELETE ANY
(20 75 KB)
BUSINESS S SENT DURING THIS TALK

25. New search experiences raised user expectations
4/11/2017 9:45 AM
New search experiences raised user expectations
Relevant refiners
Suggestions
Instant results
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

26. 4/11/2017
Users struggle to find what they are looking for, re-searching frequently
PAIN POINTS
Too slow
Refining
is difficult & inefficient
Pre-Organizing doesn’t improve search success
Recall - Difficult to remember context
Source: OXG Search Focus Groups
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Introducing personalized search
4/11/2017 9:45 AM
Introducing personalized search
Lightning fast results that utilize the new indexing and query pipeline
Personalized suggestions that help interpret user intentions
Content based refiners that help finding best results with minimal effort
Hit highlighting to easily find relevant sections in long conversations
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. 4/11/2017 9:45 AM
Demo
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Additional details Availability Suggestions Refiners
4/11/2017 9:45 AM
Additional details
Availability
Hit-highlighting is already available today
Lighting search, dynamic refiners and suggestions available for service customers first, on-prem with next release
Suggestions
Keyword suggestions are populated from search history and mailbox content
People suggestions are a combination of matches from recipient cache and directory
To: search suggestions in “Sent Items”
More suggestions sources and types are being planned
Refiners
From:, Folder, Attachment and Date refiners
More refiner categories being planned
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. 4/11/2017 9:45 AM
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Query troubleshooting demo

33. Query troubleshooting demo

34. Query troubleshooting demo

35. Query troubleshooting demo

36. Query troubleshooting demo

37. Query troubleshooting demo

38. Query troubleshooting demo

39. 4/11/2017 9:45 AM
© 2014 Microsoft Corporation. All rights reserved. Microsoft, Windows and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.
© 2013 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.