Microsoft SQL Server 7.0 Full-Text Search What is full-text search and how do I troubleshoot it? John Kane, MCSE PSS SQL Server Support Microsoft Corporation

2 SQL Server 7.0 Full-Text Search   Full-text search (FTS) is a new component of Microsoft® SQL Server™ 7.0 that allows for faster and more flexible searching of character- based columns in SQL Server tables and file- based documents.   Full-text search is integrated with the new Microsoft Search Service for Microsoft® Windows NT® Server 4.0.

3 SQL Server 7.0 Full-Text Search   Full-text search is installed under the custom installation type and is available on the following SQL Server versions: SQL Server 7.0 Enterprise Edition SQL Server 7.0 Standard SQL Server 7.0 Small Business Server Both Windows NT Workstation and Microsoft® Windows® 9.x clients can access Full-Text Search.

4 SQL Server 7.0 Full-Text Search   Real-world applications for full-text search: Internet/intranet search interface Product catalog searching Document management Data Warehouse / Business Intelligence (DW/BI) search interface Any application requiring a fast and flexible search interface, possibly combining SQL data and file- based documents

5 SQL Server 7.0 Full-Text Search   How is full-text search different than using the LIKE operator? Full-text search can search on words or phrases, including the use of wildcards such as asterisk (*). Full-text search can search for words in close proximity. Full-text search can search on inflectional forms (different tenses of words) such as drive, drove, or driving. A full-text search query is much faster than a LIKE query.

6 SQL Server 7.0 Full-Text Search   What do I need to run full-text search queries? Transact-SQL predicates: CONTAINS or FREETEXT For rowset and ranked results use: CONTAINSTABLE or FREETEXTTABLE.

7 SQL Server 7.0 Full-Text Search   How do I install full-text search? Full-text search is not set up by default in the standard installation of SQL Server 7.0. You must choose Custom Install during setup and select Full-Text under Server Components.

8 SQL Server 7.0 Full-Text Search

9   How do I install full-text search? (cont.) After setup is complete, a new service (Microsoft Search Service) has been added. This service can be managed under the Enterprise Manager Support Services folder.

10 SQL Server 7.0 Full-Text Search

11 SQL Server 7.0 Full-Text Search   What tools do I need to set up a full-text index? Use the Full-Text Indexing Wizard in SQL Server Enterprise Manager. Use system stored procedures: sp_fulltext_catalog sp_fulltext_table sp_fulltext_column Note: The table must have a one-column, non-nullable unique index to be considered for full-text indexing.

12 SQL Server 7.0 Full-Text Search

13 SQL Server 7.0 Full-Text Search  What else do I need to do to use a full-text index? After you have established the full-text index, you must populate it using one of the following methods: After you have established the full-text index, you must populate it using one of the following methods: Enterprise Manager Database-specific Full-Text Catalogs’ Catalog Content menu. Enterprise Manager Database-specific Full-Text Catalogs’ Catalog Content menu. System Stored Procedure: sp_fulltext_catalog with start_full or start_incremental actions. System Stored Procedure: sp_fulltext_catalog with start_full or start_incremental actions. Note: You should also set up a SQL Server Agent job for scheduled full-text indexing.

14 SQL Server 7.0 Full-Text Search

15 SQL Server 7.0 Full-Text Search  New full-text search functions and objects: Metadata functions: Metadata functions: FULLTEXTCATALOGPROPERTY FULLTEXTCATALOGPROPERTY FULLTEXTSERVICEPROPERTY FULLTEXTSERVICEPROPERTY SQL-DMO Objects: SQL-DMO Objects: FullTextCatalog FullTextCatalog FullTextService and more… FullTextService and more…

16 SQL Server 7.0 Full-Text Search  Are the full-text indexes stored within SQL Server? No. The full-text indexes are stored externally in system folders and files called catalogs. No. The full-text indexes are stored externally in system folders and files called catalogs. By default, these full-text catalogs are located under the parent folder Mssql7\Ftdata. By default, these full-text catalogs are located under the parent folder Mssql7\Ftdata. Full-text catalogs are maintained as a collection of folders and files, such as Sql Full-text catalogs are maintained as a collection of folders and files, such as Sql

17 SQL Server 7.0 Full-Text Search  full-text search  Simple full-text search SQL code examples: Obtain a list of articles where the description mentions both “Picabo Street” and “downhill racing.” SELECT article_id FROM Hockey_News WHERE CONTAINS (description, ' “ Picabo Street “ AND “ downhill racing “ ') This matches the following...

18 SQL Server 7.0 Full-Text Search  Results of a simple full-text search CONTAINS query: “Picabo Street is scheduled to return to downhill racing.” “Picabo Street's super G win during downhill racing...” “Downhill Champions: Lindh and Picabo Street excel in downhill skiing. Racing is their passion…”

19 SQL Server 7.0 Full-Text Search  full-text search  Simple full-text search SQL code examples: (cont.) Why not just use LIKE? A: Performance !! Results for searching on “dropdown” in DB (~ 150,000 rows):

20 SQL Server 7.0 Full-Text Search  full-text search  Simple full-text search SQL code examples: (cont.) Proximity search: words or phases close to one another (approximately 50 words): SELECT Company, Price FROM Travel_Loc WHERE CONTAINS(Description, 'skiing NEAR Aspen'

21 SQL Server 7.0 Full-Text Search  Simple full-text search SQL code examples (cont.): Word generation or inflectional forms of words, such as swim, swam, swimming… Word generation or inflectional forms of words, such as swim, swam, swimming… SELECT colActivity, colClass FROM HERC_Club WHERE CONTAINS(colExercise, 'FORMSOF(INFLECTIONAL, swim)') 'FORMSOF(INFLECTIONAL, swim)')

22 SQL Server 7.0 Full-Text Search   Simple full-text search SQL code examples (cont.): The FREETEXT predicate supports a simple form of natural language query, for example, searches for similar meanings, not exact words: SELECT help_text_col FROM on_line_help_tbl WHERE FREETEXT ( help_text_col, 'Download Service Packs' )

23 SQL Server 7.0 Full-Text Search   Relevance ranking of results: Rank is a derived column value between 1 and 1,000. Indicates how well the row matches the selection criteria. Applies to all FTS queries and is useful with both the ISABOUT and NEAR functions. Must use CONTAINSTABLE and FREETEXTTABLE rowset-valued functions.

24 SQL Server 7.0 Full-Text Search   Relevance ranking SQL code examples: Return the abstract column for articles on text processing where those articles that are most relevant are listed first: SELECT M.article_number, M.abstract, FT.RANK FROM magazines, CONTAINSTABLE(magazines, article,'ISABOUT ( ("DB2" NEAR "text extender") WEIGHT(0.9), ("SQL Server" NEAR "text") WEIGHT(0.3), ("SQL Server" NEAR ”fulltext") WEIGHT(0.9))') AS FT WHERE M.article_number = FT.[KEY] ORDER BY FT.RANK DESC

25 SQL Server 7.0 Full-Text Search   Full-text search stored procedure calling sequence: sp_fulltext_database 'enable' sp_fulltext_catalog 'PubInfo', 'create' sp_fulltext_table 'pub_info', 'create', 'PubInfo', 'UPKCL_pubinfo' sp_fulltext_column 'pub_info', 'pr_info', 'add' sp_fulltext_table 'pub_info', 'activate' sp_fulltext_catalog 'PubInfo', 'start_full'... wait... Use sp_help_fulltext_* for full-text metadata queries...

26 SQL Server 7.0 Full-Text Search   Removing FTS from a SQL database stored procedure calling sequence: sp_fulltext_table 'pub_info', 'drop' sp_fulltext_catalog 'PubInfo', 'drop' sp_fulltext_database 'disable' sp_fulltext_service 'clean_up'

27 SQL Server 7.0 Full-Text Search   Full versus incremental full-text population   Full full-text index population Every row of every table associated with a full-text catalog is re-indexed. Full-text catalog is discarded and rebuilt from scratch. Most useful when there are many changes or inserts.

28 SQL Server 7.0 Full-Text Search   Full versus incremental full-text population (cont.):   Incremental full-text index population: Only changed rows of tables within an full-text c atalog are re-indexed. Row will be reindexed even if the other columns are changed. Requires presence of a timestamp column on a table. If the table schema version of table is altered, may be a full population. Most useful on large tables with smaller turnover of rows.

29 SQL Server 7.0 Full-Text Search   Full-text search and distributed queries: Issue SQL queries against SQL data and system file documents and properties. OLE DB Provider for Index Server: MSIDXS OPENQUERY OPENROWSET

30 SQL Server 7.0 Full-Text Search   Full-text search and distributed query examples find: Titles, filenames, file sizes, and URLs of the Excel files under the IIS virtual root of /Excel/Revenue such that: The file names start with ‘OBOS’. The file sizes are less than 10,000 bytes. Include the words “Index Server” near “SQLServer”

31 SQL Server 7.0 Full-Text Search   FTS and distributed query examples (cont.): SELECT DocTitle, FileName, size, Vpath FROM SCOPE(' "/Excel/Revenue" ') WHERE FileName LIKE ‘OBOS%.xls’ AND size < AND CONTAINS (' "Index Server" NEAR "SQL Server” ')

32 SQL Server 7.0 Full-Text Search   Full-text performance tuning from practical experience: Full-text queries are faster than queries using LIKE “%test%. For SQL tables with fewer than 1 million rows, both full-text indexing and search have acceptable performance and no special performance tuning is usually required.

33 SQL Server 7.0 Full-Text Search  Full-text performance tuning from practical experience (cont.): For SQL tables with more than 1 million rows: Hardware considerations Windows NT system configuration considerations SQL Server 7.0 configuration considerations Full-text indexing and searching considerations

34 SQL Server 7.0 Full-Text Search   Full-text performance for more than 1 million rows: Hardware considerations: Fast and multiple CPUs: one to four 500- MHz Xeon III processors Lots of memory: 1 GB to 4 GB of physical RAM Multiple disk controllers with several channels Fast disk I/O subsystems, RAID0, and RAID5 FTS is not currently supported on clusters

35 SQL Server 7.0 Full-Text Search   Full-text performance for more than 1 million rows (cont.): Windows NT system configuration considerations: Pagefile.sys needs to be sized one and one-half to two times the amount of available physical RAM. Pagefile.sys files need to be placed on their own drives (RAID0), preferably on a separate controller, or least on a separate channel off of a shared controller. Microsoft Search Service can be set to dedicated during full-text index periods and then reset to background or normal when not full-text indexing.

36 SQL Server 7.0 Full-Text Search   Full-text performance for more than 1 million rows (cont.): SQL Server configuration considerations: While full-text indexing or populating the full-text catalogs is ongoing, SQL Server’s maximum memory may need to be limited to half of the available physical RAM. Support for language neutral full-text can be configured using the sp_configure system stored procedure.

37 SQL Server 7.0 Full-Text Search   Full-text performance for more than 1 million rows (cont.): Full-text indexing and search considerations: Full-text Indexing or populating the full-text catalogs should be done during periods of low system activity, typically during database maintenance windows. The full-text indexing of SQL tables with more than 4 million rows can take many hours or days to complete. Consider the options offered in KB article Q240867, “INF: How to Move, Copy & Backup SQL Server 7.0 Full-Text Catalog Folders & Files.”

38 SQL Server 7.0 Full-Text Search   Full-text performance for more than 1 million rows (cont.): Full-text indexing and search considerations: Apply the latest service pack (SP), currently SP1. When designing full-text search queries, consider using the CONTAINSTABLE and FREETEXTTABLE functions and their new Top_N_Rank parameter. See KB Article Q240833, “FIX: Full-Text Search Support for TOP via CONTAINSTABLE and FREETEXTTABLE Clauses.” Currently available as a hotfix and is included in SP2.

39 SQL Server 7.0 Full-Text Search   Full-text search basic troubleshooting techniques: Identify the trouble area(s): Full-text indexing (populating full-text catalogs) and/or Full-text searching Gather information about: Hardware, Windows NT, and SQL Server configurations. Review the Windows NT application log using Event Viewer. SQL tables and full-text catalogs involved.

40 SQL Server 7.0 Full-Text Search   Full-text search basic troubleshooting techniques (cont.): Review the SQL Server Full-Text Search Troubleshooter on the Microsoft Support Web site at: /sql7fts.asp Are you getting some type of FTI or FTS error? Most are recorded in the Windows NT application log as a “Microsoft Search” source. Are you having FTI or FTS performance problems? Review the previous slides on full-text performance issues.

41