Part 3B: Text Indexing, Term Lists & Taxonomies

Value space continuum of expressivity… Text indexing Thesauri Ontology Term lists Faceted Classification Less More Taxonomies Analytico-synthetic Classification Tagging Enumerated Classification Increasing control over form, relationships and meaning…

Text Indexing ▪ Full-text and inverted files/indexes

Inverted files… Primary form of index developed for use in information systems for full-text retrieval It is called an “inverted file” because the normal rows (documents) and columns (words) of a database are inverted with rows representing words and columns representing documents.

Example inverted file… Main Data File ID HOUSE PRICE 1 1208 Twin Oaks Way $100,000 2 100 Sutton Heights $200,000 3 10 Pine Street $150,000 4 8539 Billings Circle 5 9537 Highway 101 North 6 10 Capitol Hill Avenue North For example, assume you have a database of houses (rows) and one field (column) for each of those houses is the price. If you want to do rapid search by house price, build an inverted file with the rows being the prices and the columns the houses. You look up the price once and harvest the row’s columns for the houses. Inverted File or Inverted Index $100,000 1 4 5 $150,000 3 6 $200,000 2

Inverted file (document level)… Text 1 Gold silver truck 2 Shipment of gold damaged in a fire 3 Delivery of silver arrived in a silver truck 4 Shipment of gold arrived in a truck Number Term Times; Documents 1 a <3; 2,3,4> 2 arrived <2; 3,4> 3 damaged <1; 2> 4 delivery <1; 3> 5 fire 6 Gold <3; 1,2,4> 7 of 8 in 9 shipment <2; 2,4> 10 silver <2; 1,3> 11 truck <3; 1,3,4>

Inverted file (term-level)… Document Text 1 Gold silver truck 2 Shipment of gold damaged in a fire 3 Delivery of silver arrived in a silver truck 4 Shipment of gold arrived in a truck Number Term Times; Documents Words 1 a <3; (2;6),(3;6),(4;6)> 2 arrived <2; (3;4),(4;4)> 3 damaged <1; (2;4)> 4 delivery <1; (3;1)> 5 fire <1; (2;7)> 6 gold <3; (1;1),(2;3),(4;3)> 7 of <3; (2;2),(3;2),(4;2)> 8 in <3; (2;5),(3;5),(4;5)> 9 shipment <2; (2;1),(4;1)> 10 silver <2; (1;2),(3;3,7)> 11 Truck <3; (1;3),(3;8),(4;7)>> Proximity operator support

Inverted file (document level)… Text 1 Gold silver truck 2 Shipment of gold damaged in a fire 3 Delivery of silver arrived in a silver truck 4 Shipment of gold arrived in a truck Number Term Times; Documents 1 a <3; 2,3,4> 2 arrived <2; 3,4> 3 damaged <1; 2> 4 delivery <1; 3> 5 fire 6 Gold <3; 1,2,4> 7 of 8 in 9 shipment <2; 2,4> 10 silver <2; 1,3> 11 truck <3; 1,3,4> Stop words With very sophisticated full-text retrieval systems, the aggregate size of the inverted files necessary to support search can be larger than the text files they index.

Term Lists

Term lists… The simplest forms of controlled value spaces are term lists—lists of controlled terms ordered by some principle (frequently alphabetical) Infants Ankle biters Rug rats The list of authorized U.S. state abbreviations An alphabetic list of enumerated subject terms Infants (preferred term) Don’t underestimate the power of these simple, controlled lists

Simple (yet powerful) lists… A list (also sometimes called a pick list) is a limited set of terms arranged as a simple alphabetical list or in some other logically evident way. Lists are used to describe aspects of entities that have a limited number of possibilities. Examples include geography (e.g., country, state, city), language (e.g., English, French, Swedish), or format (e.g., text, image, sound) Simple alphabetical list: Alabama Alaska Arkansas California Connecticut Delaware Simple logical list: Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune Pluto*

Taxonomies ▪ Yahoo! Directory

Dominant form on the Web… Hierarchical tree structure Example: Yahoo! Directory Frequently permit polyhierarchy (multiple parents) No general principles guiding design of taxonomies “A collection of controlled vocabulary terms organized into a hierarchical structure. Each term in a taxonomy is in one or more parent/child (broader/narrower) relationships to other terms in the taxonomy.” [NISO/Z39.19] [emphasis added] This is not intended to imply that Web taxonomies are necessarily unprincipled. It just means that as a form, they do not have the same guiding principles as we will see with thesauri and classifications. Individual designs of taxonomies can be rigorously structured and with consistent, intelligent, well thought out forms of cross referencing and other devices.

Polyhierarchy

Polyhierarchy… [NISO/Z39.19] Based on generic relationship Based on whole-part relationship Based on multiple types of relationship musical instruments stringed instruments percussion instruments piano biology chemistry biochemistry bones head skull

Node Labels milk . . <milk by source animal> .. buffalo milk Non-indexable concepts used for purposes of organizing other concepts in meaningful ways milk . . <milk by source animal> .. buffalo milk .. cow milk .. goat milk .. sheep milk . <milk by region> .. United States .. India ..China

End • Part 3B: Text Indexing, Term Lists & Taxonomies