1. Thesaurus Design and Development
University of California, Berkeley
School of Information Management and Systems
SIMS 202: Information Organization and Retrieval
8/28/97
Information Organization and Retrieval

2. Information Organization and Retrieval
Review
Origins and Uses of Controlled Vocabularies for Information Retrieval
Types of Indexing Languages, Thesauri and Classification Systems
8/28/97
Information Organization and Retrieval

3. Information Organization and Retrieval
Indexing Languages
An index is a systematic guide designed to indicate topics or features of documents in order to facilitate retrieval of documents or parts of documents.
An Indexing language is the set of terms used in an index to represent topics or features of documents, and the rules for combining or using those terms.
8/28/97
Information Organization and Retrieval

4. Information Organization and Retrieval
Thesauri
A Thesaurus is a collection of selected vocabulary (preferred terms or descriptors) with links among Synonymous, Equivalent, Broader, Narrower and other Related Terms
8/28/97
Information Organization and Retrieval

5. Information Organization and Retrieval
Thesauri (cont.)
Examples:
The ERIC Thesaurus of Descriptors
The Art and Architecture Thesaurus
The Medical Subject Headings (MESH) of the National Library of Medicine
8/28/97
Information Organization and Retrieval

6. Classification Systems
A classification system is an indexing language often based on a broad ordering of topical areas. Thesauri and classification systems both use this broad ordering and maintain a structure of broader, narrower, and related topics. Classification schemes commonly use a coded notation for representing a topic and it’s place in relation to other terms.
8/28/97
Information Organization and Retrieval

7. Automatic Indexing and Classification
Automatic indexing is typically the simple deriving of keywords from a document and providing access to all of those words.
More complex Automatic Indexing Systems attempt to select controlled vocabulary terms based on terms in the document.
Automatic classification attempts to automatically group similar documents using either:
A fully automatic clustering method.
An established classification scheme and set of documents already indexed by that scheme.
8/28/97
Information Organization and Retrieval

8. Information Organization and Retrieval
Today
Thesaurus design
Steps in Thesaurus development
Indexing
8/28/97
Information Organization and Retrieval

9. Why develop a thesaurus?
To provide a conceptual structure or “space” for a body of information
To make it possible to adequately describe the topical contents of informational objects at an appropriate level of generality or specificity
To provide enhanced search capabilities and to improve the effectiveness of searching (I.e., to retrieve most of the relevant material without too much irrelevant material).
8/28/97
Information Organization and Retrieval

10. Why develop a thesaurus?
To provide vocabulary (or terminological) control.
When there are several possible terms designating a single concept, the thesaurus should lead the indexer or searcher to the appropriate concept, regardless of the terms they start with.
8/28/97
Information Organization and Retrieval

11. Preliminary considerations
What is used now?
Continue using an existing thesaurus?
Ad hoc modification of existing thesaurus?
Develop a new well-structured thesaurus?
What is the scope and complexity of the subject field?
What kind of retrieval objects or data will be dealt with?
How exhaustive and specific is the desired description of objects?
8/28/97
Information Organization and Retrieval

12. Preliminary Considerations
The scope and complexity of the field will provide some indication of the scope and complexity of the thesaurus.
It is better to plan for a larger and more comprehensive system than a smaller system that rapidly will become inadequate as the database grows.
Development of a good thesaurus requires a major intellectual effort as well as clerical operations like data entry and production of sorted lists.
8/28/97
Information Organization and Retrieval

13. Development of a Thesaurus
Term Selection.
Merging and Development of Concept Classes.
Definition of Broad Subject Fields and Subfields.
Development of Classificatory structure
Review, Testing, Application, Revision.
8/28/97
Information Organization and Retrieval

14. Information Organization and Retrieval
1. Term Selection
Select sources for the collection of terms.
Prearranged Sources
Open-ended Sources
Assign codes to each source.
Selection of terms
For part of pre-arranged and for all open-ended sources
Enter terms into database with all information.
8/28/97
Information Organization and Retrieval

15. Information Organization and Retrieval
1.1 Kinds of Sources
Prearranged Sources
Existing descriptor lists, classification schemes thesauri. This includes universal schemes like DDC or LCSH.
Nomenclatures of single disciplines
Treatises on the terminology of a field
Encyclopedias, lexica, dictionaries and glossaries.
Tables of contents of textbooks and handbooks
Indexes of journals or abstracting journals
Indexes of other publications in the field
8/28/97
Information Organization and Retrieval

16. Information Organization and Retrieval
1.1 Kinds of Sources
Open-ended sources
Lists of search requests or interest profiles
Description of projects/activities to be served by the information retrieval system.
Discussion with specialists in the field
Sample of documents in the field
Ask users why and how these documents relate to the field.
Have documents indexed by experts in the field
Lists of titles of documents in the field
Abstracts and reviews of documents
Your own knowledge
8/28/97
Information Organization and Retrieval

17. Information Organization and Retrieval
Selection of sources
Prearranged sources require less effort in gathering the material, and may already indicate some relationships between terms and concepts and relationships among terms.
Open-ended sources can reflect current terminology and may provide more complete coverage.
Choose a set of sources that are current, as complete as possible, and considered authoratative.
8/28/97
Information Organization and Retrieval

18. Information Organization and Retrieval
Selection of Sources
Each selected source is assigned an ID for tracking its use in the development of the thesaurus.
Useful when making decisions about which terms to prefer
Useful for backtracking when questions arise (where did this come from?)
8/28/97
Information Organization and Retrieval

19. Information Organization and Retrieval
Selection of Terms
Terms can be transferred directly from prearranged sources to the recording medium (cards or database)
Have to decide which terms and references to include, or to take the whole source
8/28/97
Information Organization and Retrieval

20. Information Organization and Retrieval
Selection of Terms
In open-ended sources you read through the source and pick out terms (I.e. words and phrases) that might be useful in retrieval or as references to other terms.
Alternatively, use keyword and phrase extraction software to create lists of terms and select from those.
Transfer selected terms to the recording medium (cards or database).
8/28/97
Information Organization and Retrieval

21. 2. Merging and Development of Concept Classes
Sort Term DB into alphabetical order.
First Round: Merge information for Identical terms -- possibly pulling info from additional sources.
Second Round: Merge synonyms or terms in the same concept class.
8/28/97
Information Organization and Retrieval

22. 3. Definition of Broad Subject Fields and Subfields
Work out the detailed structure
Select Preferred Terms
Merge information for terms in the same concept class
Repeat these steps
for each subfield within a broad field
and for each broad field
Until all terms have been consolidated and preferred terms selected
Define Broad Subject fields and sort terms into these broad fields
Define subfields within each broad field and sort terms into these subfields.
8/28/97
Information Organization and Retrieval

23. 4. Development of Classificatory Structure
Produce preliminary version of classified index and update the working database.
Improve classificatory structure
Reality check: produce and distribute a version of the classified index. Distribute to users/experts.
8/28/97
Information Organization and Retrieval

24. Information Organization and Retrieval
5. Final Stages
Review
Testing
Application
Revision
8/28/97
Information Organization and Retrieval

25. Information Organization and Retrieval
Review
Discuss classified index with users/experts.
Select descriptors and checklist descriptors.
Assign Notational Symbols
Produce Main Thesaurus & Indexes
8/28/97
Information Organization and Retrieval

26. Information Organization and Retrieval
Review (cont.)
Check cross references and insert where needed
Produce Test Version
Test by Indexing
Modify as needed
Produce Production Version.
8/28/97
Information Organization and Retrieval

27. Information Organization and Retrieval
Testing a Thesaurus
Assign descriptors to a sample set of NEW documents (use enough to get an idea of any gaps in the thesaurus.
Test retrieval using sample questions and seeing how effectively the thesaurus maps to the appropriate descriptor
8/28/97
Information Organization and Retrieval

28. Information Organization and Retrieval
The Indexing Process
Concept identification
term selection (via thesaurus)
term assignment
8/28/97
Information Organization and Retrieval

29. Application: The Indexing Process (Manual)
Select Alternative
term to represent
Concept NO
Would
Concept be
better represented
by one of
these
terms Is
There
Another
Concept
Consider
Preferred
Term
Select
Establish Term
Denoting
Examine Document
and Identify
Significant
Concepts
First
Term?
Start NO
YES
Does
Thesaurus
contain term
for
Consider any
associated terms in
Thesaurus (NT,BT)
Admit New Term
Into Thesaurus
Can Concept
be expressed
combining terms?
Consider Each of
These Terms
Assign Terms to
Document
Prefer
Alternative
Term(s)
End Is
Term
suitable
Adapted from ISO 5963, p.5
8/28/97
Information Organization and Retrieval

30. Thesaurus Revision and Updates
There will always be new concepts, products, or expressions that need to be added to the thesaurus.
Set a regular schedule of reviews and revisions.
Collect complaints, problems, etc. and fold into revision of the thesaurus
8/28/97
Information Organization and Retrieval

31. Information Organization and Retrieval
References
Soegel, D. Indexing Languages and Thesauri: Construction and Maintenance. Los Angeles : Melville Publishing Co., 1974
Foskett, A.C. The Subject Approach to Information. London: Clive Bingley, 1982.
Standards:
ANSI/NISO z American National Standard Guidelines for the Construction, Format and Management of Monolingual Thesauri
ANSI/NISO Draft Standard Z x -- American National Standard Guidelines for Indexes in Information Retrieval
ISO Documentation -- Guidelines for the establishment and development of monolingual thesauri
ISO Documentation -- Guidelines for the establishment and development of multilingual thesauri
8/28/97
Information Organization and Retrieval