1. THE ANDREW W. MELLON FOUNDATION CLiMB: Computational Linguistics for Metadata Building U. of Maryland College of Information Studies

2. CLiMB and the VRA CLiMB-1 at Columbia University, 2002-2004 CLiMB project first presented at VRA 2004 Participants for user study from VRA attendees CLiMB-2 at University of Maryland, 2005-2007 Demonstrate cataloging toolkit VRA member participation

3. CLiMB’s approach and CLiMB-1 Angela Giral, former Director, Avery Art and Architecture Library, Columbia University Goals and objectives of CLiMB-2 Marilyn Domas White, U. of Maryland Demonstration of CLiMB-2 Cataloging Toolkit Jimmy Lin, U. of Maryland Next phase: Testing with partners Judith Klavans, PI, CLiMB-2 Project, U. of Maryland Today’s Program

5. Problems CLiMB Addresses  Minimal cataloging for digital images  Traditional approach Requires expertise Time intensive Expensive  Can automated techniques help?

6. The Essence of CLiMB Improving subject access to images by mining scholarly publications for metadata

7. CLiMB-1 Achievements Built and tested prototype CLiMB image cataloging toolkit with VRA Determined collection features most appropriate for CLiMB’s approach

8. CLiMB-1 Project Team Judith Klavans, PI Stephen Davis Angela Giral Patricia Renfro Bob Wolven Roberta Blitz Rebecca Passonneau David Elson

9. CLiMB-2 Project Team Judith Klavans, PI Marilyn Domas White, UMD Jimmy Lin, UMD Eileen G. Abels, UMD Dagobert Soergel, UMD Rachel Stazi, UMD Angela Giral, CU (ret) Rebecca Passonneau, CU

11. The next steps: CLiMB-2 The toolkit Catalogers as users Searchers as users of CLiMB-2 search terms

12. Making the toolkit usable Refine interface Extend to unstructured text, such as art history texts Allow for match to AAT and LC thesauri Make toolkit system-independent Develop useful documentation for catalogers

13. Catalogers as users (1) Test toolkit in lab setting for operability and usability Observe toolkit in use in regular work setting Evaluate ease of use and effectiveness in meeting toolkit objectives

14. Catalogers as users (2) Develop benchmarks for toolkit use, e.g., time Get feedback, feedback, feedback! Have CLiMB-2 data generated for real collections

15. Image searchers as users How do CLiMB-2 terms affect image searching? Test retrieval by actual image searchers working on their own tasks Note: Potential long-term effects may not be visible immediately.

17. CLiMB Cataloging Toolkit An interactive cataloging workbench Key features: Analyzes texts about images Proposes subject index terms Maps extracted terms to existing controlled vocabularies Exports to standard catalog record formats

18. Process Flow Albert Bierstadt (American, born Germany [Prussia], 1830-1902) Bridal Veil Falls, Yosemite, about 1871-73 Oil on canvas, 36 1/8 x 26 3/8 in. (91.7 x 67.0 cm.) Purchased with funds from the North Carolina Art Society (Robert F. Phifer Bequest) and various donors, by exchange, 87.9 German-born Albert Bierstadt gave definitive expression to America’s westward expansionism in the 1860s and 70s. His vast panoramas of the Rocky and Sierra Nevada Mountains introduced Americans to a majestic wilderness, awesome but unthreatening, and well worth possessing. The artist found his greatest subject in California's Yosemite Valley, which he first visited in the summer of 1863. So spectacular was the remote and secluded valley that early visitors readily imagined it the Promised Land. Bierstadt's many paintings of Yosemite, such as this view of Bridal Veil Falls, are indeed biblical in their grandeur, imbued with the sense that divinity dwells within the wilderness. Ironically, the popularity of Bierstadt's paintings attracted droves of tourists to Yosemite, compelling the artist to look elsewhere for untrammeled scenery. Winslow Homer (American, 1836-1910) Weaning the Calf, 1875 Oil on canvas, 24 x 38 in. (61.0 x 96.5 cm.) Purchased with funds from the State of North Carolina, 52.9.16 At first glance, Homer's Weaning the Calf seems simple enough: a reminiscence of an idyllic summer down on the farm. Though the artist frequently addresses themes of childhood in his art, his pictures generally avoid the cloying sentimentality common to other Victorian painters. In this painting, Homer pays close attention to the play of sunlight and shadow, endowing the scene with the heightened realism of a vivid memory. Such nostalgic visions were obviously escapist: they comforted Americans eager to forget four years of civil war as well as the disruptions of rapid industrialization. However, some scholars have suggested that Homer also intended other, more complex meanings. Weaning the Calf is one among a remarkable group of pictures from the 1870s depicting African-Americans with an understanding and sympathy rare for the time. These paintings have been interpreted as a sustained meditation on the unsettled position of blacks in post-war American society. In this picture, the central image of a black boy in tattered clothes straining to separate a stubborn calf from its mother has been read as a visual metaphor for the painful weaning of African-Americans from slavery. Though intriguing, there is little evidence that Homer himself intended such a reading. His contemporaries certainly preferred to enjoy the picture simply as "a farmyard scene of a very spirited character. "* Albert Bierstadt (American, born Germany [Prussia], 1830-1902) Bridal Veil Falls, Yosemite, about 1871-73 Oil on canvas, 36 1/8 x 26 3/8 in. (91.7 x 67.0 cm.) Purchased with funds from the North Carolina Art Society (Robert F. Phifer Bequest) and various donors, by exchange, 87.9 German-born Albert Bierstadt gave definitive expression to America’s westward expansionism in the 1860s and 70s. His vast panoramas of the Rocky and Sierra Nevada Mountains introduced Americans to a majestic wilderness, awesome but unthreatening, and well worth possessing. The artist found his greatest subject in California's Yosemite Valley, which he first visited in the summer of 1863. So spectacular was the remote and secluded valley that early visitors readily imagined it the Promised Land. Bierstadt's many paintings of Yosemite, such as this view of Bridal Veil Falls, are indeed biblical in their grandeur, imbued with the sense that divinity dwells within the wilderness. Ironically, the popularity of Bierstadt's paintings attracted droves of tourists to Yosemite, compelling the artist to look elsewhere for untrammeled scenery. Segment + Associate Analyze Select Export

19. Toolkit Behavior Input: a collection consisting of Images of art objects Text about the art objects “Bare bones” catalog records Output: Catalog records augmented with subject index terms

20. Segment + Associate Analyze Select Export Albert Bierstadt (American, born Germany [Prussia], 1830-1902) Bridal Veil Falls, Yosemite, about 1871-73 Oil on canvas, 36 1/8 x 26 3/8 in. (91.7 x 67.0 cm.) Purchased with funds from the North Carolina Art Society (Robert F. Phifer Bequest) and various donors, by exchange, 87.9 German-born Albert Bierstadt gave definitive expression to America’s westward expansionism in the 1860s and 70s. His vast panoramas of the Rocky and Sierra Nevada Mountains introduced Americans to a majestic wilderness, awesome but unthreatening, and well worth possessing. The artist found his greatest subject in California's Yosemite Valley, which he first visited in the summer of 1863. So spectacular was the remote and secluded valley that early visitors readily imagined it the Promised Land. Bierstadt's many paintings of Yosemite, such as this view of Bridal Veil Falls, are indeed biblical in their grandeur, imbued with the sense that divinity dwells within the wilderness. Ironically, the popularity of Bierstadt's paintings attracted droves of tourists to Yosemite, compelling the artist to look elsewhere for untrammeled scenery. Winslow Homer (American, 1836-1910) Weaning the Calf, 1875 Oil on canvas, 24 x 38 in. (61.0 x 96.5 cm.) Purchased with funds from the State of North Carolina, 52.9.16 At first glance, Homer's Weaning the Calf seems simple enough: a reminiscence of an idyllic summer down on the farm. Though the artist frequently addresses themes of childhood in his art, his pictures generally avoid the cloying sentimentality common to other Victorian painters. In this painting, Homer pays close attention to the play of sunlight and shadow, endowing the scene with the heightened realism of a vivid memory. Such nostalgic visions were obviously escapist: they comforted Americans eager to forget four years of civil war as well as the disruptions of rapid industrialization. However, some scholars have suggested that Homer also intended other, more complex meanings. Weaning the Calf is one among a remarkable group of pictures from the 1870s depicting African-Americans with an understanding and sympathy rare for the time. These paintings have been interpreted as a sustained meditation on the unsettled position of blacks in post-war American society. In this picture, the central image of a black boy in tattered clothes straining to separate a stubborn calf from its mother has been read as a visual metaphor for the painful weaning of African-Americans from slavery. Though intriguing, there is little evidence that Homer himself intended such a reading. His contemporaries certainly preferred to enjoy the picture simply as "a farmyard scene of a very spirited character. "* Albert Bierstadt (American, born Germany [Prussia], 1830-1902) Bridal Veil Falls, Yosemite, about 1871-73 Oil on canvas, 36 1/8 x 26 3/8 in. (91.7 x 67.0 cm.) Purchased with funds from the North Carolina Art Society (Robert F. Phifer Bequest) and various donors, by exchange, 87.9 German-born Albert Bierstadt gave definitive expression to America’s westward expansionism in the 1860s and 70s. His vast panoramas of the Rocky and Sierra Nevada Mountains introduced Americans to a majestic wilderness, awesome but unthreatening, and well worth possessing. The artist found his greatest subject in California's Yosemite Valley, which he first visited in the summer of 1863. So spectacular was the remote and secluded valley that early visitors readily imagined it the Promised Land. Bierstadt's many paintings of Yosemite, such as this view of Bridal Veil Falls, are indeed biblical in their grandeur, imbued with the sense that divinity dwells within the wilderness. Ironically, the popularity of Bierstadt's paintings attracted droves of tourists to Yosemite, compelling the artist to look elsewhere for untrammeled scenery. Winslow Homer (American, 1836-1910) Weaning the Calf, 1875 Oil on canvas, 24 x 38 in. (61.0 x 96.5 cm.) Purchased with funds from the State of North Carolina, 52.9.16 At first glance, Homer's Weaning the Calf seems simple enough: a reminiscence of an idyllic summer down on the farm. Though the artist frequently addresses themes of childhood in his art, his pictures generally avoid the cloying sentimentality common to other Victorian painters. In this painting, Homer pays close attention to the play of sunlight and shadow, endowing the scene with the heightened realism of a vivid memory. Such nostalgic visions were obviously escapist: they comforted Americans eager to forget four years of civil war as well as the disruptions of rapid industrialization. However, some scholars have suggested that Homer also intended other, more complex meanings. Weaning the Calf is one among a remarkable group of pictures from the 1870s depicting African-Americans with an understanding and sympathy rare for the time. These paintings have been interpreted as a sustained meditation on the unsettled position of blacks in post-war American society. In this picture, the central image of a black boy in tattered clothes straining to separate a stubborn calf from its mother has been read as a visual metaphor for the painful weaning of African-Americans from slavery. Though intriguing, there is little evidence that Homer himself intended such a reading. His contemporaries certainly preferred to enjoy the picture simply as "a farmyard scene of a very spirited character. "*

21. Segment + Associate Determine which texts “go with” which images Locate references to art objects in text Determine which linguistic expressions refer to the same object Find spans of text that discuss the referenced object Challenges: Varying degrees of correspondence between text and images The phenomenon of anaphora Andy Warhol… he David… the sculpture Segment + Associate Analyze Select Export

22. Considerations Types of collections: Structured: collection catalogs Unstructured: textbooks, lectures Availability: Of electronic text Of digitized images Of existing catalog records Copyright issues Segment + Associate Analyze Select Export

23. Analyze Segment + Associate Analyze Select Export System processes text associated with the art object System identifies potential index terms System employs techniques from computational linguistics Identify noun phrases Filter out inappropriate terms using heuristics Map phrases to controlled vocabularies

24. Analyze: Example Segment + Associate Analyze Select Export Market Scene on a Quay, about 1635 Frans Snyders (Flemish, 1579-1657) Oil on canvas, 79 5/16 x 135 in. (201.5 x 343.5 cm.) Purchased with funds from the State of North Carolina, 52.9.113 At 135 inches in width, Snyders’ still life cannot fail to impress, as it must have when he became the first to paint market scenes on such a scale. Snyders' native Antwerp was then the leading commercial and artistic center of Flanders. This type of subject had been initiated by Pieter Aertsen in the same city during the previous century with such paintings as the Museum's Butcher's Stall…

25. Identify noun phrases Some noun phrases are good index terms How do we identify noun phrases? Step 1: Tag words with part of speech Systems “learn” how to do this from large collections of manually tagged text Step 2: Extract patterns that correspond to noun phrases At/IN 135/CD inches/NNS in/IN width/NN,/, Snyders’/NNS still/RB life/NN can/MD not/RB fail/VB to/TO impress/VB,/, as/IN it/PRP must/MD have/VB when/WRB he/PRP became/VBD the/DT first/JJ to/TO paint/VB market/NN scenes/NNS on/IN such/PDT a/DT scale/NN./. Segment + Associate Analyze Select Export

26. Filter noun phrases Not all noun phrase make good index terms Heuristic rules for filtering noun phrases Remove certain terms: e.g., “type”, “kind”, “width” Remove noun phrases we already know about: e.g., artist’s name, name of work, etc. Do not repeat terms User can control the application of these rules Precision vs. recall tradeoff Segment + Associate Analyze Select Export

27. Map noun phrases Map noun phrases to controlled vocabulary terms How? Basic keyword searching Open research question Segment + Associate Analyze Select Export Still life AAT Term still lifes (,,... Visual and Verbal Communication) ID: 300015638 Images in which the focus is a depiction of inanimate objects, as distinguished from art in which such objects are…

28. Select User decides which of the proposed subject descriptors are actually good index terms Segment + Associate Analyze Select Export Market Scene on a Quay, about 1635 Frans Snyders (Flemish, 1579-1657) Oil on canvas, 79 5/16 x 135 in. (201.5 x 343.5 cm.) Purchased with funds from the State of North Carolina, 52.9.113 At 135 inches in width, Snyders’ still life cannot fail to impress, as it must have when he became the first to paint market scenes on such a scale. Snyders' native Antwerp was then the leading commercial and artistic center of Flanders. This type of subject had been initiated by Pieter Aertsen in the same city during the previous century with such paintings as the Museum's Butcher's Stall…

29. Export Augment catalog records based on selected terms Automatically create new records from scratch if metadata available Standard catalog formats (examples) USMARC VRA Core Categories, Version 3.0 Dublin Core Metadata Element Set, Version 1.1 Segment + Associate Analyze Select Export

30. Record Type=work Type=painting Title=Market Scene on a Quay Measurements.Dimensions= 79 5/16 x 135 in. (201.5 x 343.5 cm.) Material.Medium=oil on canvas Creator.Personal name= Frans Snyders Creator.Role=artist Date.Creation=1635 Location.Current Repository=North Carolina Museum of Art ID Number.Current Accession= 52.9.113 CLiMB.term=still life CLiMB.term=market scene VRA CORE 3.0

31. From CLiMB-1 to CLiMB-2 A client-server implementation is not ideal Techniques from computational linguistics help(!) Mapping from natural language phrases to controlled vocabularies is difficult Users like control over filtering heuristics Non-technological issues present problems –Availability of collections, text, cataloging expertise –Copyright The CLiMB-2 toolkit is a reimplementation from the ground up

32. Technical Specifications Java Application Platform-independent Runs locally on a standalone machine Supports richer interaction models XML extensively employed To describe input text To store intermediate results To encode output (when appropriate)

33. What you’ll see A prototype of the CLiMB-2 toolkit Two sample images Simulated filtering heuristics Simulated mapping from natural language phrases to AAT terms

41. Building Partners Image collections Museums Libraries Users and user groups Catalogers Image searchers of many types Tool builders for image libraries

42. University of Maryland Jack Sullivan, Landscape Architecture Bryn Mawr and University of Pennsylvania Jeff Cohen, Vernacular Architecture Current Partners

43. Partner Scenario Partner has images, catalogers, image system, some electronic texts CLiMB team provides toolkit with a manual Partner catalogers use toolkit and report periodically on: How are they using the toolkit? How easy is the toolkit to use? How is the toolkit affecting their normal workflow? How would they improve/change the toolkit? CLiMB maintains a collaborative area online for CLiMB community

44. Building CLiMB Partnerships Collections Texts Catalogers Community input

45. If you are interested... Contact Project Manager, Rachel Wadsworth Stazi rstazi@umd.edu www.umiacs.umd.edu/~climb