1. Sept. 25-26, 2008 City of St. Louis, Missouri 2008 Taxonomy Boot Camp Organizing Information for Search & Discovery San Jose McEnery Convention Center San Jose CA Planning Made All The Difference: City of St. Louis Case Study

2. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

3. Sept. 25-26, 2008City of St. Louis, Missouri Background  Started as a Community Information Network not a municipal website in 1995.  Focus was on general useful content (history, Neighborhoods, Education, Housing, Services, and Government). Municipal information came later.  By 2006 site had over 10,000 pages of content, 120+ stand-alone dynamic applications and no overall way to stitch everything together  Organizing and finding information was an exercise in faith and frustration. Complaints became increasingly common.

4. Sept. 25-26, 2008City of St. Louis, Missouri Site Consolidation Goals Decided to overhaul website and consolidate web presence using a content management solution. Goals: 1.Classification and organization of content 2.Integration of 120+ dynamic applications 3.Use standard metadata and create a Standardized Municipal Topic Classification Challenge: 1.How?

5. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

6. Sept. 25-26, 2008City of St. Louis, Missouri Research & Review of Practices  No standardized US model for government available  International models reviewed:  Australia: Thesaurus of Australian Government Subjects http://www.naa.gov.au/records-management/create- capture-describe/describe/agls/index.aspx  Canada: Government of Canada Core Subject Thesaurus http://en.thesaurus.gc.ca/intro_e.html  United Kingdom: Integrated Public Service Vocabulary (IPSV) http://www.esd.org.uk/standards/ipsv/

7. Sept. 25-26, 2008City of St. Louis, Missouri British IPSV as model Top Terms Deep Structure http://www.esd.org.uk/standards/ipsv/viewer/ The British central government developed and promotes the Integrated Public Services Vocabulary as the standard for all British Governmental bodies.

8. Sept. 25-26, 2008City of St. Louis, Missouri How to Integrate Our Early Classification Effort  User Survey  Services A to Z  Random Site Topics  Category Agreement Assessment

9. Sept. 25-26, 2008City of St. Louis, Missouri Survey: Category Agreement Assessment (CAA)  400+ terms on site  Random presentation of 20 terms  User asked to type primary and secondary category  Results generated list used for our A to Z pages

10. Sept. 25-26, 2008City of St. Louis, Missouri Category Agreement Assessment: A to Z  Resulted in 45 categories  Created A-Z list  Topics  Categories  Beginning of a Folksonomy

11. Sept. 25-26, 2008City of St. Louis, Missouri Tagged sample contents with IPSV  Mapped our A to Z topics and categories against British IPSV

12. Sept. 25-26, 2008City of St. Louis, Missouri Findings  Our Content topics mapped well to IPSV  IPSV hierarchy was too deep and complex  Content classification needed to expand beyond topics  Effort was beyond our team’s expertise  Outside Information Architecture services was needed to create a robust but simpler classification

13. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

14. Sept. 25-26, 2008City of St. Louis, Missouri Retaining IA Expertise  Identified possible resources mentioned in reputable resources with a mandate toward standards and Dublin Core  Developed list of questions  Interviewed contacts to identify good match and clarify scope  Selected Taxonomy Strategies to filter all information and devise a well structured Information Architecture document to guide our CMS implementation

15. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

16. Sept. 25-26, 2008City of St. Louis, Missouri Materials Used to Create Classification Schema  IPSV  List and frequency of Search Terms  User Survey Results:  Usability Assessment Results  List of information people could not find on website  Category Agreement Assessment (folksonomy)  A to Z content items and categories  Audience/user scenarios  CMS Application requirements list  Site Architecture and Navigation Requirements

17. Sept. 25-26, 2008City of St. Louis, Missouri Summary of St. Louis Metadata Standard Text Fields: Title, Description, Date, Headline, Subhead, … Useful for search, and for linking from one page to another. Typical Vocabulary-based Fields: Access, Audience, Content Type, Organization, Topic Content Type provides the basis for page templates. Organization and Topic provide basis for dynamic navigation. Special Vocabulary-based Fields: Event Type, Neighborhood, Procurement Type, Ward These are key to improving the dynamic navigation. Vocabulary Name Vocabulary Purpose and Description Access Groups List of group names for different access control groups. Currently only contains "Public" until we understand the various access control groups better. Audience Describes a small number of target audiences for content. This list may grow in the future but the amount of content on the site now does not justify it, esp. since this is likely to be highly correlated with Topic and Organization. Content Type List of genres of content produced by the City of St. Louis, such as press releases, RFPs, meeting agendas and minutes, etc. Page templates are based on Content Type, and many queries will look for specific content types. Event Types List of types of events, such as public meetings, alderman meetings, exhibitions, etc. Used to categorize calendar events so that more tailored calendar listings can be shown. Neighborhood Simple list of Neighborhoods within City of St. Louis. Used for geographic tagging. Organization Hierarchical list of entities within the government of the City of St. Louis, or affiliated with the government. Organization Type Simple list of different type of organizations that provide content which may be linked to from the site. Procurement Types Sub-content types for "RFPs, Bids and Notices." Topic Hierarchical list of topics relevant to city and county governments within the United States. This is the main subject categorization scheme. Ward Simple list of Wards within City of St. Louis. Used for geographic tagging.

18. Sept. 25-26, 2008City of St. Louis, Missouri Summary of St. Louis Content Types.CV LabelSynonyms A-Z Pages Calendars and SchedulesEvent Calendar, Events, Public Meetings Contact Info and BiographiesBios, Directory Facts, Figures, and DemographicsCharts, Graphs, Population, Data, Statistics Gallery pagesVideo, Audio, Images Group Home Pages Help pages Informational Pages Job Openings Jobs, Help wanted, job annoucements, employment opportunities Laws, Bills, and Policies Regulations, Ordinances, Procedures, Rules, Laws, Legislation, Bills, Board Bills News and AnnoucementsNewsletters, Bulletins Maps and GIS Meeting MaterialsAgendas, Minutes, Handouts Online Forms and ApplicationsOnline Surveys, Find an Elected Official, Permits Plans and Reports CAFR, Comprehensive Annual Financial Reports, Budget reports, Departmental status reports, Program performance reports Printed Forms and TemplatesSurveys, Permits, Tax Forms, Records Request RFPs, Bids, and Notices RFP, RFI, RFQ, Proposals, Bids, Contract Awards, Procurement Subscriptions

19. Sept. 25-26, 2008City of St. Louis, Missouri Business & Industry Business Assistance, Incentives and Promotion Business Regulation and Licensing Business Taxation Labor Market Facts & Figures Local Economy Small Business Workforce Training and Development Community Births, Deaths, Weddings, and other Life Events Beautification and Nuisance Reduction Community Facilities Facts, Figures & Community Spirit History Neighborhoods Neighborhood Associations Neighborhood Programs & Events Social and Support Groups Volunteering Non-Governmental Organizations Immigration and Nationality Education & Skills Training Adult and Community Education Further and Higher education Homeschooling Public and Private Schools Special Educational Needs and Additional Support Employment, Jobs & Careers Employment Rights and Regulations Local Government Employment Opportunities Self employment Unemployment and jobseeking Youth Employment Environment Environmental Protection Hazardous Materials Waste Management & Recycling Climate Government Local Government Records Local Governmental Units Complaints and Service Requests Elections & Voting Elected Officials Federal and State Government Services Fees & Payment Information Local Government Services Permits, Inspections & Certifications Policy Making Procurement Taxation & Revenue Wards and Neighborhoods Account & Annual Budget & Operating Plan Employees Health Animal Control Immunizations and Public Health Food Safety Nutrition and Preventative Care Housing & Community Development Home Ownership Home Rental Housing Financial Programs Home Repairs and Renovation Relocation Temporary Housing Homelessness Leisure & Culture Arts and Entertainment Events Children's and Teens Activities Libraries, Museums & Cultural Venues Gambling and Lotteries Parks and Gardens Pets and Animals Shopping & Dining Sports and Recreation Tourism Planning & Urban Development Land Acquisition Preservation Planning Property Assessment Zoning Public Safety Police Fire Dept. Ambulance and Emergency Services Building Inspections & Permits Courts and Justice System Legal Assistance Reporting & Operation Planning Transport & infrastructure Airports Construction, Maintenance & Improvements Electric, Gas, Sewer & Water Utilities Telephone, Cable TV & Internet Utilities Motor Vehicles Public Transportation Parking Rail and Water Transportation Streets, Sidewalks & Streetlighting St. Louis Topics.CV Before Usability Testing

20. Sept. 25-26, 2008City of St. Louis, Missouri Dublin Core Metadata Standard http://www.dublincore.org  Followed the Dublin Core Metadata Element, a vocabulary of fifteen properties for use in resource description  Mapped Dublin Core properties to corresponding metadata fields in the proprietary CMS  Coded to output DC metadata in source code  Examples: DC.TermDescriptionCMS Equivalent dc.coverage.wardLocation related to contentcustom metadata perhaps attached to the base template dc.terms.audienc e Audience content is geared toward request.page.audienc ecategory dc.typeType of content (e.g. Press Release, Fiscal Report, Meeting Minutes, etc request.Page.Categor yID dc.subjectRelevant terms from Topics.CV Page.CategoryID

21. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

22. Sept. 25-26, 2008City of St. Louis, Missouri Taxonomy Testing 3 testing approaches used:  Tagging representative content  Card-sorting and use-based scenario testing with focus groups  Benchmarks and metrics to evaluate usability test results, identify coverage gaps, and provide guidance for changes.

23. Sept. 25-26, 2008City of St. Louis, Missouri Tagging Representative Content Test Cases:  A to Z topics  Random URLs  Human Services content pages

24. Sept. 25-26, 2008City of St. Louis, Missouri Card Sorting and Use-based Scenarios with Focus Groups  22 representative questions were selected  42 participants were tested  Testers were asked “where would you look for …”  Testers were asked to “think aloud”  Pre- and post questions were asked  Initial findings indicated that 8 of the 22 scenarios may require revisions  Governance Team agreed on revisions  8 of the original scenarios were re-tested using 8 new participants  Modifications to the proposed taxonomy were based on the findings and approved by Governance Group.

25. Sept. 25-26, 2008City of St. Louis, Missouri Task-based Card Sort Representative Question Example: You are planning to build a new deck on your house. You want to know if you need a permit for this. Which topic would you select to get started? Topics 1.Business & Industry 2.Community 3.Education & Skills Training 4.Employment, Jobs & Careers 5.Environment 6.Government 7.Health 8.Housing & Community Dev 9.Leisure & Culture 10.Planning & Urban Development 11.Public Safety 12.Transport & infrastructure 6.Government Local Government Records Local Governmental Units Complaints and Service Requests Elections & Voting Elected Officials Federal and State Government Services Fees & Payment Information Local Government Services Permits, Inspections & Certifications Policy Making Procurement Taxation & Revenue Wards and Neighborhoods Account & Annual Budget Employees 1.Topics 2.Content Types

26. Sept. 25-26, 2008City of St. Louis, Missouri Analysis of Task-based Card Sort Scenario 1. Busi ness and Indu stry 2. Com muni ty 3. Educ ation and Traini ng 4. Employ ment, Jobs, and Careers 5. Enviro nment 6. Gover nment 7. He alt h 8. Housi ng 9. Leis ure and Cult ure 10. Urba n Devel opme nt and Plann ing 11. Law, Safety, and Justice 12. Transpo rtation, Infrastru cture, and Utilities 1. Historic development requirements 1 1 6 2. Lead Remediation Initiatives 1 1 32 1 3. Jury Duty 3 5 4. Permits 1 2 2 3 5. Street light 3 5 6. Complaints 1 21 4 7. Assess Property1 4 2 1 8. Census Data 2 6

27. Sept. 25-26, 2008City of St. Louis, Missouri Summary Usability Testing Process  Card Sorting and Use-based Scenarios  Governance Team Members Involvement  Re-testing and Validation  Changes to Taxonomy and Metadata based on results For Example:  ‘Public Safety’ became ‘Law, Safety, and Justice’  Words like “Development” and “Community” repeated in different terms were eliminated  Identified additional ‘See Also’ association

28. Sept. 25-26, 2008City of St. Louis, Missouri Business & Industry Business Assistance, Incentives and Promotion Business Regulation and Licensing Business Taxation Labor Market Facts & Figures Local Economy Small Business Workforce Training and Development Community Births, Deaths, Weddings, and other Life Events Beautification and Nuisance Reduction Community Facilities Facts, Figures & Community Spirit History Neighborhoods Neighborhood Associations Neighborhood Programs & Events Social and Support Groups Volunteering Non-Governmental Organizations Immigration and Nationality Education & Skills Training Adult and Community Education Further and Higher education Homeschooling Public and Private Schools Special Educational Needs and Additional Support Employment, Jobs & Careers Employment Rights and Regulations Local Government Employment Opportunities Self employment Unemployment and jobseeking Youth Employment Environment Environmental Protection Hazardous Materials Waste Management & Recycling Climate Government Local Government Records Local Governmental Units Complaints and Service Requests Elections & Voting Elected Officials Federal and State Government Services Fees & Payment Information Local Government Services Permits, Inspections & Certifications Policy Making Procurement Taxation & Revenue Wards and Neighborhoods Account & Annual Budget & Operating Plan Employees Health Animal Control Immunizations and Public Health Food Safety Nutrition and Preventative Care Housing & Community Development Home Ownership Home Rental Housing Financial Programs Home Repairs and Renovation Relocation Temporary Housing Homelessness Leisure & Culture Arts and Entertainment Events Children's and Teens Activities Libraries, Museums & Cultural Venues Gambling and Lotteries Parks and Gardens Pets and Animals Shopping & Dining Sports and Recreation Tourism Planning & Urban Development Land Acquisition Preservation Planning Property Assessment Zoning Public Safety Police Fire Dept. Ambulance and Emergency Services Building Inspections & Permits Courts and Justice System Legal Assistance Reporting & Operation Planning Transport & infrastructure Airports Construction, Maintenance & Improvements Electric, Gas, Sewer & Water Utilities Telephone, Cable TV & Internet Utilities Motor Vehicles Public Transportation Parking Rail and Water Transportation Streets, Sidewalks & Streetlighting Topics.CV Before Usability Testing

29. Sept. 25-26, 2008City of St. Louis, Missouri Finalized Topics.CV After Usability Testing Business and Industry Business Assistance, Incentives, and Promotion Business Regulation and Licensing Business Taxation Labor Market Facts and Figures Local Economy Small Business Workforce Training and Development Community Births, Deaths, Weddings, and other Life Events Beautification and Nuisance Reduction Community Facilities Facts and Figures History Neighborhoods Neighborhood Associations Neighborhood Programs and Events Social and Support Groups Volunteering Non-Governmental Organizations Immigration and Nationality Education and Skills Training Adult and Community Education Further and Higher education Homeschooling Public and Private Schools Special Educational Needs and Additional Support Employment, Jobs, and Careers Employment Rights and Regulations Local Government Employment Opportunities Self employment Unemployment and Jobseeking Youth Employment Environment Environmental Protection Hazardous Materials Waste Management and Recycling Climate Government Local Government Records Local Government Offices, Agencies, and Departments Complaints and Service Requests Elections and Voting Elected Officials Federal and State Government Services Fees and Payment Information Local Government Services Permits, Inspections, and Certifications Policy Making Procurement Taxation and Revenue Wards and Neighborhoods Account and Annual Budget and Operating Plan Employees Health Animal Control Immunizations and Public Health Food Safety Nutrition and Preventative Care Housing Home Ownership Home Rental Housing Financial Programs Home Repairs and Renovation Relocation Temporary Housing Homelessness Law, Safety, and Justice Police Fire Department Ambulance and Emergency Services Courts and Justice System Legal Assistance Tourism Leisure and Culture Arts and Entertainment Events Children and Teens Activities Libraries, Museums, and Cultural Venues Gambling and Lotteries Parks and Gardens Pets and Animals Shopping and Dining Sports and Recreation Transportation, Infrastructure, and Utilities Airports Construction, Maintenance, and Improvements Electric, Gas, Sewer, and Water Utilities Telephone, Cable TV, and Internet Utilities Motor Vehicles Public Transportation Parking Rail and Water Transportation Streets, Sidewalks, and Streetlighting Urban Development and Planning Land Acquisition, Reutilization, and Development Preservation Planning Zoning

30. Sept. 25-26, 2008City of St. Louis, Missouri Process Summary Review and filtering of all relevant documentation (A to Z classification, International Best Practices, search terms and frequency, use scenarios, requirements, etc.) = Draft Information Architecture Document + Sample Tagging of pages + Wireframes = Proposed Information Architecture Volume 1.0 + Taxonomy Testing = Adopted Information Architecture Volume 1.0

31. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

32. Sept. 25-26, 2008City of St. Louis, Missouri Wireframe Tagging of Page Populated by querying calendar system with Org =“Cultural Resources Office” OR Topic= ”Planning & Urban Development/Preservation” OR Topic=“Community/History” WHERE Date > Today ASCENDING ORDER BY Date Related folders box dynamically populated from query AROUND(“Planning & Urban Development/Preservation”, “Community/History”). Same as See also references Spotlights and other sidebars can be manually filled at page creation/edit time. Alternatively, they can be dynamically filled with queries like: SELECT (Title, URL, Description, Thumbnail) WHERE Topic = AROUND (”Planning & Urban Development/Preservat ion”, “Community/History) AND Thumbnail not null. Page Index with Metadata for content placement Office contact info dynamically populated from contacts database WHERE Org = Planning&UrbanDesign Agency/CulturalResources Office Tagging of Page: Type=Group Home Pages Org=Planning & Urban Design Agency / Cultural Resources Office Topic=“Planning & Urban Development/Preservation”, “Community/History” Coverage.Wards=“MANY_WARDS” Coverage.Neighborhoods=“City Wide”

33. Sept. 25-26, 2008City of St. Louis, Missouri Taxonomy & Applications Integration Challenge: How to leverage Information Architecture for outside applications (120+)? Solution:  Create homepage in CMS  Login Permissions managed by CMS  Re-coded to have one single page output  Select Metadata and taxonomy during page creation Example: Gatewood CemeteryGatewood Cemetery  Application data outside of CMS tables  Permissions set inside CMS  Display page is tagged using CMS metadata.  Topic relationships visible in “Browse by Topic” and “ See Also…”

34. Sept. 25-26, 2008City of St. Louis, Missouri Browse by, See Also… Tagged Content Page Topic association with other facets External Application Integrated in CMS using Metadata and Taxonomy

35. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

36. Sept. 25-26, 2008City of St. Louis, Missouri Governance  Information Architecture Governance Team (IAGT) created to institute a process for changes.  9 members, representatives of major sections of government (Public Safety, Community Services, Parks, Health) and Web Team Manager as Lead  IAGT Responsible for Policies Governing Process Change. Policy Documents to-date:  Process Guide  Editorial Rules  IAGT Responsible for Testing and Validation

37. Sept. 25-26, 2008City of St. Louis, Missouri Agenda 1.Background 2.Research and Review of practices 3.Information Architecture Expertise 4.Metadata and Taxonomy Development Process 5.Taxonomy Testing 6.Implementation 7.Governance 8.Conclusions

38. Sept. 25-26, 2008City of St. Louis, Missouri Conclusions  Goal to standardize Metadata and Taxonomy for Municipal Government proved difficult.  Taxonomy Testing was critical  Testing with user groups forced customization of taxonomy that reflected local flavor  Process helped identified areas of missing content and classification. (i.e. Included audience facet for human services type classifications)  Benefits Created:  Strengthened Relationships between content  Extensible to other section of city government  Improved ability to find information and search  More discreet filtering  Established process for future site/content growth.

39. Sept. 25-26, 2008City of St. Louis, Missouri Thank you! For more information contact: Sonya Pelli Internet Services Manager Information Technology Services Agency City of St. Louis, Missouri pellis@stlouiscity.com http://stlouis.missouri.org 314 622-3400 x258