SUSHI A beginner’s guide to NISO’s Standardized Usage Statistics Harvesting Initiative Breakout Sessions: Group B UKSG Conference and Exhibition Torquay April 7-9, 2008 Oliver Pesch EBSCO Information Services opesch@ebsco.com

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

Why do librarians collect usage statistics? Because they must Government and funding bodies may require them E.g. ARL statistics To inform renewal decisions Overall use Cost-per-use Support cancellation decisions Generally manage e-resources and the tools and programs that support them One simple reason for collecting usage data is because they are obliged to. Organizations like ARL require their members to submit detailed statistics on the library, the library collection and its use. Other agencies and funding bodies demand the same. The big challenge with collecting statistics for online resources is that the library does not control the collection and, depending on the resource, the majority of the use can come from users who do not “pass through” the library (physically or virtually). Other than the “because they must”, usage statistics are being used as one input into renewal decisions. In the case of databases, the number of searches performed on a given database, compared to others is an indicator of the usefulness of that database, similarly, journals with low use, or high cost per use may undergo additional scrutiny. Many librarians work to maximize their budgets; therefore, they may cancel journals or databases that are no longer as relevant and use that money for other materials. In the case where the library budge it under pressure, usage statistics become a tool for isolating those materials that could be considered for cancellation. Note that just because one journal or database does not have the same level of use as another, does not mean it is less valuable. The librarian will also take into consideration the discipline and programs the resource is supporting. Low use may be as a result of the database being hard to find on the library web page and thus can be used to prompt action to better highlight and promote. Then usage can be measured over time to rate the effectiveness of the change.

Why do librarians collect usage statistics? Because they must Government and funding bodies may require them E.g. ARL statistics To inform renewal decisions Overall use Cost-per-use Support cancellation decisions Generally manage e-resources and the tools and programs that support them This screen shot is courtesy Innovative Interfaces, Inc. and shows one of the reports from their Usage Consolidation module. Note the “cost per use” column – a simple calculation of price paid divided by number of full text downloads from the journal.

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

Timeline for usage related standards efforts

Timeline for usage related standards efforts Online collections and their use grows …..… Online collections continue to grow and become a significant part of the libraries collection. As a result, libraries need to measure usage to control these resources that are taking a growing percentage of their budget. Not all vendors provide usage statistics for a variety of reasons. And when they do, the reporting is not consistent from one to the next. The problem becomes critical for consortia who want the statistics to evaluate the effectiveness of their purchases.

Timeline for usage related standards efforts ICOLC Guidelines for Usage Data The International Coalition of Library Consortium became the first to address this growing problem. In 1999 they created their guidelines for reporting usage data. They normalized the terminology, and set expectations as to what elements a vendor was to report on. Significant because for the first time vendors were given a yardstick to be measured against.

Timeline for usage related standards efforts ICOLC Guidelines: Release 2 Two years later came the second release of the guidelines which included refinements to the first.

Timeline for usage related standards efforts Project COUNTER formed The ICOLC guidelines helped; however, inconsistencies in counting and formatting continued to be a problem. Something beyond a set of definitions was needed. Publishers, Librarians and Aggregators teamed together to solve the problem and formed COUNTER… the goal was to create a code of practice that would lead to consistent, comparable and credible usage statistics.

Timeline for usage related standards efforts COUNTER Code of Practice Release 1 Release 1 of the COUNTER Code of Practice was published within a year. It clarified terminology; identified specific reports that were needed, addressed common problems with web logs and double-clicking, and specified the format and methods of delivery for the reports.

Timeline for usage related standards efforts ERMI committee formed In 2002 the E-Resource Management Initiative was formed under the sponsorship of the Digital Library Foundation. This was an outcome of the research published the prior year by Tim Jewell. Tim was investigating the growing challenge of managing e-resources. He discovered that many libraries were developing their own solutions. The goal of ERMI was to come up with some standard approaches and guidelines for managing e-resources and as a result in 2004 the committee published its report, which included functional specifications, a data dictionary and an entity relationship diagram. The ERMI work became the blue-print for commercial ERM systems.

Timeline for usage related standards efforts First commercial ERM released In 2004 Innovative Interfaces released the first commercial ERM. The ERM is intended to offer the library a single place to store and access all information about their e-resources. The reason we are talking about ERM systems is that, as the single place to store all information about accessing and administering e-resources, the natural extension to this system was to incorporate usage data.

Timeline for usage related standards efforts ERM Usage Consolidation Module As a result Innovative, I believe, was the first to attempt to add the usage consolidation module. Their goal was to leverage the work of COUNTER so that they could load the full text usage information in a standard format comparable across vendors. Unfortunately there were variation in how the code of practice was being applied, AND it was a lot of work to gather reports so something else had to be done.

Timeline for usage related standards efforts SUSHI committee formed By mid 2005, it was clear that a method of automatically harvesting usage data was needed and thus SUSHI was born. SUSHI stands for the Standardized Usage Statistics Harvesting Initiative – we will get into more about SUSHI in a minute.

Timeline for usage related standards efforts COUNTER Code of Practice release 2 Shortly after SUSHI was created COUNTER updated its code of practice for journals and databases. They addressed some of the issues that were uncovered by early usage consolidation work by becoming much more specific with the formats and introducing some additional elements, such as “Publisher”/ They also introduced the notion of an audit to verify compliance.

Timeline for usage related standards efforts SUSHI released as draft standard SUSHI was released as a draft standard some 14 months after the committee was formed, and…

SUSHI certified by ANSI as Z39.93 Timeline SUSHI certified by ANSI as Z39.93 It was approved by the NISO members and officially became NISO Z39.93 by the end of last year.

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

Usage data importance grows with e-collections Usage Statistics Usage data importance grows with e-collections Collection management Budget management The simple fact that as online takes an increasingly larger role in the library collection, so does the need to measure usage… for collection management (weeding) and budget management

Usage data importance grows with e-collections Usage Statistics Usage data importance grows with e-collections Collection management Budget management Credibility and consistency… Different vendors using different terminology Inconsistencies in processing lead to over counting Formatting differences make comparison challenging So with the need for usage data and the fact that the usage is not gathered by the library, surfaced the first major problem. Those vendors that were providing usage (and many did not), were not consistent in terminology, formatting and even the basic techniques for counting.

Usage data importance grows with e-collections Usage Statistics Usage data importance grows with e-collections Collection management Budget management Credibility and consistency… Different vendors using different terminology Inconsistencies in processing lead to over counting Formatting differences make comparison challenging …COUNTER… A number of standards initiatives, like ICOLC, NISO and others contributed to solving these problems, but it was COUNTER that really made the difference

Usage data importance grows with e-collections Usage Statistics Usage data importance grows with e-collections Collection management Budget management Credibility and consistency… Different vendors using different terminology Inconsistencies in processing lead to over counting Formatting difference comparison challenging Consolidation and meaningful reporting… Many vendors and reports to process Collection-level views needed …COUNTER… Usage data is gathered at the vendor sites… The library must retrieve and process to create meaningful reports.

…Usage Consolidation tools (ERM)… Usage Statistics Usage data importance grows with e-collections Collection management Budget management Credibility and consistency… Different vendors using different terminology Inconsistencies in processing lead to over counting Formatting difference comparison challenging Consolidation and meaningful reporting Many vendors and reports to process Collection-level views needed …COUNTER… It was not until COUNTER came in to play that library application vendors and service providers saw that they could create a consolidation systems that would effectively consolidate the reports. Around this time the ERMs were beginning to appear on the market and thus the foundation was there to support such consolidation. …Usage Consolidation tools (ERM)…

…Usage Consolidation tools (ERM)… Usage Statistics Usage data importance grows with e-collections Collection management Budget management Credibility and consistency… Different vendors using different terminology Inconsistencies in processing lead to over counting Formatting difference comparison challenging Consolidation and meaningful reporting… Many vendors and reports to process Collection-level views needed Retrieving and processing… Obtaining reports is time consuming Formatting and other adjustments still needed …COUNTER… Now we have the standard in place to get somewhat uniform stats from vendors and the place to load them, then next problem is uncovered. The time it takes to retrieve reports is significant and the process convoluted. And even with COUNTER, some manual tweaking is often needed …Usage Consolidation tools (ERM)…

…Usage Consolidation tools (ERM)… Usage Statistics Usage data importance grows with e-collections Collection management Budget management Credibility and consistency… Different vendors using different terminology Inconsistencies in processing lead to over counting Formatting difference comparison challenging Consolidation and meaningful reporting… Many vendors and reports to process Collection-level views needed Retrieving and processing… Obtaining reports is time consuming Formatting and other adjustments still needed …COUNTER… Resulting from this challenge came SUSHI… which we will discuss later. …Usage Consolidation tools (ERM)… …SUSHI…

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

Goals Codes of practice Audit Coming in release 3 http://www.projectcounter.org/ COUNTER, or “Counting Online Usage of Networked Electronic Resources” Formed in 2002, Project COUNTER is a non-profit organization that was formed with the participation of publishers, librarians and aggregators. This collaboration was key for the success of this group. While many publishers were providing usage statistics for their product for years, they were counting different things in different ways. Project COUNTER has lead in the standardization of the usage of electronic resources and focuses on how things are counted and how they are reported. The ultimate goal for this standardization can be summed up in three Cs: Usage reports should be consistent, they should be credible, and they should be comparable across products. I’ll get into examples in the next few minutes and then conclude with some caveats.

Libraries and consortia need online usage statistics Why COUNTER? Goal: credible, compatible, consistent publisher/vendor-generated statistics for the global information community Libraries and consortia need online usage statistics To assess the value of different online products/services To support collection development To plan infrastructure Publishers need online usage statistics To experiment with new pricing models To assess the relative importance of the different channels by which information reaches the market To provide editorial support

COUNTER Codes of Practice Definitions of terms used Specifications for Usage Reports What they should include What they should look like How and when they should be delivered Data processing guidelines Auditing Compliance

COUNTER: current Codes of Practice 1) Journals and databases Release 1 Code of Practice launched January 2003 Release 2 published April 2005 replacing Release 1 in January 2006 Now a widely adopted standard by publishers and librarians Almost 100 vendors now compliant 10,000+ journals now covered Librarians use it in collection development decisions Publishers use it in marketing to prove ‘value’

Journal and Database Code of Practice: Reports Journal Report 1 Full text article requests by month and journal Journal Report 2 Turnaways by month and journal Database Report 1 Total searches and sessions by month and database Database Report 2 Turnaways by month and database Database Report 3 Searches and sessions by month and service

COUNTER: current Codes of Practice 2) Books and reference works Release 1 Code of Practice launched March 2006 10 vendors now compliant Relevant usage metrics less clear than for journals Different issues than for journals Direct comparisons between books less relevant Understanding how different categories of book are used is more relevant

Books and Reference Works: Reports Book Report 1 Number of successful requests by month and title Book Report 2 Number of successful section requests by month and title Book Report 3 Turnaways by month and title Book Report 4 Turnaways by month and service Book Report 5 Total searches and sessions by month and title Book Report 6 Total searches and sessions by month and service

Specific Formats Here is an example of a Journal Report. I have highlighted the required metadata in the top left-hand corner that needs to preceed the report. This includes when the report was run. You will notice that there is a distinction between the Publisher and the Platform. This is not a concern for most publishers, but is a concern for an aggregator like EBSCO Host or ProQuest. Services like HighWire, Ingenta and Metapress also host multiple publishers on a single platform. The other distinction is that yearly totals come in three flavors: A simple yearly total for all fulltext requests, and two additional columns that divide up HTML requests from PDF requests. The rationale for this was that some publishers provide multiple format versions of the same article and that readers tend to browse an HTML version before downloading the PDF version of the same article. Without breaking these two formats down, it would have been difficult to compare the usage of a publisher that provided both versions with another that provided only say PDF.

Explicit report layout – “consistent” The “consistency” goal is achieved by creating very explicit standards for how reports are presented. Not much is left to interpretation or imagination in the latest COUNTER release. All cells in the report are defined and described so there is no question on what goes into them. The main impetus for this change was the development of electronic management systems for libraries that could ingest these reports.

Credibility: COUNTER Audit Independent audit required within 18 months of compliance, and annually thereafter Audit is online, using scripts provided in the Code of Practice Auditor can be: Any Chartered Accountant Another COUNTER-approved auditor ABCE is the first COUNTER-approved auditor Industry-owned Not-for-profit Independent and impartial Part of ABC (Audit Bureau of Circulations) Providing website traffic audits for over 150 companies and certifying over 1400 domains Have successfully completed test audits on COUNTER usage reports

Coming soon… Release 3 of the Journals and Databases Code of Practice Key features… Consortium reports Sets expectations for handling of: Federated searching Internet robots and archives like LOCKSS Browser prefetching Reports must be available in XML format Revised COUNTER XML Schema SUSHI support becomes a requirement for compliance

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

A more efficient data exchange model SUSHI: Objectives COUNTER statistics provides an excellent model and rules for usage statistics counting Libraries needed: A more efficient data exchange model Current model is file-by-file spreadsheet download Background query and response model is more efficient and scalable

SUSHI: What it is and Isn’t A web-services model for requesting data Replaces the user’s need to download files from vendor’s website A request for data where the response includes COUNTER data Using COUNTER’s schema What it isn’t: A model for counting usage statistics A usage consolidation application

SUSHI: COUNTER Reports Usage Reports Journal Report 1 Full text article requests by month and journal Journal Report 2 Turnaways by month and journal Database Report 1 Total searches and sessions by month and database Database Report 2 Turnaways by month and database Database Report 3 Searches and sessions by month and service

Web Services: the chosen approach for SUSHI Web services combine the best aspects of component-based development and the Web. Commercially accepted Widely supported (W3C) Secure … but first some definitions NISO has Web Services committee Amazon use web service to integrate book buying into other sites Thousands of others…

Definitions XML Schema (XSD) A language for describing the structure and constraining the contents of XML documents. (reactivity.com glossary)

Definitions XML Schema (XSD) A language for describing the structure and constraining the contents of XML documents. (reactivity.com glossary)

Definitions Web Services Open, standard (XML, SOAP, etc.) based Web applications that interact with other web applications for the purpose of exchanging data. (lucent.com)

Simple Object Access Protocol (SOAP) Definitions Simple Object Access Protocol (SOAP) SOAP is a lightweight XML based protocol used for invoking web services and exchanging structured data and type information on the Web. (oracle.com)

Definitions Web Services Description Language (WSDL) is an XML format published for describing Web services. (wikipedia.org)

Web Services: An example System A provides online information about companies. System B provides real-time stock quotations. Using Web Services, System A can integrate real-time stock quotes into the company information they provide.

System A sends the stock symbol to System B. Internet Online Company Data Real Time Stock Quotes (web service) Stock symbol

System B returns the quote. All of this happens in milliseconds. System A System B Internet Online Company Data Real Time Stock Quotes (web service) Stock symbol Stock quote

“Messages” are formatted in XML, and the protocol used to communicate is SOAP (Simple Object Access Protocol). System A System B Internet Online Company Data Real Time Stock Quotes Stock symbol SOAP SOAP: SOAP Version 1.2 is a lightweight protocol intended for exchanging structured information in a decentralized, distributed environment. Simple Object Access Protocol. SOAP is a lightweight XML based protocol used for invoking web services and exchanging structured data and type information on the Web. (Oracle) Web Service: Open standard (XML, SOAP, etc.) based Web applications that interact with other web applications for the purpose of exchanging data. (lucent) XML Schema: XML Schema is a language for describing the structure and constraining the contents of XML documents. (reactivity.com glossary) Stock quote

SUSHI : The Exchange Report Request Report Response <Requester> <Customer Reference> <Report Definition> Report Response <Requester> <Customer Reference> <Report Definition> <Report as payload>

SUSHI: Architecture The next series of slides graphically show a SUSHI transaction Library’s ERM system requests a usage report SUSHI client makes the request SUSHI server processes request SUSHI server prepares COUNTER report SUSHI server “packages” and returns response SUSHI client processes COUNTER report

The Library’s ERM and Content Provider’s systems are both connected to the internet. Library Content Provider Internet

The SUSHI client is software that runs on the library’s server, usually associated with an ERM system. Library Content Provider ERM Internet SUSHI Client

The SUSHI server is software that runs on the Content Provider’s server, and has access to the usage data. Library Content Provider ERM Internet SUSHI Client SUSHI Server (web service) Usage Data

When the ERM system wants a COUNTER report, it sends a request to the SUSHI client, which prepares the request. Library Content Provider ? ERM Internet Request SUSHI Client SUSHI Server (web service) Usage Data

The SUSHI request is sent to the Content Provider The SUSHI request is sent to the Content Provider. The request specifies the report and the library the report is for. Library Content Provider ? ERM Request Internet Request SUSHI Client SUSHI Server (web service) Usage Data

The SUSHI server reads the request then processes the usage data. Library Content Provider ? ERM Internet Request SUSHI Client SUSHI Server (web service) Usage Data

The SUSHI server creates the requested COUNTER report in XML format. Library Content Provider ? ERM Internet SUSHI Client SUSHI Server (web service) COUNTER Usage Data

A response message is prepared according to the SUSHI XML schema. Library Content Provider ? ERM Internet Response SUSHI Client SUSHI Server (web service) COUNTER Usage Data

The COUNTER report (XML) is added to the Response as its payload The COUNTER report (XML) is added to the Response as its payload. The response is sent to the client. Library Content Provider ? ERM Internet Response SUSHI Client SUSHI Server (web service) COUNTER Usage Data

The COUNTER report (XML) is added to the Response as its payload The COUNTER report (XML) is added to the Response as its payload. The response is sent to the client. Library Content Provider ? ERM Response Internet SUSHI Client SUSHI Server (web service) COUNTER Usage Data

The SUSHI client processes the response and extracts the COUNTER report. Library Content Provider ? ERM Response Internet SUSHI Client SUSHI Server (web service) COUNTER Usage Data

The extracted COUNTER report is passed to the ERM system for further processing. Library Content Provider  ERM COUNTER Internet SUSHI Client SUSHI Server (web service) Usage Data

Overview Background on usage statistics Why librarians collect them Timeline of standards Progression of improvements COUNTER SUSHI What it is How it works SUSHI and COUNTER: why they are important To libraries To publishers

Why COUNTER and SUSHI are important For libraries and publishers Usage statistics are being used to inform decisions They need to be consistent, credible and comparable And, easy to obtain SUSHI

More thoughts on usage statistics should enlighten rather than obscure should be practical are only part of the story should be used in context should be reliable

COUNTER and SUSHI Questions and answers

SUSHI What effect will release 3 of the COUNTER Code of Practice have on SUSHI? Rapid adoption of SUSHI due to it being a COUNTER compliance requirement New COUNTER schema will allow all COUNTER reports to be delivered through SUSHI using one schema Additional reports will help consortia

How many vendors are compliant with COUNTER codes of practice? SUSHI How many vendors are compliant with COUNTER codes of practice? Almost 100 vendor/products are compliant with Journals and Databases COP 10 vendors are compliant with Books and Reference Works See… http://www.projectcounter.org/compliantvendors.html

Where do I find the standard and more information about it? SUSHI Where do I find the standard and more information about it? NISO web site for SUSHI: http://www.niso.org/ (Select “Standards” and search for Z39.93) SUSHI Schemas: http://www.niso.org/schemas/sushi

What help is there for developers? SUSHI What help is there for developers? Toolkits for .NET (courtesy EBSCO) and JAVA (courtesy Swets) available on the NISO web site Recorded Webinars on the NISO web site Developer email list Contact either Oliver Pesch [opesch@ebsco.com] or Adam Chandler [alc28@cornell.edu] to be added

How big a project is it to create a SUSHI Server? If data COUNTER data is available, and developers are familiar with implementing web services in .NET or JAVA; then, the project is relatively small (weeks not months)

Thank you! Oliver Pesch opesch@ebsco.com