Presentation is loading. Please wait.

Presentation is loading. Please wait.

Understanding the Methodological Foundations of Public Library National Rating Systems Ray Lyons Library Statistics for the 21st Century World IFLA Satellite.

Similar presentations


Presentation on theme: "Understanding the Methodological Foundations of Public Library National Rating Systems Ray Lyons Library Statistics for the 21st Century World IFLA Satellite."— Presentation transcript:

1 Understanding the Methodological Foundations of Public Library National Rating Systems Ray Lyons Library Statistics for the 21st Century World IFLA Satellite Meeting - Montréal, Québec August , 2008

2 Milestones in Statistical Thinking Historical account by Alain Desrosières: Historical account by Alain Desrosières: The Politics of Large Numbers: A History of Statistical Reasoning (Harvard University Press, 1998) Traces statistical ideas from 15 th century to present day Traces statistical ideas from 15 th century to present day Evolutionary ideas from German descriptive statistics and English political arithmetic pertinent to library statistical collection Evolutionary ideas from German descriptive statistics and English political arithmetic pertinent to library statistical collection

3 Desrosières describes two key statistical practices: Desrosières describes two key statistical practices: Creation of equivalencesCreation of equivalences − Establishing standard classifications to describe phenomena by focusing on similarities and ignoring differences Encoding Encoding − Specification and use of definitions to assign individual cases to classifications Milestones in Statistical Thinking

4 Purpose of this Presentation To suggest that creating equivalent classes and other aspects of statistical data collection limit the accuracy and validity of national and local public library comparisons. To suggest that creating equivalent classes and other aspects of statistical data collection limit the accuracy and validity of national and local public library comparisons.

5 National Collection of Public Library Statistics in the USA Federal-State Cooperative System (FSCS) Federal-State Cooperative System (FSCS) Collaborative initiated in 1980’s Collaborative initiated in 1980’s First FSCS data published by U.S. Department of Education’s National Center for Educational Statistics First FSCS data published by U.S. Department of Education’s National Center for Educational Statistics

6 National Collection of Public Library Statistics in the USA Collection system renamed Public Library Statistics Cooperative (PLSC) in 2007 Collection system renamed Public Library Statistics Cooperative (PLSC) in 2007 PLSC recently transferred to Institute of Museum and Library Services, an agency of the federal government (www.imls.gov) PLSC recently transferred to Institute of Museum and Library Services, an agency of the federal government (www.imls.gov)

7 National Public Library Ratings Introduced in USA Hennen’s American Public Library RatingsHennen’s American Public Library Ratings(HAPLR) Used FSCS input and output statistics for public librariesUsed FSCS input and output statistics for public libraries Created by library consultant Thomas Hennen (library director in Wisconsin)Created by library consultant Thomas Hennen (library director in Wisconsin) Issued annually since 1999 except for 2000 and 2007)Issued annually since 1999 (except for 2000 and 2007)

8 Published (and endorsed) by American Library Association (ALA) Published (and endorsed) by American Library Association (ALA) Calculation methods were controversial among profession Calculation methods were controversial among profession Highly-rated public libraries were delighted with calculation methods Highly-rated public libraries were delighted with calculation methods “Methodologically indefensible and politically priceless” - US Library Director “Methodologically indefensible and politically priceless” - US Library Director Hennen’s American Public Library Ratings (HAPLR)

9 Utilizes 5 enabling (input) indicators and Utilizes 5 enabling (input) indicators and 3 use (output) indicators 3 use (output) indicators Enabling (Input) Total staff Materials expenditures Total operating expenditures Number of printed volumes Serial subscriptions Use (Output) Visits Circulation (loans) Reference transactions Hennen’s American Public Library Ratings (HAPLR)

10 Recombines 8 indicators into 15 rates (ratios), similar in style to BIX Recombines 8 indicators into 15 rates (ratios), similar in style to BIX (1) Total expenditures per capita (2) Materials expenditures per total expenditures (3) Materials expenditures per capita (4) Staff per 1000 population (5) Periodical subscriptions per 1000 population... and 10 others See See

11 Study Conducted at NCLIS 2006 research on HAPLR ratings methodology overseen by U.S. National Commission for Library and Information Science (NCLIS)2006 research on HAPLR ratings methodology overseen by U.S. National Commission for Library and Information Science (NCLIS) “Unsettling Scores: An Evaluation of the Hennen American Public Library Ratings, Public Library Quarterly, Volume 26, Numbers 3/4, “Unsettling Scores: An Evaluation of the Hennen American Public Library Ratings, Public Library Quarterly, Volume 26, Numbers 3/4, (ISSN ) (ISSN )

12 Library Journal’s Public Library National Ratings Announced LJ Index introduced June 2008 * LJ Index introduced June 2008 * Rates 9200 public libraries using PLSC data Rates 9200 public libraries using PLSC data Co-designed with Keith Lance ( Library Research Service, State Library of Colorado) Co-designed with Keith Lance ( Library Research Service, State Library of Colorado) Emphasizes disclosing limitations of rating methodology Emphasizes disclosing limitations of rating methodology Keith Curry Lance and Ray Lyons, “The New LJ Index,” Library Journal, June 15, 2008, p

13 Library Journal’s Public Library National Ratings Announced Encourages responsible interpretation of rating results Encourages responsible interpretation of rating results LJ Index based upon 4 use (output) indicators: LJ Index based upon 4 use (output) indicators: – Visits – Loans (circulation) – Internet terminal use – Program attendance

14 Public Library Ratings Use Composite Scoring HAPLR, BIX, and LJ Index each calculate a single composite score summarizing each library’s performance HAPLR, BIX, and LJ Index each calculate a single composite score summarizing each library’s performance Circulation per capita + FTE staffing + Subscriptions + Internet computers Total program attendance Calculation Algorithm 635 (Composite Score)

15 Context of Comparative Library Statistics Comparative measures are: Comparative measures are: − Measures used as part of a more general process to assess library value and effectiveness − Measures intended for use in an ongoing process of performance measurement

16 Performance Measurement Model Resources Services Utilized Intermediate Outcomes End Outcomes Outcome Measures Outcome Measures RESULTS EFFORTS

17 Planning-for-Results approach to management Planning-for-Results approach to management Abandonment of established operational and performance standards Abandonment of established operational and performance standards ALA / PLA 1987 publication, Output Measures for Public Libraries: A Manual of Standardized Procedures, defines standard statistics and collection procedures ALA / PLA 1987 publication, Output Measures for Public Libraries: A Manual of Standardized Procedures, defines standard statistics and collection procedures Performance Management as Envisioned by the Public Library Association (PLA)

18 Comparative Performance Measurement Public sector management practice Public sector management practice Used by state and local governments for: Used by state and local governments for: − Accountability − Planning/budgeting − Program monitoring − Operational improvement Uses established standards and benchmarking (Ammons, 2001) Uses established standards and benchmarking (Ammons, 2001)

19 Lack of reliable methods for identifying peer librariesLack of reliable methods for identifying peer libraries −Lack of instruments for measuring comparability Comparisons are either approximate or inaccurateComparisons are either approximate or inaccurate Can result in incorrect or misleading conclusionsCan result in incorrect or misleading conclusions Key Problems with Library Statistics

20 National rating systems use simplistic or imprecise criteria for identifying peers (library type, community size, etc.)National rating systems use simplistic or imprecise criteria for identifying peers (library type, community size, etc.) Ignore library mission, unique community needs, institutional context, etc.Ignore library mission, unique community needs, institutional context, etc. Accuracy and validity of comparisons are compromisedAccuracy and validity of comparisons are compromised Key Problems with Library Statistics

21 Lack of criteria for evaluating measures Lack of criteria for evaluating measures There are no ‘right’ or ‘wrong’ scores on an output measure; ‘high’ and ‘low’ values are relative. The scores must be interpreted in terms of library goals, scores on other measures, and a broad range of other factors. - Van House, Weill, and McClure (1990) Key Problems with Library Statistics

22 FIG NEWTON COOKIES Versus GRANOLA BAR SNACK 100% more fruit!!! A Diversionary ‘Visual Aid’

23 National rating systems apply the National rating systems apply the “More-is-Better Rule” “More-is-Better Rule” − Views higher numbers as favorable performance, lower as unfavorable “More activity does not necessarily “More activity does not necessarily mean better activity” - Van House, Weill, and McClure (1990) Striving to earn higher numbers may Striving to earn higher numbers may compromise service quality compromise service quality Key Problems with Library Statistics

24 Collection of standard statistics assumes all library resources/activities counted to be equivalent Collection of standard statistics assumes all library resources/activities counted to be equivalent Standardization ignores differences in: Standardization ignores differences in: Key Problems with Library Statistics - Complexity - Sophistication - Relevance - Quality (Merit) - Value (Worth) - Effectiveness - Efficiency - Significance

25 National ratings systems add, subtract, and multiply these non-equivalent ‘units’ of library resources, services, and products if they were equivalent National ratings systems add, subtract, and multiply these non-equivalent ‘units’ of library resources, services, and products if they were equivalent Final scores imply appear arithmetically consistent and correct even though they count unequal units Final scores imply appear arithmetically consistent and correct even though they count unequal units Key Problems with Library Statistics

26 Data imprecision due to Data imprecision due to − Inconsistent collection methods − Mistakes − Sampling error − “Gaming” Imprecision makes individual library comparisons less accurate Imprecision makes individual library comparisons less accurate Key Problems with Library Statistics

27 Variety of reasons for insufficient scores:Variety of reasons for insufficient scores: Key Problems with Library Statistics - Inadequate knowledge of community needs - Staff skill deficiencies - Inadequate staffing - Inefficient workflows - Inadequate planning - Limited user competencies... and others Adapted from Poll and te Boekhorst (2007)

28 Output measures “reflect the interaction of users and library resources, constrained by the environment in which they operate. The meaning of a specific score on any measure depends on a broad range of factors including the library’s goals, the current circumstances of the library and its environment, the users, the manner in which the measure was constructed, and how the data were collected.” [emphasis added] Output measures “reflect the interaction of users and library resources, constrained by the environment in which they operate. The meaning of a specific score on any measure depends on a broad range of factors including the library’s goals, the current circumstances of the library and its environment, the users, the manner in which the measure was constructed, and how the data were collected.” [emphasis added] - Van House, Weill, and McClure (1990) - Van House, Weill, and McClure (1990)

29 Improvements Needed Fuller understanding of limitations of statistical indicators and comparison methods Fuller understanding of limitations of statistical indicators and comparison methods “The [input and output] measures are best used with other information about the library.” “The [input and output] measures are best used with other information about the library.” - Van House, Weill, and McClure (1990) - Van House, Weill, and McClure (1990)

30 Relate amounts and types of resources and services to verifiable levels of community need Relate amounts and types of resources and services to verifiable levels of community need Increased understanding of measurement and interpretation Increased understanding of measurement and interpretation − Draw reasonable conclusions and interpretations − Study behavioral science measurement practices Improvements Needed

31 Behavioral Science Measurement Model Conceptualization Nominal Definition Operational Definition Measurement in Real World Babbie (2007)

32 Recommendations Promote view of public library national rating systems as ‘contests’ with simple, arbitrary rules Promote view of public library national rating systems as ‘contests’ with simple, arbitrary rules Ratings are: Ratings are: − Quite approximate indicators of performance − Inadequate indicators of quality, excellence, and effectiveness

33 Recommendations Advise libraries to interpret national ratings cautiously and conscientiously Advise libraries to interpret national ratings cautiously and conscientiously Develop and test measurement instruments for identifying peer libraries Develop and test measurement instruments for identifying peer libraries Use ratings to inspire more robust measurement of public library performance Use ratings to inspire more robust measurement of public library performance


Download ppt "Understanding the Methodological Foundations of Public Library National Rating Systems Ray Lyons Library Statistics for the 21st Century World IFLA Satellite."

Similar presentations


Ads by Google