Waves of Numbers: How to Use and Interpret Benchmarking Tools Presentation given at MLA-San Diego, May, 2003 Revised as a PowerPoint Show for instructional purposes by Rosalind. F. Dudden, AHIP Health Sciences Librarian Gerald Tucker Memorial Medical Library National Jewish Medical and Research Center Denver, CO

MLA Benchmarking Network2May, 2003 Waves of Numbers This slide show will move forward at a steady pace. Text will fade in and out. I have moved my words into the slide to mimic my speech. I hope it works out for you. Lets start the show! You can go faster using the right arrow, repeat with the left arrow. The show lasts 12 minutes.

MLA Benchmarking Network3May, 2003 Waves of Numbers In todays world, one is bombarded with numbers, waves of them coming at you. I thought this title was appropriate since the Benchmarking Network has just provided you with over 46 million (46,089,864) new numbers. You can use these numbers to defend or improve your library service. My subtitle mentions use and interpretation of these numbers and I am attempting to cover both. It is much easier to show people how to use the tools than it is to interpret them.

MLA Benchmarking Network4May, 2003 Background From December 2001 to March 2002, MLA members were encouraged to provide data on library services and resources using a Web intake form. The data was edited. Tables representing the aggregate data of 344 hospital libraries were developed and made available to members on MLANET in September, 2002 In March 2003, an interactive site became available to survey participants on MLANET. (For a fee to all members)

MLA Benchmarking Network5May, 2003 MLA Benchmarking Network Outcomes Team, Rosalind Dudden, Chair Statistical Consultation Jeff Magouirk, Dept of Bio-statistics, National Jewish Medical and Research Center, Denver, CO Jim Mills, Jim Mills Engineering Services, Denver, CO Data Collection and Report Programming MLA Headquarters team, lead by Kate Corcoran Benchmarking Implementation Task Force, Debbie Rand, Chair Benchmarking Network Task Force, Bernie Todd Smith, Chair Lets give some credit to all these hardworking MLA volunteers, Right up front!

MLA Benchmarking Network6May, 2003 MLA Benchmarking Network Several people have asked: Why do you want to know this? or What is your question? … My response to Why: To have data to present if I am asked by the administration to prove that the library operations are similar to other libraries of comparable size. or To improve services through process improvement

MLA Benchmarking Network7May, 2003 MLA Benchmarking Network My response to What is your question? is that everyone has a different question and we need to be prepared to answer it. One question is: What do librarians do? For instance, you could ask On average, how many monographs do hospital libraries circulate?

MLA Benchmarking Network8May, 2003 What do we do? On average that is… Problem: Very large/very small libraries distort the average. Number of Responses MeanMedianMinimumMaximum Quartile 1 Quartile 2 Quartile 3 Quartile 4 To find an average, one could divide the data by the size of the library and then find the average, or mean, of each group.

MLA Benchmarking Network9May, 2003 Annual Circulation qualified answers meanmedianthird quartile All2421, ,918 ! But I have 4 FTEs in my library and I circulate 2,835! ! But I work in a 1 person library and I circulate 471! Problem: Very large/very small libraries distort the average. The answer for all 242 libraries that responded: The outcomes team developed a system of tables to display the data to take this variation into account.

MLA Benchmarking Network10May, 2003 Vocabulary - Parameters Parameter A numerical value describing a characteristic of a population Hospital parameters Physicians (pa05) Hospital FTE (pa06) Patient Discharges (pa07) Outpatient Visits (pa08) Staffed Beds (pa09) Patient Admissions (pa10) Library parameters Total Library FTE (a01) Total Expenditures (a21) 12 parameters of size are used. First we identified parameters of size for a hospital, training programs, and the hospital library itself.

MLA Benchmarking Network11May, 2003 Distribution of a Parameter Number of libraries in each parameter range for Hospital FTEs: Ranges 1 and 8: Extreme Outliers, top and bottom - 2.5% Ranges 2 and 7: More Outliers, top and bottom - 10 % Ranges 3 to 6: Remaining 75% divide into Quatriles The parameters were distributed over 8 ranges to account for the outliers but not eliminate them. It is an acceptable statistical method to eliminate outliers but these very large and very small hospitals need a chance to benefit from the statistics. The middle ranges need more comparable statistics. No outliers were eliminated from the parameters.

MLA Benchmarking Network12May, 2003 Vocabulary - Measures Examples of the 73 measures of library activity collected, as developed by the Benchmarking Content Team. For each measure, this data is reported: Number Mean Median Third Quartile Maximum Minimum Reference Questions (ps01) Mediated Searches (ps02) Educational Program Sessions (ps04) Monographs Circulated (ps06) Item Received from Outside Sources (Borrows) (ps08) Item Sent to Outside Sources (Lends) (ps09) Expenditures - Monographs (a14) Expenditures - Print Serials (a15) Print Monographs (ts01) Current Print Serials Titles (ts02) Print Serials Titles/Electronic Full-Text Access (ts03) Serials Titles/Electronic Full-Text Access Only (ts04) Measures Developed: The third quartile or 75% mark, was recommended for use in TQM or QI programs.

MLA Benchmarking Network13May, 2003 Measures - by - Parameters You put these two concepts together: number of ILL borrows (ps08) by number of hospital FTEs (pa06) You produce a table that looks like the next slide. Note that the each of the 4 middle parameters have about 60 libraries. The total, or All, line shows 311, which means that 311 libraries out of 344 participants answered BOTH questions, how many FTEs and how many ILL borrows.

MLA Benchmarking Network14May, 2003 Number of ILL borrows (ps08) by Number of hospital FTEs (pa06) Hospital FTEsqualified answers meanmedianthird quartile maximumminimum Range 1: 0 to ,50072 Range 2: 500 to ,0482,64745 Range 3: 850 to 1,349591, ,2174,20096 Range 4: 1,350 to 1,824621,2821,1001,5734, Range 5: 1,825 to 2,599611,7261,3142,0006, Range 6: 2,600 to 4,249582,2061,6172,9877, Range 7: 4,250 to 7,999263,0382,8364,1556, Range 8: 8,000 or more86,2987,0319,46110, All3111,7041,2002,00210,69745

MLA Benchmarking Network15May, 2003 How many tables are there? 12 Parameters Multiplied by 73 Measures 876 Tables on the MLANET Members Only Website Available to all MLA members! Each table has 6 columns and 9 rows 12 x 73 x 6 x 9 = 46,089,864 numbers Which table is going to prove your case, or inspire you to improve you system?

MLA Benchmarking Network16May, 2003 Demonstration To demonstrate the Benchmarking tools on the MLA Website, what follows are some slides and screen shots from the Website. First we need a question. And we need to know how the answer is going to be used. You can often find an answer to a question, but to interpret the answer, one usually needs to know why it is being asked.

MLA Benchmarking Network17May, 2003 Question: How does the number of Interlibrary Loan Borrows relate to the number of : Physicians Hospital FTEs Training Positions? Which is the best parameter to use when describing my library to the administration? Which parameter, if it changes, will impact the library workload?

MLA Benchmarking Network18May, 2003 Benchmarking Access Get out your membership card Use Internet Explorer as a browser Go to URL: Click on Members Only Enter UserID Enter Password

MLA Benchmarking Network19May, 2003 Benchmarking Access Choose Benchmarking Network Note other statistical surveys: MLA/Hay Group Study Salary Survey

MLA Benchmarking Network20May, 2003 Benchmarking Access Interactive Site Aggregate Tables - First demonstration Choose -->

MLA Benchmarking Network21May, 2003 Benchmarking Access Aggregate Data Collected December 2001 to March 2002 Composite Hospital Library Tables Index to the tables Explanation of Statistics Used In-depth explanation of the tables Methodology Description to use when asked how the figures were gathered Worksheet of 2001 Benchmarking Questions To review to see what data was gathered Choose Resources available to you:

MLA Benchmarking Network22May, 2003 Choose a measure Administrative Services Measures (Financial), by Administrative Services Measures (Staffing and Other), by Public Services Measures, by Technical Services Measures, by Special Services Measures, by Composite Hospital Library Tables This page is divided into five parts; We assume that ILL questions are under Public Service Measures and scroll down to that area. We get a list that looks like the next slide.

MLA Benchmarking Network23May, 2003 Choose a measure This screen shot shows that the tables on the page selected will show the number of borrows.

MLA Benchmarking Network24May, 2003 Worksheet of 2001 Benchmarking Data Public Service Measures Resource Use PS07. Interlibrary loaning/borrowing: indicate whether or not your library is an official LOANSOME DOC provider. PS08. Indicate the approximate NUMBER of items OF ALL TYPES your library borrows or receives from outside sources annually (include ILL AND commercial document delivery services. PS09. Indicate the approximate NUMBER of items OF ALL TYPES your library lends or sends to outside sources annually. You could also have found out which section the question was in by looking at the Data Worksheet.

MLA Benchmarking Network25May, 2003 Choose a parameter To find the data on these three parameters you need to link into each parameter separately. These are the twelve parameters, 6 for hospital size, 4 for teaching hospitals, and two for library size.

MLA Benchmarking Network26May, 2003 To Review the Question: How does the number of Interlibrary Loan Borrows relate to the number of : Hospital FTEs Physicians Training Positions? Assumptions Hospital FTEs 1500 Physicians 600 Training Positions 20 Borrowed items 1,245

MLA Benchmarking Network27May, 2003 To Review: Choose a measure 1. Administrative Services Measures (Financial), by 2. Administrative Services Measures (Staffing and Other), by 3. Public Services Measures, by 4. Technical Services Measures, by 5. Special Services Measures, by

MLA Benchmarking Network28May, 2003 To Review: Choose a parameter Number of Physicians in the Hospital (pa05) Number of Hospital Full Time Equivalent (FTE) Employees (pa06) Number of Patient Discharges Annually (pa07) Number of Outpatient Visits Annually (pa08) Number of Staffed Beds (pa09) Number of Patient Admissions Annually (pa10) Number of Postgraduate Training Positions (pa11) Number of Medical School Clerkships (pa12) Number of Nursing Student Positions (pa14) Number of Allied Health Student Positions (pa16) Number of Total Library FTEs (a01) Total Annual Expenditures for the Library (a21)

MLA Benchmarking Network29May, 2003 Interlibrary Loan Borrows by Hospital FTEs The average for this hospital (Hospital FTEs = 1500) is 1,282.

MLA Benchmarking Network30May, 2003 Interlibrary Loan Borrows by Hospital FTEs RangeHospital FTEsqualified answers meanmedianthird quartile maxmin 10 to , to ,0482, to 1,349591, ,2174, ,350 to 1,824621,2821,1001,5734, ,825 to 2, ,7261,3142,0006, ,600 to 4,249582,2061,6172,9877, ,250 to 7,999263,0382,8364,1556, ,000 or more86,2987,0319,46110, All3111,7041,2002,00210, libraries answered both questions. The middle 4 ranges have 58 libraries and over. The merged data of 58 libraries gives the answers a high confidence level. The librarian could use the mean or median. When the data is spread as widely as this, from 216 to 4,400, generally the median is a more representative number. In her range of Hospital FTEs, 1500, the median number is 1,100 borrows.

MLA Benchmarking Network31May, 2003 Collecting the Data Remember the librarian wants to compare three tables. She can do this visually, or print them out, or she can copy them and put them in a spread sheet. The next slides show how to export the data to Excel.

MLA Benchmarking Network32May, 2003 Moving data to a Spreadsheet Using Internet Explorer -- Highlight the table and Copy

MLA Benchmarking Network33May, 2003 Moving data to a Spreadsheet Open up an Excel Spreadsheet -- Put your cursor in the first cell -- Paste After you paste the three tables into Excel, you can move the data around to produce comparative tables.

MLA Benchmarking Network34May, 2003 Working in a Spreadsheet

MLA Benchmarking Network35May, 2003 Median for Each Range Number of physicians N=57 Number of hospital FTEs N=62 Number of training positions N=42 Range ,162 Range Range ,224 Range 41,2001,1001,392 Range 51,1001,3141,622 Range 61,2801,6171,536 Range 72,4362,8363,218 Range 85,5157,0316,136 all1,1681,2001,439 This is the data that produced that chart on the previous slide.

MLA Benchmarking Network36May, 2003 Interpretation - The difficult part! - I may be wrong! Question (review): How does the number of Interlibrary Loan Borrows relate to the number of Hospital FTEs, Physicians, Training Positions? Interpretation: Given that the training program figure was not only higher but closer to the librarys own figure of 1245, any personnel cuts or additions in that program would most likely impact the workload more than the other two parameters.

MLA Benchmarking Network37May, 2003 Benchmarking Access Part 2 - Interactive Site Go to URL:

MLA Benchmarking Network38May, 2003 Benchmarking Access Choose -->

MLA Benchmarking Network39May, 2003 Benchmarking Access This takes you to a page where, if you are a participant, you can login. Any MLA member can buy access to the site.

MLA Benchmarking Network40May, 2003 Step One: Demographic Selection Benchmarking Network Report Selection You can chose to match your fiscal year to the results. You can choose to use only your state or region. But if you are from Utah, only one person participated and that person was you. But if you are from the Midcontinental region, 29 libraries participated.

MLA Benchmarking Network41May, 2003 Benchmarking Network Report Selection Step Two: Select One or More Criteria Similar to Your Institution You can chose to look as system data. 99 hospital classified themselves as systems. If being, or not being, a teaching hospital is important to your operation, this is the only place to choose this option. It was not possible to choose this parameter on the tables.

MLA Benchmarking Network42May, 2003 Benchmarking Network Report Selection TeachingNon- Teaching An example of output from the interactive tables. By choosing teaching hospital, you are automatically limiting your retrieval to less than 225 hospitals.

MLA Benchmarking Network43May, 2003 Benchmarking Network Report Selection Step Two: Select One or More Criteria Similar to Your Institution Physicians Range 5: 500 to 699 Hospital FTEs Range 4: 1,350 to 1,824 Training positions Range 3: 15 to 29 By choosing the parameter for training positions, we are limiting ourselves to teaching hospitals, so we chose I dont care. We scroll down and choose the three parameters we want. Here we have chosen range 5 for number of physicians. We would then scroll down to choose range 4 for number of hospital FTEs and range 3 for number of training positions.

MLA Benchmarking Network44May, 2003 Benchmarking Network Report Selection Step Three: Data Category Selection Here you select your set of measures. All the measures for each section are displayed on the results page. The results page is too long to display in a slide show.

MLA Benchmarking Network45May, 2003 Benchmarking Network Report Results Based on your selections, there were 3 institutions that matched your criteria. If fewer than five institutions have met your criteria, they are not identified here, for confidentiality reasons. To increase the number of institutions matching specific criteria you choose, go back to the selection page and use fewer criteria.

MLA Benchmarking Network46May, 2003 Benchmarking Network Report Results Total Number of items received from outside sources annually (borrows) ps08 Mean (average) MedianThird Quartile (75%) Maximum With the three hospitals, one of them must be you, so you are comparing yourself to only two other hospitals. With only three results and the median and maximum the same, one can hypothesize that two libraries are reporting 1245 and one fewer than 904. So you go back to the selection page.

MLA Benchmarking Network47May, 2003 Benchmarking Network Report Selection Physicians Range 5: 500 to 699 Hospital FTEs Range 4: 1,350 to 1,824 We chose all teaching hospitals instead of a parameter for number of training positions. We scroll down and choose the other two parameters we want. Here we have chosen range 5 for number of physicians. We would then scroll done choose range 4 for number of hospital FTEs.

MLA Benchmarking Network48May, 2003 Benchmarking Network Report Results Based on your selections, there were 9 institutions that matched your criteria, including your own (marked by an asterisk). They are: 1Pomona Valley Hosp. Medical Ctr.Pomona, CA 2Presbyterian Intercommunity HospitalWhittier, CA 3St. Vincent HospitalWorcester, MA 4Durham Regional HospitalDurham, NC 5Medical Center at PrincetonPrinceton, NJ 6Mercy Fitzgerald HospitalDarby, PA 7The Westerly HospitalWesterly, RI 8Primary Children's Medical CenterSalt Lake City, UT 9Waukesha Memorial HospitalWaukesha, WI This time you get nine libraries from around the country. They are similar in size but do not necessarily have the same number of training positions. From this list and others that you produce, you could select benchmarking partners to do process improve projects, as described in the Fact Sheet available on the MLA Benchmarking Website.

MLA Benchmarking Network49May, 2003 Benchmarking Network Report Results Total Number of items received from outside sources annually (borrows) ps08 Teaching Hospitals; Physicians - Range 5; Hospital FTEs - Range 4 Mean (average) MedianThird Quartile (75%) Max N=9 In these results, even with the large maximum, the mean and median are not that far apart so here the median again shows that your number of borrows compares of others. It is by chance the same number. Nine teaching hospitals with the same range of Hospital FTEs and Physicians have data comparable with your activity. Unfortunately, the chart does not work out as a visual.

MLA Benchmarking Network50May, 2003 Benchmarking Network Report Results Total Number of items received from outside sources annually (borrows) ps08 All Hospitals; Physicians - Range 5; Hospital FTEs - Range 4 Mean (average) MedianThird Quartile (75%) Max N=12 If you go back to the interactive site and try again, here are some different results. This time you choose all hospitals to see what the result will be. This time, 12 hospitals with the same range of Hospital FTEs and Physicians have data comparable with yours activity, but slightly higher.

MLA Benchmarking Network51May, 2003 Interpretation - The difficult part! - I may be wrong! Given 9 teaching hospitals with the same range of numbers of hospital FTEs and physicians, the number of borrows was similar to ours. Given 12 hospitals, teaching or not, the indicators are higher. Still use the Training Positions as an indicator

MLA Benchmarking Network52May, 2003 Conclusion Have your question in mind all the time. Know what questions were asked. Take you time surveying the aggregate tables that answer your question. Use the interactive site to refine your parameters of size. Use the interactive site to identify Benchmarking Partners for process analysis.

MLA Benchmarking Network53May, 2003 Assessment and Benchmarking Special Interest Group First Meeting Mon, May 5, 2003 Co-Conveners: Linda Garr Markwell Lyn Dennison 1. Provide a forum for exchange of knowledge about assessment, benchmarking and outcome measures. 2. Promote the exchange of ideas and experience in implementing assessment, benchmarking, and outcome measures. 3. Collaborate with Sections to encourage MLA annual programming relevant to the interests of this SIG and promote professional development for librarians in this area.

MLA Benchmarking Network54May, 2003 MLA Benchmarking Network