Reading Discussion – s519 by Peter Hall – 1/21/09 Ryan, J., McClure, C. R., & Bertot, J. C. (2001). Choosing measures to evaluate networked information resources and services: Selected Issues. In C. R. McClure & J. C. Bertot (Eds.), Evaluating networked information services: Techniques, policy and issues (pp ). Medford, NJ: Information Today Inc.

Abstract IT and information managers see the importance in measuring their organizations networks and information flow important Two questions to consider – What are the key elements of a strategy that measures these networks? What are the key issues that may arise when employing such a strategy? Research suggests successful managers develop a strategy and consider key issues prior to implementing these measures and continuing with data collection, analysis, and use procedures

Examples of Measurements Quantitative (Statistics) Performance measures (Statistical ratios) Qualitative (Interviews, surveys, observations, shadowing, etc.) Any others?

The Research Sponsored by United States Institute for Museum and Library Services Multi-method approach – Site visits, case studies, interviews, focus groups, surveys, observation, path analysis Iterative approach – Study team adapted collection and analysis based on learning Limited study scope – Study participants were all library information managers Drawbacks and/or limitations?

Selection Issues What were some potential concerns and issues that these library information managers identified prior to adopting data collection, measures, and analysis? Why are you measuring something?  What are the potential uses?  Tradeoffs – New measures bring additional costs, time, and resources of an organization What will you do with your findings?  May have different outcomes and effects for different stakeholders in organization (i.e. management, employees, shareholders)

Measure Types Capacity – Ability of an organization to use or deliver information through network Use – Utilization measure of these resources or services Efficiency – Relates services or resources used to services or resources provided Impact – Effect of resource or service on an activity or situation Outcome – Are goals achieved? Examples of measure types? Other considerations  Defining measures, data collection, and analysis  Fair measures – All of an organizations operations

Data Collection Sources – Are all relevant stakeholders’ input collected? Can data be collected uniformly for accurate analysis? Which data collection measures should be used? Qualitative, quantitative, both? Any new collection techniques that should be used? Privacy – Are stakeholders free to give an accurate portrayal? Balancing data collection across organization Proper instruction and preparation is key Any other potential issues?

Data Analysis and Use Combine new data with existing information – Internal, external  Example – Your new data with national studies or internal performance or composite measures  Is this information compatible for comparison?’ How do you analyze the results generated by a new measure?  Is further instruction necessary?  Who interprets and how?  How is this data presented?  How do you use it? Does the end cost outweigh the benefit?  Table Examples

Conclusions / Recommendations Information managers should choose network measures based on context of change, complexity, utility, and resources Proper strategy required prior to implementation of network measures, data collection, and analysis for success Reducing and balancing staff burden of collection and analysis Be able to provide proper instruction/training to properly educate important stakeholders Future and additional considerations and research

Discussion Topics Has anyone been involved in this process in their organizations?  Success/failures, most important metrics, time consumption, cost/benefits?  Did anything come out of this data? Any other analytics tools used?  i.e. External, web-based tools  Internal work flow documents Thoughts on the reading – Agree/disagree, glaring deficiencies, techniques used, research methods?

Work Flow Example