1. Qualitative Methods in Assessing Libraries, Users and ICT Applications Dr. R. Sevukan Associate Professor & Head Dept. of Library & Information Science Pondicherry University Puducherry, India sevukan2002@yahoo.com NACLIN, 9-11 December, 2014

2. What is an Assessment? It is the process of gathering information from the stakeholders using various methods to systematically gauge the effectiveness of an institution. It is the process of gathering information from the stakeholders using various methods to systematically gauge the effectiveness of an institution.

3. Why is assessment important? Are we delivering what we think we are delivering? Are we delivering what we think we are delivering? Are customers getting what they are supposed to be getting? Are customers getting what they are supposed to be getting? Are customers satisfied with what they are provided? Are customers satisfied with what they are provided? Is there a way to deliver the information better, thereby promoting satisfaction? Is there a way to deliver the information better, thereby promoting satisfaction?

4. Answers to these questions Information sources needed for research, teaching, and learning; Information sources needed for research, teaching, and learning; Reasons and frequency of library use; Reasons and frequency of library use; Campus computer network connectivity; Campus computer network connectivity; Use of electronic resources; Use of electronic resources; Instructional needs and effectiveness; Instructional needs and effectiveness; Satisfaction on the services available; and Satisfaction on the services available; and Enhance the quality of services rendered. Enhance the quality of services rendered.

5. Methods of Assessment (Source: Niglas, 1999)

6. Qualitative Vs. Quantitative Qualitative Research is primarily exploratory research used to gain an understanding of underlying reasons, opinions, and motivations. It provides insights into the problem or helps to develop ideas or hypotheses for potential quantitative research. Qualitative Research is primarily exploratory research used to gain an understanding of underlying reasons, opinions, and motivations. It provides insights into the problem or helps to develop ideas or hypotheses for potential quantitative research. Qualitative Research Qualitative Research Quantitative Research is used to quantify the problem by way of generating numerical data or data that can be transformed into useable statistics. It is used to quantify attitudes, opinions, behaviors, and other defined variables – and generalize results from a larger sample population Quantitative Research is used to quantify the problem by way of generating numerical data or data that can be transformed into useable statistics. It is used to quantify attitudes, opinions, behaviors, and other defined variables – and generalize results from a larger sample population Quantitative Research Quantitative Research

7. Qualitative Methods: Characteristics  Naturalistic – Investigate social processes as they unfold naturally rather than reasoning from either the conditions under which they occur or the outcomes that correlate with them  Emergent – Willingness to adapt methods of analysis as knowledge and understanding deepens  Purposeful – Cases are selected because they are “information rich” Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3 rd ed. (Thousand Oaks, CA: Sage, 2002), pp. 40- 41.

8. Data Collection Characteristics of Qualitative Method  Thick description – The goal is deep understanding about phenomena and experiences.  Personal experience and engagement – The researcher is the primary instrument of data collection and analysis.  Empathic stance – Sensitivity, respect, awareness, responsiveness, and openness.  Attention to process – Situational and contextual changes occur throughout the life of the study.

9. Analytical Characteristics of Qualitative Method  Unique case orientation – Every case is special and unique  Inductive and creative – Research process guided by analytical principles rather than rules  Holistic perspective – Phenomena under study are complex and cannot be reduced to a few variables  Context sensitivity – Avoid broad generalizations; investigation does not rest in isolation from the larger social/ cultural/ economic/political environment  Reflexivity – Critical self-awareness of one’s own presence in the research process

10. Selective Applications in Library Settings  To assess individualized outcomes  To explore internal dynamics of programs and services  To evaluate critical cases of program or service utilization  To acquire details about program implementation  To obtain information about the nuances of program or service quality  To personalize the evaluation process and/or outcomes assessment  To discover effects of specific programs or services  To add depth, detail, and meaning to statistical data  To provide privileged access to processes, causes, and effects. Source: Patton, Michael Q. How to Use Qualitative Methods in Evaluation. 1 st ed. (Thousand Oaks, CA: Sage, 1987), pp. 41-42.

11. Some Techniques for Collecting Assessment Data In-Depth Interviews – speaking with and listening to our users Observation – witness what is going on in a particular setting and explore the experiences of others Qualitative Case Studies – holistic understanding of social phenomena in library settings

12. In-Depth Interviews Outcomes for Assessment What is meaningful or important in the respondent’s own voice? What is meaningful or important in the respondent’s own voice? Researcher’s knowledge, expertise, and interpersonal skills can be used to explore interesting or unexpected concepts or issues raised by respondents Researcher’s knowledge, expertise, and interpersonal skills can be used to explore interesting or unexpected concepts or issues raised by respondents Opportunity for deep probing of issues Opportunity for deep probing of issues Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3 rd ed. (Thousand Oaks, CA: Sage, 2002)

13. Observation: Outcomes for Assessment Can facilitate multidimensional and long-term imbeddedness in the “natural setting” Can facilitate multidimensional and long-term imbeddedness in the “natural setting” Melds together aspects of looking and listening and watching and asking (connect what people are saying to what they are actually doing) Melds together aspects of looking and listening and watching and asking (connect what people are saying to what they are actually doing) Unmasks everyday interaction between people and organizations; reveals the mundane Unmasks everyday interaction between people and organizations; reveals the mundane Cases can be revisited repeatedly Cases can be revisited repeatedly

14. Case Studies Design Considerations  Individuals – comparative sample of individual experiences  Groups – sample of individuals who share a common identity  Communities – sample of individuals with ethnic, cultural, spiritual or other types of characteristics in common  Programs – comparative sample of projects or services intended to meet a public need  Policies – comparative sample of plans of action  Organizations – comparative sample of a formal group of people with one or more shared goals

15. Qualitative Case Studies Applications for Assessment Illustrative: Descriptive and intended to add “realism” and in-depth examples to other information about a program or policy. Illustrative: Descriptive and intended to add “realism” and in-depth examples to other information about a program or policy. Exploratory: Also descriptive, but aimed at generating hypotheses for later investigation rather than for purposes of illustrating. Exploratory: Also descriptive, but aimed at generating hypotheses for later investigation rather than for purposes of illustrating. Deviant/Critical Instance: Examines a single instance of unique interest or serves as a critical test of an assertion about a program, problem, or strategy. Deviant/Critical Instance: Examines a single instance of unique interest or serves as a critical test of an assertion about a program, problem, or strategy. Program Implementation: Investigates operations, often at several sites, and with a subjective, value laden, and/or affective emphasis. Program Implementation: Investigates operations, often at several sites, and with a subjective, value laden, and/or affective emphasis. Program Effects: This application uses the case study to examine causality and usually involves multi-site, multi-method assessments. Program Effects: This application uses the case study to examine causality and usually involves multi-site, multi-method assessments. Cumulative: This brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause- and-effect. Cumulative: This brings together findings from many case studies to answer an evaluation question, whether descriptive, normative, or cause- and-effect.

16. Assessment Tools: A Glance

19. The Challenge of Assessment in Libraries Traditional statistics emphasize inputs, expenditures, acquisitions, holdings, etc. Traditional statistics emphasize inputs, expenditures, acquisitions, holdings, etc. Helping funding agencies understand success of investment Helping funding agencies understand success of investment No demonstrable relationship between expenditures and service quality No demonstrable relationship between expenditures and service quality Lack of metrics describing outcomes: success from the user’s point of view Lack of metrics describing outcomes: success from the user’s point of view Need to redesign library services to better meet changing patterns of use Need to redesign library services to better meet changing patterns of use Building the climate, tools, and skill set for library assessment Building the climate, tools, and skill set for library assessment

20. No Single Method!!! “The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable – i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two.” “The difficulty lies in trying to find a single model or set of simple indicators that can be used by different institutions, and that will compare something across large groups that is by definition only locally applicable – i.e., how well a library meets the needs of its institution. Librarians have either made do with oversimplified national data or have undertaken customized local evaluations of effectiveness, but there has not been devised an effective way to link the two.” Sarah Pritchard, Library Trends, 1996

21. Successful Model Mixed Methods Assessment Tools Qualitative Methods Quantitative Methods

22. “Different is not always better but better is always different”

23. “Let us replicate but not duplicate research” --- H.K.Kaul