L643: Evaluation of Information Systems Week 4: January 28, 2008

Measurement Measurement is “the assignment of values to outcomes following a set of rules” E.g., Measurement of tastiness Good Bad E.g., measurement of size 2.5 inches 1 inch

Conceptualization (Chambliss & Schutt, 2006) Define the concept (or purpose) e.g., PDA vs. Laptop in a hospital Identify variables that correspond to the concept e.g., ???? Determine how we can measure these variables [use available data, construct Qs, make observations, content analysis]

Scales of Measurement (Salkind, 2007) Nominal – attributes are named E.g.: nationality (American, Canadian, British) Ordinal – attributes can be ordered E.g., level of use (Low, High) Interval – distance is meaningful E.g., temperature in Fahrenheit (11, F, 30 F, 50 F) Ratio – absolute zero is included E.g., # of clients in 6 months (0, 10, 30)

Reliability of Measurement A reliable measurement is one that gives consistent results Test-retest reliability Parallel forms reliability Internal consistency reliability Interrater reliability

To Increase Reliability Standardized instructions Adequate sample size Avoid unclear items/vague statement Adjust difficulties Minimize external factors

Validity of Measurement Validity = It measures what it is supposed to measure Content validity (by expert) Criterion validity (by a set of existing criteria) Construct validity (by conceptual theory)

Reliability vs. Validity

Population & Sample (Chambliss & Schutt, 2006) Others A B C Others

Sample Size Large Errors Small Small Large Size of Sample

Sampling What is a sample? In a research context, “any group on which information is obtained” The larger group to which one hopes to apply the results is called the population E.g., All 700 students at a State University who are majoring in mathematics, constitutes a population E.g., 50 of those students constitute a sample Random sampling vs. non random sampling

Random Sampling the premise that a sample represents a population is based on the assumption that the sample has been selected at random from the population of interest

Definition of An Information System (Davis, 1994) A set of hardware, software, data, procedural, and human components that work together to generate, collect, store, retrieve, process, analyze, and/or distribute information The purpose of an IS is to get the right information to the right people at the right time Davis, W. S. (1994). Business Systems Analysis and Design. Belmont, CA: Wadsworth.

Information System Objectives Why do we need to consider system objectives? Each system will be designed to meet certain performance objectives These objectives affect the design of evaluation: They are the evaluation criteria

Information System Objectives Which objectives are mentioned or implied in each of the cases? John Fluevog Boots & Shoes Zipcar LibraryThing

A Taxonomy of Information Systems Success System Quality Measures of the information processing system itself Information Quality Measures of information system output , e.g., accuracy, meaningfulness, and timeliness Information/System Use Recipient consumption of the output of an information system

A Taxonomy of Information Systems Success User Attitudes Satisfaction Recipient response to the use of the output of an information system Individual Impact The effect of an information system on the behavior of the recipient Organizational Impact The effect of the information product on organizational performance

Observations by DeLone & McLean (1992) There is no consensus on particular measure of information systems success Simplifying different dependent variable measures would contribute to the MIS research (the results can be compared) Few field studies to measure organizational performance 6 success categories indicate MIS success is a multidimensional construct & that it should be measured as such (Only 28/100 studied multiple categories)

Independent vs. Dependent Variables (presumed or possible cause) Dependent Variable(s) (presumed results) affects Examples of variables Survey format—online, paper-based (IV) Response rates (DV)

Measuring the value of information (Ahituv, 1989) Real World Data Information system Decision maker Decisions, actions Outcomes Point of measurement perceived value of information Point of measurement realistic value of information Ahituv, N. (1989). Assessing the value of information: Problems and approaches. Proceedings of the Tenth International Conference on Information Systems, December 4-6, Boston, 315-325.

IS Success Model (DeLone & McLean, 1992)

I/S Success Model, Expanded (based on Hwang, Windsor, & Pryor, 2000) 0.37 System quality Use Individual impact Organizational impact InfoSys Information quality User satisfaction 0.37 0.43 0.47 0.36 User environment Organizational environment

Updated D&M IS Success Model (2002, 2003) Information Quality Intention To Use Use Net Benefits System Quality User Satisfaction Service Quality Creation Use Consequences

Evaluation The man with the tusk states that an elephant is like a spear. The man with the tail argues that the elephant is like a rope. The man with the trunk says, no, its like a snake. The man with the side thinks its more like a wall. But the man with the leg is sure the elephant is like a tree. The flaw in all their reasoning is that speculating on the WHOLE from too few FACTS can lead to VERY LARGE errors in judgment. How does this fable relate to System Evaluation?

Values in Evaluation (Davidson, 2005) Suppose that a client does not like the findings of your evaluation and says, “Well, that’s just your opinion about the program. Evaluations are always just so subjective.” How would you respond?

Class Activities: Evaluation of IUCAT Write down the assigned category on the worksheet. Individually fill out the worksheet #1 - #3 (p. 1) Work with a group of 5 people to fill out the worksheet for the category (p.2) Present your discussions to the whole class Use the worksheet (p.2) to jot down the main points