1. L643: Evaluation of Information Systems
Week 4: January 28, 2008

2. Measurement
Measurement is “the assignment of values to outcomes following a set of rules”
E.g., Measurement of tastiness
Good
Bad
E.g., measurement of size
2.5 inches
1 inch

3. Conceptualization (Chambliss & Schutt, 2006)
Define the concept (or purpose)
e.g., PDA vs. Laptop in a hospital
Identify variables that correspond to the concept
e.g., ????
Determine how we can measure these variables [use available data, construct Qs, make observations, content analysis]

4. Scales of Measurement (Salkind, 2007)
Nominal – attributes are named
E.g.: nationality (American, Canadian, British)
Ordinal – attributes can be ordered
E.g., level of use (Low, High)
Interval – distance is meaningful
E.g., temperature in Fahrenheit (11, F, 30 F, 50 F)
Ratio – absolute zero is included
E.g., # of clients in 6 months (0, 10, 30)

5. Reliability of Measurement
A reliable measurement is one that gives consistent results
Test-retest reliability
Parallel forms reliability
Internal consistency reliability
Interrater reliability

6. To Increase Reliability
Standardized instructions
Adequate sample size
Avoid unclear items/vague statement
Adjust difficulties
Minimize external factors

7. Validity of Measurement
Validity = It measures what it is supposed to measure
Content validity (by expert)
Criterion validity (by a set of existing criteria)
Construct validity (by conceptual theory)

9. Reliability vs. Validity

10. Population & Sample (Chambliss & Schutt, 2006)
Others A B C
Others

11. Sample Size
Large
Errors
Small
Small
Large
Size of Sample

12. Sampling What is a sample?
In a research context, “any group on which information is obtained”
The larger group to which one hopes to apply the results is called the population
E.g., All 700 students at a State University who are majoring in mathematics, constitutes a population
E.g., 50 of those students constitute a sample
Random sampling vs. non random sampling

13. Random Sampling
the premise that a sample represents a population is based on the assumption that the sample has been selected at random from the population of interest

14. Definition of An Information System (Davis, 1994)
A set of hardware, software, data, procedural, and human components that work together to generate, collect, store, retrieve, process, analyze, and/or distribute information
The purpose of an IS is to get the right information to the right people at the right time
Davis, W. S. (1994). Business Systems Analysis and Design. Belmont, CA: Wadsworth.

15. Information System Objectives
Why do we need to consider system objectives?
Each system will be designed to meet certain performance objectives
These objectives affect the design of evaluation: They are the evaluation criteria

16. Information System Objectives
Which objectives are mentioned or implied in each of the cases?
John Fluevog Boots & Shoes
Zipcar
LibraryThing

17. A Taxonomy of Information Systems Success
System Quality
Measures of the information processing system itself
Information Quality
Measures of information system output , e.g., accuracy, meaningfulness, and timeliness
Information/System Use
Recipient consumption of the output of an information system

18. A Taxonomy of Information Systems Success
User Attitudes Satisfaction
Recipient response to the use of the output of an information system
Individual Impact
The effect of an information system on the behavior of the recipient
Organizational Impact
The effect of the information product on organizational performance

19. Observations by DeLone & McLean (1992)
There is no consensus on particular measure of information systems success
Simplifying different dependent variable measures would contribute to the MIS research (the results can be compared)
Few field studies to measure organizational performance
6 success categories indicate MIS success is a multidimensional construct & that it should be measured as such (Only 28/100 studied multiple categories)

20. Independent vs. Dependent Variables
(presumed or
possible cause)
Dependent
Variable(s)
(presumed
results)
affects
Examples of variables
Survey format—online, paper-based (IV)
Response rates (DV)

21. Measuring the value of information (Ahituv, 1989)
Real
World
Data
Information
system
Decision
maker
Decisions,
actions
Outcomes
Point of measurement
perceived value of
information
Point of measurement
realistic value of
information
Ahituv, N. (1989). Assessing the value of information: Problems and
approaches. Proceedings of the Tenth International Conference on Information
Systems, December 4-6, Boston,

22. IS Success Model (DeLone & McLean, 1992)

23. I/S Success Model, Expanded (based on Hwang, Windsor, & Pryor, 2000)
0.37
System
quality
Use
Individual
impact
Organizational
impact
InfoSys
Information
quality
User
satisfaction
0.37
0.43
0.47
0.36
User
environment
Organizational
environment

24. Updated D&M IS Success Model (2002, 2003)
Information
Quality
Intention
To Use
Use
Net
Benefits
System Quality
User
Satisfaction
Service
Quality
Creation
Use
Consequences

26. Evaluation
The man with the tusk states that an elephant is like a spear. The man with the tail argues that the elephant is like a rope. The man with the trunk says, no, its like a snake. The man with the side thinks its more like a wall. But the man with the leg is sure the elephant is like a tree.
The flaw in all their reasoning is that speculating on the WHOLE from too few FACTS can lead to VERY LARGE errors in judgment.
How does this fable relate to System Evaluation?

28. Values in Evaluation (Davidson, 2005)
Suppose that a client does not like the findings of your evaluation and says, “Well, that’s just your opinion about the program. Evaluations are always just so subjective.” How would you respond?

29. Class Activities: Evaluation of IUCAT
Write down the assigned category on the worksheet. Individually fill out the worksheet #1 - #3 (p. 1)
Work with a group of 5 people to fill out the worksheet for the category (p.2)
Present your discussions to the whole class
Use the worksheet (p.2) to jot down the main points