Copyright, 2001, Starr Roxanne Hiltz  [How to] EVALUATE (and improve)  the EFFECTIVENESS of  [Organizational] INFORMATION SYSTEMS:  A HUMAN FACTORS Perspective (acceptance and impacts on individuals, organizations, and society)  Information Systems Research

2 Information Systems, defined…  Information Systems research investigates phenomena where technological and social systems interact (Alan Lee, former editor of MIS Quarterly, the top rated journal in IS)  IS research studies the interaction of technology and organizations. It draws on economics, sociology, computer science, cognitive science and other disciplines” (AIS panel on IS Research, 2001).

3 ORGANIZATIONAL INFORMATION SYSTEMS  are computer-based systems which support managers and professionals in their work. They include –"Management Information Systems" (e.g., Personnel systems for salary administration, Finance and Accounting systems, Production planning and scheduling) –Communication systems (electronic mail or computer conferencing, web sites) –Marketing Systems (e.g., E-commerce)

4 ORGANIZATIONAL INFORMATION SYSTEMS Include  Decision Support systems (e.g., expert systems, Strategic planning systems)  CSCW (computer-supported cooperative work, including Group Decision Support Systems)  “Learning Systems” (computer mediated instruction, knowledge bases..)  Social Networks – people working with people in groups and relationships

5 EFFECTIVENESS  Refers to results for end users and the organization as a whole. It includes:  1. Is the system USED? If not, why not?  2. Are users SATISFIED? (What makes them lose time and causes stress?)  3. What are the benefits or IMPACTS (negative as well as positive) for the organization as a whole?

6 To assess EFFECTIVENESS  You must first determine the OBJECTIVES of the organization or community, and then develop CRITERION measures to assess how well the system helps the organization or community obtain these objectives.

7 EFFECTIVENESS VS. EFFICIENCY  Effectiveness can be contrasted with EFFICIENCY. Efficiency has to do with how well the MIS development and operations processes utilize resources (staff, money, machines) to provide the information system to the user.

8 EFFECTIVENESS VS. EFFICIENCY  Efficiency vs. Effectiveness "Driving a car that gets 100 miles per gallon is not EFFECTIVE if you are driving around without purpose... Information productivity must be defined within the proper context. Information, except as an aesthetic experience, is not a consumer good.  Information has economic value only when it is used by people to help something be more effective

9 EFFECTIVENESS VS. EFFICIENCY  "Effectiveness is a social concept: it applies to groups. Efficiency is an individualistic concept: it applies to isolated acts." It does not matter how efficiently something is carried out if the end product is not used, or has negative effects.

10 EVALUATION METHODS  The course will include a review of social- science based methodologies which can be used to conduct empirical studies of information systems, generally from the USER'S point of view.  These are the research methods that are used in most articles published in the core research journals, which include MISQ, ISR, JMIS, and CACM.

11 QUALITATIVE methods:  Result in largely descriptive data rather than numerical data  "Protocol Analysis:" Individual users are given a task to accomplish using the system, and recorded as they "think out loud"  Unstructured or "depth" interviews  Group techniques-- e.g., focus groups, Delphi technique

12 QUANTITATIVE methods:  The sample survey  Controlled Experiments  Appropriate statistical analyses of the results

13 COURSE CONTENT  Review course objectives and syllabus  Research is a lot of work: reading professional journal articles, carrying out studies of users and systems. About 8-10 hours a week outside of class will be required.

14 Requirements  1. Critically read a number of research articles  Text on research methods (mainly a reference)  2. Develop your research skills through assignments  3. Prepare your PROJECT.DOC following academic style based on logical thinking  4. Final project: Needs some theory, some data, some analysis and conclusions, and a discussion section

15 Logistics:  Read the text chapters before the lecture  Do the assignments  Attend class – presentation to your peers, and commenting on others work, is part of learning research  If you are going to be away when an assignment is due, get it done before you leave…

16 Expectations  You will do your work as an ethical, responsible professional, and turn it in on time.  If you “cheat” in any way (turning in somebody else’s work as your own, including plagiarism) you will fail the course  Information Technology professionals must be absolutely honest or trustworthy, or they do not belong in the profession

17 The first steps in any research are:  1. Do a “literature search” for applicable theories, constructs, prior findings  2. Choose a theoretical framework- a set of organizing constructs– or a theory ( a model that posits relationships among the constructs)  3. Define the constructs and choose measures- build on prior research.

18 CONSTRUCTS AND VARIABLES  A CONSTRUCT IS A DEFINITION OF AN ABSTRACT QUALITY OR STATE OF AN INDIVIDUAL, GROUP, OR SOCIETY. – examples: “Use” – “user acceptance" – "productivity" – You cannot "see" these things.

19 Theoretical Frameworks  Building and testing theory should be the main purpose of any empirical study.  Theory consists of a set of constructs, the relationships among them, and most importantly, the “why” that explains those relationships.  A good theory provides the framework, the story line, that holds together the entire study, from design of measures and data collection methods to the presentation of results.

20 Example: "FUNCTIONALITY AND USABILITY"  FUNCTIONALITY refers to the features of a system... what it can DO.  USABILITY or ease of use can only be defined within the context of characteristics of the task and of the user.

21 "FUNCTIONALITY AND USABILITY"  THESIS: There is no functionality without usability.  Corollary: It is not true that the more functionality, the better! "Richer" but less usable systems provide less "effective" (actual) functionality."

22 GOODWIN: REASONS WHY SYSTEMS ARE NOT USED  1. Limited functionality: if the functions provided do not match task requirements, a system will not be usable (ex.: system with no editor!)  2. Poor interface design (e.g., inadequate flexibility (won't take abbreviations, provide defaults, words or commands as well as Icon menus, etc.)  Poor consistency and integration within the system (Different parts look and work in different ways, very confusing!)

23 GOODWIN: REASONS WHY SYSTEMS ARE NOT USED  3.ACCESSIBILITY: Access or Availability problems – But: this interacts with motivation.  4.Start-stop hassles (losing some of the work you have done if you have to stop in the middle and do something else)

24 GOODWIN: REASONS WHY SYSTEMS ARE NOT USED  5.Poor response time  6. Poor systems dynamics  7. Inadequate training and user aid

25 GOODWIN: REASONS WHY SYSTEMS ARE NOT USED  8. Poor or non-existent documentation  9. Poor integration with other systems (non-transferability of data; negative interference with learned interface conventions)  10. COSTS (Not mentioned by Goodwin)-- including learning time, exceed expected benefits

26 End of Lecture 1