1. Research Design
Week 4 Lecture 1
Thursday, Apr. 1, 2004

2. ISYS3015 Analytical Methods for IS Professionals
Agenda
Triangulation
What is research design
Two broad categories of research methodology
The data source
Measurement instruments
Ethical issues
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

3. Questions about assignment 1
Where to find the research problem?
What do I expect to review?
How do I write the review?
Word limit?
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

4. ISYS3015 Analytical Methods for IS Professionals
Triangulation
Triangulation of measures
Triangulation of observers
Triangulation of theory
Triangulation of method
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

5. Examples of triangulation
Topic: The amount of violence in popular Hollywood films
Measures
Observers
Theory
Method
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

6. What is research design
A detailed blueprint specifying how the research will be carried out
Basic format of all research process
General criteria for a research project
Universality
Could be carried out by any competent person
Replication
Control
Measurement
Data can be measured
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

7. Quantitative and qualitative orientation
Different research problems
=>Different research design and method
Collection of different types of data
Different interpretation of data
Two broad categories
Quantitative research
Used to answer questions about relationships among measured variables with the purpose of explaining, predicting and controlling phenomena
Qualitative research
Used to answer questions about the complex nature of phenomena, often with the purpose of describing and understanding the phenomena from the participant’s point of view
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

8. ISYS3015 Analytical Methods for IS Professionals
Research Process
Quantitative
Concepts are in the form of distinct variables
Relationships are expressed in hypotheses
Measures are systematically created before data collection and are standardized
Data are in the form of numbers from precise measurement
Procedures are standard, and replication is assumed
Qualitative
Concepts are in the form of themes, motifs, generalizations, and taxonomies
Measures are created in an ad hoc manner and are often specific to the individual setting or researcher
Data are in the form of words and images from documents, observations and transcripts
Procedures are particular and replication is very rare
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

9. ISYS3015 Analytical Methods for IS Professionals
Types of Analysis
Quantitative
Findings are typically expressed in terms of relationships that are presented in tables and graphs
Qualitative
Findings are typically expressed by quoting interviews or relating experiences the researcher has had in the field
Which approach should I use?
Guidelines from “practical application” on page 112 of textbook
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

10. Validity of the methodology
Internal validity
The extent to which its design and the data that it yields allow the researcher to draw accurate conclusions about cause and effect and other relationships within the data
Concerns
Lack of control
Reactivity: Hawthorne effect
Experimenter expectancy
3 cases from textbook page 104
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

11. Validity of the methodology (cont)
External validity
The extent to which the conclusions drawn can be generalized to other context
Real-life setting
A representative sample
Replication In a different context
Validity in qualitative research
triangulation
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

12. Planning for data collection
Four fundamental questions about the data
What data are needed?
Where are the data located?
How will the data be secured?
How will the data be interpreted?
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

13. ISYS3015 Analytical Methods for IS Professionals
Data source
Secondary data
Information collected by others for a purpose other than the project at hand, but may be useful for the project
Primary data
Original data collected for the purpose of the research problem
Observation, questionnaire, interview and tape recording, experiment results
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

14. ISYS3015 Analytical Methods for IS Professionals
Use of secondary data
Answering research questions or solving some or all of the research problems
Helping in problem formulation
Providing benchmarking measures and other findings that can be compared later on with the results of the study at hand
Building and testing analytical (mathematical, econometric) models
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

15. An example of data source
From this week’s paper
“last-minute bidding and the rules for ending second-price auctions: evidence from eBay and Amazon auction” by Roth A. and Ockenfels A.
What types of data were collected in the study?
How did the authors use those data for?
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

16. Where to find secondary data
Sources may be internal or external
Internet sites
Central and local government studies and report
Australian Bureau of Statistics
Australian Stock Exchange
Commercial research companies selling data
Studies and reports of institutions and departments such as universities, telecommunication departments, marketing and other research institutes
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

17. Measurement instruments
Use existing measurement instruments
A clock to measure execution time
A published personality test to measure a person’s tendency to be either shy or outgoing
Develop our own measurement instruments
A survey to assess people’s opinion about welfare reform
A final exam to test students’ understanding of a certain subjects
Determine clearly and definitively the nature of the measurement instruments
Provide evidence that the the instruments have a reasonable degree of validity and reliability for the research purpose
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

18. Validity of measurement instrument
Face validity
A personal judgment that a measure seems to be reasonable.
Content validity
The extent to which a measure reflects the dimension(s) implied by the concept
Always in connection with achievement testing
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

19. Validity of measurement instrument
Criterion validity
The extent to which the results of a measurement instrument correlate with another, presumably related measure (criterion)
Construct validity
The extent to which the measure reflects the theoretical concept it is supposed to measure
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

20. Reliability of measurement instrument
Test-retest reliability (stability over time)
The extent to which the same instrument yields the same result on two different occasions
Inter-rater reliability (representative across subgroups)
The extent to which two or more individuals evaluating the same product or performance give identical judgments
Internal consistency reliability (equivalence across indicators)
The extent to which all the items with in a single instrument yield similar results.
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

21. ISYS3015 Analytical Methods for IS Professionals
Ethical Issues
Researcher’s ethical responsibility to respondents:
Voluntary participation
People not be coerced into participating in research
Informed consent
Prospective research participants must be fully informed about the procedures and risks involved in research and must give their consent to participate.
Do not place pressure on respondents
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

22. ISYS3015 Analytical Methods for IS Professionals
Ethical Issues (cont)
Confidentiality and Anonymity
Identifying information will not be made available to anyone who is not directly involved in the study.
Debrief subjects
Where experiments or field studies involve deception, subjects should have the study explained to them after the session
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

23. ISYS3015 Analytical Methods for IS Professionals
Ethical Issue(cont)
Researcher’s responsibility to his or her discipline and society
Represent research literature fairly
Acknowledge all your sources
Be aware of potential sources of bias
Avoid of falsifying data
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney

24. ISYS3015 Analytical Methods for IS Professionals
Summary
Triangulation
General criteria for a research project
Two major types of research design
quantitative and qualitative design
Two different data sources
Find and evaluate measurement instruments
Ethical issues
Thursday, Apr. 1, 2004
ISYS3015 Analytical Methods for IS Professionals
School of IT, The University of Sydney