1. 南京理工大学信息管理系 Research Methodology 科学研究方法
Class 6A
April 12, 2017
RM-Class 6

2. Overview of Today’s Class
Class 6A
Major data collection techniques IV
Think aloud protocols
Transaction log analysis
Research diaries
Content analysis
Other methods
Preparations for data analysis
Quantitative research vs. qualitative research
Class 6B (see PowerPoint slides for Class 6B)
Quantitative data analysis (aka statistics)
Q&A only
RM-Class 6

3. Think Aloud Protocol（出声思考法）
Herbert Simon and Peter Ingwersen’s contributions to this technique
Mainly used for collecting data on subjects’ cognitive and thinking processes
Must be used together with another technique (e.g., experiment, observation, performing a task) to be functional
Need a recording facility to capture the think alouds
Example: Berg, et al., 2010
RM-Class 6

4. Think Aloud Protocol: Features
Pros
One of the few methods that can collect data on people’s thinking and cognitive activities
Gather rich data for qualitative analysis
Cons
Actual cognitive/thinking process is faster than what one can verbalize.
Performing a task while thinking aloud could affect the quality of both.
There is always the tacit and unconscious which can’t be verbalized.
When to use it in our field?
RM-Class 6

5. Transaction Log Analysis
What are transaction logs (系统使用记录)?
Server-side vs. client-side
Features
No biases, recall, or incorrect memory involved
Log data can be analyzed quantitatively & qualitatively
No data on motivations, feelings, …
How to obtain transaction log data?
Access issue
Ethical issue
Sampling issue
Cleansing issue
RM-Class 6

6. More on Transaction Log Analysis
Dos and don’ts
Do observe ethics guidelines in obtaining and using log data
Do use an appropriate sampling method for collection data
Be aware of the limitations regarding what log data can record
Don’t be over interpreting
When to use transaction log analysis?
Examples
Hollink, 2011; Waller, 2011
RM-Class 6

7. Research Diary What is research diary Features Major types
A varied set of data collection instruments and techniques, ranging from descriptive event logs to narrative personal accounts
Features
Information is captured at or close to the time of occurrence
The diary keeper is familiar with what is to be recorded
It is unobtrusive
No transcription is needed (cf: interview, …)
Major types
Solicited, unsolicited
Unstructured, semi-structured, structured
Interval-contingent, signal-contingent, event-contingent
RM-Class 6

8. More on Research Diary Dos & don’ts
Choose reasonable prompt or interval for diary keeping
Provide clear instructions as to how to keep diaries and also specify entry length, …
Stay in touch with participants during data collection
Don’t ask participants to keep diaries for too long or long diaries
When to use this method in our field?
Examples
St. Jean, et al., 2011; Sun, et al., 2011
RM-Class 6

9. Content Analysis: An Overview
What is it?
Careful, detailed, objective, systematic examination and interpretation of the target source (e.g., publications, texts, passages, tags, courses in curriculum, theories applied) for collecting qualitative data
As a data collection technique, it is utilized via analyzing the target data source
Similar techniques
Text analytics
Discourse analysis (i.e., conversational data)
Secondary analysis (i.e., use what has been collected for another purpose/study as target data)
RM-Class 6

10. More on Content Analysis
Features
No fixed, sequential procedures
Many analytical techniques (e.g., comparison, induction, deduction, abstraction, summarization) are used
Consider all message characteristics
Incorporate all contexts; …
Content types
Manifest (显性) vs. latent (隐性)
Text vs. non-text
Example
Chu, 2015; Ke & Cheng, 2015
RM-Class 6

11. Content Analysis: Myths & Approaches
Content analysis is easy
Anyone can do content analysis as it does not take any special preparations
Content analysis is for academic use only
Common approaches
Domain analysis  class 类目
Taxonomic analysis  category 类别
Componential analysis  attribute 属性
RM-Class 6

12. Domain vs. Category vs. Attribute
Categories of Contrast
Signed
Action
Feeling
Junk Mail No
(attribute)
Throw away
Disgust
Personal
Letters
Yes
Read & keep
Delighted
Bills
Read & pay
Don’t like
RM-Class 6

13. Content Analysis: Dos & don’ts
Include all the context in the data collection process
Have another person perform the same analysis to validate your content analysis results
Take proper measures to ensure interconsider consistency
Don’t ignore the quantitative component of the target data
Don’t lose patience in the process
RM-Class 6

14. Theoretical approach What? Features
A technique for gathering data through conceptual analysis, theoretical examination, or other similar activities with a purpose of theory or model building
Features
Similar to content analysis, especially latent content analysis
Its focus is on theory, modeling, …
Distinguish it from reviews, opinion pieces or random discussion
RM-Class 6

15. Other Data Collection Techniques
Action research
Aims to solve an immediate problem or to produce guidelines for best practice by using multiple data collection techniques
Similar to fieldwork with an emphasis on actions needed to carry out research
Card sorting
Information horizon
Drawing …
RM-Class 6

16. Data Collection Techniques -- Summary --
Different techniques can be used for collecting different kinds of data.
Always choose a technique that is the most appropriate for collecting what is needed to answer your research question(s).
Normally, more than one technique is needed for collecting all the data for a study.
Mixed methods, triangulation, or plural methods
A taxonomy of research methods discussed so far …
Case study of Slone: Data collection techniques
RM-Class 6

17. Research Methods: A Taxonomy by Applicability
Super-methods (超级方法)
content analysis, …
Meta-methods (元方法)
experiment, ethnography/fieldwork, …
Stand-alone methods (独立方法)
bibliometrics, focus groups, historical method, interview, observation, questionnaire, research diaries, theoretical approach, transaction log analysis, webometrics & altmetrics, …
Contingent methods (附随方法)
card sorting, Delphi study, think aloud protocol, …
Pseudo-methods (虚名方法)
case study, cross-sectional study, longitudinal study, literature study, grounded theory (扎根理论), …
RM-Class 6

18. Data Coding (标注) Nominal data: Code the categories
Numerals could be used to represent data categories but they don’t have any property of numerals.
Ordinal data: Code the rankings
Numerals could be used to represent data ordering but they don’t have the equal distance property.
Interval/Ratio data: Code them as they are
A few notes
Code the data to the highest level possible
Missing data?
Levels of measurement vs. data analysis
Nominal, ordinal → Mainly qualitative
Interval, ratio → Mainly quantitative
RM-Class 6

19. Quantitative Research vs. Qualitative Research
What is quantitative research?
Research that collects and analyzes quantitative data
What is qualitative research?
Research that collects and analyzes qualitative data
What differentiates between the two?
Type of data collected
Type of data analysis techniques used
What is the relationship between the two?
RM-Class 6

20. Qualitative Coding Practice
Work in groups of two.
Code all the research articles in JASIS&T V57, N1, available at the class website, individually using the coding schema provided to identify the methods used in each.
Compare your coding results with your group partner and compute intercoder consistency in percentage.
Discuss and revise your coding results accordingly.
Compute again your intercoder consistency in percentage.
Bring your coding results to Class 7 for discussion
RM-Class 6

21. Next Class (4/13, 8:30-11:30am) Overview of qualitative research
Major qualitative data analysis techniques
Qualitative coding
Discussion on the Qualitative Coding Practice
Content analysis, comparison, …
Software for qualitative data analysis
Research design: A preview
RM-Class 6