2. © De Montfort University, 20011 Research Methods Howell Istance Department of Computer Science De Montfort University

3. © De Montfort University, 20012 This session…. Why do you need this…. Online resources Literature reviews The nature of research Questionnaire design Structured investigations

4. © De Montfort University, 20013 Why do you need this…? To understand content of directed reading within taught modules To be able to write essays or reviews of published literature To be able to plan and conduct your project where there is an element of research in the investigation or the evaluation of what you have done

5. © De Montfort University, 20014 Library student study skills books… http://www.library.dmu.ac.uk/Skills/Study/ Collection of self study books relevant to research and projects Most available in.pdf or.html format (free download) Series includes… –Research Methods –Designing a questionnaire –Information Citation and Control –Dissertation preparation and presentation –Research interviews –Planning a sample survey –Thinking it through: an introduction to critical thinking

6. © De Montfort University, 20015 MSc projects All projects are required to have at least one academic objective, usually an investigation of literature relevant to some aspect of your project Most projects will include some form of evaluation –E.g the production of a computer-based tool to teach letters of the alphabet to nursery level children –A piece of software which purports to do this will be of limited value without some evidence of how effective it is in doing so. –Demonstrating its effectiveness requires a structured study

7. © De Montfort University, 20016 Project Skills… There is a non-assessed module in the second semester, which covers –Personal time management –Project management –Project Selection –Written and verbal communication –Giving demonstrations –Learning Strategies

8. © De Montfort University, 20017 Literature reviews Conducted to ensure a researcher is familiar with ‘all’ of the what is known about a particular field Often published in order to bring other researchers (and MSc students) up to speed quickly in an unfamiliar field Need to have the scope of the review carefully defined –Not too big such that adequate coverage is infeasible, and there is too much literature to review and the review becomes unfocussed (and thereby not useful) –Not too narrow such that there are too few papers to include

9. © De Montfort University, 20018 Form of a literature review… Requires reading literature from a variety of sources Forming some form of taxonomy or structure for your review Identifying where in your taxonomy the various contributions from the literature fall Critically reviewing the literature –Identifying different approaches, contradictions between contributions, analysis of strengths and weaknesses –Not simply pasting quotes from different papers Drawing your own conclusions, particularly concerning completeness of coverage Highlight implications for your work (if appropriate)

10. © De Montfort University, 20019 Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets.

11. © De Montfort University, 200110 Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. rationale

12. © De Montfort University, 200111 Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. Scope of review

13. © De Montfort University, 200112 Example of a literature review Hand, C. "A Survey of 3D Interaction Techniques". Computer Graphics Forum, 16(5): 269-281. (Dec 1997) (Abstract) Recent gains in the performance of 3D graphics hardware and rendering systems have not been matched by a corresponding improvement in our knowledge of how to interact with the virtual environments we create; therefore there is a need to examine these further if we are to improve the overall quality of our interactive 3D systems. This paper examines some of the interaction techniques which have been developed for object manipulation, navigation and application control in 3D virtual environments. The use of both mouse-based techniques and 3D input devices is considered, along with the role of feedback and some aspects of tools and widgets. Indication of taxonomy

14. © De Montfort University, 200113 Sources of information Scientific journals Conference proceedings (refereed and unrefereed) Magazines, newspapers WWW Important to differentiate between reported investigations and reported opinion Conclusions of any investigation contain some element of informed opinion or judgement Sources of information must be cited appropriately

15. © De Montfort University, 200114 Project Academic Objectives For example.. ‘Review methods of usability evaluation appropriate to the evaluation of internet-based groupware systems for use by students undertaking a course by distance learning’ –Requires review of methods described in the literature, using the original sources (not just someone else’s review) –Implications of problems/needs associated with evaluation of groupware systems –Comparisons and contrasts between methods in light of identified needs of groupware applications –Your conclusions about which method(s) to use and how

16. © De Montfort University, 200115 How to do it… Collect and read current papers and reviews of a field From the list of references cited in these, get copies of those which appear relevant Start to classify the papers you collect in one or several ways to form the basis of a taxonomy Identify authors who are prominent and check what else they have published (WWW useful here) Read contents of recent relevant journals in library Subscribe to mailing lists for coming conferences and look at contributors search on-line bibliographic archives (e.g bids.ac.uk)

17. © De Montfort University, 200116 Constructively criticising research.. Researchers have a vested interest in making a piece of work appear significant and worthy of publication They may not be as forthcoming about the limitations of the work as about the benefits Results should be reported in sufficient detail to enable the reader to draw his/her own conclusions and thereby judge the validity of the conclusions drawn by the author Are the conclusions drawn justified by the evidence provided? Is the method of investigation appropriate or flawed? Are the investigators aware of other similar contributions in the field?

18. © De Montfort University, 200117 What is research? A systematic enquiry, which is reported in a form that allows the research methods and outcomes to be accessible to others Concerned with seeking solutions to problems or answers to meaningful questions Meaningful questions are expressed in a way that indicates what you will accept as an answer Non-meaningful (in research terms) questions are not answerable as a result of enquiry alone (eg judgemental or metaphysical questions) Positivism versus phenomalism

19. © De Montfort University, 200118 Nature of research - positivist Deals with positive facts and observable phenomena Subscribes to the ‘scientific method’ Primary goal is not only description but prediction and explanation Classification of substances and events, and observation of these, provide the basis for descriptive laws based on consistencies in patterns and properties Characterised by absolute or varying degree of generalisability Quantitative, as it draws on measurable evidence

20. © De Montfort University, 200119 Postulates in Positivist Research postulate of natural kinds: all instances of classes and categories of phenomena exhibit the same properties postulate of constancy: all phenomena remain the same or change only very slowly over time postulate of determinism: there is orderliness and regularity in nature, constancy in terms of cause and effect

21. © De Montfort University, 200120 Nature of research - phenomenalist Considers that each phenomena is unique and is controlled by variables such as time, location and culture No two situations are identical No reliance on postulates of natural kinds, constancy or determinism Essentially subjective, where the content of research and the way it is pursued is indicative of researchers intention Outcomes are descriptions which are expressed in narrative and mainly in qualitative terms

22. © De Montfort University, 200121 Stockholm archipelago

23. © De Montfort University, 200122 Example of both approaches… Can the study of critical incidents (as opposed to accidents) in marine navigation in the Stockholm archipelago provide the basis for improvements in sea safety in the area? Positivist approach: collect data via interview, classify types of incidents, produce analyses, make recommendations based on analysis Phenominalist approach: analyse interviews in depth, seek to draw conclusions about causal factors

24. © De Montfort University, 200123 Positivist research methods include... Descriptive research Anything that is variable, varies to a defined degree, and thus can be measured Surveys, case studies, causal comparative studies, correlational studies, developmental studies, trend studies Experimental research Deliberate manipulation of certain factors under highly controlled conditions Purpose is to identify causal connections through keeping the levels of some variables constant and manipulating others

25. © De Montfort University, 200124 Questionnaire design Relevant to requirements gathering for, and evaluation of, projects Self study pack by Arthur Rothwell covers: –Planning and logistics of questionnaire design –Layout issues –Forms of questions –Contents of questions

26. © De Montfort University, 200125 Planning and logistics of questionnaire design Quantitative or qualitative? Legal requirements: the Data Protection Act Confidentiality and anonymity Sample size Volunteer respondents Identifying subject areas Determining appropriate length Typical time scale Main components of questionnaires

27. © De Montfort University, 200126 Forms of question Open and closed questions Dichotomous items Scaled items Mid-point or no mid-point? Ranking to show preference

28. © De Montfort University, 200127 Content of items Avoiding response set Components of attitudes Common types of faulty items –leading questions –context effects –double barelled questions –vague and ambiguous terminology –hidden assumptions –social desirability

29. © De Montfort University, 200128 Leading questions and context effects Would you agree that the governments policies on health are unfair? Item wordings should not contain value judgements How many pints of beer did you drink last night? Think how the context of the study would affect the response, say in a –survey of young peoples life styles –survey of health behaviour and heart disease

30. © De Montfort University, 200129 Double barreled questions Do you believe the training programme was a good one and effective in teaching you new skills? avoid questions that involve multiple premises

31. © De Montfort University, 200130 Vague and ambiguous terminology How often do you clean your teeth? –Frequently –often –infrequently –never what does ‘frequently’ mean? Give quantifiers to ensure all respondents understand the same thing by the response categories

32. © De Montfort University, 200131 Hidden assumptions, social desirability When did you last borrow a video tape? Avoid hidden assumptions - what are these? Do you ever give to charity? May lead to a positive response as otherwise something negative about the respondent is being conveyed

33. © De Montfort University, 200132 Structured Investigations - 1 An MSc project has produced a computer-based tool to teach letters of the alphabet to nursery level children Talk to the person next to you and discuss how you would demonstrate the effectiveness of the tool through some form of structured investigation Consider the resources required and the feasibility of conducting the study within a 3-month project

34. © De Montfort University, 200133 Ways of approaching this…. You need access to people with expertise and, hopefully, a group of nursery children The product needs to be tested, and has to be ready in time for this to happen Subjective assessment by teachers comparing the tool with –Other computer based tools –Paper-based methods Objective approaches –Before and after letter recognition tests with two matched groups of children –Use of non-parametric statistical techniques

35. © De Montfort University, 200134 Structured Investigations - 2 A project has produced a web-based presentation of a college offering educational courses both as a 2D collection of pictures of the building, and as a 3D virtual building which you can ‘walk through’ How would you compare the relative efficiencies of each approach? Again, discuss with the person next to you –What you are trying to assess –How to carry out the assessment

36. © De Montfort University, 200135 Ways of approaching this…. What to assess? –How much information a visitor to the site obtains with each type of presentation –Usability issues – e.g ease of navigation through the representations of the college building How to assess this? –Unstructured user trials, where the amount of building explored is recorded –Structured user trials, where a task is given to find pieces of information –Subjective assessment of visit by participant –Observed usability problems