HOPE Research Methods Collecting, Processing and Analyzing Data.

HOPE Research Methods Collecting, Processing and Analyzing Data

HOPE Research Methods Tony Fleet 2 Aims of the Session The purpose of this session is to: Alert you to the different types of methodology available to you in your research Make you aware of the different techniques that you might use in collecting, presenting and analysing data Discuss the different kinds of problems that you might encounter in pursuing your research.

HOPE Research Methods Tony Fleet 3 Contents 1.Developing Your Research QuestionsDeveloping Your Research Questions 2.The Different Types of ResearchThe Different Types of Research 3.Selecting Appropriate Research MethodsSelecting Appropriate Research Methods 4.Robustness of MethodsRobustness of Methods 5.Structuring Your MethodologyStructuring Your Methodology 6.Data AnalysisData Analysis 7.Problems with The Research ProcessProblems with The Research Process 8.SummarySummary

HOPE 1. Developing your Research Questions This section of the presentation examines what you need to do in order to focus your research.

HOPE Research Methods Tony Fleet 5 The Purpose of Research The purpose of research is to contribute to a current academic debate, and possibly to advance knowledge in some manner. This means that the research that you undertake has to be : 1.Embedded in a recognisable field of study, taking account of, and drawing on, past research 2.Of interest to other researchers working in the same field, and possibly to the wider community 3.Generalisable to more than one individual experience or circumstance.

HOPE Research Methods Tony Fleet 6 Typical Research Structure Conduct Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions The process of research is well-documented. This diagram more or less describes the activities you need to undertake. What we will do in this session, is to look at some of the elements, and how they fit together.

HOPE Research Methods Tony Fleet 7 Getting Started Conduct Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions The first thing that you will do is to make sure you are well-informed, and to pose pertinent questions which your research will answer.

HOPE Research Methods Tony Fleet 8 Defining Your Field of Study (1) You undertake a Literature Review in a particular field, in order to ensure that your research is embedded within that field, and that you are taking account of the methods, issues, results, theories and conventions which apply. At the end of the literature review, you will have narrowed the field down to a relatively small topic within the field, and will have some ‘unanswered questions’ which your research will try to investigate. These focussed questions form the basis of your Research.

HOPE Research Methods Tony Fleet 9 Defining Your Field of Study (2) Your Research Questions are crucial, as they effectively define both the content of your research and the manner in which you carry it out.  Your research methodology is designed specifically to attempt to answer these questions  The data that you collect will be focussed on issues relevant to these questions  Your analysis of the data will seek to provide answers to the questions  Your conclusions will summarise the answers.

HOPE Research Methods Tony Fleet 10 Research Questions Typical research questions might be: –To what extent is the business community in India aware of the potential of Bluetooth? –Is the software currently available for teaching arithmetic to 5 year olds appropriate and effective? –Are there differences in the way that men and women approach the task of writing software? –How can historical events be modelled effectively using VRML?

HOPE Research Methods Tony Fleet 11 Defining Your Population When framing your Research questions, you need to be clear about what set of objects, people or events forms the ‘background population’ in your study. Are you saying, for example that the CAL software you produce is designed for all English-Speaking people, all men, all Afro-Caribbeans, all children under 5, all those who have been diagnosed as dyslexic, or simply to Afro- Caribbean boys under 5 with specific learning difficulties? If you are investigating whether on-line learning is effective, is your population: students, University students, UK University students, Liverpool Hope Students, Liverpool Hope Computing Students or Liverpool Hope MSc. Computing Students?

HOPE Research Methods Tony Fleet 12 Research Questions References (SW Library) Lewis, Ian. - So you want to do research! : a guide for beginners on how to formulate research questions. - 2nd ed. - Edinburgh : Scottish Council for Research in Education, 1997. - (SCRE publication ; 2 ;.. - 1860030327

HOPE 2. The Different Types of Research Here we look at the different options available to us in carrying out the research.

HOPE Research Methods Tony Fleet 14 Your Research Focus The main focus of your research can be: Product-based Research, focusing on producing a piece of hardware or software (or the designs for them) which is at the ‘cutting edge of a discipline, drawing on other researchers’ ideas, best practice and what is feasible. In doing this, you may need to explore how the product will ‘enmesh’ with current systems, and existing and future technologies. People-based research, focusing on the people who interact with the hardware or software, looking at issues such as usability, user behaviour, compatibility of software with current user systems and other HCI issues. Both of these approaches are legitimate, and it is possible that in carrying out your research you might need to use elements of each one.

HOPE Research Methods Tony Fleet 15 Product v. People Which focus? The focus of your research is decided by you. It will depend upon how confident you are in creating a product at the ‘cutting edge’, or how comfortable you will be as a researcher in dealing with people. When you frame your research questions, you need to ensure that their focus leads you into the kind of research that you want to do. There is no right answer, but you may find that your research will be best carried out by using a ‘main’ focus of one element, with a subsidiary focus of another. For example producing a piece of software which securely encodes personal data as a self-encrypting and decrypting file stored on an ID card, may well need trialling with real people.

HOPE Research Methods Tony Fleet 16 Approaches to Research There are two main approaches to doing research: Quantitative Research looks for ‘hard’ numerical data; it proceeds by counting, measuring and summarising. The goal is to look for statistical significance in the results. Qualitative Research takes a ‘soft’ approach to data, and has a more descriptive ‘feel’. It attempts to get to the heart of the matter, exploring individual cases in detail, and seeking the reasons for behaviour and explanations for events. Both of these approaches are legitimate, and it is possible to combine elements.

HOPE Research Methods Tony Fleet 17 Quantitative v. Qualitative Which approach? The approach you use will depend upon your topic and your research questions. It will also depend upon how comfortable you as a researcher feel about using these methods. There is no right answer here, and, as we shall see in the rest of this presentation, there may be good reasons for adopting a variety of methods, which encompass both quantitative and qualitative approaches

HOPE Research Methods Tony Fleet 18 Types of Research There are four main types of research that you might consider:  Experimental Research  Survey Research  Evaluative Research  Observational Research All four of these types can incorporate both quantitative and qualitative approaches

HOPE Research Methods Tony Fleet 19 Experimental Research This is normally quantitative, but can take two forms: An attempt to produce a piece of hardware, software or a combination of both, which is at the ‘cutting edge’ of a discipline. An attempt to investigate and document the performance of a particular piece technology in specific circumstances. This might involve: Creating hardware or software applications Devising detailed tests and evaluation procedures Carrying out rigorous testing Evaluating performance or usability

HOPE Research Methods Tony Fleet 20 Survey Research This research can be qualitative or quantitative; in the widest sense, you are interviewing people. This might involve: An unstructured interview A semi-structured interview An structured interview based on questionnaire (face to face, or by telephone) An administered questionnaire A Postal Questionnaire

HOPE Research Methods Tony Fleet 21 Evaluative Research This is primarily qualitative. Here you are trying to assess whether something is of value, whether it meets its specifications, or whether it is fit-for-purpose. This might involve: Developing a list of criteria on which to make judgements Examining the object against each of the criteria to judge to what extent it conforms to expectations Weighing the positives and the negatives, coming to overall conclusions Matching these judgments against similar judgements made elsewhere in the literature or in real life.

HOPE Research Methods Tony Fleet 22 Observational Research This research normally uses a qualitative approach; in the widest sense, you are recording people’s behaviour. This might involve: Participating in a task or situation Making field notes of experiences Creating and using an Observation Schedule Making a check-list of occurrences of particular events or items.

HOPE Research Methods Tony Fleet 23 Research possibilities QualitativeQuantitative Experiment Field- testingBench-testing and simulations Survey Unstructured Interviews Written Questionnaires Evaluation Using expert judgesEvaluation criteria and checklists. Observation Participant Observation Observation schedules

HOPE Research Methods Tony Fleet 24 Other Forms of Research Historical & Documentary Research proceeds by scrutinising existing materials, both written and artefacts, using them as sources of evidence. Action Research is normally conducted in an educational or political context. Action is taken, monitored, evaluated and then modified for the next cycle. Ethnographic Research consists of an in-depth study of a cultural phenomenon, in order to generate new theory. Case Study Research selects a whole range of research methods in scrutinising one particular context or situation.

HOPE Research Methods Tony Fleet 25 Research methods 1 References (SW Library) Crabtree, Benjamin F. - Doing qualitative research. - London : Sage, 1992. - (Research Methods for Primary Care ; 3). - 0803943121 Creswell, John W.. - Research design : qualitative and quantitative approaches / John W. Creswell. - Thousand Oaks, Calif; London : Sage, 1994. - 0803952554 Creswell, John W.. - Qualitative inquiry and research design : choosing among five traditions / J. - Thousand Oaks, Calif.; London : SAGE, 1998. - 0761901434

HOPE Research Methods Tony Fleet 26 Research methods 2 References (SW Library) Miller, Delbert Charles. - Handbook of research design and social measurement. - 3rd ed. - New York : David McKay Co. Inc, 1977. - m0859739 Research methods in education and the social sciences / [Research Methods in. - Block 3B : Research design. - Milton Keynes : Open University Press, 1983. - (DE304, Block 3B ; 3B). - 0335074235 Yin, Robert K.. - Case study research : design and methods. - Rev. ed. - Newbury Park; London : Sage, 1989. - (Applied social research methods series ; v.5). - 080393470x

HOPE 3. Selecting Appropriate Research Methods The next few slides discuss how you might go about selecting your research methods from those available

HOPE Research Methods Tony Fleet 28 Selecting Your Methodology Your research methodology consists of : Research Methods (experiment, survey etc.) Research Instruments (questionnaire, tests etc.) Analytical Tools (statistics, inductive or deductive methods) When selecting the methodology, you need to be aware of: The Research Questions you are trying to answer The Population you are trying to generalise to.

HOPE Research Methods Tony Fleet 29 Factors to Consider Undertake Literature Review Select Research Questions How has previous research in this area been done? Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis What methods have been used? What research instruments have been devised? How will the methods & instruments be applied? What Statistical tests can be carried out? Test Hypotheses & Draw Conclusions What Research Hypotheses can be tested?

HOPE Research Methods Tony Fleet 30 Appropriate methodology Do your Research Questions involve impressions, attitudes, opinions, beliefs or knowledge held by people? If so, then survey research is appropriate Do your Research Questions involve behaviour, actions, reactions to events, circumstances or objects? If so, then observational research is appropriate

HOPE Research Methods Tony Fleet 31 Appropriate methodology Do your Research Questions involve the reliability or robustness of hardware, software, systems or infrastructure? If so, then an evaluative study is appropriate Do your Research Questions involve the testing of hardware or software at the technical level, speed, accuracy, security etc? If so, then experimentation is appropriate

HOPE Research Methods Tony Fleet 32 A Mix of Methods It may be that your research questions overlap some of these categories, or different questions address more than category. If so, you should consider a mix of methods, that ensure that you cover all eventualities. This may bring added benefits. (See Triangulation)

HOPE Research Methods Tony Fleet 33 Mixed Methods References (SW Library) Mixing methods : qualitative and quantitative research / edited by Julia Bra. - Aldershot : Avebury, 1995. - 1859721168 Tashakkori, Abbas. - Mixed methodology : combining qualitative and quantitative approaches / Abba. - Thousand Oaks, Calif.; London : Sage, 1998. - (Applied social research methods series ; v.46). - 0761900705

HOPE 4. Robustness of Methods As well as ensuring that your questions are well-focussed, and your methods relevant and appropriate, you need to ensure that your methods are also Reliable and Valid

HOPE Research Methods Tony Fleet 35 Reliability & Validity Research is Reliable if different methods, researchers and sample groups would have produced the same results. Research is Valid if the results produced by the research are accurate portrayals of the situation, explanations are effective, and predictions from the research are actually borne out by observation.

HOPE Research Methods Tony Fleet 36 Reliability Research can have poor reliability, if it on one or two cases only, or if personal judgement or opinion is included. Reliability can be improved if data collection methods are made more precise, we have controlled experimentation, and we can produce statistical summaries.

HOPE Research Methods Tony Fleet 37 Validity Research can have poor validity if the data produced is too far removed the object under study, or the respondent. Using detailed, highly structured research instruments can lead to distortions, by forcing observations into categories when they do not fit. Data summaries and averaging can also lead to distortions, and meaningless generalities. Graphical representations and percentages can be highly selective and produce biased findings. Validity can be improved by working directly with individuals or objects, focusing on specific cases, making detailed observations, conducting face to face interviews, taking detailed measurements in specific circumstances etc.

HOPE Research Methods Tony Fleet 38 Reliability v. Validity Valid, but not reliable

HOPE Research Methods Tony Fleet 39 Reliability v. Validity Reliable, but not Valid

HOPE Research Methods Tony Fleet 40 Methodological Trade-Off If you improve validity, you will almost certainly reduce reliability. If you improve reliability it will be at the cost of reducing your validity. The trade-off is to balance the two so that the benefits of using particular methods outweigh the losses incurred.

HOPE Research Methods Tony Fleet 41 Triangulation Triangulation takes its name from the navigational method of positioning a ship at sea by making two independent observations. The purpose here is to use two distinct methodologies, independent of one another to confirm that the effects which we are observing are real, and not artefacts of the research process.

HOPE Research Methods Tony Fleet 42 Triangulation Triangulation attempts to counter the methodological trade-off by using a mix of methodologies. If you are using highly structured, statistical or measurement-based research, you supplement this with detailed observations or face to face interviewing. If your research is mainly based on individuals or on single items, you ensure that at least part of it has some statistical summaries, structured observations or questionnaires.

HOPE Research Methods Tony Fleet 43 Validity & Reliability References (SW Library) Kirk, Jerome. - Reliability & validity in qualitative research. - Beverly Hills Calif : Sage Pubns, 1986. - (Qualitative Research Methods Series ; 1). - 0803924704 Litwin, Mark S. - How to measure survey reliability and validity. - London : Sage, 1995. - (The Survey Kit ; 7). - 0803957041

HOPE 5. Structuring Your Methodology This section looks in detail at the techniques that you might employ in the Research Activity itself.

HOPE Research Methods Tony Fleet 45 Research Structure Conduct Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions After framing your Research Questions, and selecting your methodology, you should test that this is going to work by conducting a small- scale Pilot Study

HOPE Research Methods Tony Fleet 46 5a Pilot Studies A pilot study is a set of preliminary investigations & procedures carried out prior to the main research, to ensure that the research is possible and can proceed without hitches.

HOPE Research Methods Tony Fleet 47 Pilot Study (1) In almost every type of Research, you will need to devise or adapt some sort of Research Method or Instrument. This might be a measuring procedure, a set of evaluation criteria, an interview procedure or questionnaire, or an observational method or schedule.

HOPE Research Methods Tony Fleet 48 Pilot Study (2) Your procedure or instrument should be based on best practice from previous research. It is unlikely that you will find exactly what you need; you will be forced to adapt or amend it. This means that you will need to conduct a Pilot Study to test whether the new instrument is fit-for-purpose.

HOPE Research Methods Tony Fleet 49 Pilot Study (3) The main purpose of a Pilot Study is to iron out ‘bugs’ in procedures, or to check that instruments work. The size of the pilot study will depend on how ‘inventive’ you needed to be. If your procedures are almost entirely of your own devising, then you will need a fairly extensive pilot study to check them. If you have ‘lifted’ methods from the literature, than your pilot can be quite small.

HOPE Research Methods Tony Fleet 50 Pilot Study (3) With questionnaires & observation schedules, you will need to check that individual items are giving you expected results. With test procedures, you need to check that you can actually do what you have said that you are going to do. With evaluation criteria, you need to use these in a limited context, to see that they are workable and effective.

HOPE Research Methods Tony Fleet 51 Pilot Study (4) As a result of the Pilot Study, you need to evaluate procedures and instruments, making amendments where necessary. The Pilot Study stage will be part of your research; you will need to write this up, reporting on how your methods were adapted and improved as a result.

HOPE Research Methods Tony Fleet 52 Carrying Out the Research Undertake Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Here we will examine how particular methods and instruments can be applied in the research situation.

HOPE Research Methods Tony Fleet 53 5b A detailed look at some Research methods Experimental Design Evaluative Research Observational Research Survey Research

HOPE Research Methods Tony Fleet 54 5c.1 Experimental Design This section looks at the different ways in which you can conduct experiments. Note that you do not need to be doing pure ‘experimental’ research to adopt these methods.

HOPE Research Methods Tony Fleet 55 Experimental Designs You may need to think about experimental design even if you are doing types of research other than ‘Experiments’ These designs occur where you have made some change, and are trying to find out its effect. The net result is that you are comparing one thing, or one group with another thing or group.

HOPE Research Methods Tony Fleet 56 Experimental Designs The terminology for experimental designs comes from agricultural experiments. We have different ‘treatments’ which we apply to different groups We control the groups for different ‘factors’

HOPE Research Methods Tony Fleet 57 Experimental Designs Experiments normally involve ‘independent’ and ‘dependent’ variables’ Independent variables are factors that can be controlled for, like temperature, file sizes, age, gender etc. Dependent variables are those factors which will change as a result of altering the independent variables. For example, download times will increase as a result of increasing file sizes. –Download time = dependent variable, –File size = independent variable

HOPE Research Methods Tony Fleet 58 Pre-Test/Post Test Control Group Design This is the classic experimental design. It allows you to split your sample into two distinct parts (A & B) You give the same test to the two groups before you start You treat one of the groups You apply the same test afterwards. Group A (treated) Group B (untreated) Pre-test Treatment Post-test

HOPE Research Methods Tony Fleet 59 Control Group The Control Group (B), is one which is the same in all respects, except for the fact that we make no changes. We can use the test measurements on the control group as a benchmark for any changes we make to the treatment group (A)

HOPE Research Methods Tony Fleet 60 Example 1: Server Testing Suppose we wish to check whether a firewall is effective in blocking external attacks. Set up System A & System B on two different servers, running ‘near- equivalent’ internal & external programs. Devise a test which includes a full range of possible attacks; apply to both A & B. Take series of measurements & observations. Incorporate firewall into System A Reapply test to both systems, which are again running ‘near-equivalent’ programs. System A (treated) System B (untreated) Pre-test Apply Firewall Post-test

HOPE Research Methods Tony Fleet 61 Example 2: Software Design Suppose we wish to find out whether anthropomorphic agents are useful in communicating with novice users of a website. Set up Website A & Website B with ‘near- equivalent’ structure, but different content. Devise a procedure which asks two equivalent sets of users to make explorations of the websites; apply to both A & B. Make a series of observations Incorporate Agents into Website A Re-apply procedure, asking two more equivalent sets of users to make the same explorations of both Websites. Again make observations. Website A (treated) Website B (untreated) Pre-test Apply Agents Post-test

HOPE Research Methods Tony Fleet 62 Example 3: On-Line Learning Suppose we wish to find out whether training is effective in helping students cope with the demands of on-line learning. Set up group A & group B with ‘near- equivalent’ members. The two groups to undertake a short programme of learning on-line, which incorporates a short evaluation and a knowledge test. Give Training to Group A The two groups to undertake a further short programme of learning on-line, again incorporating a short evaluation and knowledge test. Group A (treated) Group B (untreated) Pre-test Apply Training Post-test

HOPE Research Methods Tony Fleet 63 Factorial Designs This is where we examine the effects of two or more independent factors simultaneously. For two factors, we would need 4 groups –Group A (Control: no treatment) –Group B (Factor 1 treatment only) –Group C (Factor 2 Treatment only) –Group D (Factors 1 & 2 Treatments) Clearly, this is going to increase the complexity and size of the research, but it has the added benefit of producing verifiable results in cases where two factors interact to produce interesting effects.

HOPE Research Methods Tony Fleet 64 Other Design Variants Post-Test only Control Group Design Here we assume that both groups are the same (but do not test that assumption). We simply apply the treatment to one group, and apply the test to both groups. Matched Pairs Design The individuals we use in the test groups (subjects) are matched for characteristics which are likely to affect the outcome (gender, age, level of education, ethnicity etc.)

HOPE Research Methods Tony Fleet 65 Experimental Designs: Reliability Issues For high reliability, we need to build in strict controls over each of the independent variables affecting the outcomes of the experiment. We will also need to ensure that any measurements taken of the dependent variables are as accurate as possible. We also need to ensure that our methodology is clear and replicable.

HOPE Research Methods Tony Fleet 66 Experimental Designs: Validity Issues For high validity, we need to conduct the experiment in as ‘natural’ a setting as possible, and in as near as exact a match to the circumstances in which the events or objects would normally operate. If a sample is being used, then the sample (whether time, events, people or objects) should be as representative of the ‘background’ population as possible, and we should make detailed observations of how the events unfold as well as the final measurements.

HOPE Research Methods Tony Fleet 67 Experimental Design References (SW Library) Campbell, Donald T., Donald Thomas, 1916-. - Experimental and quasi-experimental designs for research / Donald T. Campbell. - Boston; London : Houghton Mifflin, 1963. – 0395307872 Field, Andy. - How to design and report experiments / Andy Field, Graham Hole. - London : SAGE, 2003. - 0761973826 Miller, Steve. - Experimental design and statistics. - London : Methuen, 1975. - (Essential Psychology ; A8). - m0805407

HOPE Research Methods Tony Fleet 68 5c.2 Evaluative Research This section looks at the methods and issues surrounding evaluation. You may need to use such techniques if evaluation is implicit in your research questions

HOPE Research Methods Tony Fleet 69 Evaluative Research Evaluative Research covers those cases where you are attempting to compare whether one procedure or object is better or more effective than another procedure or object, or to determine whether a particular procedure or object is fit-for- purpose. Evaluation needs to be done against a set of criteria, which have been established as valid and reliable in this context. You would normally produce an evaluation form to be completed by respondents.

HOPE Research Methods Tony Fleet 70 Evaluation Criteria To establish criteria for evaluation, we need to break the topic down into individual elements, and state the different sub-topics on which the item is to be evaluated. Alongside this, we will normally state specific questions which will need to be answered in order to judge the item against the criterion. The answers to the questions may involve measurements, counts, assessments on subjective scales, statements of fact or possibly even opinion.

HOPE Research Methods Tony Fleet 71 Example: Website Evaluation Criteria Design Is the use of colour acceptable? Are the elements in harmony? Navigation Do the links work? How many links per page? Content How many words & graphics on page? Is the text interesting & informative? Interactivity What interactive features are used? Do these improve communication with the user? Coding What scripting is used? Is it clearly annotated?

HOPE Research Methods Tony Fleet 72 Example: Website Evaluation Form 1. Design Colour Safe Pallet used? Yes/ No Harmony of elements on page:good/ neutral/ poor 2. Navigation Number of links on the page:____ Number of working links____ 3. Content Total size of graphics files on page:____ MB Is the text: boring/ dull/ neutral/ interesting/ fascinating 4. Interactivity Tick and/or name all interactive features used: rollovers; dynamic images; image maps; ______, ______ 5. Coding What DTD has been used? _______________________

HOPE Research Methods Tony Fleet 73 Evaluation Criteria: Validity Issues For criteria to be valid, each of the criteria elements need to have face or content validity; the questions posed by the criterion needs to be relevant to the object, and relates to a feature of it We also need to be able to answer the questions in an objective manner, without recourse to guessing, or giving an ‘impressionistic’ response.

HOPE Research Methods Tony Fleet 74 Evaluation Criteria: Reliability Issues For criteria to be reliable, each of the criteria elements need to be clear and unambiguous, so that different assessors would interpret the criteria in the same way. We also need to ensure that repetition of the evaluation exercise will yield results which are not dissimilar to one another.

HOPE Research Methods Tony Fleet 75 Evaluation Methods References (SW Library) Britain, Sandy. - A framework for pedagogical evaluation of virtual learning environments. - Manchester : Joint Information Systems Committee, 1999. - (JISC Technology Applications Programme r.. - M0000712EL Broadbent, George Ernest. - The role of evaluation in user interface design : a practical implementation. - Liverpool : University of Liverpool, 1997. - p7270369 Redmond-Pyle, David. - Graphical user interface design and evaluation (GUIDE) : a practical process. - London : Prentice Hall, 1995. - 013315193x Smeltzer, Nicholas. - Critical analysis of the design and evaluation of a computer-based project. - Liverpool : University of Liverpool, 2001. - M0002704LO

HOPE Research Methods Tony Fleet 76 5c.3 Observational Research This section describes what you need to do if your research involves making detailed observations of people, events or objects.

HOPE Research Methods Tony Fleet 77 Observational Research In this case you are trying to determine Either: How an individual or group of individuals react, interact or behave in particular circumstances Or: How software or hardware performs when used by particular groups of people

HOPE Research Methods Tony Fleet 78 Participant Observation In this case, the researcher becomes one of the subjects, and works alongside the subject, monitoring behaviour and interacting with them. Pros:Researcher can fully ‘understand’ what the issues are. Can get real validity to research. Cons: Researcher can influence the research, alter opinions. Highly subjective & unreliable. Taking notes is difficult; relies on good memory.

HOPE Research Methods Tony Fleet 79 Non-Participant Observation In this case, the researcher studies the situation apart from subjects, taking notes, monitoring behaviour and observing interactions. The use of audio & video recording is useful here. Pros:Researcher can get overview of the situation, and achieve objectivity Cons: Researcher can only see resulting behaviours, not what is causing it or why it is happening.

HOPE Research Methods Tony Fleet 80 Observation Schedules An observation schedule can be a simple list of things to look for in a particular situation It can be far more complex; a minute by minute count of events such a mouse-clicks or verbal interactions between subjects.

HOPE Research Methods Tony Fleet 81 Observation Schedule An Example Observation of subject using Information Portal Subject: M / FAge: 18-21 / 21-30 / 31-50 / 50+ Date: _________Time: _________ Selected Navigation Tool : Mouse /Keyboard/ TouchScreen First 5 pages visited in order: __ __ __ __ __ Time to obtain to obtain required information: ___ min ___ sec Total Number of Pages visited: __ Feedback from subject: very positive / positive / neutral / negative / very negative

HOPE Research Methods Tony Fleet 82 Observation Schedules: Some Reliability Issues For an observation schedule to be reliable, it should require structured documentation of events. This will involve such things as checklists, minute-by-minute categorisation of activity and numerical data such as frequencies of occurrence and time intervals. Timings should have a clear start and end points. The schedule should leave little room for subjective judgement.

HOPE Research Methods Tony Fleet 83 Observation Schedules: Some Validity Issues For an observation schedule to be valid, it should refer to events which actually happen; each of the events on the sheet should be possible, and likely to occur. Timings should be possible to take, and not interfere with other observations which should be made. There should be room for observations which ‘enrich’ the data by adding detail to the numbers, offering explanation and illumination.

HOPE Research Methods Tony Fleet 84 Observational Research References (SW Library) Harding, Jacqueline. - How to make observations & assessments / Jackie Harding and Liz Meldon- Smith. - 2nd ed. - London : Hodder & Stoughton, 2000. - 034078038x Robertson, Kevin. - Observation, analysis and video / editors: Anne Simpkin and Penny Crisfield. - Leeds : National Coaching Foundation, 1999. - 1902523164 Simpson, Mary. - Using observations in small- scale research : a beginner's guide / Mary Simpson. - Glasgow : Scottish Council for Research in Education, 1995. - (SCREpublications ; 16 ; 130). - 1860030122

HOPE Research Methods Tony Fleet 85 5c.4 Survey-Type Research This section describes what you need to do in order to use human respondents to provide you with information.

HOPE Research Methods Tony Fleet 86 Survey methods There are two distinct elements here: The interview techniques that you adopt in order to elicit information from peopleinterview techniques The sampling methods that you adopt in order to select respondents for interviewsampling methods You will need to make rational choices for both of these elements, depending upon the focus of your research and the population under study.

HOPE Research Methods Tony Fleet 87 Interviewing We interview people face-to-face in order to try to find out exactly what they think. With the right questions, people respond with high quality information. The data that you get can be of high validity, since you have access to respondents’ own words.

HOPE Research Methods Tony Fleet 88 Types of Interviews Unstructured Interview Structured Interview Semi-Structured Interview Administered Questionnaire

HOPE Research Methods Tony Fleet 89 Open or Closed? When interviewing respondents, the main choice facing the researcher is whether to use open questions, which leave the respondent free to answer in any way they think fit, or closed questions which force the respondent to make particular choices, pre-determined by the researcher. OPEN: What is your experience of chat rooms? CLOSED: Do you think chat rooms should be monitored? (Yes/No/Maybe)

HOPE Research Methods Tony Fleet 90 Open or Closed? Validity & Reliability Issues In general, the data from open questions is richer, more illuminating, and more valid, as it can illustrate clearly why particular subjects think the way that they do, or why they behave in particular ways. Data from closed questions however, is more amenable to statistics. It is easier to summarise, spot trend and make comparisons. In general the data is more reliable. A good strategy is to use a mixture of both open and closed questions

HOPE Research Methods Tony Fleet 91 Unstructured Interviews Unstructured Interview Interviewer has no set agenda; the object is to get the respondent to talk freely about various topics. Pros: Can be high quality data, often get real insights into what people really think. Cons: Can be difficult for the novice researcher to carry out; need to be good at steering the conversation without forcing it. Time consuming; can only carry out a small number. Difficult to analyse

HOPE Research Methods Tony Fleet 92 Semi- Structured Interview Interviewer has a formal list of topics, which sets the agenda; however, the interviewer is free to take these in different orders, or to return to topics at different points. Here we include the idea of ‘focus’ groups, where a facilitator encourages the discussion of a particular topic. Pros: Data provided can be almost as good as unstructured interview, but in a more focussed manner. Cons: Can miss important ideas, because agenda set beforehand; interviews can also be time consuming & difficult to manage & analyse afterwards.

HOPE Research Methods Tony Fleet 93 Structured Interviews Interviewer has a list of topics to be taken in a particular order; questions are written, and read out. Pros: You can make good comparisons between different respondents; you also will have access to respondents’ thinking. Cons: Unless you have done some preliminary investigations, and extensively piloted the interviews, you may miss lots of important data. Takes time to do properly; can only carry out a few.

HOPE Research Methods Tony Fleet 94 Administered Questionnaire Interviewer has devised a list of questions which are read out; some multiple choice, some open-ended. Filled in either by respondent or interviewer. Pros: Interview can be quite brief; easy to make comparisons, data amenable to statistical analysis. Cons: Data may be ‘warped’ by the choice of questions. Respondent may be forced into giving fabricated answers.

HOPE Research Methods Tony Fleet 95 Questionnaires There are many different types of questionnaire, depending upon what you are trying to find out. As well as gathering factual data about the person, you may be trying to explore their: –Knowledge –Beliefs –Attitudes –Opinion –Behaviour If you need to design a questionnaire, see for example: http://www.leeds.ac.uk/iss/documentation/ top/top2/index.html http://www.leeds.ac.uk/iss/documentation/ top/top2/index.html

HOPE Research Methods Tony Fleet 96 Question Types Factual Data Binary (Yes/No) Single Selection from Categories Multiple Selection from Categories Attitude Scale items Focussed items to elicit categories Conditional Questions for routing Open-ended Questions Self-Reporting

HOPE Research Methods Tony Fleet 97 Writing Questions Put Factual Data (demographics) at end Put Explanations and Disclaimers at the start Keep questions as brief as possible Use non-technical language where possible Avoid leading respondents towards particular answers Use direct questions, no hypotheticals Use simple questions, no portmanteaus Use a mixture of positively and negatively worded questions Verify data by asking same question in different ways. See: http://www.analytictech.com/mb313/principl.htmhttp://www.analytictech.com/mb313/principl.htm

HOPE Research Methods Tony Fleet 98 Questionnaire Example 1. Which operating system are you currently using: Linux Windows Other 2. How would you rate the operating system? very poor poor good very good 3. Circle the tasks which you use your computer to do: Word processing Internet Access Program Development 4. Estimate the number of hours you use the computer for each week: ______ hours 5. Do you use broadband? Yes No

HOPE Research Methods Tony Fleet 99 Protocol Whatever your interviewing method, as part of the ethical constraints on Hope Researchers, you are required to do the following: Explain to the respondents what the research is about. Tell them that they will not be identified by name in the final report, and that any views that they express will be in confidence. Explain to them that if there are questions with which they feel uncomfortable, they do not have to answer.

HOPE Research Methods Tony Fleet 100 Attitude Measurement You may wish to incorporate ‘attitude measurement’ into your questionnaire as either a major or a minor feature. There are several different types of scales which can be used to elicit numerical measurements of attitude: –Likert Scaling –Thurstone Scaling –Guttman Scaling –Semantic Differential Scaling See: http://www.socialresearchmethods.net/kb/scalgen.htmhttp://www.socialresearchmethods.net/kb/scalgen.htm

HOPE Research Methods Tony Fleet 101 Likert Scales (1932) This consists of items like “I would not trust a bank’s website to keep my details secure” Respondents are asked to use a scale such as 1=Strongly disagree; 2=disagree; 3=neutral; 4=agree; 5=strongly agree Scales can vary: 0-4, 1-7 etc. Some proponents suggest removing the neutral category to force a choice, but this can reduce validity. Scores on individual items are totalled to obtain the respondent’s score; this is often shown as an average.

HOPE Research Methods Tony Fleet 102 Thurstone Scales (1928) This pre-ranks 11 statements with numerical values 1-11; each statement carries a numerical score, for example: 1 = “Teleworking is an impossible concept” 2 =“Teleworking may be OK in exceptional cases” 8 = “If offered the opportunity, I would try teleworking” 11 = “In 20 years’ time, everyone will be teleworking” Clearly, lots of preparatory work needs to be done to generate and rank the statements. A respondent’s score, is the average numerical value of all the items they agree with.

HOPE Research Methods Tony Fleet 103 Guttman Scales (1944) This pre-ranks statements in order, very similar to Thurstone scaling, except here we attempt to construct the scale so that if a person agrees with item 4 on the scale, they will also agree with items 1,2 and 3. When the questions are administered, the items are muddled, but each retains a ranking. The respondent’s score is the sum of the ranks associated with the items he or she agreed with.

HOPE Research Methods Tony Fleet 104 Osgood’s (1957) Semantic Differential Scales Respondents are asked to rate an idea or an object against a series of opposing adjectives or descriptions, for example: Using Learnwise Exciting [ ] [ ] [ ] [ ] [ ] [ ]Dull Hard [ ] [ ] [ ] [ ] [ ] [ ]Easy Frustrating[ ] [ ] [ ] [ ] [ ] [ ]Stimulating The scale asks respondents to tick the box nearest the descriptor that they agree with. Each box has a numerical value: e.g. 1,2,3… 7 The respondent’s score can be portrayed graphically, or as a total of numerical values.

HOPE Research Methods Tony Fleet 105 Questionnaires & Interviews: Reliability Issues For questionnaire results to be reliable, you need closed questions that are precisely framed, unambiguous, with response categories that are well-defined, exhaustive and exclusive. You need to ask the same question in different ways, and you need to collate the information into statistical summaries and expose the data to rigorous statistical testing. You need to use as large a sample as possible, and even out any random fluctuations in the data by using summary statistics and hypothesis testing.

HOPE Research Methods Tony Fleet 106 Questionnaires & Interviews: Validity Issues For questionnaire results to be valid, you need response categories which enable the respondent to express their views accurately; this can best be done with open questions. You should take time with the respondent and respect the data that they provide for you. You need to ensure that the sample is representative of the population, and large enough for the results to be statistically significant.

HOPE Research Methods Tony Fleet 107 Questionnaire References On-Line http://www.tardis.ed.ac.uk/~kate/qmcweb/q cont.htmhttp://www.tardis.ed.ac.uk/~kate/qmcweb/q cont.htm http://www.leeds.ac.uk/iss/documentation/t op/top2.pdfhttp://www.leeds.ac.uk/iss/documentation/t op/top2.pdf http://www.statpac.com/surveys/ http://www.fao.org/docrep/W3241E/w3241 e05.htm#chapter%204:%20questionnaire %20designhttp://www.fao.org/docrep/W3241E/w3241 e05.htm#chapter%204:%20questionnaire %20design http://www.surveysystem.com/sdesign.htm

HOPE Research Methods Tony Fleet 108 Questionnaire References (1) (SW Library) Foddy, William. - Constructing questions for interviews and questionnaires : theory and practice. - Cambridge : Cambridge University Press, 1993. - 0521467330 Fowler, Floyd J.. - Improving survey questions : design and evaluation. - London : Sage, 1995. - 0803945833 Frazer, Lorelle. - Questionnaire design and administration : a practical guide / Lorelle Frazer. - Brisbane : Wiley, 2000. - 0471342920 Gillham, W. E. C., William Edwin Charles, 1936-. - Developing a questionnaire. - London : Continuum, 2000. - (Real world research). - 0826447953

HOPE Research Methods Tony Fleet 109 Questionnaire References (2) (SW Library) Oppenheim, A. N., Abraham Naftali, 1924-. - Questionnaire design, interviewing and attitude measurement / A.N. Oppenheim. - New ed. - London : Continuum, 2000. - 0826451764 Wengraf, Tom. - Qualitative research interviewing : biographic narrative and semi-structured interviews - London : SAGE, 2001. - 0803975007 Young, Pauline V. - Scientific social surveys and research : an introduction to the background,. - Englewood Cliffs, N.J. : Prentice-Hall, 1966. - (Prentice- Hall Sociology Series). - m0859969 Youngman, Michael Brendan. - Designing and analysing questionnaires. - Maidenhead, Berks : TRC Rediguide. - (Rediguide ; 12). - m0891542

HOPE Research Methods Tony Fleet 110 Sampling The question of how to select the respondents for the sample is always tricky. The principle behind sampling, is that you should ensure that the sample is appropriate in order for you to generalise your results to the population under study. There are essentially three ways in which this is done: –Random Sampling –Quota Sampling –Stratified Random Sampling

HOPE Research Methods Tony Fleet 111 Random Sampling This method requires you to get a list of all the population, as near complete as you can find, then use some random selection method (such as shutting your eyes and stabbing a pen at the list, or allocating using random numbers) The idea is that every person in the population has an equal chance of ending up in the sample. Most statistical methods assume that you are sampling randomly, and it is the only method which overall is guaranteed to ensure that samples are free from bias, and therefore provide validity.

HOPE Research Methods Tony Fleet 112 Quota Sampling Here, you identify particular sectors of the population, such as men, women, those under 21, over 21, employed, unemployed etc., and put quotas on the number of people in each category. For example, in a quota sample of 12, we might have: –2 males under 21; 2 females under 21 –4 males over 21 ; 4 females over 21 The purpose here is not to get a sample which represents the population in its entirety, but to ensure that views from important sections of the population are represented.

HOPE Research Methods Tony Fleet 113 Stratified Random Sampling One of the problems with Random Sampling, is that if you take small samples, you can very well end up with a biased sample, by chance. In Stratified Random sampling, you would measure what proportions of the population lie in each category or strata, and select your sample so that you get precisely those proportions in those strata For example in your population you might have: –15% males under 21; 10% females under 21 –30% males over 21 ; 45% females over 21 If you select a sample of 200 you would need stratified samples: –30 males under 21; 20 females under 21 –60 males over 21 ; 90 females over 21 The allocation of subjects to samples should be done randomly. The purpose here is to get a valid sample which which represents the population in its entirety.

HOPE Research Methods Tony Fleet 114 Other Sampling Methods Snowball Sampling: This method is used whenever you are dealing with a tricky subject, where people may be doing devious or illegal activities (for example hacking, virus creation etc.). In this case, you would use one respondent to suggest the name of another who might be willing to be interviewed; that one would lead to several others and so on. Cluster Sampling: This method might be used if you have a diverse population (such as those living in African urban communities). Here you would randomly select particular cities, and sample within neighbourhoods within those cities. Convenience Sampling: This is used mainly for investigative research. It relies on the fact that people appearing at a particular location or time will do so, ‘at random’ (may not be true). We use this fact to stand in one place and interview them.

HOPE Research Methods Tony Fleet 115 Sampling: On-Line References This is an easy introduction: http://www.csm.uwe.ac.uk/~pwhite/SURVEY1/node26. htmlhttp://www.csm.uwe.ac.uk/~pwhite/SURVEY1/node26. html This is more complex and technical: http://www.socialresearchmethods.net/kb/external.htm

HOPE 6. Data Analysis This section of the presentation looks at the different types of data available, and how this can be analysed

HOPE Research Methods Tony Fleet 117 Analysis of Research Data Undertake Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Here we examine how the data produced by the research can be presented effectively and statistically analysed.

HOPE Research Methods Tony Fleet 118 Data Analysis 6aTypes Of DataTypes Of Data 6bExtracting Data for AnalysisExtracting Data for Analysis 6cPresentation of DataPresentation of Data 6cPrinciples of Statistical TestingPrinciples of Statistical Testing 6dSelected TestsSelected Tests 6eAnalytical ToolsAnalytical Tools

HOPE Research Methods Tony Fleet 119 6aTypes of Data In this section we describe the different types of data that you might encounter, and what you might do with it in your study.

HOPE Research Methods Tony Fleet 120 Types of Data ‘Measurement’-type data, given in units measured on a decimal scale. e.g. MB/sec, times, averages and other statistical summary values. ‘Counted’ data: number of occurrences of events or objects: e.g. number of pages visited ‘Ranking’ data: subjective impressions, such as marks out of 10, rankings etc. e.g. Interface rated as very poor (1), poor (2), good (3) or very good (4). Categorical data: Names of different things or categories, such as those named by an interviewee (NB these may be counted) e.g. “Manager A discussed 3 concerns: security, access and reliability” Descriptive data, such as that produced by interviews or observational field-notes: e.g. “Subject A tried to click on all the blue text, even though it was not underlined”.

HOPE Research Methods Tony Fleet 121 ‘Measured’ Data Such data is called “continuous”, “ratio scale”, or “real number” data This type of data is easiest to use; it can be arithmetically manipulated (added, subtracted, multiplied, divided, summarised) it is also likely to have good, predictable statistical features. There are lots of methods to pick from. The inclusion of valid measured data will add reliability to your study.

HOPE Research Methods Tony Fleet 122 ‘Counted’ Data Such data is normally called “discrete”, “interval scale”, or “integer” data It can be added, subtracted and averaged. ‘Counted’ data, has almost all the features of measured data, especially if the numbers of the count are large (above 30 or so). For small numbers (e.g. less than 30 observations in total), methods are limited. The inclusion of valid counted data will almost certainly improve the reliability of your study.

HOPE Research Methods Tony Fleet 123 ‘Ranking’ Data Such data is normally called “interval scale data” data. In order to use it effectively, there are several techniques; the usual one is to convert the data into numerical values (e.g. “Yes” = 1, “No” = 0) When this has been done, it can be added, subtracted and averaged. ‘Interval’ data needs care; the data ‘looks like’ counts where the numbers are small, but are simply subjective impressions. To deal with such data we need to summarise large volumes of it data. The inclusion of valid counted data, suitably summarised can add reliability to your study.

HOPE Research Methods Tony Fleet 124 ‘Categorical’ Data Such data is normally called “nominal” data. In order to use such data effectively, the objects, classes or ideas that these categories represent should have the items within them counted, evaluated on an interval scale and/or summarised. A preliminary study may have found, for example that there are 4 main uses for spreadsheets in business: Finance, Timesheets, Inventory and General Calculation. Businesses may be asked to rank these in order of importance, or to rate their usefulness. ‘Categorical data is much more difficult to deal with. Processes are time consuming and highly subjective; There are relatively few statistical tools which deal effectively with categorical data. The inclusion of valid ‘Categorical’ data, suitably summarised can add both validity and reliability to your study.

HOPE Research Methods Tony Fleet 125 ‘Descriptive’ Data Descriptive data is simply a verbal description of something that happened, or what a respondent did, or a quote from an interview. It is the most difficult to deal with effectively. There are methods for extracting summary information from such data (see Grounded Theory: Glazer & Strauss); however, these are not for the novice researcher in IT, as they are very time consuming. The best use for such data is for illustration and supporting evidence elsewhere. The inclusion of appropriate ‘Descriptive’ data, suitably selected can improve the overall validity of your study.

HOPE Research Methods Tony Fleet 126 Types Of Data References (SW Library) Research methods in education and the social sciences / [Research Methods - Block 5 : Classification and measurement. - Milton Keynes : Open University Press, 1979. - (DE304 : a third.. - 0335074405 Research methods in education and the social sciences / [Research Methods. - Block 6 : Making sense of data. - Milton Keynes : Open University Press, 1979. - (DE304 : a third level course. - 0335074413

HOPE Research Methods Tony Fleet 127 Types of Data References (SW Library) Chatfield, Christopher. - Statistics for technology : a course in applied statistics. - 2nd. - London : Penguin Books, 1978. - 0412157500 Kranzler, Gerald. - Statistics for the terrified / Gerald D. Kranzler, Janet P. Moursund. - Englewood Cliffs, N.J. : Prentice Hall; London : Prentice-Hall International, 1995. – 0131838318 Pentz, Mike. - Handling experimental data. - Milton Keynes : Open U.P., 1988. - 0335158242 Salkind, Neil J.. - Statistics for people who (think they) hate statistics. - Thousand Oaks, Calif.; London : Sage, 2000. - 0761916210

HOPE Research Methods Tony Fleet 128 6bExtracting Data For Analysis Data produced by the research process does not have to be numerical in order to be of value. However, statistical techniques work best where the data uses numerical values, or can be converted to numerical scales.

HOPE Research Methods Tony Fleet 129 Survey Data Questionnaire data should be the easiest data to extract. Ideally the questionnaire will have been designed with the methods of analysis in mind, so that questions are framed in such a way as to facilitate graphical representation and statistical testing. Interview Data can be more problematic; you may need to ‘invent’ categories, and to count up instances of different cases, both in the same interview, and with different subjects.

HOPE Research Methods Tony Fleet 130 Questionnaire Data Examples Percentage of Respondents using Broadband: 82% In the survey, 93% of people used their computer to access the internet, compared to 37% who used it for word-processing, and 7% who used it to develop programs. 87% of Linux users rated their operating system as good or very good, compared to 64% of Windows users. The average number of hours spent on the computer each week by men was 9.3, compared to 3.7 for women.

HOPE Research Methods Tony Fleet 131 Interview Data Examples 5 out of the 6 interviewees were using Broadband. During the interview, all interviewees mentioned the positive benefits of moving to broadband. For example one respondent said “My daughter can use the phone while I am replying to emails”. Altogether there were 25 instances in the interviews where benefits were named, as opposed to only three cases where drawbacks were noted. These cases broke down as follows: –20% of cases referred to accessibility issues –60% of cases referred to improved speed –20% of cases referred to improved compatibility However, it must be noted that all the drawbacks were cited by a respondent who reportedly used the internet for over 70 hours per week. The other five respondents’ use was an average of 15.7 hours, with totals ranging from 4 hours to 25 hours.

HOPE Research Methods Tony Fleet 132 Observational Data Pure observational field notes are unlikely to contain numerical data. As with interview data, you may have to ‘invent’ appropriate categories and count up instances or occurrences. Observation schedules are easier to deal with. As with questionnaires, these should have been designed with the methods of analysis in mind, so that items are framed in such a way as to facilitate graphical representation and statistical testing.

HOPE Research Methods Tony Fleet 133 Observational Data Examples Out of the 40 people observed in the study, 36 of them (90%) used the mouse as the first point of contact with the portal. After the entry page, 70% visited page 3, 20% visited page 7 and the rest visited pages 2, 4 and 9. The mean time taken to access the information by novice users was 163.5 seconds (SD = 24.8 secs); however, those who were subjected to the brief training session were able to access the information in a mean time of 48.3 seconds (SD = 13.9 secs) 95% of the users rated the interface good or very good.

HOPE Research Methods Tony Fleet 134 Experimental Data Experiments should have been designed with specific comparisons in mind, where independent variables are identified & controlled, and dependent variables are measured in some way. The methods of analysis should have been identified beforehand, and the whole is susceptible to graphical representation and statistical testing.

HOPE Research Methods Tony Fleet 135 Experimental Data Examples System A 4.5MB: 1.8 secs 25.3MB: 7.2 secs 185MB: 35.4 secs 387MB: 71.8secs 875MB: 165.3 secs 1.56GB 299.5 secs 4.76GB timed out Using two identical machines, System A using Windows XP and System B using Linux, accessing the website using a 56Mb/sec modem. Download times were: System B 4.5MB: 1.6 secs 25.3MB: 6.4 secs 185MB: 36.2 secs 387MB: 74.3secs 875MB: 155.8 secs 1.56GB 287.5 secs 4.76GB 889.3 secs

HOPE Research Methods Tony Fleet 136 Evaluation Data Evaluation is unlikely to yield lots of numerical data. Categories on the Evaluation Sheet may well involve numerical values: counts, times, file sizes etc., and these are amenable to graphical representation & testing. Other categories may require judgement on ‘descriptors’, or on quality. It is possible to convert some of these to simple scales (0,1,2,3 etc), and to rank items in order, for example. Yes/No can be encoded as 1 and 0.

HOPE Research Methods Tony Fleet 137 Evaluation Data Examples The 5 evaluators rated the 4 packages as follows: Evaluator 12345 BestCCDCC |ADBBD |DBCDB WorstBAAAA Package C performed best overall; it achieved higher grades on 6 out of 7 criteria, and received a total rating of 8.3, compared to ratings of 6.1, 4.5 and 2.8 for packages B, D and A The reasons for Package A’s poor performance were cited by the evaluators as: poor readability, confused navigation, and lack of interactivity. 4 out of 5 of the evaluators rated the interface design for Package A as poor or very poor.

HOPE Research Methods Tony Fleet 138 Data Collection References (SW Library) Research methods in education and the social sciences. – Block 4 : Data collection procedures. - Milton Keynes : Open University Press, 1979. - (DE304 ; block 4). - 0335074251 Research methods in education and the social sciences / [Research Methods. - Block 6 : Making sense of data. - Milton Keynes : Open University Press, 1979. - (DE304 : a third level course. - 0335074413

HOPE Research Methods Tony Fleet 139 6c Presenting Data This section of the presentation looks at some of the different ways in which you might present data, and the different ways in which you can draw meaning from it.

HOPE Research Methods Tony Fleet 140 Presenting Numerical Information Frequency Tables Cross Tabs Time Series Data

HOPE Research Methods Tony Fleet 141 Frequency Tables In this example the number of errors cited when 3 different websites are run through an html code checker are tabulated. It is normal where percentages are given, to quote the actual numbers involved, OR the total number of cases. The percentages allow an easy comparison to be made on a ‘standard scale’

HOPE Research Methods Tony Fleet 142 CrossTabulations The purpose of a Cross-tabulation is to examine the relationships between two different variables. In this example, the number of men and women of different age groups expressing an interest in Blue Tooth are cross-tabulated. Here the information does not really show what is happening. Here we have used percentages, and it is now clear that although there is a reduction in interest in both males and females as they get older, it is far more dramatic for females.

HOPE Research Methods Tony Fleet 143 Time Series Data A Time Series simply logs the value of a particular variable over a period of seconds, days or weeks. Because of amount of numerical information, it is very difficult to make sense of time series data unless it is summarised or presented in graphical form. This data shows the result of an experiment to randomly access websites, and download any.jpg files on the first page onto a hard drive. All we can see here is that the numbers are declining, but we cannot really get a sense of how this is happening

HOPE Research Methods Tony Fleet 144 Presenting Graphical Information Simple Bar Charts & Histograms Comparative Charts Scattergrams Time Series Data

HOPE Research Methods Tony Fleet 145 Simple Bar Charts & Histograms Simple graphs can be highly effective in making a point. This one shows the fact that Package A clearly has been rated higher than the other two packages.

HOPE Research Methods Tony Fleet 146 Comparative Charts All three of these charts display the same information. Which makes it clearer?

HOPE Research Methods Tony Fleet 147 Scattergrams These are useful if you are trying to demonstrate a relationship between two variables. We could add a ‘regression line’ to make the relationship explicit.

HOPE Research Methods Tony Fleet 148 Time Series A variant on the scattergram with lines added, records of what happened over a period of time to the quantity measured. Here the data shows an almost uniform rate of loss of sectors, minute- by-minute.

HOPE Research Methods Tony Fleet 149 Simple Statistics Data needs to be summarised using a variety of standard tools: Proportions, expressed in percentages; Averages, such as means, medians etc. Measures of variability, such as ranges, standard deviations etc. In this example, 3 different websites have been given gradings out of 10 by a panel of 6 experts. In the summaries, it is clear that A outperforms the others, and C is the worst. However, there is a lot more inconsistency in A’s scores than any of the others.

HOPE Research Methods Tony Fleet 150 St. John, P. R.. - Methods of presenting fieldwork data / P.R. St. John, D.A. Richardson. - Sheffield : The Geographical Association, 1989. - 0948512164

HOPE Research Methods Tony Fleet 151 6d Statistical Testing This section of the presentation explains the principles behind statistical testing, and describes the some of the different types of statistical tests available. It also explores probability theory.

HOPE Research Methods Tony Fleet 152 Hypothesis Testing Undertake Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses; Draw Conclusions If possible, the final section of your study should include some statistical tests to provide rigour to your arguments.

HOPE Research Methods Tony Fleet 153 Probability Theory (1) If your data consists of measurements, measured then you can probably assume that you have a ‘Normal’ Distribution, such as IQ scores. Lots of people with average IQ scores in the middle, but with very low numbers with High IQs or Low IQs. http://www.socialresearchmethods.net/kb/sampstat.htm

HOPE Research Methods Tony Fleet 154 Probability Theory (2) Even if your data is not measurement data, you can still make use of ‘Normal’ Distribution techniques, provided that: Your data is counted, and you have high numbers (over 50 observed in each category) You are working with averages of 10 items or more.

HOPE Research Methods Tony Fleet 155 Probability Theory (3) Probability tries to measure how likely a particular observed event is, before it occurs. For example, if we picked an undergraduate student at random, and asked what is the likelihood of that student winning the lottery, it would be a very low probability event (prob = 0.0000001) On the other hand, if we asked what is the likelihood of that student graduating, the probability would be very high. (prob = 0.95)

HOPE Research Methods Tony Fleet 156 Random Chance In time, all things are possible. Mountains might crumble, pigs might fly, you might win the lottery. If we wait around long enough these events will happen, because each event has a small, but finite probability. Random chance will dictate the day, the year, or the millennium in which these occur.

HOPE Research Methods Tony Fleet 157 Making predictions Because of the way that statistics works, we can use the features of data, and the idea of random chance to make predictions. If we were to walk into the dining hall, and select a student at random, we could predict the likelihood that: 1.The student has an IQ between 85 and 115 (about 70%) 2.The student has an IQ over 130 (about 2.5%) The first is a high probability event, and that’s what we would normally expect to happen. Clearly the second is a low probability event, and therefore we would be very surprised if it occurred.

HOPE Research Methods Tony Fleet 158 Hypothesis Testing We all work on assumptions every day. Let us suppose that you have come to college on what you think is a normal day, only to find the rooms deserted, no lecturers, no students. What is the explanation? Is it that: 1.You came in on a Sunday by mistake, thinking it was Monday 2.The college has been evacuated due to a leak at a nearby toxic waste incinerator.

HOPE Research Methods Tony Fleet 159 Hypothesis Testing Most people would immediately suggest (1), because (2) is a very low probability event. However, we can use the situation to reason thus:  If we assume that Today is Monday, and we come in and find the college deserted, then the reason must be (2).  However, this is such a low probability event that we would not normally expect to witness it.  This means that something is wrong.  It is more likely that our original assumption is incorrect, and Today is NOT Monday.

HOPE Research Methods Tony Fleet 160 Hypothesis Testing This is formalised into a process which goes like this: We set up two competing hypotheses (normally called H0 and H1). We make observations in data, and calculate the probability of these observations occurring by random chance. We base this calculation on the assumption that H0 is true. If the probability is very low, we reject H0 (the null hypothesis), and accept H1, the alternative hypothesis.

HOPE Research Methods Tony Fleet 161 Hypothesis Testing An Example Let us set up two competing hypotheses:  H0: MSc. Computing is equally attractive to men and women.  H1: MSc Computing is more attractive to one gender than the other. We now observe the number of students in a group, and count the number of males and the number of females, and ask what is the likelihood of observing such, if the subject were really equally attractive..

HOPE Research Methods Tony Fleet 162 Hypothesis Testing An Example Suppose in the class there are 8 men 2 women If H0 is true, then the probability of this event is p=0.01; this is 1%, extremely low. We therefore reject H0, and accept H1, concluding that MSc. Computing is more attractive to one gender than the other.

HOPE Research Methods Tony Fleet 163 How low can you go? The question now comes, how low does a probability have to be in order for you to conclude that the event is not simply occurring as a result of random chance. For most circumstances, we would use p<.05 or less than 5%, and we would say such a result is statistically significant If p <.005, then the result is highly significant. If p<.0005, then the result is very highly significant.

HOPE Research Methods Tony Fleet 164 What does this all prove? Your results may not be statistically significant, that does not mean that they are not true; it simply means that the data you collected was not sufficient to decide things one way or another. If your results are significant, then you need to decide whether the interpretation that you are putting in them is actually the correct one; there may be other explanations for the results than the one that you have stated as H1. In other words, you should be cautious when you write this up; be analytical, and question everything.

HOPE Research Methods Tony Fleet 165 6e Examples of Statistical Tests This section looks at some examples of statistical tests that you can undertake in different contexts. It distinguishes between Parametric and non-Parametric testing

HOPE Research Methods Tony Fleet 166 Parametric Testing (1) Parametric Testing assumes that the data you are working with follows a Normal Distribution. You can use these techniques if Your data is measured Your data is counted, but there is a large number in the count Your data is in numerical categories, or categories which can be translated into numbers, and the overall shape of the data is a ‘bell’ curve. You are working with averages, based on samples of 10 or more.

HOPE Research Methods Tony Fleet 167 Parametric Testing (2) These are some samples of Parametric Tests Z-Test –Single sample: Null hypothesis is that the average of a sample is equal to some theoretical value. Paired Sample T-Testing –Two samples: Null hypothesis is that there is no difference between the two samples ANOVA –Three or more samples; null hypothesis is that there is no difference between the samples. Correlation –Two measurements taken on the same data; null hypothesis is that the measurements are not related in any measurable way.

HOPE Research Methods Tony Fleet 168 Non-Parametric Testing (1) Non-Parametric Testing assumes that the data you are working with does not follow a Normal Distribution. You need to use these techniques if Your data is counted, with low numbers in each category Your data is in categories which cannot be translated into a numerical scale Data where the overall shape of the data is not a ‘bell’ curve. You are working with medians or other statistical summary values which are not averages.

HOPE Research Methods Tony Fleet 169 Non-Parametric Testing (2) These are some samples of Non-Parametric Tests: Chi-Squared Test –Crosstabs or Contingency Tables showing the effect of two different factors: Null hypothesis that the two factors are independent. Mann-Whitney U-test –Two samples: Null hypothesis that there is no difference between the two samples Phi Test –2 x 2 Crosstab; Null hypothesis is that the factors are unrelated. Spearman’s Correlation –Two judges asked to rank a series of items; null hypothesis is that the rankings are statistically unrelated.

HOPE Research Methods Tony Fleet 170 Selected Test Examples t-Test ANOVA Correlation & Regression Chi-Squared Test

HOPE Research Methods Tony Fleet 171 t-Test This is used to test whether a the mean of an observed sample is significantly different from an hypothesised value. In the example shown, 12 respondents were given a Likert-scaled questionnaire, to determine their attitude towards chat-room ‘behaviour guidelines’. A neutral score would be 2.5 A t-test can be used to test the hypothesis: Ho: Respondents disagree with (or are neutral to) the guidelines The t-statistic is calculated to be t = 2.72 at p<.05; so we reject Ho, and conclude that the respondents agree with the guidelines.

HOPE Research Methods Tony Fleet 172 ANOVA This is used to test whether a the means of three or more samples are significantly different from one another. The example above shows the average ratings from 6 different evaluators of three different web-authoring packages, A, B & C on a variety of ten-point scales. We can test Ho: There is no difference between the ratings of the packages, using an ANOVA (Analysis of Variance) test. The results below show a p value of 0.004881 We reject Ho, and conclude that the packages ARE different. The result is highly significant.

HOPE Research Methods Tony Fleet 173 Correlation & Regression This is used to determine whether there is a relationship between two variables, and if so, what? The data collected shows the download times for 10 files of different sizes. There is a correlation of 0.998864 between the two variables, showing that a very strong relationship exists. The regression analysis below shows that this can be expressed as: Download Time = 70.65 x File Size - 1.49

HOPE Research Methods Tony Fleet 174 Chi-Squared Test This is used to determine whether the effects of two or more factors are independent. The data above shows the number of respondents of each gender in different age categories who expressed interest in using Blue Tooth technology within the next 12 months. A Ch-Squared test is used to test the hypothesis: Ho: There is no relationship between age and gender in potential Blue Tooth use. The probability value of the Chi-Squared Statistic is p = 0.00893 We would reject Ho, and conclude that there are differences between the age profiles of men and women potential users of Blue Tooth.

HOPE Research Methods Tony Fleet 175 Which test? http://research.med.umkc.edu/tlwbios tats/choosetest.htmlhttp://research.med.umkc.edu/tlwbios tats/choosetest.html http://staff.harrisonburg.k12.va.us/~g corder/stats_Index_Page.htmlhttp://staff.harrisonburg.k12.va.us/~g corder/stats_Index_Page.html

HOPE Research Methods Tony Fleet 176 Statistical Testing http://www.statsoft.com/textbook/stat home.htmlhttp://www.statsoft.com/textbook/stat home.html http://faculty.vassar.edu/lowry/webtex t.htmlhttp://faculty.vassar.edu/lowry/webtex t.html http://www.tufts.edu/~gdallal/LHSP.H TMhttp://www.tufts.edu/~gdallal/LHSP.H TM

HOPE Research Methods Tony Fleet 177 Statistical Testing References (SW Library) Ferguson, George A. - Statistical analysis in psychology and education. - 3rd. - Tokyo : McGraw-Hill Kogakusha, 1971. - m0891134 Peers, Ian S.. - Statistical analysis for education and psychology researchers. - London : Falmer Press, 1996. - 0750705051 St. John, P. R.. - Methods of statistical analysis of fieldwork data / P.R. St. John, D.A. Rich. - Sheffield : The Geographical Association, 1989. - 1899085165

HOPE Research Methods Tony Fleet 178 6f Analytical Tools In order to analyse the data, you will need a package that is capable of arithmetic and other data manipulation, and will allow you to display the data in the form of tables and charts, as well as to perform statistical tests. There are several packages that you can use in order to analyse your data, but two of the main ones are: –Microsoft Excel –SPSS

HOPE Research Methods Tony Fleet 179 Excel or SPSS? Excel will be OK to use if all your data is numeric, and that you have a clear idea of what you want to do, and how to do it. Excel is good at data display, but can only perform limited statistical testing, mainly parametric. SPSS is better at statistical testing, is highly versatile and can do lots of clever stuff. You will need this if you intend to analyze questionnaire data, or have lots of complex, categorical data and need to do non- parametric testing.

HOPE Research Methods Tony Fleet 180 Using Excel Excel is available on all computers, and there is an extensive help facility embedded in it. You will probably have worked with it before, so your ‘learning curve’ is fairly gentle. If you have only a few items to display, and you are not going to do any heavy statistical testing, then Excel is probably the easiest tool to use.

HOPE Research Methods Tony Fleet 181 Example of Excel Use (1) You can type data directly into an Excel worksheet. Excel has formulae to calculate statistics, such as: = Average(C3:C8) = StDev(C3:C8)

HOPE Research Methods Tony Fleet 182 Example of Excel Use (2) This is an example of a chart easily produced from the data on the previous slide using Excel.

HOPE Research Methods Tony Fleet 183 Example of Excel Use (3) Excel can produce a range of descriptive statistics very easily.

HOPE Research Methods Tony Fleet 184 Using SPSS SPSS is “Statistical Package for the Social Sciences” and is an industry standard tool for data analysis. If you are doing lots of work with numerical data (e.g. analysing questionnaires) then you will need to use SPSS. SPSS is available via the F: drive, and it is also available for £5 from the technician in the Psychology Lab (GLB)

HOPE Research Methods Tony Fleet 185 SPSS Tutorial

HOPE Research Methods Tony Fleet 186 Example of SPSS Use Part of a study looks at file upload times under 10 different conditions, using two types of browser: Netscape or Internet Explorer. The same circumstances relate to each pair of readings NetscapeInt Exp 13.4012.90 13.8015.70 12.9011.70 15.3014.60 16.4011.40 14.7014.20 15.8015.30 17.1016.10 12.7011.50 11.8012.10

HOPE Research Methods Tony Fleet 187 Setting Up Variables First of all we set up descriptors for each of the two variables: –Netscape –Internet Explorer

HOPE Research Methods Tony Fleet 188 Entering Values Then we Enter the 10 values for each variable, corresponding to the different cases.

HOPE Research Methods Tony Fleet 189 Analysing the data We can select graphical and numerical summaries

HOPE Research Methods Tony Fleet 190 Summaries Results: It looks as if Netscape generally takes longer to upload files than IE: 14.39 secs for NS as opposed to 13.55 secs for IE We might try to test this as a hypothesis.

HOPE Research Methods Tony Fleet 191 Results of a T-Test We test: H0: there is no difference between sample means. H1: There is a difference between sample means. P = 0.158. Unfortunately this is greater than.05, and not low enough to reject the null hypothesis; the result is not significant.

HOPE Research Methods Tony Fleet 192 Using SPSS http://www.utexas.edu/its/rc/tutorials/ stat/spss/spss1/index.htmlhttp://www.utexas.edu/its/rc/tutorials/ stat/spss/spss1/index.html http://www.strath.ac.uk/IT/Docs/SPS S/spss.htmlhttp://www.strath.ac.uk/IT/Docs/SPS S/spss.html http://www.leeds.ac.uk/iss/document ation/math.htmlhttp://www.leeds.ac.uk/iss/document ation/math.html http://www.indiana.edu/~statmath/stat /spss/win/giant.htmlhttp://www.indiana.edu/~statmath/stat /spss/win/giant.html

HOPE Research Methods Tony Fleet 193 Using SPSS References (SW Library) Dancey, Christine P.. - Statistics without maths for psychology : using SPSS for Windows / Christine. - 2nd ed. - Harlow : Prentice Hall, 2002. - 0130336335 Kinnear, Paul R.. - SPSS for Windows made simple : release 10 / Paul R. Kinnear, Colin D. Gray. - [New ed.]. - Hove : Psychology Press, 2002. - 1841691186 Pallant, Julie. - SPSS survival manual : a step by- step guide to data analysis using SPSS for Windows. - Buckingham : Open University Press, 2001. - 0335208908 West, Robert. - Computing for psychologists : statistical analysis using SPSS and MINITAB. - London : Harwood Academic, 1991. - 371865086x

HOPE 7. Problems with The Research Process

HOPE Research Methods Tony Fleet 195 Research Pitfalls Finally, we look at some problems that you might encounter which invalidate your research – and how to prevent these.

HOPE Research Methods Tony Fleet 196 Some Research Pitfalls (1) Researcher Bias Problem: Introducing your personal prejudices or preconceptions into the research, and obtaining invalid results. Solution: Counter by an attitude shift: we do not try to prove a hypothesis is true, rather we test whether or not a hypothesis is true, and keep an open mind as to the result.

HOPE Research Methods Tony Fleet 197 Some Research Pitfalls (2) The Hawthorne Effect Problem: Subjects behave differently because they know they are participating in Research. Solution: Use of control group in experiments; concealed observation techniques; surveillance methods; naturalistic data collection methods

HOPE Research Methods Tony Fleet 198 Some Research Pitfalls (3) The Halo Effect Problem: If Subjects rate something as good or bad overall, they are likely to rate all features of it as good or bad, without discriminating between them. Solution: In questionnaires & evaluations, use a variety of similar questions, positively & negatively worded.

HOPE Research Methods Tony Fleet 199 Some Research Pitfalls (4) Pseudoscience Problem: Setting out to prove a hypothesis by only looking for evidence which confirms it, and disregarding evidence which contradicts it. Solution: Explicitly stating what would be disconfirming evidence for a hypothesis, and setting out to look for it.

HOPE Research Methods Tony Fleet 200 Some Research Pitfalls (5) Sample Bias Problem: Getting misleading results, because the sample on which you have based your analysis is consistently unrepresentative. Solution: Identifying the group of people, objects or events you are trying to generalise about, and selecting a sample which has the same characteristics as that group, or is a random selection from it.

HOPE Research Methods Tony Fleet 201 Some Research Pitfalls (6) Overgeneralisation Problem: Making sweeping claims that the results that you have observed from one or two cases will be present in the population at large. Solution: Be cautious in your claims, and use moderate language, such as ‘likely’ or ‘probable’ causes; only claim that your results are ‘significant’ when they are actually statistically significant, after carrying out a statistical test and rejecting the null hypothesis.

HOPE Research Methods Tony Fleet 202 Some Research Pitfalls (7) Mistaking Association for Causality Problem: Thinking that because one variable increases (or reduces) as another increases (or reduces) that we have found a ‘cause’ of an event. For example, the variable ‘difficulty of navigation’ in a multimedia package may be correlated with the number of buttons. However, it is the underlying navigational structure, which is at the root of the problem, which coincidentally generates the number of buttons. Solution: Understanding that correlation is about variation, and that multiple factors often act together to cause particular effects, and seeking such factors.

HOPE Research Methods Tony Fleet 203 Some Research Pitfalls (8) Ex Post Facto Hypothesizing Problem: Deciding on your research hypotheses after you have seen the data, and noticed some results which might be significant. In any data set, there are bound to be ‘blips’ which are freak results, and you are in danger of basing your results on these. Solution: Decide on your research hypotheses and the test you will do before you collect data. That way the tests will be fair, and your results valid.

HOPE 8. Summary

HOPE Research Methods Tony Fleet 205 Summary Things to consider Devising your Research Questions Selecting Your Methodology Deciding on Your Methods of Analysis Some Do’s & Don’ts

HOPE Research Methods Tony Fleet 206 Relationship between Elements To a large extent, this process is iterative. You need a good idea of the data you intend to collect, and how you will use it, before you devise your research questions. Conduct Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions

HOPE Research Methods Tony Fleet 207 Devising Research Questions Your Research Questions should arise naturally out of the Literature Review. However, in your Research Project you should only devise questions that you are capable of answering. That means when you ‘phrase’ them, you should have a good idea of the methods that you will use to answer them, the data that you will collect, and how that data will be analysed. What you should not know,is the answer to those questions!

HOPE Research Methods Tony Fleet 208 Selecting your Methodology Your methodology should be devised specifically to answer your Research Questions It should draw on the methods used by the studies that have formed a part of your Literature Review. It should also be created in such a way that the data that it produces is amenable to analysis That means you should be able to take advantage of the different forms of analysis, and the different tools available, in order to enhance your research.

HOPE Research Methods Tony Fleet 209 Deciding on the Analysis The method of Analysis needs to be appropriate to the data collected. It also clearly needs to be sufficient in order to answer the Research Questions. It should also use methods that have been established by previous researchers that you have highlighted in your Literature Review.

HOPE Research Methods Tony Fleet 210 Advice – some Dos Plan ahead; think through your methodology and your analysis before framing your Research Questions. Include at least one ‘Research Hypothesis’, which you know you are going to test; this adds credibility to your dissertation. Spend time looking at how previous researchers have pursued their research; draw on their experience, adapt their methods and modify their research instruments… but don’t forget to credit them in the write-up! Consult with your supervisor, take advice.

HOPE Research Methods Tony Fleet 211 Advice – some Don’ts Do not set out to ‘prove’; set out to ‘investigate’. Do not try to take on too wide a topic; this is an MSc. Dissertation, and you should be working in a confined area, but achieving true depth of analysis Do not expect to have statistically significant results; most Masters’ dissertations fail to prove anything one way or the other. However, you are expected to know whether the results are statistically significant or not.

HOPE Research Methods Tony Fleet 212 … and Finally If you are doing a questionnaire, or data analysis, you may need to have specialist advice. You can sign up to see me, even though I am not your supervisor. However, if you come for a consultation, I expect you to know what you want to do, and have a good idea of how to do it; you should come to me to check that you are on the right lines, not to find out what to do. Contact Tony Fleet in FML 213, telephone: 0151- 291-3525 or email me at fleeta@hope.ac.ukfleeta@hope.ac.uk

HOPE Research Methods Tony Fleet 213 References and Bibliography Comprehensive accounts of Research methods: http://www.socialresearchmethods.net/kb/ http://www.socialpsychology.org/methods.htm Huge list of resources & links: http://www.geocities.com/Athens/3238/bookmark.htm Undertaking a Research Project: Guidelines http://dec.bournemouth.ac.uk/staff/scrowle/Teaching/ FinalYear/ProjectWorkshops/index.htmlhttp://dec.bournemouth.ac.uk/staff/scrowle/Teaching/ FinalYear/ProjectWorkshops/index.html

HOPE Research Methods Tony Fleet 214 Research Methods Books in the Sheppard-Worlock: Fowler, Floyd J.. - Survey research methods. - 2nd ed. - Newbury Park, Calif.; London : Sage Publications, 1993. - (Applied social research methods series ; v.1). - 0803950489 Lehman, Richard S. - Statistics and research design in the behavioural sciences. - Pacific Grove, Calif. : Brooks/Cole, 1991. - 0534138780

HOPE Research Methods Tony Fleet 215 Other Research Design Books in the Sheppard-Worlock: Handbook of applied social research methods / edited by Leonard Bickman and. - London : SAGE, 1997. - 076190672x Human centred methods in information systems : current research and practice. - Hershey; London : Idea Group Pub, 2000. - 1878289640 Qualitative research : theory, method and practice / edited by David Silverman. - London : Sage, 1997. - 0803976666 Research design. - A. - Milton Keynes : Open University Press, 1979. - (Social sciences, a third level course research methods in education and Social Sciences ). - 0335074227

HOPE Research Methods Tony Fleet 216 Past MSc. Dissertations You should examine what other students have done, especially in an area which is akin to yours. Remember: if it is on the shelves, it passed! Examples: Broadbent, George Ernest. - The role of evaluation in user interface design : a practical implementation. - Liverpool : University of Liverpool, 1997. - p7270369 Lomas, Joanne. - The research and design of a computer based training package for children. - Liverpool : University of Liverpool, 2001. - M0002708LO

HOPE Research Methods Collecting, Processing and Analyzing Data.

Similar presentations

Presentation on theme: "HOPE Research Methods Collecting, Processing and Analyzing Data."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

HOPE Research Methods Collecting, Processing and Analyzing Data.

Similar presentations

Presentation on theme: "HOPE Research Methods Collecting, Processing and Analyzing Data."— Presentation transcript:

Similar presentations

About project

Feedback