Collecting, Processing and Analyzing Data

Collecting, Processing and Analyzing Data
Research Methods Collecting, Processing and Analyzing Data

Research Methods Tony Fleet
Aims of the Session The purpose of this session is to: Alert you to the different types of methodology available to you in your research Make you aware of the different techniques that you might use in collecting, presenting and analysing data Discuss the different kinds of problems that you might encounter in pursuing your research. Research Methods Tony Fleet

Contents Developing Your Research Questions The Different Types of Research Selecting Appropriate Research Methods Robustness of Methods Structuring Your Methodology Data Analysis Problems with The Research Process Summary Research Methods Tony Fleet

1. Developing your Research Questions
This section of the presentation examines what you need to do in order to focus your research.

The Purpose of Research
The purpose of research is to contribute to a current academic debate, and possibly to advance knowledge in some manner. This means that the research that you undertake has to be : Embedded in a recognisable field of study, taking account of, and drawing on, past research Of interest to other researchers working in the same field, and possibly to the wider community Generalisable to more than one individual experience or circumstance. Research Methods Tony Fleet

Typical Research Structure
Conduct Literature Review The process of research is well-documented. This diagram more or less describes the activities you need to undertake. What we will do in this session, is to look at some of the elements, and how they fit together. Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

Conduct Literature Review
Getting Started The first thing that you will do is to make sure you are well-informed, and to pose pertinent questions which your research will answer. Conduct Literature Review Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

Defining Your Field of Study (1)
You undertake a Literature Review in a particular field, in order to ensure that your research is embedded within that field, and that you are taking account of the methods, issues, results, theories and conventions which apply. At the end of the literature review, you will have narrowed the field down to a relatively small topic within the field, and will have some ‘unanswered questions’ which your research will try to investigate. These focussed questions form the basis of your Research. Research Methods Tony Fleet

Defining Your Field of Study (2)
Your Research Questions are crucial, as they effectively define both the content of your research and the manner in which you carry it out. Your research methodology is designed specifically to attempt to answer these questions The data that you collect will be focussed on issues relevant to these questions Your analysis of the data will seek to provide answers to the questions Your conclusions will summarise the answers. Research Methods Tony Fleet

Research Questions Typical research questions might be: To what extent is the business community in India aware of the potential of Bluetooth? Is the software currently available for teaching arithmetic to 5 year olds appropriate and effective? Are there differences in the way that men and women approach the task of writing software? How can historical events be modelled effectively using VRML? Research Methods Tony Fleet

Defining Your Population
When framing your Research questions, you need to be clear about what set of objects, people or events forms the ‘background population’ in your study. Are you saying, for example that the CAL software you produce is designed for all English-Speaking people, all men, all Afro-Caribbeans, all children under 5, all those who have been diagnosed as dyslexic, or simply to Afro-Caribbean boys under 5 with specific learning difficulties? If you are investigating whether on-line learning is effective, is your population: students, University students, UK University students, Liverpool Hope Students, Liverpool Hope Computing Students or Liverpool Hope MSc. Computing Students? Research Methods Tony Fleet

Research Questions References (SW Library)
Lewis, Ian. - So you want to do research! : a guide for beginners on how to formulate research questions. - 2nd ed. - Edinburgh : Scottish Council for Research in Education, (SCRE publication ; 2 ; Research Methods Tony Fleet

2. The Different Types of Research
Here we look at the different options available to us in carrying out the research.

Your Research Focus The main focus of your research can be: Product-based Research, focusing on producing a piece of hardware or software (or the designs for them) which is at the ‘cutting edge of a discipline, drawing on other researchers’ ideas, best practice and what is feasible. In doing this, you may need to explore how the product will ‘enmesh’ with current systems, and existing and future technologies. People-based research, focusing on the people who interact with the hardware or software, looking at issues such as usability, user behaviour, compatibility of software with current user systems and other HCI issues. Both of these approaches are legitimate, and it is possible that in carrying out your research you might need to use elements of each one. Research Methods Tony Fleet

Product v. People Which focus?
The focus of your research is decided by you. It will depend upon how confident you are in creating a product at the ‘cutting edge’, or how comfortable you will be as a researcher in dealing with people. When you frame your research questions, you need to ensure that their focus leads you into the kind of research that you want to do. There is no right answer, but you may find that your research will be best carried out by using a ‘main’ focus of one element, with a subsidiary focus of another. For example producing a piece of software which securely encodes personal data as a self-encrypting and decrypting file stored on an ID card, may well need trialling with real people. Research Methods Tony Fleet

Approaches to Research
There are two main approaches to doing research: Quantitative Research looks for ‘hard’ numerical data; it proceeds by counting, measuring and summarising. The goal is to look for statistical significance in the results. Qualitative Research takes a ‘soft’ approach to data, and has a more descriptive ‘feel’. It attempts to get to the heart of the matter, exploring individual cases in detail, and seeking the reasons for behaviour and explanations for events. Both of these approaches are legitimate, and it is possible to combine elements. Research Methods Tony Fleet

Quantitative v. Qualitative Which approach?
The approach you use will depend upon your topic and your research questions. It will also depend upon how comfortable you as a researcher feel about using these methods. There is no right answer here, and, as we shall see in the rest of this presentation, there may be good reasons for adopting a variety of methods, which encompass both quantitative and qualitative approaches Research Methods Tony Fleet

Types of Research There are four main types of research that you might consider: Experimental Research Survey Research Evaluative Research Observational Research All four of these types can incorporate both quantitative and qualitative approaches Research Methods Tony Fleet

Experimental Research
This is normally quantitative, but can take two forms: An attempt to produce a piece of hardware, software or a combination of both, which is at the ‘cutting edge’ of a discipline. An attempt to investigate and document the performance of a particular piece technology in specific circumstances. This might involve: Creating hardware or software applications Devising detailed tests and evaluation procedures Carrying out rigorous testing Evaluating performance or usability Research Methods Tony Fleet

Survey Research This research can be qualitative or quantitative; in the widest sense, you are interviewing people. This might involve: An unstructured interview A semi-structured interview An structured interview based on questionnaire (face to face, or by telephone) An administered questionnaire A Postal Questionnaire Research Methods Tony Fleet

Evaluative Research This is primarily qualitative. Here you are trying to assess whether something is of value, whether it meets its specifications, or whether it is fit-for-purpose. This might involve: Developing a list of criteria on which to make judgements Examining the object against each of the criteria to judge to what extent it conforms to expectations Weighing the positives and the negatives, coming to overall conclusions Matching these judgments against similar judgements made elsewhere in the literature or in real life. Research Methods Tony Fleet

Observational Research
This research normally uses a qualitative approach; in the widest sense, you are recording people’s behaviour. This might involve: Participating in a task or situation Making field notes of experiences Creating and using an Observation Schedule Making a check-list of occurrences of particular events or items. Research Methods Tony Fleet

Research possibilities
Qualitative Quantitative Experiment Field- testing Bench-testing and simulations Survey Unstructured Interviews Written Questionnaires Evaluation Using expert judges Evaluation criteria and checklists. Observation Participant Observation Observation schedules Research Methods Tony Fleet

Other Forms of Research
Historical & Documentary Research proceeds by scrutinising existing materials, both written and artefacts, using them as sources of evidence. Action Research is normally conducted in an educational or political context. Action is taken, monitored, evaluated and then modified for the next cycle. Ethnographic Research consists of an in-depth study of a cultural phenomenon, in order to generate new theory. Case Study Research selects a whole range of research methods in scrutinising one particular context or situation. Research Methods Tony Fleet

Research methods 1 References (SW Library)
Crabtree, Benjamin F. - Doing qualitative research. - London : Sage, (Research Methods for Primary Care ; 3) Creswell, John W.. - Research design : qualitative and quantitative approaches / John W. Creswell. - Thousand Oaks, Calif; London : Sage, Creswell, John W.. - Qualitative inquiry and research design : choosing among five traditions / J. - Thousand Oaks, Calif.; London : SAGE, Research Methods Tony Fleet

Research methods 2 References (SW Library)
Miller, Delbert Charles. - Handbook of research design and social measurement. - 3rd ed. - New York : David McKay Co . Inc, m Research methods in education and the social sciences / [Research Methods in. - Block 3B : Research design. - Milton Keynes : Open University Press, (DE304, Block 3B ; 3B) Yin, Robert K.. - Case study research : design and methods. - Rev. ed. - Newbury Park; London : Sage, (Applied social research methods series ; v.5) x Research Methods Tony Fleet

3. Selecting Appropriate Research Methods
The next few slides discuss how you might go about selecting your research methods from those available

Selecting Your Methodology
Your research methodology consists of : Research Methods (experiment, survey etc.) Research Instruments (questionnaire, tests etc.) Analytical Tools (statistics, inductive or deductive methods) When selecting the methodology, you need to be aware of: The Research Questions you are trying to answer The Population you are trying to generalise to. Research Methods Tony Fleet

Factors to Consider Undertake Literature Review
How has previous research in this area been done? Undertake Literature Review What methods have been used? Select Research Questions What research instruments have been devised? Devise Methodology & Research Instruments How will the methods & instruments be applied? Apply Methods & Instruments What Statistical tests can be carried out? Perform Statistical Analysis What Research Hypotheses can be tested? Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

Appropriate methodology
Do your Research Questions involve impressions, attitudes, opinions, beliefs or knowledge held by people? If so, then survey research is appropriate Do your Research Questions involve behaviour, actions, reactions to events, circumstances or objects? If so, then observational research is appropriate Research Methods Tony Fleet

Appropriate methodology
Do your Research Questions involve the reliability or robustness of hardware, software, systems or infrastructure? If so, then an evaluative study is appropriate Do your Research Questions involve the testing of hardware or software at the technical level, speed, accuracy, security etc? If so, then experimentation is appropriate Research Methods Tony Fleet

A Mix of Methods It may be that your research questions overlap some of these categories, or different questions address more than category. If so, you should consider a mix of methods, that ensure that you cover all eventualities. This may bring added benefits. (See Triangulation) Research Methods Tony Fleet

Mixed Methods References (SW Library)
Mixing methods : qualitative and quantitative research / edited by Julia Bra. - Aldershot : Avebury, Tashakkori, Abbas. - Mixed methodology : combining qualitative and quantitative approaches / Abba. - Thousand Oaks, Calif.; London : Sage, (Applied social research methods series ; v.46) Research Methods Tony Fleet

4. Robustness of Methods As well as ensuring that your questions are well-focussed, and your methods relevant and appropriate, you need to ensure that your methods are also Reliable and Valid

Reliability & Validity
Research is Reliable if different methods, researchers and sample groups would have produced the same results. Research is Valid if the results produced by the research are accurate portrayals of the situation, explanations are effective, and predictions from the research are actually borne out by observation. Research Methods Tony Fleet

Reliability Research can have poor reliability, if it on one or two cases only, or if personal judgement or opinion is included. Reliability can be improved if data collection methods are made more precise, we have controlled experimentation, and we can produce statistical summaries. Research Methods Tony Fleet

Validity Research can have poor validity if the data produced is too far removed the object under study, or the respondent. Using detailed, highly structured research instruments can lead to distortions, by forcing observations into categories when they do not fit. Data summaries and averaging can also lead to distortions, and meaningless generalities. Graphical representations and percentages can be highly selective and produce biased findings. Validity can be improved by working directly with individuals or objects, focusing on specific cases, making detailed observations, conducting face to face interviews, taking detailed measurements in specific circumstances etc. Research Methods Tony Fleet

Reliability v. Validity
Valid, but not reliable Research Methods Tony Fleet

Reliability v. Validity
Reliable, but not Valid Research Methods Tony Fleet

Methodological Trade-Off
If you improve validity, you will almost certainly reduce reliability. If you improve reliability it will be at the cost of reducing your validity. The trade-off is to balance the two so that the benefits of using particular methods outweigh the losses incurred. Research Methods Tony Fleet

Triangulation Triangulation takes its name from the navigational method of positioning a ship at sea by making two independent observations. The purpose here is to use two distinct methodologies, independent of one another to confirm that the effects which we are observing are real, and not artefacts of the research process. Research Methods Tony Fleet

Triangulation Triangulation attempts to counter the methodological trade-off by using a mix of methodologies. If you are using highly structured, statistical or measurement-based research, you supplement this with detailed observations or face to face interviewing. If your research is mainly based on individuals or on single items, you ensure that at least part of it has some statistical summaries, structured observations or questionnaires. Research Methods Tony Fleet

Validity & Reliability References (SW Library)
Kirk, Jerome. - Reliability & validity in qualitative research. - Beverly Hills Calif : Sage Pubns, (Qualitative Research Methods Series ; 1) Litwin, Mark S. - How to measure survey reliability and validity. - London : Sage, (The Survey Kit ; 7) Research Methods Tony Fleet

5. Structuring Your Methodology
This section looks in detail at the techniques that you might employ in the Research Activity itself.

Research Structure Conduct Literature Review
After framing your Research Questions, and selecting your methodology, you should test that this is going to work by conducting a small-scale Pilot Study Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

5a Pilot Studies A pilot study is a set of preliminary investigations & procedures carried out prior to the main research, to ensure that the research is possible and can proceed without hitches. Research Methods Tony Fleet

Pilot Study (1) In almost every type of Research, you will need to devise or adapt some sort of Research Method or Instrument. This might be a measuring procedure, a set of evaluation criteria, an interview procedure or questionnaire, or an observational method or schedule. Research Methods Tony Fleet

Pilot Study (2) Your procedure or instrument should be based on best practice from previous research. It is unlikely that you will find exactly what you need; you will be forced to adapt or amend it. This means that you will need to conduct a Pilot Study to test whether the new instrument is fit-for-purpose. Research Methods Tony Fleet

Pilot Study (3) The main purpose of a Pilot Study is to iron out ‘bugs’ in procedures, or to check that instruments work. The size of the pilot study will depend on how ‘inventive’ you needed to be. If your procedures are almost entirely of your own devising, then you will need a fairly extensive pilot study to check them. If you have ‘lifted’ methods from the literature, than your pilot can be quite small. Research Methods Tony Fleet

Pilot Study (3) With questionnaires & observation schedules, you will need to check that individual items are giving you expected results. With test procedures, you need to check that you can actually do what you have said that you are going to do. With evaluation criteria, you need to use these in a limited context, to see that they are workable and effective. Research Methods Tony Fleet

Pilot Study (4) As a result of the Pilot Study, you need to evaluate procedures and instruments, making amendments where necessary. The Pilot Study stage will be part of your research; you will need to write this up, reporting on how your methods were adapted and improved as a result. Research Methods Tony Fleet

Carrying Out the Research
Undertake Literature Review Select Research Questions Devise Methodology & Research Instruments Here we will examine how particular methods and instruments can be applied in the research situation. Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

5b A detailed look at some Research methods
Experimental Design Evaluative Research Observational Research Survey Research Research Methods Tony Fleet

5c.1 Experimental Design This section looks at the different ways in which you can conduct experiments. Note that you do not need to be doing pure ‘experimental’ research to adopt these methods. Research Methods Tony Fleet

Experimental Designs You may need to think about experimental design even if you are doing types of research other than ‘Experiments’ These designs occur where you have made some change, and are trying to find out its effect. The net result is that you are comparing one thing, or one group with another thing or group. Research Methods Tony Fleet

Experimental Designs The terminology for experimental designs comes from agricultural experiments. We have different ‘treatments’ which we apply to different groups We control the groups for different ‘factors’ Research Methods Tony Fleet

Experimental Designs Experiments normally involve ‘independent’ and ‘dependent’ variables’ Independent variables are factors that can be controlled for, like temperature, file sizes, age, gender etc. Dependent variables are those factors which will change as a result of altering the independent variables. For example, download times will increase as a result of increasing file sizes. Download time = dependent variable, File size = independent variable Research Methods Tony Fleet

Pre-Test/Post Test Control Group Design
This is the classic experimental design. It allows you to split your sample into two distinct parts (A & B) You give the same test to the two groups before you start You treat one of the groups You apply the same test afterwards. Group A (treated) Group B (untreated) Pre-test Pre-test Treatment Post-test Post-test Research Methods Tony Fleet

Control Group The Control Group (B), is one which is the same in all respects, except for the fact that we make no changes. We can use the test measurements on the control group as a benchmark for any changes we make to the treatment group (A) Research Methods Tony Fleet

Example 1: Server Testing
Suppose we wish to check whether a firewall is effective in blocking external attacks. Set up System A & System B on two different servers, running ‘near-equivalent’ internal & external programs. Devise a test which includes a full range of possible attacks; apply to both A & B. Take series of measurements & observations. Incorporate firewall into System A Reapply test to both systems, which are again running ‘near-equivalent’ programs. System A (treated) System B (untreated) Pre-test Apply Firewall Post-test Research Methods Tony Fleet

Example 2: Software Design
Website A (treated) Website B (untreated) Pre-test Apply Agents Post-test Suppose we wish to find out whether anthropomorphic agents are useful in communicating with novice users of a website. Set up Website A & Website B with ‘near-equivalent’ structure, but different content. Devise a procedure which asks two equivalent sets of users to make explorations of the websites; apply to both A & B. Make a series of observations Incorporate Agents into Website A Re-apply procedure, asking two more equivalent sets of users to make the same explorations of both Websites. Again make observations. Research Methods Tony Fleet

Example 3: On-Line Learning
Suppose we wish to find out whether training is effective in helping students cope with the demands of on-line learning. Set up group A & group B with ‘near-equivalent’ members. The two groups to undertake a short programme of learning on-line, which incorporates a short evaluation and a knowledge test. Give Training to Group A The two groups to undertake a further short programme of learning on-line, again incorporating a short evaluation and knowledge test. Group A (treated) Group B (untreated) Pre-test Apply Training Post-test Research Methods Tony Fleet

Factorial Designs This is where we examine the effects of two or more independent factors simultaneously. For two factors, we would need 4 groups Group A (Control: no treatment) Group B (Factor 1 treatment only) Group C (Factor 2 Treatment only) Group D (Factors 1 & 2 Treatments) Clearly, this is going to increase the complexity and size of the research, but it has the added benefit of producing verifiable results in cases where two factors interact to produce interesting effects. Research Methods Tony Fleet

Other Design Variants Post-Test only Control Group Design Here we assume that both groups are the same (but do not test that assumption). We simply apply the treatment to one group, and apply the test to both groups. Matched Pairs Design The individuals we use in the test groups (subjects) are matched for characteristics which are likely to affect the outcome (gender, age, level of education, ethnicity etc.) Research Methods Tony Fleet

Experimental Designs: Reliability Issues
For high reliability, we need to build in strict controls over each of the independent variables affecting the outcomes of the experiment. We will also need to ensure that any measurements taken of the dependent variables are as accurate as possible. We also need to ensure that our methodology is clear and replicable. Research Methods Tony Fleet

Experimental Designs: Validity Issues
For high validity, we need to conduct the experiment in as ‘natural’ a setting as possible, and in as near as exact a match to the circumstances in which the events or objects would normally operate. If a sample is being used, then the sample (whether time, events, people or objects) should be as representative of the ‘background’ population as possible, and we should make detailed observations of how the events unfold as well as the final measurements. Research Methods Tony Fleet

Experimental Design References (SW Library)
Campbell, Donald T., Donald Thomas, Experimental and quasi-experimental designs for research / Donald T. Campbell. - Boston; London : Houghton Mifflin, – Field, Andy. - How to design and report experiments / Andy Field, Graham Hole. - London : SAGE, Miller, Steve. - Experimental design and statistics. - London : Methuen, (Essential Psychology ; A8). - m Research Methods Tony Fleet

5c.2 Evaluative Research This section looks at the methods and issues surrounding evaluation. You may need to use such techniques if evaluation is implicit in your research questions Research Methods Tony Fleet

Evaluative Research Evaluative Research covers those cases where you are attempting to compare whether one procedure or object is better or more effective than another procedure or object, or to determine whether a particular procedure or object is fit-for-purpose. Evaluation needs to be done against a set of criteria, which have been established as valid and reliable in this context. You would normally produce an evaluation form to be completed by respondents. Research Methods Tony Fleet

Evaluation Criteria To establish criteria for evaluation, we need to break the topic down into individual elements, and state the different sub-topics on which the item is to be evaluated. Alongside this, we will normally state specific questions which will need to be answered in order to judge the item against the criterion. The answers to the questions may involve measurements, counts, assessments on subjective scales, statements of fact or possibly even opinion. Research Methods Tony Fleet

Example: Website Evaluation Criteria
Design Is the use of colour acceptable? Are the elements in harmony? Navigation Do the links work? How many links per page? Content How many words & graphics on page? Is the text interesting & informative? Interactivity What interactive features are used? Do these improve communication with the user? Coding What scripting is used? Is it clearly annotated? Research Methods Tony Fleet

Example: Website Evaluation Form
1. Design Colour Safe Pallet used? Yes/ No Harmony of elements on page: good/ neutral/ poor 2. Navigation Number of links on the page: ____ Number of working links ____ 3. Content Total size of graphics files on page: ____ MB Is the text: boring/ dull/ neutral/ interesting/ fascinating 4. Interactivity Tick and/or name all interactive features used: rollovers; dynamic images; image maps; ______, ______ 5. Coding What DTD has been used? _______________________ Research Methods Tony Fleet

Evaluation Criteria: Validity Issues
For criteria to be valid, each of the criteria elements need to have face or content validity; the questions posed by the criterion needs to be relevant to the object, and relates to a feature of it We also need to be able to answer the questions in an objective manner, without recourse to guessing, or giving an ‘impressionistic’ response. Research Methods Tony Fleet

Evaluation Criteria: Reliability Issues
For criteria to be reliable, each of the criteria elements need to be clear and unambiguous, so that different assessors would interpret the criteria in the same way. We also need to ensure that repetition of the evaluation exercise will yield results which are not dissimilar to one another. Research Methods Tony Fleet

Evaluation Methods References (SW Library)
Britain, Sandy. - A framework for pedagogical evaluation of virtual learning environments. - Manchester : Joint Information Systems Committee, (JISC Technology Applications Programme r.. - M EL Broadbent, George Ernest. - The role of evaluation in user interface design : a practical implementation. - Liverpool : University of Liverpool, p Redmond-Pyle, David. - Graphical user interface design and evaluation (GUIDE) : a practical process. - London : Prentice Hall, x Smeltzer, Nicholas. - Critical analysis of the design and evaluation of a computer-based project. - Liverpool : University of Liverpool, M LO Research Methods Tony Fleet

5c.3 Observational Research
This section describes what you need to do if your research involves making detailed observations of people, events or objects. Research Methods Tony Fleet

Observational Research
In this case you are trying to determine Either: How an individual or group of individuals react, interact or behave in particular circumstances Or: How software or hardware performs when used by particular groups of people Research Methods Tony Fleet

Participant Observation
In this case, the researcher becomes one of the subjects, and works alongside the subject, monitoring behaviour and interacting with them. Pros:Researcher can fully ‘understand’ what the issues are. Can get real validity to research. Cons: Researcher can influence the research, alter opinions. Highly subjective & unreliable. Taking notes is difficult; relies on good memory. Research Methods Tony Fleet

Non-Participant Observation
In this case, the researcher studies the situation apart from subjects, taking notes, monitoring behaviour and observing interactions. The use of audio & video recording is useful here. Pros:Researcher can get overview of the situation, and achieve objectivity Cons: Researcher can only see resulting behaviours, not what is causing it or why it is happening. Research Methods Tony Fleet

Observation Schedules
An observation schedule can be a simple list of things to look for in a particular situation It can be far more complex; a minute by minute count of events such a mouse-clicks or verbal interactions between subjects. Research Methods Tony Fleet

Observation Schedule An Example
Observation of subject using Information Portal Subject: M / F Age: / / / 50+ Date: _________ Time: _________ Selected Navigation Tool : Mouse /Keyboard/ TouchScreen First 5 pages visited in order: __ __ __ __ __ Time to obtain to obtain required information: ___ min ___ sec Total Number of Pages visited: __ Feedback from subject: very positive / positive / neutral / negative / very negative Research Methods Tony Fleet

Observation Schedules: Some Reliability Issues
For an observation schedule to be reliable, it should require structured documentation of events. This will involve such things as checklists, minute-by-minute categorisation of activity and numerical data such as frequencies of occurrence and time intervals. Timings should have a clear start and end points. The schedule should leave little room for subjective judgement. Research Methods Tony Fleet

Observation Schedules: Some Validity Issues
For an observation schedule to be valid, it should refer to events which actually happen; each of the events on the sheet should be possible, and likely to occur. Timings should be possible to take, and not interfere with other observations which should be made. There should be room for observations which ‘enrich’ the data by adding detail to the numbers, offering explanation and illumination. Research Methods Tony Fleet

Observational Research References (SW Library)
Harding, Jacqueline. - How to make observations & assessments / Jackie Harding and Liz Meldon-Smith. - 2nd ed. - London : Hodder & Stoughton, x Robertson, Kevin. - Observation, analysis and video / editors: Anne Simpkin and Penny Crisfield. - Leeds : National Coaching Foundation, Simpson, Mary. - Using observations in small-scale research : a beginner's guide / Mary Simpson. - Glasgow : Scottish Council for Research in Education, (SCREpublications ; 16 ; 130) Research Methods Tony Fleet

5c.4 Survey-Type Research
This section describes what you need to do in order to use human respondents to provide you with information. Research Methods Tony Fleet

Survey methods There are two distinct elements here: The interview techniques that you adopt in order to elicit information from people The sampling methods that you adopt in order to select respondents for interview You will need to make rational choices for both of these elements, depending upon the focus of your research and the population under study. Research Methods Tony Fleet

Interviewing We interview people face-to-face in order to try to find out exactly what they think. With the right questions, people respond with high quality information. The data that you get can be of high validity, since you have access to respondents’ own words. Research Methods Tony Fleet

Types of Interviews Unstructured Interview Structured Interview Semi-Structured Interview Administered Questionnaire Research Methods Tony Fleet

Open or Closed? When interviewing respondents, the main choice facing the researcher is whether to use open questions, which leave the respondent free to answer in any way they think fit, or closed questions which force the respondent to make particular choices, pre-determined by the researcher. OPEN: What is your experience of chat rooms? CLOSED: Do you think chat rooms should be monitored? (Yes/No/Maybe) Research Methods Tony Fleet

Open or Closed? Validity & Reliability Issues
In general, the data from open questions is richer, more illuminating, and more valid, as it can illustrate clearly why particular subjects think the way that they do, or why they behave in particular ways. Data from closed questions however, is more amenable to statistics. It is easier to summarise, spot trend and make comparisons. In general the data is more reliable. A good strategy is to use a mixture of both open and closed questions Research Methods Tony Fleet

Unstructured Interviews
Interviewer has no set agenda; the object is to get the respondent to talk freely about various topics. Pros: Can be high quality data, often get real insights into what people really think. Cons: Can be difficult for the novice researcher to carry out; need to be good at steering the conversation without forcing it. Time consuming; can only carry out a small number. Difficult to analyse Research Methods Tony Fleet

Semi- Structured Interview
Interviewer has a formal list of topics, which sets the agenda; however, the interviewer is free to take these in different orders, or to return to topics at different points. Here we include the idea of ‘focus’ groups, where a facilitator encourages the discussion of a particular topic. Pros: Data provided can be almost as good as unstructured interview, but in a more focussed manner. Cons: Can miss important ideas, because agenda set beforehand; interviews can also be time consuming & difficult to manage & analyse afterwards. Research Methods Tony Fleet

Structured Interviews
Interviewer has a list of topics to be taken in a particular order; questions are written, and read out. Pros: You can make good comparisons between different respondents; you also will have access to respondents’ thinking. Cons: Unless you have done some preliminary investigations, and extensively piloted the interviews, you may miss lots of important data. Takes time to do properly; can only carry out a few. Research Methods Tony Fleet

Administered Questionnaire
Interviewer has devised a list of questions which are read out; some multiple choice, some open-ended. Filled in either by respondent or interviewer. Pros: Interview can be quite brief; easy to make comparisons, data amenable to statistical analysis. Cons: Data may be ‘warped’ by the choice of questions. Respondent may be forced into giving fabricated answers. Research Methods Tony Fleet

Questionnaires There are many different types of questionnaire, depending upon what you are trying to find out. As well as gathering factual data about the person, you may be trying to explore their: Knowledge Beliefs Attitudes Opinion Behaviour If you need to design a questionnaire, see for example: Research Methods Tony Fleet

Question Types Factual Data Binary (Yes/No) Single Selection from Categories Multiple Selection from Categories Attitude Scale items Focussed items to elicit categories Conditional Questions for routing Open-ended Questions Self-Reporting Research Methods Tony Fleet

See: Writing Questions Put Factual Data (demographics) at end Put Explanations and Disclaimers at the start Keep questions as brief as possible Use non-technical language where possible Avoid leading respondents towards particular answers Use direct questions, no hypotheticals Use simple questions, no portmanteaus Use a mixture of positively and negatively worded questions Verify data by asking same question in different ways. Research Methods Tony Fleet

Questionnaire Example
1. Which operating system are you currently using: Linux Windows Other 2. How would you rate the operating system? very poor poor good very good 3. Circle the tasks which you use your computer to do: Word processing Internet Access Program Development 4. Estimate the number of hours you use the computer for each week: ______ hours 5. Do you use broadband? Yes No Research Methods Tony Fleet

Protocol Whatever your interviewing method, as part of the ethical constraints on Hope Researchers, you are required to do the following: Explain to the respondents what the research is about. Tell them that they will not be identified by name in the final report, and that any views that they express will be in confidence. Explain to them that if there are questions with which they feel uncomfortable, they do not have to answer. Research Methods Tony Fleet

Attitude Measurement You may wish to incorporate ‘attitude measurement’ into your questionnaire as either a major or a minor feature. There are several different types of scales which can be used to elicit numerical measurements of attitude: Likert Scaling Thurstone Scaling Guttman Scaling Semantic Differential Scaling See: Research Methods Tony Fleet

Likert Scales (1932) This consists of items like “I would not trust a bank’s website to keep my details secure” Respondents are asked to use a scale such as 1=Strongly disagree; 2=disagree; 3=neutral; 4=agree; 5=strongly agree Scales can vary: 0-4, 1-7 etc. Some proponents suggest removing the neutral category to force a choice, but this can reduce validity. Scores on individual items are totalled to obtain the respondent’s score; this is often shown as an average. Research Methods Tony Fleet

Thurstone Scales (1928) This pre-ranks 11 statements with numerical values 1-11; each statement carries a numerical score, for example: 1 = “Teleworking is an impossible concept” 2 =“Teleworking may be OK in exceptional cases” 8 = “If offered the opportunity, I would try teleworking” 11 = “In 20 years’ time, everyone will be teleworking” Clearly, lots of preparatory work needs to be done to generate and rank the statements. A respondent’s score, is the average numerical value of all the items they agree with. Research Methods Tony Fleet

Guttman Scales (1944) This pre-ranks statements in order, very similar to Thurstone scaling, except here we attempt to construct the scale so that if a person agrees with item 4 on the scale, they will also agree with items 1,2 and 3. When the questions are administered, the items are muddled, but each retains a ranking. The respondent’s score is the sum of the ranks associated with the items he or she agreed with. Research Methods Tony Fleet

Osgood’s (1957) Semantic Differential Scales
Respondents are asked to rate an idea or an object against a series of opposing adjectives or descriptions, for example: Using Learnwise Exciting [ ] [ ] [ ] [ ] [ ] [ ] Dull Hard [ ] [ ] [ ] [ ] [ ] [ ] Easy Frustrating [ ] [ ] [ ] [ ] [ ] [ ] Stimulating The scale asks respondents to tick the box nearest the descriptor that they agree with. Each box has a numerical value: e.g. 1,2,3… 7 The respondent’s score can be portrayed graphically, or as a total of numerical values. Research Methods Tony Fleet

Questionnaires & Interviews: Reliability Issues
For questionnaire results to be reliable, you need closed questions that are precisely framed, unambiguous, with response categories that are well-defined, exhaustive and exclusive. You need to ask the same question in different ways, and you need to collate the information into statistical summaries and expose the data to rigorous statistical testing. You need to use as large a sample as possible, and even out any random fluctuations in the data by using summary statistics and hypothesis testing. Research Methods Tony Fleet

Questionnaires & Interviews: Validity Issues
For questionnaire results to be valid, you need response categories which enable the respondent to express their views accurately; this can best be done with open questions. You should take time with the respondent and respect the data that they provide for you. You need to ensure that the sample is representative of the population, and large enough for the results to be statistically significant. Research Methods Tony Fleet

Questionnaire References On-Line
Research Methods Tony Fleet

Questionnaire References (1) (SW Library)
Foddy, William. - Constructing questions for interviews and questionnaires : theory and practice. - Cambridge : Cambridge University Press, Fowler, Floyd J.. - Improving survey questions : design and evaluation. - London : Sage, Frazer, Lorelle. - Questionnaire design and administration : a practical guide / Lorelle Frazer. - Brisbane : Wiley, Gillham, W. E. C., William Edwin Charles, Developing a questionnaire. - London : Continuum, (Real world research) Research Methods Tony Fleet

Questionnaire References (2) (SW Library)
Oppenheim, A. N., Abraham Naftali, Questionnaire design, interviewing and attitude measurement / A.N. Oppenheim. - New ed. - London : Continuum, Wengraf, Tom. - Qualitative research interviewing : biographic narrative and semi-structured interviews - London : SAGE, Young, Pauline V. - Scientific social surveys and research : an introduction to the background,. - Englewood Cliffs, N.J. : Prentice-Hall, (Prentice-Hall Sociology Series). - m Youngman, Michael Brendan. - Designing and analysing questionnaires. - Maidenhead, Berks : TRC Rediguide. - (Rediguide ; 12). - m Research Methods Tony Fleet

Sampling The question of how to select the respondents for the sample is always tricky. The principle behind sampling, is that you should ensure that the sample is appropriate in order for you to generalise your results to the population under study. There are essentially three ways in which this is done: Random Sampling Quota Sampling Stratified Random Sampling Research Methods Tony Fleet

Random Sampling This method requires you to get a list of all the population, as near complete as you can find, then use some random selection method (such as shutting your eyes and stabbing a pen at the list, or allocating using random numbers) The idea is that every person in the population has an equal chance of ending up in the sample. Most statistical methods assume that you are sampling randomly, and it is the only method which overall is guaranteed to ensure that samples are free from bias, and therefore provide validity. Research Methods Tony Fleet

Quota Sampling Here, you identify particular sectors of the population, such as men, women , those under 21, over 21, employed, unemployed etc., and put quotas on the number of people in each category. For example, in a quota sample of 12, we might have: 2 males under 21; 2 females under 21 4 males over 21 ; 4 females over 21 The purpose here is not to get a sample which represents the population in its entirety, but to ensure that views from important sections of the population are represented. Research Methods Tony Fleet

Stratified Random Sampling
One of the problems with Random Sampling, is that if you take small samples, you can very well end up with a biased sample , by chance. In Stratified Random sampling, you would measure what proportions of the population lie in each category or strata, and select your sample so that you get precisely those proportions in those strata For example in your population you might have: 15% males under 21; 10% females under 21 30% males over 21 ; 45% females over 21 If you select a sample of 200 you would need stratified samples: 30 males under 21; 20 females under 21 60 males over 21 ; 90 females over 21 The allocation of subjects to samples should be done randomly. The purpose here is to get a valid sample which which represents the population in its entirety. Research Methods Tony Fleet

Other Sampling Methods
Snowball Sampling: This method is used whenever you are dealing with a tricky subject, where people may be doing devious or illegal activities (for example hacking, virus creation etc.). In this case, you would use one respondent to suggest the name of another who might be willing to be interviewed; that one would lead to several others and so on. Cluster Sampling: This method might be used if you have a diverse population (such as those living in African urban communities). Here you would randomly select particular cities, and sample within neighbourhoods within those cities. Convenience Sampling: This is used mainly for investigative research. It relies on the fact that people appearing at a particular location or time will do so, ‘at random’ (may not be true). We use this fact to stand in one place and interview them. Research Methods Tony Fleet

Sampling: On-Line References
This is an easy introduction: This is more complex and technical: Research Methods Tony Fleet

6. Data Analysis This section of the presentation looks at the different types of data available, and how this can be analysed

Analysis of Research Data
Undertake Literature Review Select Research Questions Devise Methodology & Research Instruments Here we examine how the data produced by the research can be presented effectively and statistically analysed. Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

Data Analysis 6a Types Of Data 6b Extracting Data for Analysis 6c Presentation of Data 6c Principles of Statistical Testing 6d Selected Tests 6e Analytical Tools Research Methods Tony Fleet

6a Types of Data In this section we describe the different types of data that you might encounter, and what you might do with it in your study. Research Methods Tony Fleet

Types of Data ‘Measurement’-type data, given in units measured on a decimal scale. e.g. MB/sec, times, averages and other statistical summary values. ‘Counted’ data: number of occurrences of events or objects: e.g. number of pages visited ‘Ranking’ data: subjective impressions, such as marks out of 10, rankings etc. e.g. Interface rated as very poor (1), poor (2), good (3) or very good (4). Categorical data: Names of different things or categories, such as those named by an interviewee (NB these may be counted) e.g. “Manager A discussed 3 concerns: security, access and reliability” Descriptive data, such as that produced by interviews or observational field-notes: e.g. “Subject A tried to click on all the blue text, even though it was not underlined”. Research Methods Tony Fleet

‘Measured’ Data Such data is called “continuous”, “ratio scale”, or “real number” data This type of data is easiest to use; it can be arithmetically manipulated (added, subtracted, multiplied, divided, summarised) it is also likely to have good, predictable statistical features. There are lots of methods to pick from. The inclusion of valid measured data will add reliability to your study. Research Methods Tony Fleet

‘Counted’ Data Such data is normally called “discrete”, “interval scale”, or “integer” data It can be added, subtracted and averaged. ‘Counted’ data, has almost all the features of measured data, especially if the numbers of the count are large (above 30 or so). For small numbers (e.g. less than 30 observations in total), methods are limited. The inclusion of valid counted data will almost certainly improve the reliability of your study. Research Methods Tony Fleet

‘Ranking’ Data Such data is normally called “interval scale data” data. In order to use it effectively, there are several techniques; the usual one is to convert the data into numerical values (e.g. “Yes” = 1, “No” = 0) When this has been done, it can be added, subtracted and averaged. ‘Interval’ data needs care; the data ‘looks like’ counts where the numbers are small, but are simply subjective impressions. To deal with such data we need to summarise large volumes of it data. The inclusion of valid counted data, suitably summarised can add reliability to your study. Research Methods Tony Fleet

‘Categorical’ Data Such data is normally called “nominal” data. In order to use such data effectively, the objects, classes or ideas that these categories represent should have the items within them counted, evaluated on an interval scale and/or summarised. A preliminary study may have found, for example that there are 4 main uses for spreadsheets in business: Finance, Timesheets, Inventory and General Calculation. Businesses may be asked to rank these in order of importance, or to rate their usefulness. ‘Categorical data is much more difficult to deal with. Processes are time consuming and highly subjective; There are relatively few statistical tools which deal effectively with categorical data. The inclusion of valid ‘Categorical’ data, suitably summarised can add both validity and reliability to your study. Research Methods Tony Fleet

‘Descriptive’ Data Descriptive data is simply a verbal description of something that happened, or what a respondent did, or a quote from an interview. It is the most difficult to deal with effectively. There are methods for extracting summary information from such data (see Grounded Theory: Glazer & Strauss); however, these are not for the novice researcher in IT, as they are very time consuming. The best use for such data is for illustration and supporting evidence elsewhere. The inclusion of appropriate ‘Descriptive’ data, suitably selected can improve the overall validity of your study. Research Methods Tony Fleet

Types Of Data References (SW Library)
Research methods in education and the social sciences / [Research Methods - Block 5 : Classification and measurement. - Milton Keynes : Open University Press, (DE304 : a third Research methods in education and the social sciences / [Research Methods. - Block 6 : Making sense of data. - Milton Keynes : Open University Press, (DE304 : a third level course Research Methods Tony Fleet

Types of Data References (SW Library)
Chatfield, Christopher. - Statistics for technology : a course in applied statistics. - 2nd. - London : Penguin Books, Kranzler, Gerald. - Statistics for the terrified / Gerald D. Kranzler, Janet P. Moursund. - Englewood Cliffs, N.J. : Prentice Hall; London : Prentice-Hall International, – Pentz, Mike. - Handling experimental data. - Milton Keynes : Open U.P., Salkind, Neil J.. - Statistics for people who (think they) hate statistics. - Thousand Oaks, Calif.; London : Sage, Research Methods Tony Fleet

6b Extracting Data For Analysis
Data produced by the research process does not have to be numerical in order to be of value. However, statistical techniques work best where the data uses numerical values, or can be converted to numerical scales. Research Methods Tony Fleet

Survey Data Questionnaire data should be the easiest data to extract. Ideally the questionnaire will have been designed with the methods of analysis in mind, so that questions are framed in such a way as to facilitate graphical representation and statistical testing. Interview Data can be more problematic; you may need to ‘invent’ categories, and to count up instances of different cases, both in the same interview, and with different subjects. Research Methods Tony Fleet

Questionnaire Data Examples
Percentage of Respondents using Broadband: 82% In the survey, 93% of people used their computer to access the internet, compared to 37% who used it for word-processing, and 7% who used it to develop programs. 87% of Linux users rated their operating system as good or very good, compared to 64% of Windows users. The average number of hours spent on the computer each week by men was 9.3, compared to 3.7 for women. Research Methods Tony Fleet

Interview Data Examples
5 out of the 6 interviewees were using Broadband. During the interview, all interviewees mentioned the positive benefits of moving to broadband. For example one respondent said “My daughter can use the phone while I am replying to s”. Altogether there were 25 instances in the interviews where benefits were named, as opposed to only three cases where drawbacks were noted. These cases broke down as follows: 20% of cases referred to accessibility issues 60% of cases referred to improved speed 20% of cases referred to improved compatibility However, it must be noted that all the drawbacks were cited by a respondent who reportedly used the internet for over 70 hours per week. The other five respondents’ use was an average of 15.7 hours, with totals ranging from 4 hours to 25 hours. Research Methods Tony Fleet

Observational Data Pure observational field notes are unlikely to contain numerical data. As with interview data, you may have to ‘invent’ appropriate categories and count up instances or occurrences. Observation schedules are easier to deal with. As with questionnaires, these should have been designed with the methods of analysis in mind, so that items are framed in such a way as to facilitate graphical representation and statistical testing. Research Methods Tony Fleet

Observational Data Examples
Out of the 40 people observed in the study, 36 of them (90%) used the mouse as the first point of contact with the portal. After the entry page, 70% visited page 3, 20% visited page 7 and the rest visited pages 2, 4 and 9. The mean time taken to access the information by novice users was seconds (SD = 24.8 secs); however, those who were subjected to the brief training session were able to access the information in a mean time of 48.3 seconds (SD = 13.9 secs) 95% of the users rated the interface good or very good. Research Methods Tony Fleet

Experimental Data Experiments should have been designed with specific comparisons in mind, where independent variables are identified & controlled, and dependent variables are measured in some way. The methods of analysis should have been identified beforehand, and the whole is susceptible to graphical representation and statistical testing. Research Methods Tony Fleet

Experimental Data Examples
Using two identical machines, System A using Windows XP and System B using Linux, accessing the website using a 56Mb/sec modem. Download times were: System A 4.5MB: secs 25.3MB: secs 185MB: secs 387MB: secs 875MB: secs 1.56GB secs 4.76GB timed out System B 4.5MB: secs 25.3MB: secs 185MB: secs 387MB: secs 875MB: secs 1.56GB secs 4.76GB secs Research Methods Tony Fleet

Evaluation Data Evaluation is unlikely to yield lots of numerical data. Categories on the Evaluation Sheet may well involve numerical values: counts, times, file sizes etc., and these are amenable to graphical representation & testing. Other categories may require judgement on ‘descriptors’, or on quality. It is possible to convert some of these to simple scales (0,1,2,3 etc), and to rank items in order, for example. Yes/No can be encoded as 1 and 0. Research Methods Tony Fleet

Evaluation Data Examples
The 5 evaluators rated the 4 packages as follows: Evaluator Best C C D C C | A D B B D | D B C D B Worst B A A A A Package C performed best overall; it achieved higher grades on 6 out of 7 criteria, and received a total rating of 8.3, compared to ratings of 6.1, 4.5 and 2.8 for packages B, D and A The reasons for Package A’s poor performance were cited by the evaluators as: poor readability, confused navigation, and lack of interactivity. 4 out of 5 of the evaluators rated the interface design for Package A as poor or very poor. Research Methods Tony Fleet

Data Collection References (SW Library)
Research methods in education and the social sciences. – Block 4 : Data collection procedures. - Milton Keynes : Open University Press, (DE304 ; block 4) Research methods in education and the social sciences / [Research Methods. - Block 6 : Making sense of data. - Milton Keynes : Open University Press, (DE304 : a third level course Research Methods Tony Fleet

6c Presenting Data This section of the presentation looks at some of the different ways in which you might present data, and the different ways in which you can draw meaning from it. Research Methods Tony Fleet

Presenting Numerical Information
Frequency Tables Cross Tabs Time Series Data Research Methods Tony Fleet

Frequency Tables In this example the number of errors cited when 3 different websites are run through an html code checker are tabulated. It is normal where percentages are given, to quote the actual numbers involved, OR the total number of cases. The percentages allow an easy comparison to be made on a ‘standard scale’ Research Methods Tony Fleet

CrossTabulations The purpose of a Cross-tabulation is to examine the relationships between two different variables. In this example, the number of men and women of different age groups expressing an interest in Blue Tooth are cross-tabulated. Here the information does not really show what is happening. Here we have used percentages, and it is now clear that although there is a reduction in interest in both males and females as they get older, it is far more dramatic for females. Research Methods Tony Fleet

Time Series Data A Time Series simply logs the value of a particular variable over a period of seconds, days or weeks. Because of amount of numerical information, it is very difficult to make sense of time series data unless it is summarised or presented in graphical form. This data shows the result of an experiment to randomly access websites, and download any .jpg files on the first page onto a hard drive. All we can see here is that the numbers are declining, but we cannot really get a sense of how this is happening Research Methods Tony Fleet

Presenting Graphical Information
Simple Bar Charts & Histograms Comparative Charts Scattergrams Time Series Data Research Methods Tony Fleet

Simple Bar Charts & Histograms
Simple graphs can be highly effective in making a point. This one shows the fact that Package A clearly has been rated higher than the other two packages. Research Methods Tony Fleet

Comparative Charts All three of these charts display the same information. Which makes it clearer? Research Methods Tony Fleet

Scattergrams These are useful if you are trying to demonstrate a relationship between two variables. We could add a ‘regression line’ to make the relationship explicit. Research Methods Tony Fleet

Time Series A variant on the scattergram with lines added, records of what happened over a period of time to the quantity measured. Here the data shows an almost uniform rate of loss of sectors, minute-by-minute. Research Methods Tony Fleet

Simple Statistics Data needs to be summarised using a variety of standard tools: Proportions, expressed in percentages; Averages, such as means, medians etc. Measures of variability, such as ranges, standard deviations etc. In this example, 3 different websites have been given gradings out of 10 by a panel of 6 experts. In the summaries, it is clear that A outperforms the others, and C is the worst. However, there is a lot more inconsistency in A’s scores than any of the others. Research Methods Tony Fleet

St. John, P. R.. - Methods of presenting fieldwork data / P.R. St. John, D.A. Richardson. - Sheffield : The Geographical Association, Research Methods Tony Fleet

6d Statistical Testing This section of the presentation explains the principles behind statistical testing, and describes the some of the different types of statistical tests available. It also explores probability theory. Research Methods Tony Fleet

Hypothesis Testing Undertake Literature Review Select Research Questions Devise Methodology & Research Instruments If possible, the final section of your study should include some statistical tests to provide rigour to your arguments. Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses; Draw Conclusions Research Methods Tony Fleet

Probability Theory (1) If your data consists of measurements, measured then you can probably assume that you have a ‘Normal’ Distribution, such as IQ scores. Lots of people with average IQ scores in the middle, but with very low numbers with High IQs or Low IQs. Research Methods Tony Fleet

Probability Theory (2) Even if your data is not measurement data, you can still make use of ‘Normal’ Distribution techniques, provided that: Your data is counted, and you have high numbers (over 50 observed in each category) You are working with averages of 10 items or more. Research Methods Tony Fleet

Probability Theory (3) Probability tries to measure how likely a particular observed event is, before it occurs. For example, if we picked an undergraduate student at random, and asked what is the likelihood of that student winning the lottery, it would be a very low probability event (prob = ) On the other hand, if we asked what is the likelihood of that student graduating, the probability would be very high. (prob = 0.95) Research Methods Tony Fleet

Random Chance In time, all things are possible. Mountains might crumble, pigs might fly, you might win the lottery. If we wait around long enough these events will happen, because each event has a small, but finite probability. Random chance will dictate the day, the year, or the millennium in which these occur. Research Methods Tony Fleet

Making predictions Because of the way that statistics works, we can use the features of data, and the idea of random chance to make predictions. If we were to walk into the dining hall, and select a student at random, we could predict the likelihood that: The student has an IQ between 85 and 115 (about 70%) The student has an IQ over 130 (about 2.5%) The first is a high probability event, and that’s what we would normally expect to happen. Clearly the second is a low probability event, and therefore we would be very surprised if it occurred. Research Methods Tony Fleet

Hypothesis Testing We all work on assumptions every day. Let us suppose that you have come to college on what you think is a normal day, only to find the rooms deserted, no lecturers, no students. What is the explanation? Is it that: You came in on a Sunday by mistake, thinking it was Monday The college has been evacuated due to a leak at a nearby toxic waste incinerator. Research Methods Tony Fleet

Hypothesis Testing Most people would immediately suggest (1), because (2) is a very low probability event. However, we can use the situation to reason thus: If we assume that Today is Monday, and we come in and find the college deserted, then the reason must be (2). However, this is such a low probability event that we would not normally expect to witness it. This means that something is wrong. It is more likely that our original assumption is incorrect, and Today is NOT Monday. Research Methods Tony Fleet

Hypothesis Testing This is formalised into a process which goes like this: We set up two competing hypotheses (normally called H0 and H1). We make observations in data, and calculate the probability of these observations occurring by random chance. We base this calculation on the assumption that H0 is true. If the probability is very low, we reject H0 (the null hypothesis), and accept H1, the alternative hypothesis. Research Methods Tony Fleet

Hypothesis Testing An Example
Let us set up two competing hypotheses: H0: MSc. Computing is equally attractive to men and women. H1: MSc Computing is more attractive to one gender than the other. We now observe the number of students in a group, and count the number of males and the number of females, and ask what is the likelihood of observing such, if the subject were really equally attractive. . Research Methods Tony Fleet

Hypothesis Testing An Example
Suppose in the class there are 8 men 2 women If H0 is true, then the probability of this event is p=0.01; this is 1%, extremely low . We therefore reject H0, and accept H1, concluding that MSc. Computing is more attractive to one gender than the other. Research Methods Tony Fleet

How low can you go? The question now comes, how low does a probability have to be in order for you to conclude that the event is not simply occurring as a result of random chance. For most circumstances, we would use p<.05 or less than 5%, and we would say such a result is statistically significant If p <.005, then the result is highly significant. If p< .0005, then the result is very highly significant. Research Methods Tony Fleet

What does this all prove?
Your results may not be statistically significant, that does not mean that they are not true; it simply means that the data you collected was not sufficient to decide things one way or another. If your results are significant, then you need to decide whether the interpretation that you are putting in them is actually the correct one; there may be other explanations for the results than the one that you have stated as H1. In other words, you should be cautious when you write this up; be analytical, and question everything. Research Methods Tony Fleet

6e Examples of Statistical Tests
This section looks at some examples of statistical tests that you can undertake in different contexts. It distinguishes between Parametric and non-Parametric testing Research Methods Tony Fleet

Parametric Testing (1) Parametric Testing assumes that the data you are working with follows a Normal Distribution. You can use these techniques if Your data is measured Your data is counted, but there is a large number in the count Your data is in numerical categories, or categories which can be translated into numbers, and the overall shape of the data is a ‘bell’ curve. You are working with averages, based on samples of 10 or more. Research Methods Tony Fleet

Parametric Testing (2) These are some samples of Parametric Tests Z-Test Single sample: Null hypothesis is that the average of a sample is equal to some theoretical value. Paired Sample T-Testing Two samples: Null hypothesis is that there is no difference between the two samples ANOVA Three or more samples; null hypothesis is that there is no difference between the samples. Correlation Two measurements taken on the same data; null hypothesis is that the measurements are not related in any measurable way. Research Methods Tony Fleet

Non-Parametric Testing (1)
Non-Parametric Testing assumes that the data you are working with does not follow a Normal Distribution. You need to use these techniques if Your data is counted, with low numbers in each category Your data is in categories which cannot be translated into a numerical scale Data where the overall shape of the data is not a ‘bell’ curve. You are working with medians or other statistical summary values which are not averages. Research Methods Tony Fleet

Non-Parametric Testing (2)
These are some samples of Non-Parametric Tests: Chi-Squared Test Crosstabs or Contingency Tables showing the effect of two different factors: Null hypothesis that the two factors are independent. Mann-Whitney U-test Two samples: Null hypothesis that there is no difference between the two samples Phi Test 2 x 2 Crosstab; Null hypothesis is that the factors are unrelated. Spearman’s Correlation Two judges asked to rank a series of items; null hypothesis is that the rankings are statistically unrelated. Research Methods Tony Fleet

Selected Test Examples
t-Test ANOVA Correlation & Regression Chi-Squared Test Research Methods Tony Fleet

t-Test This is used to test whether a the mean of an observed sample is significantly different from an hypothesised value. In the example shown, 12 respondents were given a Likert-scaled questionnaire, to determine their attitude towards chat-room ‘behaviour guidelines’. A neutral score would be 2.5 A t-test can be used to test the hypothesis: Ho: Respondents disagree with (or are neutral to) the guidelines The t-statistic is calculated to be t = 2.72 at p< .05; so we reject Ho, and conclude that the respondents agree with the guidelines. Research Methods Tony Fleet

ANOVA This is used to test whether a the means of three or more samples are significantly different from one another. The example above shows the average ratings from 6 different evaluators of three different web-authoring packages, A, B & C on a variety of ten-point scales. We can test Ho: There is no difference between the ratings of the packages, using an ANOVA (Analysis of Variance) test. The results below show a p value of We reject Ho, and conclude that the packages ARE different. The result is highly significant. Research Methods Tony Fleet

Correlation & Regression
This is used to determine whether there is a relationship between two variables, and if so, what? The data collected shows the download times for 10 files of different sizes. There is a correlation of between the two variables, showing that a very strong relationship exists. The regression analysis below shows that this can be expressed as: Download Time = x File Size Research Methods Tony Fleet

Chi-Squared Test This is used to determine whether the effects of two or more factors are independent. The data above shows the number of respondents of each gender in different age categories who expressed interest in using Blue Tooth technology within the next 12 months. A Ch-Squared test is used to test the hypothesis: Ho: There is no relationship between age and gender in potential Blue Tooth use. The probability value of the Chi-Squared Statistic is p = We would reject Ho, and conclude that there are differences between the age profiles of men and women potential users of Blue Tooth. Research Methods Tony Fleet

Which test? Research Methods Tony Fleet

Statistical Testing Research Methods Tony Fleet

Statistical Testing References (SW Library)
Ferguson, George A. - Statistical analysis in psychology and education. - 3rd. - Tokyo : McGraw-Hill Kogakusha, m Peers, Ian S.. - Statistical analysis for education and psychology researchers. - London : Falmer Press, St. John, P. R.. - Methods of statistical analysis of fieldwork data / P.R. St. John, D.A. Rich. - Sheffield : The Geographical Association, Research Methods Tony Fleet

6f Analytical Tools In order to analyse the data, you will need a package that is capable of arithmetic and other data manipulation, and will allow you to display the data in the form of tables and charts, as well as to perform statistical tests. There are several packages that you can use in order to analyse your data, but two of the main ones are: Microsoft Excel SPSS Research Methods Tony Fleet

Excel or SPSS? Excel will be OK to use if all your data is numeric, and that you have a clear idea of what you want to do, and how to do it. Excel is good at data display, but can only perform limited statistical testing, mainly parametric. SPSS is better at statistical testing, is highly versatile and can do lots of clever stuff. You will need this if you intend to analyze questionnaire data, or have lots of complex, categorical data and need to do non-parametric testing. Research Methods Tony Fleet

Using Excel Excel is available on all computers, and there is an extensive help facility embedded in it. You will probably have worked with it before, so your ‘learning curve’ is fairly gentle. If you have only a few items to display, and you are not going to do any heavy statistical testing, then Excel is probably the easiest tool to use. Research Methods Tony Fleet

Example of Excel Use (1) You can type data directly into an Excel worksheet. Excel has formulae to calculate statistics, such as: = Average(C3:C8) = StDev(C3:C8) Research Methods Tony Fleet

Example of Excel Use (2) This is an example of a chart easily produced from the data on the previous slide using Excel. Research Methods Tony Fleet

Example of Excel Use (3) Excel can produce a range of descriptive statistics very easily. Research Methods Tony Fleet

Using SPSS SPSS is “Statistical Package for the Social Sciences” and is an industry standard tool for data analysis. If you are doing lots of work with numerical data (e.g. analysing questionnaires) then you will need to use SPSS. SPSS is available via the F: drive, and it is also available for £5 from the technician in the Psychology Lab (GLB) Research Methods Tony Fleet

SPSS Tutorial Research Methods Tony Fleet

Example of SPSS Use Netscape Int Exp Part of a study looks at file upload times under 10 different conditions, using two types of browser: Netscape or Internet Explorer. The same circumstances relate to each pair of readings Research Methods Tony Fleet

Setting Up Variables First of all we set up descriptors for each of the two variables: Netscape Internet Explorer Research Methods Tony Fleet

Entering Values Then we Enter the 10 values for each variable, corresponding to the different cases. Research Methods Tony Fleet

Analysing the data We can select graphical and numerical summaries Research Methods Tony Fleet

Summaries Results: It looks as if Netscape generally takes longer to upload files than IE: secs for NS as opposed to secs for IE We might try to test this as a hypothesis. Research Methods Tony Fleet

Results of a T-Test We test: H0: there is no difference between sample means. H1: There is a difference between sample means. P = Unfortunately this is greater than .05, and not low enough to reject the null hypothesis; the result is not significant. Research Methods Tony Fleet

Using SPSS Research Methods Tony Fleet

Using SPSS References (SW Library)
Dancey, Christine P.. - Statistics without maths for psychology : using SPSS for Windows / Christine. - 2nd ed. - Harlow : Prentice Hall, Kinnear, Paul R.. - SPSS for Windows made simple : release 10 / Paul R. Kinnear, Colin D. Gray. - [New ed.]. - Hove : Psychology Press, Pallant, Julie. - SPSS survival manual : a step by-step guide to data analysis using SPSS for Windows. - Buckingham : Open University Press, West, Robert. - Computing for psychologists : statistical analysis using SPSS and MINITAB. - London : Harwood Academic, x Research Methods Tony Fleet

7. Problems with The Research Process

Research Pitfalls Finally, we look at some problems that you might encounter which invalidate your research – and how to prevent these. Research Methods Tony Fleet

Some Research Pitfalls (1)
Researcher Bias Problem: Introducing your personal prejudices or preconceptions into the research, and obtaining invalid results. Solution: Counter by an attitude shift: we do not try to prove a hypothesis is true, rather we test whether or not a hypothesis is true, and keep an open mind as to the result. Research Methods Tony Fleet

The Hawthorne Effect Problem: Subjects behave differently because they know they are participating in Research. Solution: Use of control group in experiments; concealed observation techniques; surveillance methods; naturalistic data collection methods Research Methods Tony Fleet

The Halo Effect Problem: If Subjects rate something as good or bad overall, they are likely to rate all features of it as good or bad, without discriminating between them. Solution: In questionnaires & evaluations, use a variety of similar questions, positively & negatively worded. Research Methods Tony Fleet

Pseudoscience Problem: Setting out to prove a hypothesis by only looking for evidence which confirms it, and disregarding evidence which contradicts it. Solution: Explicitly stating what would be disconfirming evidence for a hypothesis, and setting out to look for it. Research Methods Tony Fleet

Sample Bias Problem: Getting misleading results, because the sample on which you have based your analysis is consistently unrepresentative. Solution: Identifying the group of people, objects or events you are trying to generalise about, and selecting a sample which has the same characteristics as that group, or is a random selection from it. Research Methods Tony Fleet

Overgeneralisation Problem: Making sweeping claims that the results that you have observed from one or two cases will be present in the population at large. Solution: Be cautious in your claims, and use moderate language, such as ‘likely’ or ‘probable’ causes; only claim that your results are ‘significant’ when they are actually statistically significant, after carrying out a statistical test and rejecting the null hypothesis. Research Methods Tony Fleet

Mistaking Association for Causality Problem: Thinking that because one variable increases (or reduces) as another increases (or reduces) that we have found a ‘cause’ of an event. For example, the variable ‘difficulty of navigation’ in a multimedia package may be correlated with the number of buttons. However, it is the underlying navigational structure, which is at the root of the problem, which coincidentally generates the number of buttons. Solution: Understanding that correlation is about variation, and that multiple factors often act together to cause particular effects, and seeking such factors. Research Methods Tony Fleet

Ex Post Facto Hypothesizing Problem: Deciding on your research hypotheses after you have seen the data, and noticed some results which might be significant. In any data set, there are bound to be ‘blips’ which are freak results, and you are in danger of basing your results on these. Solution: Decide on your research hypotheses and the test you will do before you collect data. That way the tests will be fair, and your results valid. Research Methods Tony Fleet

8. Summary

Summary Things to consider Devising your Research Questions Selecting Your Methodology Deciding on Your Methods of Analysis Some Do’s & Don’ts Research Methods Tony Fleet

Relationship between Elements
Conduct Literature Review To a large extent, this process is iterative. You need a good idea of the data you intend to collect, and how you will use it, before you devise your research questions. Select Research Questions Devise Methodology & Research Instruments Apply Methods & Instruments Perform Statistical Analysis Test Hypotheses & Draw Conclusions Research Methods Tony Fleet

Devising Research Questions
Your Research Questions should arise naturally out of the Literature Review. However, in your Research Project you should only devise questions that you are capable of answering. That means when you ‘phrase’ them, you should have a good idea of the methods that you will use to answer them, the data that you will collect, and how that data will be analysed. What you should not know,is the answer to those questions! Research Methods Tony Fleet

Selecting your Methodology
Your methodology should be devised specifically to answer your Research Questions It should draw on the methods used by the studies that have formed a part of your Literature Review. It should also be created in such a way that the data that it produces is amenable to analysis That means you should be able to take advantage of the different forms of analysis, and the different tools available, in order to enhance your research. Research Methods Tony Fleet

Deciding on the Analysis
The method of Analysis needs to be appropriate to the data collected. It also clearly needs to be sufficient in order to answer the Research Questions . It should also use methods that have been established by previous researchers that you have highlighted in your Literature Review. Research Methods Tony Fleet

Advice – some Dos Plan ahead; think through your methodology and your analysis before framing your Research Questions. Include at least one ‘Research Hypothesis’, which you know you are going to test; this adds credibility to your dissertation. Spend time looking at how previous researchers have pursued their research; draw on their experience, adapt their methods and modify their research instruments… but don’t forget to credit them in the write-up! Consult with your supervisor, take advice. Research Methods Tony Fleet

Advice – some Don’ts Do not set out to ‘prove’; set out to ‘investigate’. Do not try to take on too wide a topic; this is an MSc. Dissertation, and you should be working in a confined area, but achieving true depth of analysis Do not expect to have statistically significant results; most Masters’ dissertations fail to prove anything one way or the other. However, you are expected to know whether the results are statistically significant or not. Research Methods Tony Fleet

… and Finally If you are doing a questionnaire, or data analysis, you may need to have specialist advice. You can sign up to see me, even though I am not your supervisor. However, if you come for a consultation, I expect you to know what you want to do, and have a good idea of how to do it; you should come to me to check that you are on the right lines, not to find out what to do. Contact Tony Fleet in FML 213, telephone: or me at Research Methods Tony Fleet

References and Bibliography
Comprehensive accounts of Research methods: Huge list of resources & links: Undertaking a Research Project: Guidelines Research Methods Tony Fleet

Research Methods Books in the Sheppard-Worlock:
Fowler, Floyd J.. - Survey research methods. - 2nd ed. - Newbury Park, Calif.; London : Sage Publications, (Applied social research methods series ; v.1) Lehman, Richard S. - Statistics and research design in the behavioural sciences. - Pacific Grove, Calif. : Brooks/Cole, Research Methods Tony Fleet

Other Research Design Books in the Sheppard-Worlock:
Handbook of applied social research methods / edited by Leonard Bickman and. - London : SAGE, x Human centred methods in information systems : current research and practice. - Hershey; London : Idea Group Pub, Qualitative research : theory, method and practice / edited by David Silverman. - London : Sage, Research design. - A. - Milton Keynes : Open University Press, (Social sciences, a third level course research methods in education and Social Sciences ) Research Methods Tony Fleet

Past MSc. Dissertations
You should examine what other students have done, especially in an area which is akin to yours. Remember: if it is on the shelves, it passed! Examples: Broadbent, George Ernest. - The role of evaluation in user interface design : a practical implementation. - Liverpool : University of Liverpool, p Lomas, Joanne. - The research and design of a computer based training package for children. - Liverpool : University of Liverpool, M LO Research Methods Tony Fleet

Collecting, Processing and Analyzing Data

Similar presentations

Presentation on theme: "Collecting, Processing and Analyzing Data"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Collecting, Processing and Analyzing Data

Similar presentations

Presentation on theme: "Collecting, Processing and Analyzing Data"— Presentation transcript:

Similar presentations

About project

Feedback