1. 26 August 2015© Academic Conferences Limited, 2008 1 Validity, Reliability and Generalisability by Dr Dan Remenyi Visiting Professor School of Systems and Data Studies Trinity College Dublin dan.remenyi@tcd.ie Copyright © 2008 Academic Conferences Limited All rights reserved. No part of this document may be reproduced or utilized in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without permission in writing from the publisher.

2. 3 dimensions of research To be credible in the academic world the researcher needs to show that he/she has addressed 3 key issues which are sometimes described as:-  Validity,  Reliability and  Generalisability These issues need to be addressed at the research design stage and in the conclusions 26 August 2015© Academic Conferences Limited, 2008 2

3. Validity Validity refers to whether the issue/s which the researcher believes are being studied are in fact what is actually being studied. Ex-Ante or external validity ▫Face Validity – On the face of it is the instrument valid and are prospective knowledgeable informants able to understand what is required by the questions ▫Content Validly – do the specific questions in the interview schedule or questionnaire etc map well on to the research questions/hypotheses 26 August 2015© Academic Conferences Limited, 2008 3

4. Validity Ex-Post Validity ▫Internal validity  Only required in cause and effect research ….. Is the cause being studied really producing the effect  http://www.socialresearchmethods.net/kb/intval.php http://www.socialresearchmethods.net/kb/intval.php ▫External validity  To what extent may be finding be generalised  http://www.socialresearchmethods.net/kb/external.php ▫Construct validity  To what extent can the operationalised variables be confidently regarded to be part of the construct and to what extent can it be generalised  http://en.wikipedia.org/wiki/Construct_validity http://en.wikipedia.org/wiki/Construct_validity 26 August 2015© Academic Conferences Limited, 2008 4

5. Validity ▫Convergent and Discriminate validity  Related concepts which measure how well the construct and its component parts are consistent  http://www.socialresearchmethods.net/kb/convdisc.php http://www.socialresearchmethods.net/kb/convdisc.php ▫Statistical conclusion validity Drawing the most appropriate statistical conclusion http://www.google.co.uk/search?sourceid=navclient&ie=UTF &rlz=1T4GPEA_enGB297GB297&q=statisitcal+conclusion+va lidity 26 August 2015© Academic Conferences Limited, 2008 5

6. Reliability In general research is said to be reliable if it may be repeated by others and produce the same or similar results This concept represents special challenges for the qualitative researcher ▫qualitative research often cannot be reproduced exactly due to there being different research participants involved ▫If the same researchers are involved then there may be problems with regards to bias 26 August 2015© Academic Conferences Limited, 2008 6

7. Reliability There are 4 dimensions to reliability ▫Inter-Observer Reliability Used to assess the degree to which different raters/observers give consistent estimates of the same phenomenon. ▫Test-Retest Reliability Used to assess the consistency of a measure from one time to another. ▫Parallel-Forms Reliability Used to assess the consistency of the results of two tests constructed in the same way from the same content domain. ▫Internal Consistency Reliability Used to assess the consistency of results across items within a test. 26 August 2015© Academic Conferences Limited, 2008 7

8. Reliability and Cronbach's alpha Cronbach's alpha measures how well a set of items (or variables) measures a single unidimensional latent construct. When data have a multidimensional structure, Cronbach's alpha will usually be low. Technically speaking, Cronbach's alpha is not a statistical test - it is a coefficient of reliability (or consistency). Cronbach's alpha can be written as a function of the number of test items AND the average inter-correlation among the items. http://www.ats.ucla.edu/stat/spss/faq/alpha.html http://www.ats.ucla.edu/stat/spss/faq/alpha.html http://www.socialresearchmethods.net/kb/reltypes.php 26 August 2015© Academic Conferences Limited, 2008 8

9. Generalisability Academic research needs to produce a contribution to the body of theoretical knowledge There are different types of theoretical contribution  Substantive level theory – minor abstraction  Midlevel theories – more abstraction  Formal theories – highest level of abstraction 26 August 2015© Academic Conferences Limited, 2008 9

10. Generalisation Generalisation is an issue in research which focuses on prediction and control. It is not an issue in research which aims to achieve understanding and explanation Dimensions of generalisation – ▫an idea or conclusion having general applicability; ▫the process of formulating general concepts by abstracting common properties of instances ▫reasoning from detailed facts to general principles i.e. induction 26 August 2015© Academic Conferences Limited, 2008 10

11. Generalisations There is a mistaken belief that a theory is either generalisable or not. This is untrue. There are degrees of generalisability. 26 August 2015© Academic Conferences Limited, 2008 11

12. Convincing results – a modern day Rosetta Stone? 26 August 2015© Academic Conferences Limited, 2008 12 http://www.socialresearchmethods.net/kb/qualval.php Generalisability is said to have its equivalence in usability

13. Credibility Is the research approach and the finding believable by those involved in the research? This is similar to the issue of validity 26 August 2015© Academic Conferences Limited, 2008 13

14. Transferability Can the result be used elsewhere? This is a similar issue to generalisability 26 August 2015© Academic Conferences Limited, 2008 14

15. Dependability Can we regard the results as being such that we can repeat the study and obtain similar results This is similar to reliability 26 August 2015© Academic Conferences Limited, 2008 15

16. Confirmability In qualitative research there is the assumption that each research brings his/her own value set with them to the research. However is there anyway that anyone else would be able to confirm the results 26 August 2015© Academic Conferences Limited, 2008 16

17. Validity, Reliability and Generalisability Research may not be perfect in all three dimensions but nonetheless these issues need to be explicitly addressed The researcher may have to compromise It is not always possible to find the most appropriate informants The research question may transpire to be difficult to answer The data can be patchy The findings may not be as conclusive as you want 26 August 2015© Academic Conferences Limited, 2008 17

18. Threats to the research project There are numerous threats to your research Bias is a continual issue. It is difficult not to project your values not to project your values into the research It is difficult to ensure that you have a suitable sample.....a representative sample Applying an inappropriate analytical technique Misinterpreting the results Arriving at the wrong conclusions 26 August 2015© Academic Conferences Limited, 2008 18