Presentation is loading. Please wait.

Presentation is loading. Please wait.

Evaluation Research into the effectiveness of university learning environments Rob Phillips Academic Consultant Educational Design Project, School of Education,

Similar presentations


Presentation on theme: "Evaluation Research into the effectiveness of university learning environments Rob Phillips Academic Consultant Educational Design Project, School of Education,"— Presentation transcript:

1 Evaluation Research into the effectiveness of university learning environments Rob Phillips Academic Consultant Educational Design Project, School of Education, Curtin University, Perth Adjunct Associate Professor, School of Education, Murdoch University, Perth

2 My Background 1980s – Research scientist – theoretical chemistry and computer science – Germany and Australia 1990s – Interactive multimedia designer and project manager 2000s – Educational designer 2010s – Learning and Teaching Policy work – Academic consultant – mentoring and educational design

3 Introductions Why are you here?

4 Book advert

5 Disclaimer The book is largely about e-learning The ideas apply well to any learning environment – almost all include technology these days

6 Overview Phenomena The e-learning lifecycle e-learning evaluation research Cyclical evaluation research approaches Divide and Conquer techniques Work on own project

7 Fundamentals of Research Inquiry into the behaviour of phenomena Exercise of human curiosity Attempting to solve problems in the world Attempting to understand the world

8 In Broad Terms What sort of research do you do? What do you investigate? What are the main characteristics of this research? Work in pairs

9 Fundamentals of Research Inquiry into the behaviour of phenomena Exercise of human curiosity Attempting to understand the world Attempting to solve problems in the world

10 What is a phenomenon? Inquiry into the behaviour of phenomena What is a phenomenon? – an instance of something – an artefact or event that is known through the senses

11 Natural Phenomena They exist naturally in our world – the seasons, gravity and waves on the ocean – we can observe and measure them Other natural phenomena cannot be directly observed – the inner workings of atoms and molecules – their behaviour can be measured

12 Artificial Phenomena Phenomena/ artefacts which we create through a design activity Tangible – aircraft, cars, etc. Intangible – computer software

13 Event Phenomena An artefact or event that is known through the senses In Education – a coming-together of teacher, learners and various resources in a particular setting, usually following some design activity – Learning Environment

14 Summary Natural phenomena 1 phenomenon – many instances Artificial phenomena – Many phenomena – many instances Event Phenomena – Many phenomena – single instances

15 Paradigms vs Ideology Paradigms The set of practices that define a scientific discipline at any particular period of time (wikipedia) A world view underlying the theories and methodology of a particular scientific subject (http://oxforddictionaries.com)

16 Ideology The set of beliefs characteristic of a social group or individual (http://oxforddictionaries.com) A comprehensive and coherent set of basic beliefs about political, economic, social and cultural affairs that is held in common by a sizable group of people within a society. (http://www.socialsciencedictionary.com)

17 Paradigms vs Ideology Is your research approach based on a paradigm or an ideology?

18 Overview Phenomena The e-learning lifecycle e-learning evaluation research Cyclical evaluation research approaches Divide and Conquer techniques Work on own project

19 E-learning Artefacts and Environments We design and use e-learning artefacts (intangible artificial phenomena) – learning objects, learning management systems, e-portfolios e-learning environments (event phenomena) add the human context – an interaction between learners, teachers and resources in a set context

20 Inquiring About the World (IATW) New science unit for a new primary and early childhood degree – First offered in 2013 Teaching problem – How to make science interesting and engaging Design – Authentic learning approach – Students investigate a self-chosen environmental issue over the whole semester

21 Learning Outcomes Develop lifelong learning skills including: analytical thinking, information management, technology skills and communication skills Apply processes of inquiry to carry out a scientific investigation Report on the outcomes of the processes of inquiry using a range of Information and Communication Technologies Evaluate the outcomes of the inquiry processes in the context of the science and humanities content from the Australian Curriculum

22 Assessment 1Project Plan15% 2Report Draft20% 3Report45% 4Presentation20%

23 Video IATW Flash videoFlash video

24 Baseline The Learning Lifecycle Phase Design Develop Evaluate Trial learning env’t Learning environment Mature Learning environment Learning lifecycle

25 The IATW Lifecycle On- campus and Open Universities Australia (OUA) DateOffering July 2013Curtin Semester 2 May 2014OUA Study Period 2 July 2014Curtin Semester 2 Nov 2014OUA Study Period 4 May 2015OUA Study Period 2

26 Overview Phenomena The e-learning lifecycle e-learning evaluation research Cyclical evaluation research approaches Divide and Conquer techniques Work on own project

27 My Invention The Kugelschreiber A better way to write!

28 Researching the Kugelschreiber I know that this technology is better than old, pencil technology Compare the two, in writing a test Treatment and control groups

29 Researching the Kugelschreiber I know that this technology is better than old, pencil technology Compare the two, in writing a test Treatment and control groups Kugelschreiber results were slightly worse than pencil results, but the difference was not significant at the 90% confidence level. The Kugelschreiber is no worse than a pencil.

30 Inquiry into Phenomena Natural – Goal: observing how natural phenomena work and ultimately understand them Artificial – Goal: design artefacts, improve their design and use them and ultimately understand them Event – Goal: design events, improve their design and ultimately understand them

31 Goals of Research Be clear about the phenomenon you are investigating Clearly define the goals and questions of your research Choose the research approach accordingly

32 Summary so Far We have thought about the phenomenon of e-learning We have different research goals at different stages of the e-learning lifecycle How will we conduct this research?

33 General Research Approaches Pure versus applied research Hard versus soft research Research and development Research then development Becher, T. (1989). Academic tribes and territories: intellectual enquiry and the cultures of disciplines. Buckingham: Society for Research into Higher Education. Jones, C., Zenios, M., & Griffiths, J. (2004). Academic use of digital resources: Disciplinary differences and the issue of progression Proceedings of the 4th Networked Learning Conference. Lancaster University, England, UK.

34 Boyer’s Four Scholarships Scholarship of discovery Scholarship of integration Scholarship of application Scholarship of teaching Boyer, E. L. (1990). Scholarship reconsidered - Priorities of the professoriate. New Jersey: Carnegie Foundation for the Advancement of Teaching. Boyer, E. L. (1996). The scholarship of engagement. Journal of Public Outreach, 1(1),

35 Pasteur’s Quadrant Stokes, D. E. (1997). Pasteur's Quadrant: Basic science and technological innovation. Washington, D.C.: The Brookings Institution.

36 What is evaluation? The term is used imprecisely Evaluation implies judgement and decisions Formative evaluation: – judgements about improving something Summative evaluation: – judgements about the merit or worth of something

37 Investigating a New Learning Environment Questions: – Does the environment function as designed? – How can the environment be improved? These are evaluation judgements Is it any good???

38 Investigating a Mature Learning Environment A study of a mature learning environment may shed light on – how learners engaged with the designed learning environment – why some learners achieved at different levels – how some learners used the learning environment to achieve a deeper understanding These are educational research questions The goal is increased understanding

39 Evaluation vs Research Studies of learning environments involve a mixture of – Evaluation: making judgements about improvement and value and worth – Research: increasing understanding Continuum between judgements and understanding – Does it function as designed? – How can it be improved? – How does this lead to improved student learning? Similar methods – different goals Judgments Understanding

40 IATW - currently Pre- and post-test – Confidence with ICT – Knowledge of scientific inquiry processes Anonymous survey of students – Did they like it?

41 IATW questions to ask Was the website usable? What things didn’t work as well as they should? Was there evidence that students engaged with the authentic task? Will the learning design scale up for 2000 OUA students?

42 Overview Phenomena The e-learning lifecycle e-learning evaluation research Cyclical evaluation research approaches Divide and Conquer techniques Work on own project

43 Baseline The Learning Lifecycle Phase Design Develop Evaluate Trial learning env’t Learning environment Mature Learning environment Learning lifecycle

44 Baseline analysis Evaluation Research across the Learning Lifecycle Design evaluation Design Develop Evaluate Formative evaluation Summative evaluation Effectiveness research Summative evaluation Effectiveness research Learning lifecycle

45 e-learning Research Approaches Need to apply to an artificial phenomenon Reflect the e-learning life cycle Cyclical research approaches – Design-based research – Action Inquiry

46 Design-based Research Focus on a designed learning environment A cyclical approach that originated in engineering and other design fields Consistent with – a mixture of evaluation and research – Stokes’ views on quest for understanding and consideration of use

47 Action Inquiry Focus on personal practice Local focus Little concern with theory

48 Drawing the Threads Together e-learning environments are designed phenomena – they follow a lifecycle Cyclical research approaches Mix of evaluation and research – make sure it works first Now – onto the practical stuff

49 Overview Phenomena The e-learning lifecycle e-learning evaluation research Cyclical evaluation research approaches Divide and Conquer techniques Work on own project

50 Breaking Down the Complexity How to use all this information: – It’s a complex situation, with – many elements, which – evolves over time It requires multiple research methods Use divide and conquer techniques and a template to simply the issue

51 Divide and Conquer Techniques Evaluation research process diagram Five evaluation-research ‘forms’ Break down the e-learning lifecycle – To develop appropriate research questions Use an evaluation-research matrix Use template to combine into an evaluation- research plan

52 Evaluation- research Process

53 Evaluation- research Process 1

54 Design Principles relatively humble ‘proto-theories’ not ‘grand’ theories of learning Two roles for theory: – Drawing on theory to support evaluation-research goals and questions across the e-learning lifecycle – Building theory from research outcomes

55 Builds theory Draws on theory Baseline analysis Evaluation Research across the Learning Lifecycle Design evaluation Design Develop Evaluate Formative evaluation Summative evaluation Effectiveness research Summative evaluation Effectiveness research Learning lifecycle

56 Evaluation- research Process 1

57 The LEPO Framework Learning Environment, Process, Outcomes Educational Context Learning environments Learning processes Learning outcomes Students Teacher s facilitate design facilitate assess lead todetermine work within engage in demonstrate interact with

58 IATW Conceptual Framework Model for inquiry-based learning Model for information literacy Herrington’s 9-stage authentic learning model

59 Broad Evaluation-Research Questions The broadest questions that can be asked in the study Open-ended, exploratory questions IATW examples: – How can the unit design be improved for a large, online OUA cohort? – How effective is the unit at developing inquiry skills in students?

60 Evaluation- research Process 2

61 IATW Stakeholders – Centre for Teaching and Learning – Head of School – Standards bodies Participants – Students – Teaching staff – LMS support staff

62 Activity Fill in page 1 of the template Discuss it with your partner

63 Divide and Conquer Techniques Evaluation research process diagram Five evaluation-research ‘forms’ Break down the e-learning lifecycle – To develop appropriate research questions Use an evaluation-research matrix Use template to combine into an evaluation- research plan

64 Evaluation-research Forms Baseline analysis: Document the teaching and learning context; needs analysis; assumptions, literature Design evaluation: Evaluate the documented design of the learning environment Formative evaluation: Make judgments about, and suggest improvements to, the e-learning environment Effectiveness research: Develop understanding of how learners engage with learning tasks to demonstrate learning outcomes Project management evaluation: judgements about the conduct of a project

65 Evaluation- research Process 3

66 Specific Evaluation-research Questions Broad evaluation-research questions are hard to answer Specific evaluation-research questions should be answerable – identify the evidence which is needed to answer them Synthesise the evidence from the specific questions to answer the broad questions

67 Examples of Specific Research Questions Formative evaluation of the e-learning environment How do learners use the learning environment? To what extent do learners engage with learning tasks as intended? Are the learning tasks appropriate for the target learners? How useful do learners find the e-learning environment?

68 Divide and Conquer Techniques Evaluation research process diagram Five evaluation-research ‘forms’ Break down the e-learning lifecycle – To develop appropriate research questions Use an evaluation-research matrix Use template to combine into an evaluation- research plan

69 Evaluation- research Process 3

70 Evaluation-research Matrices To map data sources and methods against the research questions Examples from IATW

71 Baseline Analysis

72 Design Evaluation

73 Formative Evaluation

74 Effectiveness Research

75 Divide and Conquer Techniques Evaluation research process diagram Five evaluation-research ‘forms’ Break down the e-learning lifecycle – To develop appropriate research questions Use an evaluation-research matrix Use template to combine into an evaluation- research plan

76 Evaluation-research Plan Use the template Work through the evaluation matrices and develop questions which are relevant to your project Specify your data sources – Interview schedules – Survey questions

77 Do the evaluation research

78 Overlap between Forms

79 Overview Phenomena The e-learning lifecycle e-learning evaluation research Cyclical evaluation research approaches Divide and Conquer techniques Work on own project

80 Evaluation-research Plan Use the template to start to develop your own evaluation research plan Work through the evaluation matrices and develop questions which are relevant to your project

81 Application of the Approach PhD research Teaching improvement Preparation for grant applications

82 Where to publish your work? Journal of Perspectives in Applied Academic Practice International Journal for SoTL Invited essays, Research papers, essays MERLOT Journal of Online Learning and Teaching Journal of University Teaching and Learning Practice International Journal of Innovation in Science and Mathematics Education IJISME is dedicated to encouraging and supporting early career researchers as well as academics in science and mathematics who are new to education research.


Download ppt "Evaluation Research into the effectiveness of university learning environments Rob Phillips Academic Consultant Educational Design Project, School of Education,"

Similar presentations


Ads by Google