1. A Domain-specific Modeling Approach to the Development of Online Peer Assessment Yongwu Miao and Rob Koper Educational Technology Expertise Centre Open University of The Netherlands TENCompetence Workshop Barcelona, June 22th, 2007

2. Overview Problem Definition Domain-specific Modelling Peer Assessment Peer Assessment Modelling Language Target Peer Assessment Modelling Tool Conclusions and Future Work

3. Problem Definition Software development approaches to support peer assessment. A standard-based modeling approach to support peer assessment. features approaches Interoperability and Reusability Flexibility and Efficiency Integration with Learning End User Friendly Software Development Approaches No Yes Standard-based Modeling Approach Yes No Addressed problem

4. Domain-specific Modeling What is Domain-Specific Modeling (DSM) or Domain-specific language (DSL) DSM is a kind of model-driven approach to develop software applications. It raises the level of abstraction beyond programming by specifying the solution in terms of concepts and associated rules culled from the very domain of the problem being solved. The final software products are generated from these high-level specifications. Why to Use DSM DSM are more expressive than and therefore tackle complexity better, making modeling easier and more convenient. More importantly, they allow automatic, full code generation, similar to the way today's compilers generate Assembler from a programming language like JAVA.

5. What are differences between DMS and other approaches different users different languages different abstraction levels Domain-specific Modeling LD & QTI model Map to LD & QTI interpret

6. Domain-specific Modeling How to adopt DSM approach to develop software Developing a meta-model Develop executable components Developing mapping functions to transform domain model to components

7. Peer Assessment Definitions: Peer assessment is a process whereby groups rate their peers (Falchikov 2001) Peer assessment is an arrangement for peers to consider the level, value, worth, quality or successfulness of the products or outcomes of learning of others of similar status (Topping, Smith, Swanson, & Elliot, 2000). Peer assessment models: Pure peer assessment (two learners do assignment and then assess the outcome of his/her peers) With additional stages (design assessment and react to feedback) Group/collaborative assessment (more candidates and reviewers) With involvement of the tutor Different types of assignment tasks Different distribution patterns ……

8. Peer Assessment A typology of peer assessment (Topping 1998)

9. Peer Assessment Modeling Language Four kinds of stages In each stage, people with a certain role perform activities In each activity, there may be input and output documents

10. Peer Assessment Modeling Language Guide to make design decisions by showing guidance and consequence Make decisions by selecting default or pre-defined options Customize a process on any aspect (for experienced users)

11. Target Peer Assessment Authoring Tool A peer assessment authoring tool will be based on the peer assessment modeling language (meta-model). Users will be guided to specify a peer assessment model by making a series of design decisions. The consequences of the decision-making will be presented as a diagram. Users can customize the peer assessment model by editing the diagram directly. The tool is designed and will be implemented soon. The design and use of the tool will be explained (in a fake UI) by using an example

12. design assignment writing article1 reviewing article2 improving article1 writing article2 reviewing article1 improving article2 final assessment2final assessment1 learner1learner2tutor article2 feedback1 improved a2 paper1 article1 feedback2 improved a1 paper2 Target Peer Assessment Authoring Tool A peer assessment example tutor’s feedback1tutor’s feedback2 assessment form1assessment form2

13. File Edit View Tool Template Help docroleservicelanestage Peer assessment Here shows the guidance to make design decisions State bar user’s decisions Specify role structure, document, and service Show and edit peer assessment model

14. File Edit View Tool Template Help docroleservicelanestage Peer assessment How many stages in the peer assessment process? Please make a decision user’s decisions Default (three stages) With design assessment stage With additional feedback stage More complicated setting

15. File Edit View Tool Template Help docroleservicelanestage Peer assessment How many roles will be involved? Five stages have been defined design assessmentdo assignmentgive feedback react to feedbackgive feedback 2 Default (two persons) With the involvement of tutor More complicated setting

16. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to define role structure? reviewerRdesignerRcandidateR Default role setting and with the involvement of a tutor learner1RtutorRlearner2Rlearner1Rlearner2RtutorR

17. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to define a typical activity for each stage ? reviewerRdesignerRcandidateR Please make a decision learner1RtutorRlearner2Rlearner1Rlearner2RtutorR Default setting More complicated setting

18. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to define a typical activity for each stage ? reviewerRdesignerRcandidateR Nine activities are defined and roles are assigned learner1RtutorRlearner2Rlearner1Rlearner2RtutorR learner1 learner 2 learner1learner2tutor

19. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Which stages will the tutor be involved? reviewerRdesignerRcandidateR Please make a decision learner1RtutorRlearner2Rlearner1Rlearner2RtutorR learner1 learner 2 learner1learner2 Give feedback 1 Give feedback 2 More complicated setting tutor

20. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Which stages will the tutor be involved? reviewerRdesignerRcandidateR The tutor is assigned to work in the stage of “give feedback 2” learner1RtutorRlearner2Rlearner1Rlearner2RtutorR learner1 learner 2 tutor

21. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to create documents ? reviewerRdesignerRcandidateR Please make a decision learner1RtutorRlearner2Rlearner1Rlearner2RtutorR Default setting More complicated setting learner1 learner 2 tutor

22. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Would you like to create typical documents ? document are created learner1 learner 2 tutor feedback1.1Dfeedback1.2Doutcome1Doutcome2Dassignment1Dassignment2Dimproved outcome1Dimproved outcome2Dfeedback2.1Dfeedback2.2D

23. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Could you please change the titles of documents and define documents? document are created learner1 learner 2 tutor feedback1.1Dfeedback1.2Doutcome1Doutcome2Dassignment1Dassignment2Dimproved outcome1Dimproved outcome2Dfeedback2.1Dfeedback2.2D finished

24. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Could you define document? document are created learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2D finished learner 2 Review Form How do you think about this article? accept accept with minor/major modification? reject Comments: assessment form assessment editor assessment formD

25. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 Could you define document? document are created learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2D finished assessment formD

26. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 How will the documents be distributed? Please make a decision learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2Dassessment formD Default setting More complicated setting

27. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 How will the documents be distributed? documents are distributed by default learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2Dassessment formD

28. File Edit View Tool Template Help docroleservicelanestage design assessmentdo assignmentgive feedback react to feedback Peer assessment give feedback 2 How will the documents be distributed? doc-flow are customized learner1 learner 2 tutor feedback1Dfeedback2Darticle1Darticle2Dpaper1Dpaper2Dimproved article1Dimproved article2Dtutor’s feedback1Dtutor’s feedback2Dassessment formD

29. Conclusions and Future Work Conclusions: Outline an approach to apply the domain-specific modelling paradigm to the task of the development of peer assessment Develop a peer assessment modeling language Propose a method to enable the end-user to model peer assessment processes through making a series of design decisions Future work: Develop executable components Design and implement mapping algorithm Implement the tool and conduct evaluation

30. Thanks for your attention! Questions?