Presentation on theme: "Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference K. Schützler, K. Bothe."— Presentation transcript:
Reviewers comments on our paper submitted to 35th ASEE/IEEE Frontiers In Education Conference K. Schützler, K. Bothe
Kay Schützler, Baile Herculane, 28.08. - 03.09.052 Introduction In 2003 first presentations of current JCSE project state: –Information Technology Interfaces, Cavtat, Croatia –Eurocon, Ljubljana, Slovenia –1st Balkan Conference in Informatics, Thessaloniki, Greece 2005: submitted a new paper with current state to 35th IEEE/ASEE Frontiers In Education Conference, Indianapolis
Kay Schützler, Baile Herculane, 28.08. - 03.09.053 Homepage of FIE 2005
Kay Schützler, Baile Herculane, 28.08. - 03.09.054 Topics of interest on FIE 2005 Accreditation and assessment Active learning Capstone and senior design experiences Computer and Web-based software Creative design experiences CSET educational research Distance learning: Methods, technologies, and assessment Diversity: Valuing it, achieving it, and teaching it Entrepreneurship programs Ethics: Creative ways to teach and assess it Faculty development First-year courses and programs Globalization: Preparing faculty and students Innovative degree programs and curricula Innovative uses of technology in the classroom K-12 initiatives and partnerships Laboratory experiences: On-site and at a distance Learning models Lifelong learning Nontraditional students Partnerships (industry, government, university, international) Service learning Software engineering Student retention and persistence Teaming Undergraduate research experiences Women in CSET education Other (You may submit abstracts and proposals on other topics that address issues at the frontiers in CSET education.)
Kay Schützler, Baile Herculane, 28.08. - 03.09.055 K. Bothe, K. Schützler, Z. Budimac, K. Zdravkova: Collaborative Development of a Joint Web-Based Software Engineering Course Across Countries
Kay Schützler, Baile Herculane, 28.08. - 03.09.056 Aims of this presentation Document the serious and detailed review process of FIE 2005 Make reviewers comments available for all project participants Draw some conclusions
Kay Schützler, Baile Herculane, 28.08. - 03.09.057 January 10, 2005: Abstract submission February 20, 2005: Acceptance/Rejection of abstract March 21, 2005: Full paper submission April 30, 2005: Notification on review results May 30, 2005: Submission of revised paper with modification notes
Kay Schützler, Baile Herculane, 28.08. - 03.09.058 Excerpt from first accept notification email Dear Klaus, The final recommendation for FIE paper 1106 is: ACCEPT with major changes In the comments below, you will see numerical scores for different aspects of your presentation. The scale was: 1 - poor 2 - needs work 3 - good 4 - very good 5 - excellent That means: Each comment of the reviewers must be addressed (even if considered inappropriate)
Kay Schützler, Baile Herculane, 28.08. - 03.09.059 Two reviewers raise a number of questions/issues that require greater revision than the reviewers indicated - thus my recommendation. As you do have an amount of space left in the paper to addresses these, I recommend that you focus on those that would broaden the usefulness of the paper to a wider audience. In an email to me, please indicate how and where the issues raised by the reviewers are addressed. If you feel that the comments are not appropriate, please indicate that to me as well. Note from Heidi Diefes-Dux (from the organizer‘s site):
Kay Schützler, Baile Herculane, 28.08. - 03.09.0510 Reviewer's recommendation: accept_as_is New ideas: yes Comments on new ideas: It addresses collaborative development of course materials across several countries. Soundness of work: yes Contribution: 3 Comments on work contribution: project management issues Presentation quality: 4, Writing quality: 4, References: yes Gender neutral language: yes, Should anything be deleted: no Completeness of work: no Comments on completeness: still somewhat of a work in progress Reviewer 1 That’s all we received from this reviewer That’s all we received from this reviewer
Kay Schützler, Baile Herculane, 28.08. - 03.09.0511 Reviewer's recommendation: accept_revise New ideas: yes Comments on new ideas: Describes experience with a shared course in software engineering with participants in multiple countries. Approach could be useful for technical education in many locations around the world. Soundness of work: yes Reviewer 2
Kay Schützler, Baile Herculane, 28.08. - 03.09.0512 Contribution: 4 Comments on work contribution: Describes a successful project in which many participants not sharing a language use common course materials to support a course in software engineering. Insight and experience from this project could be of broad general interest, particularly in developing countries and countries partnering with them in education. Presentation quality: 4, Writing quality: 4, References: yes Gender neutral language: yes Should anything be deleted: no, Completeness of work: yes Reviewer 2 (cont.)
Kay Schützler, Baile Herculane, 28.08. - 03.09.0513 This is interesting work and the quality of the writing is fairly good. My questions address relatively minor points, but I encourage the authors to read over the paper again carefully to catch a few remaining awkward sentences, typos, and misused words. We went through the paper as requested and indeed found a few mistakes and even one incomplete sentence. Reviewer 2 comments Writing quality The reviewer’s comment Our answer
Kay Schützler, Baile Herculane, 28.08. - 03.09.0514 You describe the course as web-based, but that applies more to the instructors than student participants. (That is, students apparently attend lectures where they see powerpoint slides rather than use machines on a network.) This seems to be inconsistent with the usage of the term in my experience. Reviewer 2 comments Web-based course? We agree with reviewer's description of our course, but we still call the course web-based. In section "Overview of project management and course materials" we added "Although students are expected to attend regular lectures, the course as a whole could be called web-based. All documents the students need are published at the course website: the slides in PDF-format, case studies, assignments, tool information, literature recommendations, schedule, etc. skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0515 The abstract mentions a "plethora of interesting contributions" from the diff- erent participants across various universities and countries. Could you give some examples? I'm curious how the course has evolved over time. You allude to such changes but give essentially no concrete examples or details. Reviewer 2 comments Contributions from all participants? We added a paragraph in section that gives an example how the contributions changed the course. Besides that, we would like to draw the attention to Figure 3 for additional information on the contributions of all project partners. Translate materials to English Add new slides and other materials Include new topics Write review reports Develop slide style guides Write lecture notes Design and administrate the project web-site Contribute to the assignments Use slides in lectures and give usage reports Contribute to the pool of case studies Translate materials to native language Course development Figure 3: Contributions to the project
Kay Schützler, Baile Herculane, 28.08. - 03.09.0516 You don't say much about some of the links under course materials, specifically the examinations, tools, literature, and duration links. Are copies of all examinations shared with other participants? What about translation overhead? What kind of tools are currently available, or do you plan to make available in the future? Will the literature link simply give citations of relevant papers, or does this describe internal documents for the course? What duration information is included: just the time required in class to cover each topic or to cover each lecture? Reviewer 2 comments One comment – many questions: examinations, tools, literature, duration We extended section "Course materials" with the new subsections "V. Literature" and "VI. Tools". It covers now all relevant aspects of the materials on the project website. In section "Overview of project management and course materials" we extended the information concerning the Duration link. skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0517 Are the assignments linked in (under course materials) labs or homework style problems? What kinds of assignments have been included in the course so far, and what kind of tools or documentation do they require? Again, the missing information is addressed in two new paragraphs in subsection "IV. Assignments" of section "Course materials". "Typical tasks of the assignments include: reviews of requirements specifications, object-oriented modeling practices, performing cost estimations, applying test case determination methods, automated regression testing, and evaluating software metrics." Reviewer 2 comments Assignments
Kay Schützler, Baile Herculane, 28.08. - 03.09.0518 It seems challenging to let all participants update course mate- rials but to store only a single current file. What if you don't like the changes someone else makes? Reviewer 2 comments Concurrent updates? To address this question we added an extra paragraph at the end of subsection "IV. Update management" in section "Project management to organize collaborative work": "Of course, it is necessary to assess the results of the update process. Modified topics are reviewed by their original author. If there are any issues raised, they will be discussed and solved in a cooperative way between author and modifier." skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0519 Do you have any way of measuring the effectiveness of the class or of this approach to managing a class? To what extent has it reduced the overhead for new participants in creating a new class and getting it up and running? Reviewer 2 comments Collaboration success measured? The "Conclusions and future work" section has been extended: "Our original goals to disseminate experience and to reduce efforts in creating a new class on software engineering were reached successfully. Participants at several universities could introduce an approved and mature course with all of its parts (lectures, assignments, case studies, etc.) Both, the many years of experience and the produced materials could be reused. The partner universities in southeast Europe experienced significant reduces (compared to earlier attempts of building new courses from scratch) of the managerial efforts for introducing the course, a benefit that can hardly be overestimated." skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0520 Reviewer's recommendation: accept_revise New ideas: no Comments on new ideas: Multi-national collaboration and material sharing. Soundness of work: yes Contribution: 2 Comments on work contribution: Giving confidence to others to follow this pattern of working. Reviewer 3
Kay Schützler, Baile Herculane, 28.08. - 03.09.0521 Presentation quality: 5 Writing quality: 4 References: yes Gender neutral language: yes Should anything be deleted: no Completeness of work: no Reviewer 3 (cont.)
Kay Schützler, Baile Herculane, 28.08. - 03.09.0522 Extra commentaries by the participants and a retrospective an- alysis of the effectiveness of the transfer process would have been useful. Reviewer 3 comments Missing information As mentioned above, we discuss the effectiveness of the tran- sfer in our new version of the paper. We do not think that com- mentaries would fit in the structure of the paper. In our opinion, the paper as a whole presents the common experiences of all project partners.
Kay Schützler, Baile Herculane, 28.08. - 03.09.0523 The paper was well prepared and presented, expressed in good English and easy to read, with sufficient reference to earlier activities and relevant publications. But how was the direction and management of this project influenced by the earlier work? Reviewer 3 comments Influence of related work To clarify, why our project could not be a simple repetition of the referenced projects, we added a paragraph: "Compared to similar approaches [2, 3, 5, 6] for collaborative preparation and usage of joint course materials, our project contains a couple of unique features: multi-lingual project partners, a strict style for lecture notes, and a strong strategy for project management with review reports, update manage- ment, style guides, etc."
Kay Schützler, Baile Herculane, 28.08. - 03.09.0524 I was disappointed in the lack of critical appraisal, or pedagogic discussion concerning the effectiveness of the transfer process. I do not think, if I were to embark on a similar multi-national col- laboration, that I would rush to reread the paper for advice and guidance. There is little here which challenges common sense, and no experiential commentary. Reviewer 3 comments Addressed the right issues? As mentioned in section "Conclusion and future work", more and more partners joined us over the years and will join us in the future. From our experience the questions raised by new partners are indeed the ones addressed in our paper. Thus, we do not find this comment appropriate. skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0525 The topic and information about how to run similar projects could be of genuine value for many teachers in HE across Europe, and may interest US academics too. What key information do the authors hope to relate to readers? What are the principal benefits to be gained from reading this paper? Reviewer 3 comments Key information of the paper We added a paragraph in section "Conclusions and future work" to explicitly state that our project could serve as an guiding example to similar settings: "Besides that, our project shows that the process of transfer- ring an existing course to other environments (even with dif- ferent languages) can be performed successfully, as long as the issues we presented here are addressed appropriately."
Kay Schützler, Baile Herculane, 28.08. - 03.09.0526 Reviewer 3 comments Students cooperation Are students at the partner institutions encouraged to communicate or even cooperate on case studies and assignments? "Up to now, students do not communicate or cooperate across universities."
Kay Schützler, Baile Herculane, 28.08. - 03.09.0527 Was any appraisal undertaken to gauge the differences between the effectiveness of the course material when used in the original insti- tution, and that achieved in the acceptor universities? Did student feedback get compared? Were graded assignments and exam ques- tions shared? How was success/failure judged? Reviewer 3 comments Effectiveness of course at partner sites We extended our paper with another paragraph: "Up to now, students do not communicate or cooperate across univer- sities. However, there is already some kind of indirect cross-university influence from the students. Their solutions to the assignments are compared throughout the whole project and they are evaluating their respective course using a project-wide feedback form. The compari- son of these feedback forms is subject to future work." skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0528 How was the course material translated from German to English? Who paid for this time consuming work? How were the results checked? How long did it take? Presumably it was an irksome task because of the later reference to automating the work. Reviewer 3 comments Translation to English We changed the first paragraph of section "Project management to organize collaborative work" to cover more details on the managerial background of the project: "Because our project received substantial funding from DAAD, yearly workshops and coordination meetings were possible. Partners from SEE were enabled to stay at the German university several times in order to work on project tasks like translating existing and preparing new materials."
Kay Schützler, Baile Herculane, 28.08. - 03.09.0529 As the courses were ACM compliant, would it have been possible to use an alternative course, already available in the English language? Reviewer 3 comments Mobility issues and material reuse We do not think that using other existing courses would have been appropriate, because the partners explicitly expressed their interest in the particular course available at Humboldt University. We changed the respective sentence in the first section to reflect this situation more. How much travel, face-to-face meetings for academics, or guest-lecturing was possible during this project? The change of the first paragraph of section "Project management to organize collaborative work" does apply to this question, too. skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0530 Reviewer 3 comments Misunderstandings "the aims changed over time" - how, why and in what manner? We think, that this issue has been addressed already in the directly fol- lowing sentences. We made it a bit clearer now, by putting a colon instead of a dot. "…over time, the targets … of the project have changed considerably: With the contributions from participating partners, new materials have been developed as a response to the feedback and suggestions of professors and their students. Due to the additional information the project had to face and overcome organizational challenges to coordinate the work." "special course material problems" - what were these? We assume that this results from a misunderstanding of our bad English. We changed it to "special problems concerning course materials" and again replaced a dot by a colon. skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0531 The paper appears to have been produced by the senior part- ner, with little evidence of reported experience from the junior, accepting institutions. Views more clearly from all the instituti- ons would have been welcome. Reviewer 3 comments Paper from one partner only? We do not consider this comment as being appropriate. We are not able to imagine what kind of evidence the reviewer is requesting. The paper has been produced together by the senior partner and two of the junior partners and contains experience of both sides.
Kay Schützler, Baile Herculane, 28.08. - 03.09.0532 Well chosen case studies are often the most useful material, "borrowed" by colleagues from different universities. Little was said on this subject. Were the two original scenarios specially drafted, or inherited from the original, parent course? Mention is made of the tutors' commercial experience being brought to bear locally for the benefit of the students. How can this resource be distributed successfully to other sites? Reviewer 3 comments Case studies In subsection "III. Case studies" we added a paragraph: "Presently, we are including two case studies reused from the original course, one from an administrative application and the second from a technical application. The project participants contributed additional well suited case studies. These case studies are not included in the course so far, because teaching materials for them still have to be prepared."
Kay Schützler, Baile Herculane, 28.08. - 03.09.0533 For the reviewer, the project URL, reference , did not work when accessed. An email was dispatched to the principal author. We fixed that problem. Instead of ".../index.html" it has to be ".../index.htm". Reviewer 3 comments Technical problems and recommended literature What text books were recommended for use by students? As mentioned above, we added subsection "V. Literature" to section "Course materials" to give more information on the literature recommendations in our course. skip
Kay Schützler, Baile Herculane, 28.08. - 03.09.0534 Was any project source file control system (CVS?) employed to maintain project consistency? This question is answered in the last paragraph of subsec- tion "IV. Update management" in section "Project manage- ment to organize collaborative work“: "Since most materials are in binary formats and are not changed concurrently there was no urge to apply a complete configuration management. However, previous versions are archived to be used as fallbacks in case of problems." Reviewer 3 comments Configuration management
Kay Schützler, Baile Herculane, 28.08. - 03.09.0535 Klaus Bothe - Thank you for making those changes. I am very satisfied with your responses to the reviewers and I am looking forward to seeing you and your presentation in Indy! Heidi ************************************************************* Heidi A. Diefes-Dux, Assistant Professor Purdue University Department of Engineering Education ************************************************************* Final note from Heidi Diefes-Dux (from the organizer‘s site):
Kay Schützler, Baile Herculane, 28.08. - 03.09.0536 Conclusions Our project is interesting and useful for third parties FIE reviewers are interested in quantifiable results Commonly used real-life case studies are challenging –Still a somewhat open question in our project Cooperation between students across countries is to be considered seriously Feedback from partner universities is highly recommended – especially critiques