1. Experiences on Course Development Practices in a Power System Analysis Master Course a.k.a. tales of a journey through the valley of death in providing a high quality learning environment
Assoc. Prof. Dr. Ing. Luigi Vanfretti and Maxime Baudette
KTH SmarTS Lab
KTH Scholarship of Teaching and Learning Stockholm, March 12th 2015.

2. Acknowledgment
There has been a lot of former students involved in this efforts, through teaching assistant duties and beyond, which we can’t list here – but you know who you are, and we thank you very much! A very special thanks is expressed to Mostafa Farrokhabadi, who was the first student helping with the research work related to this talk until Maxime Baudette got involved.

3. Outline Motivation Constructive Alignment Theory
Implementation: T&L activities, Assessment Activities
Evaluation of the implementation: Feedback, R-SPQ-2F, Ranking Algorithm
Filtering Feedback: Repertory Grid
Introducing Peer Instructions with Clickers

4. EG2100 in Brief Introductory course on Power System Analysis
6 ECTS (7.5 ECTS before 2014)
Given on 2 periods (September to December)
About 85 students
The course is divided in five important topics:
Modern Power Systems with an Introduction to Sustainable Energy Technologies and Smart Grids
Fundamental Principles for Power System Analysis AC circuits
Electrical Modeling of Generators, Transmission Apparatus and Networks
Methods for Analysis and Design of Power Networks in Steady State and Unbalanced Operation
Steady State Stability Analysis using the Power-Flow Formulation
Methods for Analysis of Power Distribution Systems in Steady State

5. “I never teach my pupils, I only attempt to provide the conditions in which they can learn.” A. Einstein
Motivation
The previous course (until 2010) that was replaced by this one had several drawbacks in its pedagogic design and material:
The course material was outdated, inconsistent and lacked appeal.
No teaching method was used to encourage the learning process
 The failure rate of the course was very high
These drawbacks required a complete overhaul of the course: from preparing new material, to the design of a new implementation using teaching methods.

6. How to address a large variety of background?
Students’ Background
Students come with a very different background, and often lack any exposure to power systems at all:
Mandatory for M.Sc. Program in Electrical Power Engineering (TELPM), and M.Sc. Joint Program in Smart Electrical Networks and Systems (SENSE)
Elective in other M.Sc. Programs at the school of Electrical Engineering and some Erasmus Mundus M.Sc. Programs
Also offered to power and energy professionals within industry collaborations and PhD students (including those from other universities)
This variety in the students’ background makes it challenging to design well aligned and adequate T&L activities and proper assessment methods.

7. Redesign of the Course - Goals
In 2011 it was decided re-design the course to achieve the following goals:
Increase the success rate
Largely increase the depth and moderately the coverage of the course content
Link the course to the current developments in the industry and academia, including the “Smart grid” concept
Re-Design Approach:
Implement the constructive alignment theory (CAT) as the didactic teaching approach of choice for the course
Adopt the consensus-oriented decision-making (CODM) model to design the course
Develop effective multiple feedback channels to gather the students’ perception and select relevant feedback

8. Constructive Alignment – Basic Principles (Biggs and Tang)
The theacher’s fundamental task is to get students to engage in learning activities that are likely to result in their achieving the desired outcomes in a reasonably effective manner.
What the student does is more important in determining what is learned than what the teacher does.
Constructive Alignment Theory:
Provides a set of principles
These principles can be used to devising Teaching and Learning Activities (T&Ls)
The activities can help in achieving the Intended Learning Outcomes (ILOs).

9. Using Principles of Constructive Alignment
Course
Objectives
(Intended Learning Outcomes)
Teaching and Learning Activities
Assessment
What should the student be
able to do
as a result of the course?
Using the principles from constructive alignment:
Start by carefully aligning T&LAs and Assessments
These activities should to support the students to fulfill the ILOs.
The students role (MAJOR):
The students use the activities to construct their knowledge and achieve desired outcomes.
The teacher’s role (minor):
To design a learning environment that encourages the student to perform the T&Ls that aid the students to construct their knowledge.
What should the students do to demonstrate that they reached the objectives?
What work is appropriate
for students to do
to reach the ILOs?

10. Variety of Background – Affects the proper implementation of CAT
Question
% of correct answers 1 99 2 40 3 90 4 70 5 53 6 84 7 59 8 62 9 69 10 27 11 32 12 13 14 15 77 16 17 67 18 63 19 20 74 21 44 22 68 23 51
Total
64.8
Variety of Background – Affects the proper implementation of CAT
Affects student’s readiness to engage in the Teaching and Learning Activities Diagnostic test results show a low average in the students preparation Fundamental knowledge varies a lot.

11. CODM – Design Process (Hartnett)
Links the design group to the consensus model.
Five 2-hour sessions (10 hrs)
Allows to deliver a shared proposal:
Important if the design group will also be the implementation group (needed commitment).
Closing
Framing the Problem
Open Discussion
Identifying Underlying Concerns
Developing Proposals
Developing a Preferred Solution
Choosing a Direction

12. Design Rule for CAT Implementation
Example of using the consensus based model Step 1 (Framing the problem): design rule used for one of the T&L activities using Cause-and-Effect analysis.

13. T&L Activities Course Objectives
(Intended Learning Outcomes)
T&L Activities
1: Lectures (22) , Invited Lecture (4) 2: Reading Quiz, Conceptual Quiz 3: Homeworks (3) 4: Tests (2) 5: Final Exam
[1], [2]
[3],[4]
[5]
Teaching and Learning Activities
Assessment

14. T&L Activities - Lectures
Lectures are intended to develop conceptual understanding:

15. T&L Activities- Invited Lectures
Invited Lectures are intended to develop the interest of students by providing insights from the industry and top researchers from Academia.
Sample of Invited Lectures
Magnus Danielsson. System Design, R&D, Net Insight AB, Stockholm, Sweden.
Lecture on Communication Issues in Smart Grid Applications
Svein Harald Olsen, Statnett SF, Oslo, Norway.
2 Lectures on IEC CIM
Sonja Berlijn, SVP R&D, Statnett S, Oslo, Norway.
Lectures active R&D projects (Lean Line + Voltage upgrading)
David Petesch/Yannick Fillon, RTE - National Center for Grid Expertise, Paris La Défense – France
Lecture on EMTP and Real-Time Simulation at RTE
Prof. Federico Milano, UCD
Challenges for power system modelling and simulation

16. T&L Activities- Daily Reading Quiz, Conceptual Quiz
In total, they are worth 10 BONUS points of the final grade.
Reading Quiz consists of some very basic questions about the content of the lecture
The questions are designed in a way that a student should go through the lecture content before arriving to the lecture!
Peer Instruction Implementation
Conceptual Quizzes asked about the lecture content during the lecture.
The questions addressed different learning levels, focusing on developing deep learning. They sometimes involved a small amount of numerical calculations.
The students answered the questions using a remote clicker.

17. T&L Activities – Homeworks
For the students to practice the methods and study the concepts covered during the lectures. There are 3 Homeworks (30 points) covering the contents learned in the respective lectures. The homework are aimed to prepare the students for the assessment tasks (tests and final exam). In 2014 computer assisted assignments were introduced.
Classroom
Home!

18. Assessment – Tests and Final Exam
2 Tests during the semester .
Final exam
The tests and exam were built in two parts, with theory questions in the first part and problems in the second part. The problems were similar to those solved in the homework.
The assessment of the course is done though the different graded activities of the course.
All the grades are added to give a total grade on 100 points
Total Points
Grade Letter
94 – 100 A
87 – 93 B
81 – 87 C
74 – 80 D
65 – 73 E
60 – 64 Fx
00 – 59 F
Course Activity
Points
Daily Quiz
10 (Bonus)
Homeworks 30
Tests
Final Exam 40
Total
100 (+10 Bonus points)

19. Evaluation of the implementation
Three Course Evaluations
One after each test (end of P1 and end of P2)
One after the final exam
R-SPQ-2F Evaluation (at the beginning of the course)
Two different question types (surface and deep approach)
Students are graded using their answers
The results will reveal the students approach
Interviews using the Repertory Grid Technique
A form of structured interview to find out a participant’s preference on a given topic and the way these preferences are ordered on a rating scale
Allows to link qualitative to quantitative feedback.

20. Explanation on KTH Social
Quantitative Data
Instructions on Form
Qualitative Data

21. Sample Feedback using Google Forms
Quantitative Feedback Qualitative Feedback

22. Course Evaluations
After 2011, the form was improved for including an introductory part with instructions, explaining the students the grading scale they should use.

23. Course Evaluations

24. Course Evaluations - YOY improvements
The feedback provided by students that retook the course shows that the changes improved the course.
The acceptance of the changes comes after the students realize that it has a positive impact on their learning.
Note: very reduced population for statistical analysis
Sample Question
Is the new course structure (lectures, daily exercises, weekly exercises, and weekly tests) preferable to the previous course structure (lecture, assignments, and final examination)?
YOY = year over year

25. Activity No. 1 – Design of a course evaluation process
Instructions: Discuss in small groups (no larger than 3) the following questions. Using your answers, propose a design to evaluate a “sample” course.
What feedback do you need for your particular course?
T&L activities: do they contribute to the student’s understanding? Do the T&Ls have an impact on the Assessment Activities?
In what form do you want it and how much?
Quantitative, Qualitative, Both
How do you gather it in an efficient way?
What tools do you use.
How often do you gather it?
Number times and type of feedback to be gathered – (a) that allows to trace changes, (b) of new information (originating from the changes).
How do you use it?
How do you show the students that their opinion is considered.

26. The Revised two-factor Study Process Questionnaire: R-SPQ-2F
Allows to determine the students’ learning approach The process is exposed to some drawbacks!

27. R-SPQ-2F: A new ranking algorithm
Surface Learning
Deep Learning
1st Quarter
3rd Quarter
2nd Quarter
4th Quarter

28. Grade distribution and its correlation with students’ learning approach

29. Filtering Feedback: Repertory Grids
The repertory grid method is a “person-centered” approach that allows the student to develop their own constructs using their own perception of the subject. Repertory grids serve as a feedback mechanism where the students can assess course elements using well-defined constructs and considering different scales. The construction of each grid allows a methodological analysis of the grids to identify constructive feedback to improve the course: a formal quantitative method to select and discard feedback.
Good Feedback
Further Analysis
Bad Feedback
Discard
Good Grid
Bad Grid

30. Implementation of repertory grids
The elements of the grid were determined by the T&L activities.
The constructs were built from interviews with some students
A small group of students (about 10%) was carefully selected to represent the different learning approaches
Each student was interviewed individually about the different elements (qualitative data collection)
The constructs were identified by analyzing all interviews
An empty grid was sent to each student who should fill it in

31. Repertory grid analysis
Different software provide methods for analysis of the grids (Rep IV, Idiogrid), such as:
Principal Component Analysis (PCA): allows better interpretation of feedback.
Cluster Analysis: re-orders elements in a tree, similar objects are placed together. Allows to interpret the feedback.
Descriptive analysis (Mean, Mode)
PCA:
In Fig. 2, the position of the elements and constructs is a good
representation of their correlation. For example, this student believes
that the elements “daily exercise” and “class test solution”
are similar in the case that both are stressful and should be removed
from the course design. Note that these analyses add no
new data to the repertory grid, but simply help in the interpretation
of the grids rated by the students.
Cluster Analysis:
For example, it can be observed
that in the first level, there are twin branches of “course textbook—
lecture slides,” “Lennart’s lectures—Luigi’s lectures,” and “binder—daily exercises.”
This is quite logical as these elements
share common characteristics. Cluster analysis reveals
those elements and constructs that are in tight correlation; this
will be valuable in modifying the course. For example, this
student believes that constructs of “time consuming—stressful”
are in the same group. Other feedback or PCA analysis can be
scrutinized to identify those elements that this student sees as
stressful. Then, a decision can be made as to whether to modify
the timing of those elements.

32. Activity No. 2 – Feedback Analysis
Instructions:
Assume that you have a high volume course (many participants). In this activity, you should analyze the method for feedback analysis presented, and propose alternatives to relate the students to their learning approach and classify which feedback to address in a systematic (reproducible) way.
Are there other alternatives (methods) to analyze the course feedback and relate it to the student’s learning approach? How do you check if your implementation of CAT is working?
How can you use the proposed method to filter the different feedback and choose which to address?

33. Peer Instruction

34. Peer Instruction and the use of “Clickers” Reading Quiz and Conceptual Quiz:
To implement peer instruction there is very good technology available
These are called clickers and are used for gathering students feedback in real-time.
Clickers were given to each student and kept by them throughout the course.

35. A flexible tool for different uses!
In the EG2100 course, clickers are used for the following:
Unique identifier for each student
Attendance registration during Lectures
Pre-requisite test at the beginning of the course (not graded)
In-class questions:
Tests on the mandatory pre-readings (5 questions)
Questions during the lecture (~5 questions)
Rewarded by bonus points

36. In-class interactions
Tests on the mandatory pre-readings (5 questions)
Questions about the content of the pre-readings
Regular questions, all done in a row
Questions during the lecture (~5 questions)
Questions about the subject being taught
Regular questions, often one by one
If “average results” are obtained, students are invited to discuss with their neighbors and answer the question once again (peer instruction)
If “bad results” were obtained, the lecturer re-explained the material.

37. Preparation of Teaching Materials for Peer Instruction
Material has to be prepared to introduce the peer instruction as a change of activity.
The lectures need to be prepared in min modules
Questions need to be prepared to assess deep learning (or surface when needed)
Instead of a review at the end of the lecture, peer instruction allows for review of deep learning concepts during the lecture.
This has a cost: Your time! – Think if you can afford it before doing it!

38. Pros/Cons Pros: Greatly appreciated by the students:
Higher attendance and better preparation of the students
Better in-class attention
Helps the teacher to get direct feedback (e.g. unclear notions)
Flexible !
Cons:
Additional Data Management
Technical issues may happen in class (crash, bad reception, empty battery, etc.)
Different lecturing style (might be hard to accept by students in early phases)

39. Student’s feedback about the clickers
The feedback received is very positive and encouraging for continuing this peer-instruction implementation
The mandatory pre-readings helped your understanding of the course material.
It was helpful to discuss your answers with your classmates (peer instruction)
The daily quizzes helped you keeping up to date with the course materials.
It was fun to use the clickers. It helped you be more attentive in class

40. Activity No. 3 – Peer Instruction Method Using Clickers
Instructions: Assume that you have already your teaching prepared from previous course deliveries. In this activity, you should analyze how to modify the course materials to adapt it to the peer instruction method using clickers. Discuss the following questions, provide a proposal on how to implement peer instruction and an example slide for your particular course?
At what learning depth level do you intend to use peer instruction?
When using the peer instruction method, what factors do you need to consider so that you can use clickers to obtain quick feedback of the student’s understanding of a particular topic?
How do you transform (timing, amount of material and content) your course material to adapt it to the peer instruction method?

41. Highlights and Words of Caution
A wider variety of stakeholders can participate in CODM process – this would allow for a very good course design addressing different points of view!
CAT is a good vehicle to enhance the students’ learning. It’s implementation has to be considered carefully. Implementing CAT is extremely expensive – DO NOT BELIEVE the propaganda that says the contrary!
R-SPQ-2F new ranking algorithm efficiently classifies the students’ learning approach depth. Useful for sampling feedback from students with different learning approaches.
The two main challenges for the course designers are:
To continuously modify the course so the surface approach students move toward deep approach.
To do carry out (1) when there is absolutely no benefit to the teaching staff (a lot of work that brings no reward)

42. Discussion (1/2) – Continuity and Effort
The teaching staff changes from year to year depending on the availability of PhD students (Luigi is always involved and the only one to provide continuity)
Course planning based on consensus requires continuity, in order to focus on improvements instead of challenging the whole structure
Effort: Workload and Time
The T&L activities require a large workload for the teaching staff to prepare and correct the assignments (very expensive and not properly funded)
Course planning based on consensus requires time and involvement from the teaching staff
The feedback process requires a lot of time for gathering and analysis of the results
Integration of clickers in the lectures requires a lot of time (very expensive!) in order to provide a good experience for the students (questions should be well organized and well formulated)

43. Discussion (2/2)
Systemic factors: brace yourself to a large inertia and resistance to change
Students and other faculty will oppose different didactic approaches to those that they are used too – you will face a large resistance to change.
Not having a “single exam” to be able to pass the course has been criticized and brought up to the program coordinator year after year…
This attitude is hard to change in the 4th year. However, we have shown that the acceptance of the students improves over time when they are able to reap the benefits of a good learning environment.
You will waste your time with complaints from the student union, program coordinators, department head… just for trying to do things differently (I would say BETTER!)
There is no means to enforce student feedback at KTH, this weakens statistical analysis and the amount of samples.
Changes in rules or flexibility is needed if KTH wants to take education seriously.
No pain, No gain? More like lot’s of pain, and no gain!
The one most important thing we have learned is that there is no reward for good teaching in KTH – do it at your own risk and for your own self development.
The only benefit you could obtain, by investing even more time, is to try to generate some publications for your teaching portfolio (in case you want to find somewhere else to teach with quality).

44. Epilogue – A tale just like that of the one ring
Even though this has been a wonderful adventure in teaching and learning, all good things must come to an end.
Luigi can’t afford to continue to subsidize the delivery of the course by using time from his PhD students (which translates to use the funds from research projects!) to deliver the course.
A loss of SEK was generated in 2014.
A good learning environment comes at a cost: we do not get paid enough to deliver a high quality learning environment.
Luigi will not longer be the responsible for this course nor involved in it, and with that goes the 4 years of experience…
But at least we enjoyed the walk through the valley of death, and we have learned a lot about how to deliver a high quality learning environment!

45. Thank you!
“It's really hard to design products by focus groups. A lot of times, people don't know what they want until you show it to them.” — Steve Jobs Questions? Good luck with the rest of your day!

46. References
[1] L. Vanfretti and M. Farrokhabadi, “Consensus-Based Course Design and Implementation of Constructive Alignment Theory in a Power System Analysis Course”, European Journal of Engineering Education, [2] L. Vanfretti and M. Farrokhabadi, “Evaluating Constructive Alignment Theory Implementation in a Power System Analysis Course through Repertory Grids”, IEEE Transactions on Education, vol. 56, no. 4, pp. 443 – 452, Nov DOI: [3] L. Vanfretti and Federico Milano, “Facilitating Constructive Alignment in Power System Engineering Education using Free and Open Source Software,” IEEE Transactions on Education, vol. 55, no.3, August 2012, pp DOI: [4] S. Taylor, “Managing postgraduate research degrees”. In H. Fry, S. Ketteridge and S. Marshall (eds) The Effective Academic: A Handbook for Enhanced Academic Practice. London: Kogan Page. [5] S. Taylor and N. Beasley. A handbook for doctoral supervisors. Routledge, New York, 2005.

47. Course Objectives (Intended Learning Outcomes)
After completing the course, the students should be able to:
 Using first principles derive the basic concepts and methods used for power system analysis. 
Describe the methods used for modeling and simulation of different power system phenomena, and
Explain how computer methods affect the modeling process
Construct mathematical models for computing the steady state performance, and basic unbalanced performance of power systems.
Describe the use of the Unified Markup Language for data modeling of power system components, and interpret the Common Information Model and its relation with the power system models used in class.
Describe, derive, and compare different models of the most common equipment used in power network models.
Using different methods, to compute, analyze, and reflect on the performance of a power system under steady state and unbalanced operation, for transmission and distribution networks
Apply a power system and general purpose software tools to analyze the performance of a power system, and evaluate it for different operating conditions
Contrast different software development methods that are used in the creation of power system analysis software
Using first principles derive the basic concepts and methods used for power system analysis.
To construct data models and mathematical models for computing the steady state performance, and basic unbalanced performance of power systems.
To derive, describe and compare different models of the most common equipment used in power network models.
Using different methods, to compute, analyze, and reflect on the performance of a power system under steady state and unbalanced operation
To describe basic characteristics of renewable and distributed energy sources, as well as monitoring, analysis and control technologies used in Smart Grids.
Keep it in case of questions

48. Evaluation Diagnostic test at the beginning of the course (non graded)
Raise student awareness about their individual level of required background
In-class questions are rewarded with a very flexible bonus point system
Technical issues require to have a flexible system
Any evaluation activity requires to maintain a database of students