Presentation is loading. Please wait.

Presentation is loading. Please wait.

Thinking the unthinkable: USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-ASSISTED ASSESSMENTS Ian Harwood & Bill Warburton University of Southampton,

Similar presentations


Presentation on theme: "Thinking the unthinkable: USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-ASSISTED ASSESSMENTS Ian Harwood & Bill Warburton University of Southampton,"— Presentation transcript:

1 Thinking the unthinkable: USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-ASSISTED ASSESSMENTS Ian Harwood & Bill Warburton University of Southampton, UK.

2 Thinking the unthinkable… Introduction Overview of PRM and risk efficiency Methodology / Case background Identified risks Conclusions

3 Introduction Implementing CAA is a risky activity Limited coverage in CAA literature Aims of this paper: –Build links between CAA and PRM literature –Show how formal PRM can be beneficial

4 Viewing CAA introduction as a project Project has been defined as: 'a temporary endeavour undertaken to create a unique product or service' PMI (2000) A project is: ‘a step into the unknown, fraught with risk and uncertainty’ Lock (2000)

5 Increasing interest in Risk Now in the 'risk society' (Beck, 1992). Broad base of risk literature covering: –environmental issues (GM foods, nuclear waste) –data processing –security management (terrorism, insurance) –project management –health and safety (design criteria, disaster planning) –finance and economics (derivatives, risk/return)

6 What is ‘Risk’? ‘a choice rather than a fate’ Bernstein (1996) risk or uncertainty? –risk is present when future events occur with measurable probability –uncertainty is present when the likelihood of future events is indefinite or incalculable Knight (1921) ‘Risk, Uncertainty and Profit’

7 Thinking the unthinkable… Introduction Overview of PRM and risk efficiency Methodology / Case background Identified risks Conclusions

8 PRM interest groups Association for Project Management: Risk Management Specific Interest Group (http://www.eurolog.co.uk/apmrisksig/)http://www.eurolog.co.uk/apmrisksig/ Centre for Risk Research, School of Management, University of Southampton, UK (http://www.management.soton.ac.uk/risk/default.asp) The Project Management Institute: Risk Management Specific Interest Group (http://www.risksig.com/).

9 The Project Risk Analysis and Management process (Simon et al 1997)

10 Risk efficiency From theory of portfolio management in Economics (Markowitz, 1959) ‘Risk’ and ‘Return’ inextricably linked Now used in the project environment –CCTA ‘Project Risk Management’ document

11 Theoretical risk efficiency framework

12 Risk efficiency migration due to PRM

13 Thinking the unthinkable… Introduction Overview of PRM and risk efficiency Methodology / Case background Identified risks Conclusions

14 Overview of cases Background to Soton MLE strategy Roll out of Perception

15 Methodology Case study approach (Yin, 2003) Semi-structured interviews Cross-case analysis (Miles and Huberman, 1994) (retrospective) risk register generated

16 Thinking the unthinkable… Introduction Overview of PRM and risk efficiency Methodology / Case background Identified risks Conclusions

17 Identified risks- before & during delivery Table 2 Wrong assessment published (C3,5) Authors ‘reinvent the wheel’ (C5) Publication date missed (C1) Communication gaps (C1,3) Environmental issues (C4) Cheating (C1,2,3,5,6) Failure at point of delivery- scalability issues (C3,4) Tests submitted prematurely (C2)

18 Identified risks- during & after delivery Table 2 Informal appeals- protest about fairness of CAA (C1,5) Formal appeals: -workstation issues (C1,2,6) -unfamiliar test environment (C1) -test construction (C1) -negative marking (C6) -QA issues (C5) CAA abandoned: -changes in course architecture (C3) -staff turnover (C6) -weak implementation (C5) -perceived unfriendliness (C5)

19 ‘Proactive’ responses: mitigating actions 1 Table 2 Pre-emptive Load-test at realistic levels Allow sufficient time to fix the problem Run a formative version of the test Keep channels of communication open Help staff to comply with internal CAA procedures Ease of use- capture & address author feedback Keep channels open between authors Comply with internal CAA procedures Use prefabricated templates

20 ‘Proactive’ responses: mitigating actions 2 Table 2 Make training a prerequisite for access to the service. Help staff to comply with internal CAA procedures Use random selections from large question pools Allocate sufficient time to test workstations before exam Use a class register to book sufficient workstations Practice with formative versions Maintain tutorials & practice tests on public workstations Help tutors choose appropriate assessment methods

21 ‘Reactive’ responses: Contingency Plans Table 2 Have a paper copy Substitute marks from formative assessments Capture and address author feedback Discount compromised items Re-issue test Discount compromised items Use trained invigilators Have a spare workstation area available Monitor and address student feedback

22 Thinking the unthinkable… Introduction Overview of PRM and risk efficiency Methodology / Case background Identified risks Conclusions

23 Benefits of formal PRM Enables better informed (and more believable) plans, schedules and budgets Increases the likelihood of a project going to plan Allows more meaningful assessment of contingencies Contributes to the build-up of statistical information to assist in better management of future projects Enables a more objective comparison of alternatives Identifies, and allocates responsibility to, the best risk owner

24 Benefits of formal PRM cont… Improves general communication Leads to a common understanding and improved team spirit Assists in the distinction between good luck / good management and bad luck / bad management Helps develop the ability of staff to assess risks Focuses attention on the real and most important issues Facilitates greater risk-taking, thus increasing the benefits gained Demonstrates a responsible approach to customers

25 Risk Management Maturity (Hillson, 1997) Four levels: 1) naïve 2) novice 3) normalised 4) natural Assists in identifying areas for improvement in risk management

26 Decay of risk exposure through iterative PRM

27 Closing thoughts… Six 'first time' applications of Perception replace paper-based exams- riskier than future repeat applications –iterative approaches to formalised project risk management can build institutional corpi of CAA risk knowledge –trend of total risk exposure progresses from risk inefficient to more risk efficient –a full-scale system could be released that already catered for the greatest risks –institution becomes more risk mature (Hillson, 1997)

28 Any questions…?


Download ppt "Thinking the unthinkable: USING PROJECT RISK MANAGEMENT WHEN INTRODUCING COMPUTER-ASSISTED ASSESSMENTS Ian Harwood & Bill Warburton University of Southampton,"

Similar presentations


Ads by Google