Receiver Operator Characteristics What is it and where does it come from Statistical aspects Use of ROC

Early radar signals Is this an enemy plane? Signal Noise Ratio

The problems of decision Sound the alarm when the signal is very small –Advantages Plenty of time to get the fighters off the ground Reduce the number of bombers reaching the target –Disadvantages Lots of false alarms Waste of gasoline, wear and tear on fighter planes Exhaust fighter pilots Sound the alarm when the signal is unmistakable –Advantages No waste, no wear and tear, no exhaustion –Disadvantages More bombers get through, more bombs, more destructions

Solution to decisions Code Yellow –Signal suggests possible incoming bomber –Pilots get dressed, fighter planes get loaded with gasoline and ammunition Code Orange –Signal suggests incoming bomber likely –Fighter planes towed to runway, pilots goes to the planes Code Red –Signal is unmistakable – Fighter planes take off

Refinement to solutions Responses variable –Radar receivers varies in signal strength and noise level –Technicians operating the receiver interpret the signals differently –Each receiver and its operator must be characterised, so that their reports can produce a consistent response The Receiver Operator Characteristic (ROC) –The relationship between not missing an incoming bomber (Sensitivity) and false alarms (False Positives)

Receiver Operator Characteristics False alarms Sensitivity Useless operator Sensitivity and false alarm rate changes together Perfect operator 100% Sensitive 0% false alarms Most operators

Receiver Operator Characteristics False alarms Sensitivity Code Yellow Code Orange Code Red Increasing signal strength

False alarms Sensitivity Code Yellow Code Orange Code Red Increasing signal strength Receiver Operator Characteristics

ROC since the war The ROC was effective translating measurements into decisions A system of different level of alerts are common decision processes –Economy and company performance –Risk of fire, drought, natural disasters, emergencies –International diplomacy, risk of war Extensive developments in statistics and mathematics to enhance the method –Introduced into medical decision making in the 1960s –popularised by medical educators in the 1980s as a method of teaching decision making in medicine –Becoming a common method to evaluate the quality of predictions and tests since the 1990s

Receiver Operator Characteristics What is it and where does it come from Statistical aspects Use of ROC

Statistical ROC A measurement is normally distributed in two groups, those outcome negative and those outcome positive Using a cut off level to make a decision will create a number of TP, FN, FP, and TN. From these Sensitivity and Specificity is calculated If the cut off value changes –TP,FN,FP,TN changes –Sensitivity and Specificity changes The relationship between Sensitivity and Specificity over the range of the measurement defines the ROC

Statistical ROC

Receiver Operator Characteristics What is it and where does it come from Statistical aspects Use of ROC

Advantages of using ROC It defines the quality of a test or prediction using a measurement without specifying a cut off value for decision making Assuming Normal distribution –The mean and Standard Error can be estimated –The 95% CI can be estimated –Statistical significance can be determined –Whether one test is better than another can be determined

Use of the ROC 1 - Specificity Sensitivity Sensitivity > Specificity Cut off value for screening test Specificity > Sensitivity Cut off value for intervention decision