1. Quality and Outcomes Framework Assessor Training
Collecting and Analysing Data
Module S4

2. DATA INFORMATION KNOWLEDGE WISDOM
This is an old adage familiar to most IT people.
Practice with the keys to time the entry of each arrow and word into the slide.
If we do something with data, we can turn it into information. If we are really clever with information, it can be converted into knowledge. If we are really imaginative and intelligent we can then transform knowledge into wisdom. However, the whole process collapses if the quality of the data is poor.
WISDOM

3. QOF Review Data Collection
75% is automatically collected by QMAS
25% requires practice and PCT co-operation
Written Evidence
Observation/Interview of Practice Staff
Inspection of Equipment/Materials
Inspection of Clinical Records
Most of the data for the QOF review (Quality and Outcome Framework) will be automatically collected every month via QMAS (Quality Management Analysis System). There will be a web-based facility so practices and PCTs can view how the practices are doing. 772/1050 points will be collected in this way.
However, part of the data for the QOF review has to be collected at practice level.
Some requires the practice to provide written evidence to the PCT. This includes clinical or managerial protocols or minutes of meetings.
Some requires the PCT to observe or interview practice staff. This includes phoning the practice out-of-hours to listen to how the practice informs patients who to contact in an emergency, observing how receptionists arrange for patients to get a telephone consultation and asking the practice nurses when they last had an appraisal.
Equipment such as sterilising equipment or sphygmomonitors and materials such as drugs in the emergency bag need to be checked.
Some indicators require the PCT to inspect clinical records to ensure they are in the correct order and that the medication is linked to a disorder and so on.
The details for each indicator are to be found in the ‘blue book’ NEW GMS CONTRACT 2003, Supplementary Documents, which is essential reading for all QOF Assessors. Practices that don’t use electronic records or whose systems are incompatible with QMAS, require a manual data input outlined in the ‘blue book’.

4. Written Evidence
The practice supports smokers in stopping smoking by a strategy which includes providing literature and offering appropriate therapy. (Information 5)
The practice has undertaken a minimum of six significant event reviews in the past 3 years. (Education 2)
Person specifications and job descriptions are produced for all advertised vacancies. (Management 6)
These are examples of indicators that require written evidence from the practices. A complete check list is included in the manual. The ‘blue book’ details when these documents need to be available. Some need to be submitted at least a month in advance (grade A), some need to be available on the day of the visit (grade B) and others only need to be provided in the event of an appeal (grade C).
A forward thinking PCT will bring together the practice managers so they can share good practice and agree with the PCT what kind of documentation is acceptable.
The first example is grade A and needs submitting in advance. The PCT should have a strategy for stopping smoking and the health promotion team could provide the same material to each of the practices.
The second example is also grade A, requiring the practices to submit their minutes and action points from significant event reviews.
The third is grade B and needs to be available on the day of the visit. The PCT could help the practices produce generic person specifications and job descriptions for the different roles within a practice, which can then be adjusted to each practice’s needs when a vacancy occurs.

5. Observation/Interview of Practice Staff
The practice has a system for transferring and acting on information about patients seen by other doctors out of hours. (Records 3)
The practice has a system to allow patients to contact the out-of-hours service by making no more than two telephone calls. (Information 1)
All practice-employed nurses have an annual appraisal. (Education 3)
There are clearly defined arrangements for backing up computer data, back-up verification, safe storage of back-up tapes and authorisation for loading programmes where a computer is used. (Management 2)
Again, these are examples of indicators that have to be verified by the PCT making observations at each practice or interviewing practice staff. A complete check list is in the manual and further details are in the ‘blue book’
The majority will need to be carried out during working hours and the PCT will have to ask receptionist or other staff to demonstrate how they carry out various processes.
In the first example, the commonest arrangement is that out-of-hours services fax the practice whenever there is a contact. The PCT will need to witness how the fax is processed and who by. The question that needs to be asked is “Show me how…?”
The PCT will need to telephone the practice out-of-hours to determine if the practice fulfills the relevant quality indicators concerning the telephone message in order to contact out-of-hours help.
The PCT needs to ask direct questions of some practice staff, in this case “When was your last appraisal?” This indicator also needs grade C written evidence, which needs to available only in the event of a dispute.
The last example requires grade A written evidence that is submitted in advance but is a process that the PCT may want to observe.
The PCT staff who carry out this part of the data collection need to be calm and friendly with good interpersonal skills. It is preferable if they are already known to the practices.

6. Inspection of Equipment or Materials
The practice possesses the equipment and in-date emergency drugs to treat anaphylaxis. (Medicines 2)
The arrangements for instrument sterilisation comply with national guidelines as applicable to Primary Care. (Management 4)
The practice has systems in place to ensure regular and appropriate inspection, calibration, maintenance and replacement of equipment including: 
- A defined responsible person
- Clear recording
- Systematic pre-planned schedules
- Reporting of faults (Management 7)    
       
This check list needs to be verified by the PCT with a member of the practice staff. The best placed person is the practice manager. Again, a full list is in the manual and further details are in the ‘blue book’. They can all be carried out without interfering with the routine practice activity.
The PCT will already have staff capable of completing this task, especially those who work in health and safety or infection control.

7. Inspection of Clinical Records
Each patient contact with a clinician is recorded in the patient’s record, including consultations, visits and telephone advice. (Records 1)
The records, hospital letters and investigation reports are filed in date order or available electronically in date order. (Records 14)
The percentage of patients with cancer diagnosed from 1 April 2003 with a review by the practice recorded within six months of confirmed diagnosis. This should include an assessment of support needs, if any, and a review of coordination arrangements with Secondary Care. (Cancer 2)
Some indicators require inspection of the clinical records. The practices are most commonly required to audit fifty case notes and submit a written report at least a month in advance. The full check list is in the manual and further details are in the ‘blue book’.
For most of these indicators, The PCT is required to check the practice’s report by going through twenty randomly selected clinical records. If possible, the notes should be anaonymised. Instead of inspecting a different set of twenty records for each indicator, the PCT can inspect the same twenty records for a number of indicators. This has to be carried out by someone who is familiar with the practice’s software system and is bound by the Code of Practice on confidentiality. It is likely that clinical audits or medicines management exercises have been carried out by PCT staff who will have the trust of the practices. Any additional assessors have to be competent to do the work and should be agreed by the practices.
The first two examples in this slide are such indicators.
Other indicators are issue specific.
There are two issues concerning the clinical domains, one of which is the third example here. Although the QMAS system will identify a specific READ code for analysis, the practice needs to understand that the code should only be entered into the patient’s record under specific circumstances. In this case a cancer patient has to have their support needs assessed and a review of co-ordination of arrangements with secondary before the code is entered. These two elements need to be readily identified in the patients notes. In this case, the assessors will have to look through cancer patients case notes only.

8. Additional Data Prescribing (PPA) Hospital Activity (HES)
Patient Survey
Mortality Data (ONS)
All these streams of data are already being collected by the PCT. The medicines management team will have access to the Prescribing Pricing Authority data using the same practice codes as used in QMAS. The Hospital Episodes Statistics data relates all inpatient and outpatient activity to practices using the same code. The national patient survey and local ones will also relate their results to practices using the same code. The Office of National Statistics produces deaths data that can be related to practices but small numbers can make comparisons between practices impossible.
All of this data is available and being looked at by different PCT staff now and it will add colour and flavour to the QOF data, if similar histograms as included in the QMAS are drawn in PowerPoint with the practices along the x-axis in the same order. The QOF lead should agree with the practices what information can be shared.
This will enable PCTs and practices to ‘toggle’ between graphs and easily see how each practice is progressing in providing quality services for its population.
The manual provides list and details of the searches that can be carried scrutinising these systems that are useful in the context of QOF assessment.
All these data streams are currently available to PCTs and are used regularly by their staff

9. Interim Aspiration Scores Max 1050 points
In QMAS, all of the indicators within all of the domains will have graphs like this. PCTs and practices will be able to see how they are currently scoring against their aspiration scores. The practices will be listed by a practice identifying number.
This slide uses ‘real’ data from all the practices in one PCT. Aspiration scores is all that’s available now but by August, QMAS will be up and running. Once achievement starts to come in, it will be displayed as extra bars each month for each practice.
BEWARE SMALL NUMBERS IN SMALL PRACTICES AS ONLY A FEW PAIENTS CAN MAKE A BIG DIFFERENCE TO THESE GRAPHS.

10. CHD Max 121 points
In the CHD domain, keep your eye on practice 5 and practice 17, both of which are scoring low but are not the lowest.

11. Cholesterol <5 mmol/l Target 60% - 16 points
This target is quite difficult to achieve but practice 17 exceeds the target. In the context of an overall low score that doesn’t seem to fit.
‘Toggle’ to the next graph

12. Lipid-Lowering Drugs Items per STAR-PU
However, it is one the higher prescribers of lipid-lowering drugs.
‘Toggle’ to the previous graph and back.

13. CHD on Anti-Platelet Target 90% - 7 points
Here, practice 17 is equal lowest, some distance from target.
‘Toggle’ to next slide

14. Anti-Platelet Drugs Items per STAR-PU
Yet, it’s the highest prescriber.
‘Toggle’ with previous slide and back.
So, what’s going on? Maybe that practice is seeing everyone and making sure they prescribe aspirin but are forgetting to code for ‘takes aspirin over the counter’ or allergies and adverse reactions.

15. Acute MI Admissions per 1,000 Registered
This view is supported by the fact that practice 17 has a low admission rate for MIs

16. Angina Admissions per 1,000 Registered
AND angina. However, something else is striking about this slide and that’s practice 5.

17. PTCA or CABG per 1,000 Registered
A practice is regarded as doing well to get patients through to this end of the market, as these are operative, ‘curative’ procedures delivered by tertiary care, angioplasties and coronary artery by-pass grafts. Practice 17 gets a good number through, so this supports the view that they are clinically well-organised but a bit weak on coding onto their medical records. Practice 5 is low, suggesting a different problem. Perhaps their register isn’t accurate, their call and recall system inefficient. Maybe their practice nurse is on long term sick.
When it comes to the actual visit, the issues to be raised are different but, in each case, the PCT can be positive and supportive and offer help as part of the action plan.

18. Retinal Screening Target 90% - 5 points
QOF assessment is not just about focusing on individual practices. Some issues are to do with the whole service. This indicator within the Diabetes domain shows that only two practices expect to achieve the target. That is because there is a shortage of retinal cameras in this PCT. The solution is not within the gift of a single practice but requires the PCT to do something across the patch.
Other areas that are likely to be cross practice issues are the diagnostic test; exercise testing for angina, echocardiograms for heart failure, brain scans for stroke and spirometry for COPD

19. COPD Max 45 points
COPD is an interesting area where the additional data streams are really helpful.
Practice 15 looks to be the best and practice 7, which looks OK and practice 1 looks as though it needs some help.

20. Oxygen Prescribing Items per 1,000 Registered
Oxygen is most frequently prescribed for severe COPD cases. Practice seven has the most patients on oxygen, so they probably have the highest number with severe COPD.
Practice 15 is a very small practice with a young population, so its low level of oxygen prescribing fits with a low number with severe disease, in fact one patient only.
Practice 1 is close to average.

21. J40-J44 - COPD Non-Elective Last FCEs XYZ PCT discharge dates between January and December (Number of Discharges per 1000 List Size)
But look at this. These two practices dominate the emergency admissions. Practice 7’s admissions fits with its high number with severe disease but practice 15 should be the lowest for admissions.
‘Toggle’ with the previous graph and the one before that and back.
It turns out that practice 15’s one patient with severe disease was admitted several times.
By contrast, practice 7 has a low number of admissions.
Including these other data streams, demonstrates that some practices are doing better than the QOF scores suggest and others are doing less well.

22. Organisational Indicators Max 184 points
The organisation indicators are important because without them practices can’t deliver the clinical domains. This brings us back to practices 5 and 17

23. Records & Information Max 85 points
Within the organisational domain, it is the records and information section where they fall down. PCTs often provide IT training for everyone and the ones who turn up are the ones who least need it. Doctors, especially, fear making fools of themselves in front of their staff. Here the PCT can offer one-to-one training for the GPs in practices 5, 10 and 17. The PCT can offer practice based training for the rest of the team at these three practices as well. One-to-one training takes a lot of manpower and it would not be possible to offer it to everyone but three practices, amounting to eight GPs is feasible.
This slide and the issue of COPD demonstrate how a PCT can best target its resources to make the biggest difference.

24. Access Bonus 50 points
Access is Booleean score. This means you either get the whole score or you get none. Even if you achieve 99% of the target, you receive zero points. Here, three practices do not think they are going to achieve the access target.

25. Patient Survey Routine Appointment Same Day Or Next
The PCT has information from the patient survey which is linked to each practice. Here, the question was, “when you last made a routine appointment with the doctor, when could you be seen?” This charts the % of respondents who said the same day or the next.
‘Toggle’ between this slide and the previous one. As you can see the three lowest practices on this slide do not correspond with the three practices that do not think they will achieve the target.
The PCT will have different approaches on the practice visit. With three, the question is why do they think they are rubbish when they are not so bad. With the other three, it is why do they think they are good when they are not according to the patient survey.

26. Some Words Of Wisdom Local knowledge essential
Look at the whole picture
Beware small numbers in small practices
Offer help not criticism
Ongoing conversation between practices and PCT
The reality of the coalface
Engage with the enthusiasm the practices will have to improve the care delivered to their patients
Most of this slide is self-explanatory.
The PCTs’ relationship with practices is a long-term one and, ultimately, services will not improve unless there is co-operation between the two. This assessment needs to be carried out positively and constructively, as it is only part of an on-going conversation. The practices will be keen to see services improve for their patients and PCTs need to help achieve progress.