Presentation on theme: "Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model."— Presentation transcript:
Evaluating health informatics projects Reasons for and problems of evaluation Objective model Subjective model
Definitions Evaluate – to determine the value of (Chambers) To examine and judge carefully (Dictionary.com)
Reasons for evaluation (Friedman/ Wyatt) Promotional – –encouraging people to use systems Scholarly – –study of the impact etc. of HI systems Pragmatic (practical) – –finding out what is good and bad, improving future systems Ethical – – like any medical intervention, safe and effective Legal – –same reason. Also to inform users so they know when and when not to use it
Perspectives Stakeholders –Developers –Users –Patients –Managers –Sources of funding
Effects Structure – environment, staff, money Processes – diagnosis, investigation, treatments Outcomes – success of treatment, survival, continuing health
Complexity Combination of –Medicine & health care –Information systems / IT –Evaluation methodology Each of these is a huge area Arguably IT is the simplest, or at least the most structured
Of Medicine Extremely large and growing area of knowledge Complex structure –Equipment, staff, regulation Processes –Treatments etc. Outcomes –Long term, difficult to measure –Knock-on effects of innovation –Effect of IT particularly hard to measure
Of Information systems Difficult to fully test –Combinatorial explosion Multi-function –Has a range of effects System itself vs. impact on health care
Of Evaluation Have to measure impact –This means impact on people - difficult to study Need patients and staff to perform tests –May not be enough willing to cooperate Range of things that can be evaluated, ranging from –‘Does it work?’ to –‘Does it help patients?’
Evaluation In theory, –study situation before & after In practice, –don’t know what changes would have occurred without innovation –don’t know what interesting questions will arise during study
Tips Tailor study to problem –Not research – specific to this project Collect useful data –Data which inform final decision Look for side-effects –Effects not related to intended purpose Formative & summative –Study during & after development
Tips (continued) In vitro vs. in vivo –Evaluate on-site & off-site Don’t accept developer’s view Take account of environment – context Let questions appear during study Be prepared to use a range of methods
What can be studied Need for resource –What does it give us that we didn’t have before Development process –What methods do developers use to design their solution? Structure of resource –What does the program & spec look like? Functions of resource –How well does it work? Impact –How does it affect HCPs and patients?
Study features Focus –As previous slide Setting –Laboratory or hospital Data –Real or simulated Users –Developers, evaluators, end-users Decisions –None, simulated or real
Types of study Need validation Design validation Structure validation Laboratory function Field function Laboratory user impact Field user impact Clinical impact
Objectivist or quantitative approach Can measure things objectively and without affecting thing being measured What to measure can be agreed rationally Can use numerical data Draw definite conclusions
Objectivist approaches Comparison-based –Like randomised clinical trial Objectives-based –Does it do what the designers said? Decision facilitation –Answers questions posed by managers Goal-free approach –Evaluators not aware of project goals
Measurement studies Terminology for measurements –Object e.g. patient –Object class e.g. patient group –Attribute e.g. temperature –Instrument e.g. thermometer –Observation e.g. temperature at one time Validation – calibration of thermometer
Demonstration studies Demonstrate effect –‘Do patients who have been inoculated have a higher temperature?’ Object -> subject (patient) Attribute -> variable (temperature)
Descriptive ‘The patients in this study have a rather high temperature’. Mean, standard deviation etc.
Comparative ‘The patients in this study have a higher temperature than a control group’ Controlled environment (usually) T-test etc.
Correlational ‘We are seeing more patients with fever since we introduced inoculation’ Live situation Could still be a t-test Trying to associate one factor with another in a real situation
Subjectivist or qualitative Observations depend on observer Observations only meaningful in context Different points of view may be valid Descriptions as valuable as numbers Discussion of results
Subjectivist approaches Quasi-legal –Cf ethical debate Art criticism –Expert review Professional review –Site visit Responsive/illuminative –Immersion in environment –Questions evolve over time
Qualitative approach Attempts to understand why as well as measure differences e.g. –Is system working as intended? –How can it be improved? –Does it make a difference –Are differences beneficial? –Are the effects those expected?
Stages in qualitative study Negotiation of ground rules Immersion into environment Initial data collection to focus questions Iteration Report and feedback Final report
Methods in qualitative study Observation Interviews Document analysis Others, e.g. structured questionnaires
Mixed study Can combine qualitative and quantitative approaches