2. The ATOMS Project: Measuring Assistive Technology Outcomes Dave L. Edyburn, Roger O. Smith, Todd D. Schwanke & Kelly S. Fonner Center for Rehabilitation Sciences & Technology Closing the Gap Conference, Minneapolis, MN October 17, 2002 View: Universal Access FeaturesUniversal Access Features

3. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project2 Copyright This Microsoft PowerPoint file has been made available as an accessible, electronic handout for the participants of the presentation. You must obtain permission from the ATOMS Project before copying or further distributing this presentation.

4. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project3 ATOMS Projects Vitals Based at the University of Wisconsin – Milwaukee National consortium 5 year AT outcomes project NIDRR funded (US Dept of Education) DRRP - Disability and Rehabilitation Research Projects

5. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project4 ATOMS Project Consortium Steve Mendelsohn Helen Hayes Hospital

6. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project5 AT Outcomes Vision and Needs? What do you think an outcomes system should look like in 10 years? How would you want to use it? Audience Vision Audience Need

7. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project6 The Atoms Project Response Difficult questions for response, arent they? The ATOMS Project hopes to help clarify the fields 10 year vision.

8. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project7 Overall ATOMS Project goal Explore, Pilot, and Test AT Outcome Measurement Ideas to Recommend an AT outcomes system (Go Where No One Has Gone Before)

9. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project8 Our Overall Approach Reflected as the ATOMS Project Logo

10. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project9 Challenges to an AT outcomes system Diversity of perspectives of need for outcomes data Diversity of populations Diversity of domains for which we'd like outcomes data Requirements for a data system to use reliable and valid measures AT is only one of many interventions and it is an rarely used in isolation Outcomes methodology has advanced, but so have new outcomes measurement ideas

11. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project10 AT Outcome Model (1)- Context and Baseline Environment Task Person Function BaselineContext

12. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project11 AT Outcome Model (2)- Interventions Environment Task Person Function BaselineContext Interventions: Including the Use of Assistive Technology Intervention Approaches

13. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project12 AT Outcome Model (3) - Outcome Environment Task Person Function BaselineContext Interventions: Including the Use of Assistive Technology Intervention Approaches Enhanced Function Outcome

14. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project13 AT Outcome Model (4) - Interventions Detailed Environment Task Person Functional Performance BaselineContext Intervention Approaches Enhanced Functional Performance Outcome Reduce the Impairment Use Assistive Technology Devices and Services Redesign the Activity Compensate for the Impairment Redesign the Environment Use Personal Assistance

15. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project14 AT Outcome Model (5) - with Pre-interventions Environment Task Person Functional Performance BaselineContext Intervention Approaches Enhanced Functional Performance Outcome Reduce the Impairment Use Assistive Technology Devices and Services Redesign the Activity Compensate for the Impairment Redesign the Environment Use Personal Assistance Universal Design Health Promotion Pre-intervention

16. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project15 AT Outcome Model (6) - with Costs Environment Task Person Functional Performance BaselineContext Intervention Approaches Enhanced Functional Performance Outcome Reduce the Impairment Use Assistive Technology Devices and Services Redesign the Activity Compensate for the Impairment Redesign the Environment Use Personal Assistance Universal Design Health Promotion Pre-intervention $$

17. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project16 Face the Challenge or Run? Does that mean that a coordinated outcome system is too difficult and that we are stuck with a fragmented, haphazard, homemade, anarchistic system? Is a more cohesive and comprehensive AT outcomes approach possible?

18. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project17 Strategies Computerized data collection Centralized database Customized or multiple interfaces for various user perspectives Decision analysis data elicitation strategies

19. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project18 General ATOMS Project Timeline Year 1-2: Needs assessment & field feedback Year 2-3: Instrument/methodology exploration & development Year 3-5: Pilot ideas Year 5: Propose AT outcomes methods & system

20. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project19 ATOMS Specific Activities 1. Needs Assessment Field Scans Stakeholder Focus Groups Existing Database Analyses 2. Exploratory R&D Projects (Instrument Development) 3. Abandonment Analysis

21. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project20 Field Scans Types Current Instruments (1,2) Newer methodologies & instrumentation (4,5,6) Feedback from field (focus groups) Literature reviews (3,7,8,10,12) Legal/Policy (9) Conference scientific reviews (11)

22. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project21 Field Scan 1 – Instrument Update and Review Formal and informal tools Commercial and program specific tools Specific and general tool Instrument nomination form Identification of gaps and overlaps Searchable directory Identification of type of measurement domains addressed

23. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project22 Field Scan 2 – Coverage of AT in Current/Emerging Health & Rehabilitation Outcome Measures ~100 instruments selected for review (2 scored two- ways) Preliminary review reveals: N = 41 fail to acknowledge the use of assistive technology in their outcome scoring N = 47 lower the outcome score if AT is used (nude independence) (most examine limited types of AT) N = 24 allow for AT to elevate the outcome score, but many do not differentiate among type of aids and assistance Of the 102, 4 acknowledge that AT contributes to outcomes and isolates the outcome.

24. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project23 Field Scan 3 – Inventory of Measures Used in AT Research and Design Activity Do AT developers use appropriate outcomes instruments and methodologies for their projects? How severe is the problem? Or how is this perceived by product developers? Request methodology of grant proposals from PIs – 2001 funding (NIH n=34, NIDRR n=27) Survey of product developers Technology Exhibitors, RESNA&AOTA National Conference 2002 n=78 Random sample (n=500/1100) from ABLEDATA Directory of Manufacturers

25. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project24 Field Scan 4 – Ascertain Next Generation Data Collection Technology Factors considered for review of Technology & Methodology Content/outcomes information Scaling potentials Data collection processes/protocol Equipment (hardware/software) characteristics e.g. interface, portability, cost, durability Data handling protocols Data Reporting ~ 50 hardware & software technologies being reviewed

26. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project25 Field Scan 5 – Evaluate Cost Comparison Methods Helen Hayes Hospital: Frances Harris, Ph.D. Initiating methodology review of cost comparison methods and how various approaches match the needs for AT Outcomes measurement procedures that include cost variables. Methods of measuring costs Methods of comparing costs Cost-comparison literature review relevant to assistive technology

27. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project26 Field Scan 6 – Assess Application of Decision Analytical Approaches Multi-attribute Utility (MAU) Theory Application analysis (selected articles) 47 Engineering articles using MAU methods 60 Health-related articles using MAU methods 32 General articles on MAU methodology Bayesian Approach analysis (initial search) 3797 articles, MEDLINE (1966-present) 140 ERIC (1967-present) 1573 Engineering Village (1970-present)

28. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project27 Field Scan 7 – Review Taxonomies of Outcome What intersection of domains across taxonomies provide common language for a more universal AT outcomes discussions? e.g. Nagi Model (1965) ICIDH (1980, 1993) Rehabilitation Indicators (1983) NCMRR Research plan (1993) Quality of Life Taxonomy (Spilker & Revicki, 1996) Characterization of Rehabilitation Services (Duncan, Hoenig, Samsa, & Hamilton 1997) Institutes of Medicine Model (1997) ICIDH-2 draft (1997) ICF (2002)

29. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project28 Field Scan 8 – Identify Strategies for Isolating AT Outcomes Compare methodologies that can isolate & quantify the outcomes of assistive technology. ~410 articles e.g. Multi-variate regression analyses Structural equation modeling Direct consumer qualitative input (perceived benefit / satisfaction of a device) Randomized controlled trials Sequential Clinical Trials Time-series concurrent differential (TSCD) Qualitative

30. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project29 Field Scan 9 – Describe Legal Implications of AT Outcomes Instrumentation What are legal & ethical issues related to AT outcomes systems? -- Steven Mendelsohn Legal, responsible, & ethical data collection procedures Potential legal ramifications of AT outcomes data (positive and negative) Implications of AT outcomes for policy-making

31. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project30 Field Scan 10– Review of Special Education Technology Literature Review of 31 special education technology journals Reviews published in Journal of Special Education Technology (1999, 2000, 2001) >2700 articles Iteration with outcome filter to identify relevant articles. Summarize and report findings.

32. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project31 Field Scan 11– Technology Conference Program Review Last two years of CTG, CSUN, RESNA, and TAM conferences reviewed as fugitive literature. Identify relevant papers on AT Outcomes. Summarize and report findings.

33. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project32 Field Scan 12– Chronology of Assistive Technology Outcomes Measurement Literature review reveals that At outcomes have been measured differently over the decades This is consistent with what types of outcomes studies have been performed and the mandate for accountability This field scan will develop a chronology map and accompanying discussion

34. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project33 Quiz on the field scans: Just Kidding…. Lots of data from field scans. What do we hope to learn? Technical reports Compendium document Consensus meeting

35. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project34 Stakeholder Focus Groups Direct Feedback from field Consumer/User groups (4) Service Directors (for records & documentation assessment), Payors of AT devices & services, Researchers, Developers, & Manufacturers, and Parents & Caregivers.

36. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project35 Service Director Focus Group – April 2002 Modified NGT developed current list of identified AT Outcomes data domains (If we had a magic wand) 1. Change in performance/function (body, structure, activity) 2. Change in participation 3. Usage and why or why not 4. Consumer satisfaction (process, devices) 5. Goal Achievement 6. QOL 7. Cost 8. Demographics 9. AT interventions (services + devices) 10. Environment context List is similar to DeRuyter (1998)

37. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project36 Databases – National National Health Information Survey-Disability (NHIS-D) Assistive and information technology survey (NIDRR/RESNA/University of Michigan)

38. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project37 Databases – Clinical UW-Stout University at Buffalo (SUNY) HHH PROVAIL What data are being collected? Are available data sufficient for analysis? Is there an intersection of outcome data fields among existing service file systems? Can clinical programs adapt to collecting more data?

39. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project38 Clinical Database Update Data fields Commonalities (few) and differences (numerous) Setting & funding specific (education, vocation, hospital, university) Stakeholders (variable) Electronic records (variable & minimal) Potentials VR data Willingness of clinical programs to modify data collection

40. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project39 Exploratory R&D Projects (Instrument Development) a) AT Device Inventory b) AT Services Inventory c) Cost Identification Feasibility Study d) Assistive Technology Approach Isolation Measure (Subjective) e) Web-based visualization f) Environmental access assessment (WebAUDIT, MED- AUDIT) g) Computer branching questioning (TTSS) h) AT supplements to existing instruments (SFA-AT)

41. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project40 Public School AT Outcome Pilots & Collaborative Activity a) Ohio b) Colorado Colorado Ohio Colorado

42. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project41 NIDRR DRRP Coordination CATOR (Consortium of Assistive Technology Outcomes Research) http://www.atoutcomes.org http://www.atoutcomes.org ATOMS Project http://www.atoms.uwm.edu

43. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project42 Dissemination: What to Expect ATOMS Project Website Products Special Issues of Journals Conference Presentations 2003-2004 Symposium

44. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project43 Dissemination: ATOMS Project Products in the Works AT Outcomes Primer Study Group Outline Test Your Knowledge of AT Outcomes FAQs Course Guide (syllabi and more) Drafts of Instruments Technical Reports Compilation of Needs & Current Outcome Directions Implications for Next Generation AT Outcomes System

45. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project44 Needs Assessment Participation Opportunities Conversation Product developer survey Instrument collection

46. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project45 Contacting the ATOMS Project Completing interest survey Web: www.atoms.uwm.eduwww.atoms.uwm.edu Email: atoms@uwm.eduatoms@uwm.edu Voice: (414) 229-6568 TTY: (414) 229-5628

47. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project46 Questions, Answers and Discussion ? ? ? ? ? ? ? ?

48. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project47 Thank-you….... for your attention! Slides about the universal access features of this PowerPoint presentation follow.

49. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project48 Universal Access Features The speaker notes associated with each slide in PowerPoint are used as a Universal Access feature. They contain long text descriptions of the graphics because it was not feasible to do this with PowerPoints alternative text function. The descriptions can be used by new presenters and presentation attendees, in addition to being used for accessibility by people with vision or cognitive impairments.

50. © 2002 Center for Rehabilitation Sciences & Technology October 2002Update: The ATOMS Project49 Viewing the Speaker Notes (does not work within PowerPoint Viewer 97 or 98) In the Slide Show view within PowerPoint: Windows: right click on the slide or use the context key to bring up the context menu and then select speaker notes Mac: [Ctrl] + click on the slide to bring up the context menu and then select speakers notes The notes can also be seen as a frame or pane in the Normal view or directly by using the Notes Page view. When in Normal view, F6 is used to switch between the slide, notes and outline panes respectively. Go back to the opening presentation slide