1. FAEIS Project User Opinion Survey 2005 Thursday, June 23, 2005 Washington, D.C. H. Dean SutphinYasamin Miller ProfessorDirector, SRI Agriculture & Extension EducationCornell University Virginia Tech Universityyd17@cornell.edu sutphind@vt.eduwww.sri.cornell.edu Food and Agricultural Education Information System (FAEIS) Project User Opinion Survey 2005

2. FAEIS Project User Opinion Survey 2005 To understand from the FAEIS users, their experiences with the system and its usefulness To gather input from FAEIS users in higher education on improvements and changes that are needed in the future Purpose

3. FAEIS Project User Opinion Survey 2005 1.Solicit input about the user friendliness of FAEIS 2.How well is FAEIS meeting its goal of supporting higher education in food, agriculture, and natural resources sciences 3.How well is FAEIS meeting its goals to support the national network of universities 4.Determine the comprehensive view of current and future use of FAEIS as reported by feedback from expert in the field Objectives

4. FAEIS Project User Opinion Survey 2005 5.Determine the effectiveness of FAEIS and future direction in terms of context, input, process, and product 5.1 CONTEXT Includes the elements of the university system in which FAEIS is operating including alignment with needs of the national network of universities, reliability of FAEIS, and interactions that university personnel have with FAEIS Objectives

5. FAEIS Project User Opinion Survey 2005 5.2INPUT Asks the user to consider the human, fiscal, and other resources required for the FAEIS to operate within the university system such as commitment of staff time and variables related to data entry. Objectives

6. FAEIS Project User Opinion Survey 2005 5.3PROCESS The user is asked to evaluate process variables (i.e. timing of data entry and help desk activity) required for development, maintenance, and continuous improvement. Objectives

7. FAEIS Project User Opinion Survey 2005 5.4PRODUCT Characteristics of project outcomes and uses of the system such as quality of report generators, reliability of the system components, and need for further refinements are evaluated. Objectives

8. FAEIS Project User Opinion Survey 2005 Methodology Survey Research using Case Study Methodology Administer Survey Instrument to Population of FAEIS User Group Focus Group using Expert Panel

9. FAEIS Project User Opinion Survey 2005  Structure of evaluation distributed to FAEIS panel of experts Fall 04 for review and reaction  Review and refinement of structure based on input  Decision to delay evaluation until Spring 05.  The survey instrument was developed by SRI in collaboration with the FAEIS central project staff and USDA Higher Education administration responsible for the project. Methodology

10. FAEIS Project User Opinion Survey 2005 Population and Sample 1862 Land-Grant779 1890 Land-Grant135 1994 Land-Grant26 Non-Land-Grant419 Other (US, USDA, Priv, Org)57

11. FAEIS Project User Opinion Survey 2005 1.Outlining overall survey objectives 2.Development of survey dimensions 3.Development of items within dimensions 4.Internal testing-administration of survey to individuals on the FAEIS administrative team 5.Revision of instrument based upon internal testing Instrument Development

12. FAEIS Project User Opinion Survey 2005 6.Pilot test - via web N=35 individuals 10 pre-selected due to qualifications 25 randomly selected Personalized invitation e-mails sent with unique URL links on 5/26/05 Reminder e-mail sent to non-respondents on 5/31/05 Total: 17/35 completions Response rate: 48.6% Instrument Development

13. FAEIS Project User Opinion Survey 2005 7.Revision of questionnaire based upon pilot results Panel of experts used to determine instrument validity Variance in responses Not all questions applicable to all respondent types Additional response categories added, few minor changes made Conclusion: revised instrument was valid and reliable and able to capture the desired information Instrument Development

14. FAEIS Project User Opinion Survey 2005 8.Launching of tested instrument Personalized invitation e-mail with unique URL link sent on June 6, 2005 Data Collection

15. FAEIS Project User Opinion Survey 2005 Number of Responses by Institution Type June 17 Institution TypeTotalCompletionsRefusalsResponse Rate 1862 Land-Grant7792162827.7% 1890 Land-Grant13525318.5% 1994 Land-Grant264015.4% Non-Land-Grant419991023.6% Other (US, USDA, Priv, Org)578314.0% TOTAL1,4163524424.8% Response Rate

16. FAEIS Project User Opinion Survey 2005 Send up to five reminder e-mails to non- respondents Identify institution types with low response rates and follow-up with reminder phone calls Follow Up of Non Respondents

17. FAEIS Project User Opinion Survey 2005 Control for Non Repondent Error Comparison of non respondent to respondents data Determine reason for lack of participation in survey

18. FAEIS Project User Opinion Survey 2005 Indepth Analysis and Probing Use FAEIS Expert Panel and Focus Group Data Transcribed and Reported Internally Anonymity of Response

19. FAEIS Project User Opinion Survey 2005 Analyses to portray variables and comparison on dimensions of particular interest Conclusions Recommendations Executive Summary Final Report

20. FAEIS Project User Opinion Survey 2005 Who are the respondents: 352 respondents out of 1416 by June 17 82.4% use the FAEIS system and enter data 4.4% do not enter data, but use the system 13.2% do not use the system at all (of these significantly more NLG (20.2%) and Other (62.5%) are non-users) Preliminary Findings

21. FAEIS Project User Opinion Survey 2005 Q:Can you obtain student placement data? (significantly more NLG (30%) cannot get student placement data) Q:Are we collecting data from the institutions which you consider your peers? Q:Do a sufficient number of institutions report data to allow for meaningful and helpful analysis? Preliminary Findings: Context

22. FAEIS Project User Opinion Survey 2005 Q:How would you rate your ability to obtain the following data in the format required by FAEIS: Preliminary Findings: Input

23. FAEIS Project User Opinion Survey 2005 Q:Are the instructions in the FAEIS system clear? Preliminary Findings: Process

24. FAEIS Project User Opinion Survey 2005 Q:Is the support offered by the Help Desk adequate? Preliminary Findings: Process (continued)

25. FAEIS Project User Opinion Survey 2005 Q: How useful is the information collected in FAEIS: Preliminary Findings: Product

26. FAEIS Project User Opinion Survey 2005 Q:To what extend does FAEIS add value to other information sources that are available? Preliminary Findings: Product (continued)

27. FAEIS Project User Opinion Survey 2005 84.8% do not use Custom Report Builder Preliminary Findings: Product (continued)

28. FAEIS Project User Opinion Survey 2005 Q:Who are users of FAEIS data at your institution? Preliminary Findings: Product (continued)

29. FAEIS Project User Opinion Survey 2005 Q:In general, to what extent is it useful for your institution to participate in the FAEIS database? Preliminary Findings: Product (continued)

30. FAEIS Project User Opinion Survey 2005 Preliminary Thoughts on Conclusions and Recommendations Need further analyses to explain the high rate of “don’t know”. Is this response isolated to a particular population? How can we correct this lack of information? Preliminary data show a positive view of FAEIS on most variables Need to determine why it is more difficult for non land grants to access data Custom report builder is in early stages of release and current data may not reflect full potential

31. FAEIS Project User Opinion Survey 2005 Preliminary Thoughts on Conclusions and Recommendations FAEIS seems to be satisfactory in terms of comprehensive elements involving context, input, process, and product Respondents were not forthcoming with significant new and innovative future development recommendations FAEIS seems to be meeting its goals in terms of user friendly and meeting needs of the university community.

32. FAEIS Project User Opinion Survey 2005 Final Report Format Are there recommendations for the format, style or types of analyses? Are there recommendations on developing the web report or the written report? Any suggestions on dissemination of findings?

33. FAEIS Project User Opinion Survey 2005 Your thoughts – opinions? Questions

34. FAEIS Project User Opinion Survey 2005 Thank you.