1. The STARS Alliance Students & Technology in Academics, Research, and Service STARS Alliance Evaluation Tiffany Barnes Evaluation Team Kim Buch, Audrey Rorrer Web: Anthony Chow Qualitative & PP: Sally Berenson

2. The STARS Alliance Students & Technology in Academics, Research, and Service 2 Barnes, Eval Team NSF Site Visit 2007 Evaluation Successes: –Refined approach to evaluate Organizational, program, individual level outcomes –Instrument development –Combined qualitative and quantitative measures for: SLC Alliance effectiveness –Articulated new goals and measures (STARS/Computing Identity) Challenges: –Changing team –New research questions arise all the time –Comprehensive program has many goals - too many surveys for same people –Data collection from different people/institutions –Small team

3. The STARS Alliance Students & Technology in Academics, Research, and Service 3 Barnes, Eval Team NSF Site Visit 2007 Impact on pipeline Q1: The STARS Alliance features a broad scope (middle school to graduate school) across three categories of population (gender, ethnic, disability), with a variety of outreach, research, mentoring, and career counseling components. Characterize, quantitatively if possible, what has been attempted within this scope of activity and what has been achieved to date. To what extent have each of the target groups been reached? Each of the pipeline stages? Approximately 110 college SLC students each year –About 40% returning in Year 2 Over 500 K-12 students at outreach events Over 500 parents Over 100 teachers and counselors > 5 major community events (AU, FSU, UNCC)

4. The STARS Alliance Students & Technology in Academics, Research, and Service 4 Barnes, Eval Team NSF Site Visit 2007 Measures Q3: What new measures of impact/outcome have been developed to capture what the Alliance is seeking to achieve, i.e., beyond traditional pre- post test measures appropriately disaggregated (by gender, ethnicity, disability, and cohort) such as GPA, year-to-year retention, and degree attainment? For example, how have the observations in students' electronic journals been used by the Alliance institutions and Advisory Boards? Selected interviews each semester Journaling currently not being analyzed Climate survey to measure impact of STARS on computing culture at schools New outreach evaluation instruments

5. The STARS Alliance Students & Technology in Academics, Research, and Service 5 Barnes, Eval Team NSF Site Visit 2007 Comparison Q7: What data is the Alliance using for comparison purposes? Are they national sources, local institutional information on students, some combination, or other? Explain. Comparison of applicants, enrollments across years Comparison with NSF and Taulbee survey data

6. The STARS Alliance Students & Technology in Academics, Research, and Service 6 Barnes, Eval Team NSF Site Visit 2007 Celebration impacts Q4: Describe the benefits of the summer workshop to different categories of participant. Faculty- networking, project collaborations, information exchange, e.g. –o Junior Faculty Round Table –o Implementing Pair Programming

7. The STARS Alliance Students & Technology in Academics, Research, and Service 7 Barnes, Eval Team NSF Site Visit 2007 STARS Celebrations The STARS Celebration emphasizes STARS core values of excellence, leadership, civic engagement, service, and community – values intended to foster student success. 2007 at UNC Charlotte – 113 students – Over 40 faculty – Over 50 partners and community 2006 at Georgia Tech – 105 students – 23 faculty/staff

8. The STARS Alliance Students & Technology in Academics, Research, and Service 8 Barnes, Eval Team NSF Site Visit 2007 STARS Celebration Results Students felt the Celebration provided opportunities for community building. (2006: 97%, 2007: 92% ) Students learned how computing professionals can improve quality of life. (2006: 83%, 2007: 89%) Students learned about disparity of representation in IT: by gender and race (2006: 98%, 2007: 88%) by persons with disabilities (2006: 86%, 2007: 69%) Students had ample time to select leadership assignments to fit their professional needs (2006: 80%, 2007: 88%)

9. The STARS Alliance Students & Technology in Academics, Research, and Service 9 Barnes, Eval Team NSF Site Visit 2007 Demonstration Projects The STARS Alliance serves as an incubator for new demonstration projects and the scaling and replicating of best practices among the diverse alliance institutions. STARS Leadership Corps Pair Programming Tiered mentoring model Teaching Math to the Visually Impaired African American Researchers in Computer Science Culturally Situated Design Tools

10. The STARS Alliance Students & Technology in Academics, Research, and Service 10 Barnes, Eval Team NSF Site Visit 2007 STARS Leadership Corps Model

11. The STARS Alliance Students & Technology in Academics, Research, and Service 11 Barnes, Eval Team NSF Site Visit 2007 STARS Tiered Mentoring and Role Models High school student Middle school student Professional or Grad student Undergrad student Regional Partnerships - A computing community from Kindergarten to the Workforce

12. The STARS Alliance Students & Technology in Academics, Research, and Service 12 Barnes, Eval Team NSF Site Visit 2007 Evaluation Three-Tiered Outcomes Model: –Organizational Outcomes (e.g., Alliance) –Program Level Outcomes (e.g., Demonstration Projects) –Individual Level Outcomes (e.g., Students) Evaluation model: CIPP –Context (setting of project) –Input (project resources) –Process (activities & strategies) –Product (ultimate results)

13. The STARS Alliance Students & Technology in Academics, Research, and Service 13 Barnes, Eval Team NSF Site Visit 2007 Alliance Outcomes Increased student enrollment in computing and IT programs Increased student awareness about computing and IT Increased student readiness to enter computing and IT graduate school and workforce Increased participation of undergraduates entering computing and/or IT graduate school or workforce Increased graduate rates of computing and IT students Increased persistence and declaration of students majoring in computing or IT Increased college adjustment and GPA for students in computing or IT

14. The STARS Alliance Students & Technology in Academics, Research, and Service 14 Barnes, Eval Team NSF Site Visit 2007 Alliance Outcomes Sustained Alliance Efficacy Institutionalize Alliance Partnerships Increased national awareness of effective practices for Alliance development Serve as a model and repository for BPC

15. The STARS Alliance Students & Technology in Academics, Research, and Service 15 Barnes, Eval Team NSF Site Visit 2007 Year 1 Accomplishments Steering Committee: –Jan. 2006-2007, & Aug. 2006-2007 –Regular teleconferences for planning & update STARS Celebration –Successful Annual Conference held in August 06-07 –2006: 105 students, 23 faculty, 2007: 113 students, >40 faculty –2007: New evening for partners, >50 additional attendees Student Leadership Corps –2 universities implemented in Summer 2006 –All implemented in Fall 2006, Spring 2007, Fall 2007 Workshops –Pair Programming, Mentoring, IBPC: Jan & Aug 2007 –CSDT at Celebration Aug 2007 –Qualitative methods workshops Aug 06-07, April 2007

16. The STARS Alliance Students & Technology in Academics, Research, and Service 16 Barnes, Eval Team NSF Site Visit 2007 Accomplishments Web portal: StarsAlliance.org Evaluation –Instrument development –IRB exploration: applying for waivers for K-12 outreach evaluation –Interviews & Training Coordination with UNC Charlotte REU Site Faculty recruiting at UNC Charlotte!

17. The STARS Alliance Students & Technology in Academics, Research, and Service Results

18. The STARS Alliance Students & Technology in Academics, Research, and Service 18 Barnes, Eval Team NSF Site Visit 2007 Institutional Enrollment

19. The STARS Alliance Students & Technology in Academics, Research, and Service 19 Barnes, Eval Team NSF Site Visit 2007 SLC 06-07 Participants SLC Participants FemalesMalesTotal African American 20%23%43% Asian 7% Caucasian 29%19%48% Total 58%42%100%

20. The STARS Alliance Students & Technology in Academics, Research, and Service 20 Barnes, Eval Team NSF Site Visit 2007 2006-2007 SLC 107 students participated in 2006-2007 STARS Leadership Corps

21. The STARS Alliance Students & Technology in Academics, Research, and Service 21 Barnes, Eval Team NSF Site Visit 2007 SLC Survey Results Retention of SLC students: Over 95% are forming meaningful peer & faculty relationships 88% more committed to computing majors 94% interested in graduate education 83% felt the SLC helped them develop computing skills 91% would recommend the SLC to others Recruiting: 90% are helping others understand the value of computing Hundreds of K-12 students have participated in outreach events Building a like community: 88% felt the SLC gave them opportunities to work with people like themselves

22. The STARS Alliance Students & Technology in Academics, Research, and Service 22 Barnes, Eval Team NSF Site Visit 2007 Who is SLC? NOT the average UG student in Computing –Older, average age= 24 years –Higher GPA in Computing, M=3.3 –Higher SES 85% middle class 64% had parents with at least some college; 53% were college graduates –Much more diverse Women 51% Under-represented minorities 62% Physical disability 14% –Extremely confident that they will obtain UG degree (4.7 on 5-pt scale) –Very likely to attend graduate school 66% believe they will obtain Masters 50% believe they will obtain PhD

23. The STARS Alliance Students & Technology in Academics, Research, and Service 23 Barnes, Eval Team NSF Site Visit 2007 SLC & Computing 93% are confident that Computing is the right major 87% plan to stay in the field of Computing long-term Have a very strong Computing Identity (M=4.4/5) –Feel I fit in field of Computing M=4.2 –Expect to be treated fairly in Computing career M=4.1 –Can relate to the people around me in my classes M=4.3 –Committed to applying Computing to benefit society M=4.5 –Committed to promoting Computing to others M=4.3 Have a very strong Computing Efficacy (M=4.3/5) –Feel someone like me can succeed in a Computing career M=4.6 –I can complete the programming requirements for Computing majors M=4.5 –I can complete the math requirements for Computing majors M=4.6 –I can complete the science requirements for Computing majors M=4.6 –I can complete my current Computing degree M=4.6

24. The STARS Alliance Students & Technology in Academics, Research, and Service 24 Barnes, Eval Team NSF Site Visit 2007 SLC & Computing departments? 64% at least weekly contact with Computing faculty 77% satisfied+ with the degree of interaction with faculty 67% satisfied+ with the availability of extra-curricular activities, eg, ACM/IEEE 73% satisfied+ with welcoming atmosphere in the department 47% satisfied+ with availability of cutting- edge/innovative courses 70% feel satisfied+ that I am part of the department 73% overall satisfied+ with the department

25. The STARS Alliance Students & Technology in Academics, Research, and Service 25 Barnes, Eval Team NSF Site Visit 2007 SLC & Diversity in Computing 66% satisfied+ w/ the diversity of dept. students and faculty 64% agree that dept. students are sensitive to minority issues 77% agree that dept. faculty are sensitive to minority issues 71% agree that dept. students are sensitive to womens issues 78% agree that dept. faculty are sensitive to minority issues 69% agree that dept. students are sensitive to disability/accessibility issues 77% agree that dept. faculty are sensitive to disability/accessibility issues

26. The STARS Alliance Students & Technology in Academics, Research, and Service 26 Barnes, Eval Team NSF Site Visit 2007 SLC Concerns Computing faculty contact per semester –6% reported NO contact –5% reported they only had ONE contact –12% reported they had one contact per month Students rated their high school technology/ computing program as: –36% Not good –23% Somewhat good –14% Very good, 14% Excellent Students felt that their depts were NOT sensitive to: –36% minority issues –30% womens issues –31% disabilities/access issues

27. The STARS Alliance Students & Technology in Academics, Research, and Service 27 Barnes, Eval Team NSF Site Visit 2007 SLC quote The STARS Alliance has engaged me to use computing to make a positive impact by exposure to other areas of technology and to be of service to other students and community members. It has provided them (middle school students) access to computing resources. It has allowed them to see women and minorities as role models.

28. The STARS Alliance Students & Technology in Academics, Research, and Service 28 Barnes, Eval Team NSF Site Visit 2007 SLC Interviews Emergent themes toward careers are altruism, creativity, and mentorship. Initial early interest in computing: game playing that requires problem solving, along with a motivation to be self-sufficient. Overall, students stated that a family member or mentor had been influential in their decision to pursue a career in computing. The overwhelming majority of students indicated that their reason for persistence in computing is due to a commitment to giving back to their communities and an excitement about the creativity inherent in computing careers.

29. The STARS Alliance Students & Technology in Academics, Research, and Service 29 Barnes, Eval Team NSF Site Visit 2007 Celebration Results 97% of students felt the workshop provided adequate opportunities for community building. 83% agreed with the statement that the SLC Workshop emphasized how computing and IT professionals can use their skills to improve quality of life

30. The STARS Alliance Students & Technology in Academics, Research, and Service 30 Barnes, Eval Team NSF Site Visit 2007 Alliance structure Collaborations are beneficial to both students and faculty Participation facilitates faculty and student career development Alliance goals are being achieved and represent meaningful impact to the computing field. Training, dissemination, and sustainability goals are being met

31. The STARS Alliance Students & Technology in Academics, Research, and Service 31 Barnes, Eval Team NSF Site Visit 2007 Presentations Grace Hopper Celebration, Oct 2006 SIGCSE presentation to computing K-12 teachers, funded by Microsoft Presentation at NIU, August 2007 Panel at GHC 2007 Panel & Birds of a Feather at Tapia 2007 Paper at Frontiers in Education 2007

32. The STARS Alliance Students & Technology in Academics, Research, and Service Successes & Challenges

33. The STARS Alliance Students & Technology in Academics, Research, and Service 33 Barnes, Eval Team NSF Site Visit 2007 Alliance infrastructure Time is major issue for both faculty & students Need tools to facilitate: –Collaboration –Communication –Management –Timelines & deadlines –Tracking –Reporting –Sharing resources

34. The STARS Alliance Students & Technology in Academics, Research, and Service 34 Barnes, Eval Team NSF Site Visit 2007 Cross-institutional collaboration Faculty see this as very successful Students excited to meet students from other places Recruiting across Alliance! Looking forward to building more

35. The STARS Alliance Students & Technology in Academics, Research, and Service 35 Barnes, Eval Team NSF Site Visit 2007 Synergy with existing programs Essential to success Examples: –Charlotte REU & CRA-DMP Summer 2006 –Mentoring on NCSU campus –Intel Opportunities at Georgia Tech Share resources and minimize work

36. The STARS Alliance Students & Technology in Academics, Research, and Service 36 Barnes, Eval Team NSF Site Visit 2007 Recruiting students A challenge at first: What is STARS? Now: More students want to join! Creating ways for non-funded students to join –STARS Alliance club –Course credit, other avenues Creating ways to recognize participation

37. The STARS Alliance Students & Technology in Academics, Research, and Service 37 Barnes, Eval Team NSF Site Visit 2007 Replication of best practices A Challenge! How do you make professors do something new? You must show how it benefits them Need a cookbook approach

38. The STARS Alliance Students & Technology in Academics, Research, and Service 38 Barnes, Eval Team NSF Site Visit 2007 Sustaining changes Student STARS Alliance club STARS course added at UNC Charlotte Faculty recruiting at UNC Charlotte: –2 new black faculty, 2 new women Existence of Alliance is raising awareness: –Deans and departments are using STARS Leaders for recruiting and involving them in dept activities and policy-making, marketing

39. The STARS Alliance Students & Technology in Academics, Research, and Service 39 Barnes, Eval Team NSF Site Visit 2007 Effects of Alliance on faculty careers Writing circles –Planned for Year2 Mentoring within the Alliance –happening informally Collaborations –Landmark College, REU Site proposal, Gaming programs at UNCC, Auburn

40. STARSAlliance.org Students & Technology in Academia, Research, and Service PI: Teresa Dahlberg tdahlber@uncc.edutdahlber@uncc.edu Co-PI: Tiffany Barnes tbarnes2@uncc.edutbarnes2@uncc.edu Evaluation Team Kim Buch Audrey Rorrer Sarah Berenson Anthony Chow Jennifer Thomas