Presentation on theme: "Assessment of research: Where do we go from here? Linda Rueckert, Northeastern Illinois University (http://www.neiu.edu/~lruecker/assess.htm) Kathleen."— Presentation transcript:
Assessment of research: Where do we go from here? Linda Rueckert, Northeastern Illinois University (http://www.neiu.edu/~lruecker/assess.htm) Kathleen Morgan, Wheaton College
Who are the stakeholders in undergraduate research assessment? *Institutions/agencies *Departments and Programs *Faculty *Students
Stakeholders in Undergraduate Research: why assess? Institutions *retention *grants-seeking *grant stewardship/accountability How you ask will depend greatly on what you want to know, and why…
Stakeholders in Undergraduate Research: why assess? Departments/Programs *program improvement How you ask will depend greatly on what you want to know, and why…
Stakeholders in Undergraduate Research: why assess? Faculty *training improvement *are they learning what you think you’re teaching? *(n=61 CUR attendants) Top 3: 1. critical thinking 2. communication skills 3. applied knowledge/disciplinary literacy How you ask will depend greatly on what you want to know, and why…
Stakeholders in Undergraduate Research: why assess? Students *improve grad school prospects *improve employability *fun *develop closer relationships with faculty *we’d like to find out more! (see survey on yellow handout)—and you can help! How you ask will depend greatly on what you want to know, and why…
Stakeholders in Undergraduate Research: how assess? Institutions *Oak Ridge Associated Universities: custom-made survey instruments *University of Michigan—structured interviews (http:///www.umich.edu/~urop/facres/evalurop.html) *University of Delaware Research-based Education: (http://www.udel.edu/RAIRE) A few examples…..can you provide others?
Stakeholders in Undergraduate Research: how assess? Departments/Programs *Northern Illinois University Psychology Dept. A few examples…..can you provide others?
NEIU capstone analyses Students do independent research, field experience, or enhanced class as capstone project. All are required to write a final APA-style paper.
Capstone scoring rubric Content –Theoretical basis –Breadth and depth –Creativity –Design –Statistics –Conclusions –Relate to intro. –Broader implications Style –Flow –Spelling & grammar –Clarity –APA format
Capstone scores for content and style (old data)
Capstone scores on content subscales (old data)
Capstone scores for content and style (new data)
Capstone scores on content subscales (new data)
Stakeholders in Undergraduate Research: how assess? Faculty Critical Thinking Assessment Tools: *Watson Glaser Critical Thinking Appraisal RANRA *California Critical Thinking Skills Test *Tim van Gelder’s Critical Thinking Assessment Site (lots of links to various tools here): A few examples…..can you provide others? Communication Skills Assessment: *Liverpool Communication Skills Assessment Scale (LCSAS) *Biography of Assessment Alternatives: Oral Communication. Innovative Assessment. ERIC document ED *Assessment of Basic Oral Communication Skills: A Selected, Annotated Bibliography. ERIC document ED
Stakeholders in Undergraduate Research: how assess? Students *Kardash (2000). Self-rating instrument --used a 1-5 scale, asked research interns to rate themselves on items like “To what extent do you feel you can…… …understand contemporary concepts in your field? …make use of the primary scientific literature in your field? …formulate a research hypothesis based on a specific question?”, etc. *reflective judgment essays *goal setting and ranking *our student survey A few examples…..can you provide others?
LEARNING FROM YOUR RESEARCH EXPERIENCE* Before you begin your research work, take a few minutes and complete the statements below. This will help you to begin thinking about your learning goals for participating in research. Although at this point, you can only speculate about the research experience, the more you think about it now, the more you will learn from it. A.I am engaging in research because: B.My goals for participating in research include: C.I think I will be performing the following duties/tasks: D.I hope to develop the following skills: E.In addition to testing an hypothesis, I think I might learn about: F.What skills or perspectives do you think you already have to contribute to your research experience? Turn your completed handout in to your supervising faculty member. Sample goal setting document:
Principle Investigator (PI) Self-Evaluation Student Name:__________________________________ Semester_______ Yr_________ Title of research project you worked on:______________________________________________ ______________________________________________________________________________ Estimated # of hours spent on this project during the entire semester____________________ DIRECTIONS: The academic credit that you will receive upon satisfactory completion of your role as a PI reflects the college’s understanding that learning here also takes place outside the conventional classroom. To that end, it is useful to think about just what it is that you have learned. Using the guidelines below, write a 1-3 page reflection essay on your experience. You may find it useful to refer to your research journal as you write. Attach your essay to this cover sheet and return it to Dr. Morgan. I.Briefly describe the research project you developed or are developing and your primary responsibilities in that development. What did you do (or what do you plan to do)? What have you found out? II.Reflect (write about what you have learned): *Some of the skills that you have developed and how you learned them *What you have learned from your supervising professor, your research assistants (if any), or others who might have been involved in the project’s development *What you expected to do and what you actually did *The relationship, if any, between this out-of-the-classroom learning experience and your academic work III.Have you changed as a result of this experience? If so, in what ways? IV. What grade do you think you should receive for your efforts? Why? Sample guided reflective essay:
A Few Last Words on Assessing Undergraduate Research Experiences If you are just beginning an assessment program—whether of your classroom or your research program—here are a few suggestions to keep in mind (courtesy of Tom Angelo again): 1.Know what it is you want to know, BEFORE you begin planning your assessment. Do you want to know if student researchers go on to graduate schools more often than students who do not participate in research? Do you want to know if your student researchers are sufficiently trained in execution of a protocol to be able to manage it on their own? Do you want to know if student researchers' self-confidence increases after their experiences? Knowing what outcome you want to assess is essential to designing and implementing the right assessment. 2. Keep it simple and short. Don't make assessment into a chore or burden for either you or your students. Collect only as much data are you can reasonably use and respond to.
A Few Last Words on Assessing Undergraduate Research Experiences 3.Keep it focused. (This harkens back to knowing what you want to know!) While there are many interesting questions you might ask, focus on those most likely to improve the achievement of your own teaching, research, or program goals. Make sure that your students understand the purpose and value of your assessment. This too is an important component of assessment that we often forget. 4. Don't ask if you don't want to know, or won't be able to respond. 5. Make sure your aim is true. ALWAYS try out a new assessment technique on a colleague before trying it out on your students. 6. Close the feedback loop. Make sure to let students or other stakeholders know the outcomes of the assessment and what changes (if any) those outcomes suggest for you and them. 7. Let us know about your assessment program!