Presentation on theme: "“When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations."— Presentation transcript:
1 “When learners are at a distance, the careful design of assessments is particularly important, because society somewhat unfairly imposes higher expectations on the assessment of online learners.”Oosterhof, et al. (2008) Assessing Learners Online
2 Assessing Distance Learning: Are they learning what we think they are? Donald Staub, Title III Director Carteret Community College
3 The Lineup DL @ CCC Why Worry about DL Assessment? How We’re Doing It The DL ProgramDL in ProgramsBarriers, Conclusions & Solutionsccctitle3.wordpress.com
4 DL @ CCC 33 programs @ CCC FA10 enrollment ~1800 DL is 10+ years old 336 total sections taught243 seated sections (72%)64 Internet (20%)29 Hybrid (8%)58% FT / 42% PT
5 Why worry about DL assessment? @ the College-Level:Reason #1: Student SuccessReason #2: Persistence(btw: success vs. retention vs. persistence)
6 Reason #3: Generalizability @ the Course-Level:Reason #3: Generalizability(c.f. Oosterhof, et al, 2008)“Generalizability is the degree to which learner performances that are observed generalize to performances that are not observed.” [i.e. learning]
8 “Comparability of distance education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.”(CS 3.3.1)COC Distance Education Policy StatementComparability of distance and correspondence education programs to campus-based programs and courses is ensured by the evaluation of educational effectiveness, including assessments of student learning outcomes, student retention, and student satisfaction.
11 Q1: Is the distance education program reflected in the institution’s strategic (or master) plan? Strategic PlanWe will create a culture where all institutional actions are focused on improving student learning and success and leading to student goal completion.PM 16—Success and Withdrawal Rates by Instructional modalities
12 Q2: Is there evidence that outcomes for the program have been identified? DL-AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.DL-AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.Other possibilities: Growth rate, Success rates, Infrastructure-related….
13 Assessment: The number of internet/hybrid sections offered. Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?AO #1: Faculty demonstrate the ability to teach in the online environment in a proficient (student centered) manner.Assessment: Quality Assessment Plan, Success/ Persistence Rates, Student Surveys/Evaluations.AO #2: Students have more educational options to pursue degrees, diplomas, and certificates due to expanded online course offerings at CCC.Assessment: The number of internet/hybrid sections offered.
14 The QAP (Quality Assessment Plan) Developed in-house (using models)Peer-reviewed by content specialistStipends (T3)All current online courses have been reviewedAll new online courses must be reviewed/ certified
15 The QAP Score Cardpoints is the total possible a teacher can earn - believe its 91minimum is what they need to pass - I am happy if they get 63 or above (hybrids may do feedback in class or individually so we learned the way the QAP is now hybrid instructors may get lower scores due to no feedback to students even through they may do it during or after class - so we are modifying this for hybrids next go aroundearned is what they actually got so Laurie Freshwater may have earned a 70 and Mary Walton a 67 - below 60 we have them come back to Pre-Ah to do a 2 hour meeting to fix what needs to be addressed.
18 ILLOs (Gen Ed outcomes) Program Reviews Q3: Is there evidence that the effectiveness of the distance education program is regularly assessed and steps taken for improvement of the program?Modality ComparisonGrade DistributionsILLOs (Gen Ed outcomes)Program Reviews“Evaluate Instructional Modalities”
23 ILLOs (Gen Ed outcomes) CommunicationComputer LiteracyCritical ThinkingHumanities & Fine ArtsInformation LiteracyPersonal Growth & Responsibility
24 Computer Literacy - CIS 110 Outcome: Students demonstrate proficiency in the use of computer technology including application software, system software, and internet browsers.Assessment: Final Exam in CIS 110.
26 Computer Literacy - CIS 110 Same instructor teaching one section of each
27 Personal Growth & Responsibility ACA 115 Outcome: Students demonstrate the ability to understand and manage self effectively.Assessment: Advisement assignment in ACA 115 & 118 evaluated by rubric.Results Spring 2009: 131 total projects submitted; 81% scored 80 or above.Seated 115: 85% scored 80 or above Online 115: 81% scored 80 or above
28 Course-Level Assessment Q4: Has the institution developed student learning competencies for the courses/programs offered by distance education? If these are the same competencies for courses/programs offered by “traditional” methodologies, is assessment identified for distance learning students separate from students taking courses by “traditional” methodologies?Yes! … No?…maybe??PLLOsCourse-Level Assessment
29 PLLOs (SLOs) ACC 120 BUS 110 ECO 251/252 Same instructor teaching one section of each (take a look at a common assignment from each section)ACC 120BUS 110ECO 251/252
34 A - ( ): - You posted 3 or more times and you completed the Assignment with a well thought through response. (The more you post MEANINGFUL responses over 3 the higher your grade. Though welcome and expected, please understand that a One or Two sentence "feel good" post such as (... Great response Michaela!!) or (... I love how you put the meaning and then went down and talked about them. it made it easy to understand!!) will count for very little towards your grade.
35 HIS 111Essays (online) vs. Research Paper (seated) … same rubric
37 “I have to pull teeth to get them to post even when they get a score” Conclusions“I got rid of [the discussion board]…it’s a pain in the ass.”“I have to pull teeth to get them to post even when they get a score”20-30% of entire grade is Db…“but it’s not really a discussion board… it’s more of an assignment tool”
38 “Participation is reserved for those on the bubble.” “There’s nothing like a good old test. It averages. Students can’t argue the grade. They’re either right or wrong.”
39 Title III Director Carteret Community College firstname.lastname@example.org Donald StaubTitle III Director Carteret Community College