1. Mike Timms and Cathleen Kennedy University of California, Berkeley
FOSS: Using PADI to Develop an Online Self-Assessment System that Supports Student Science Learning
Mike Timms and Cathleen Kennedy University of California, Berkeley
AERA April 2005

2. Overview of presentation
Interactive self assessment system
Tutoring foundations of the system
Description of the system
How PADI supports the system
We will describe an interactive self assessment system that gives students feedback in the form of hints.
Developed to provide guided practice on the FOSS force and motion problems in which students apply the equation for speed (velocity!)
We will cover:
The tutoring foundations of the system – the philosophy of effective online tutoring we employed
Description of the system – how it looks and how it works
How PADI supports the system – both in the design and the implementation
© Mike Timms, 2005 2

3. Tutoring foundations of the system
Good tutors…
use hints
time and structure their feedback
use a conversational style
Selecting hints…
concrete hints for less able students
abstract hints for more able students
Good Tutors …
use hints – Tutors often offer hints as the opening to their tutorial offering.
time and structure their feedback - give students just enough feedback at the right time to allow him or her to make a correction without giving too much away. With simultaneous errors prioritize on the more significant ones. Tutors give different feedback at different times.
use a conversational style – Use “we” or “I”. For example, “ Why don’t we go back to the previous step of the problem.” “I would start at the top.” (Johnson, 2003).
Selecting hints…
The type of hint that is most beneficial depends on the learner’s cognitive ability.
Concrete hints for less able students –
need help with steps - help with steps along the way to a specific goal is effective in supporting performance throughout the task at hand
Abstract hints for more able students - the abstractness level of the hints provided should be matched to the level of cognitive development of the learner
Function oriented - explanations of how a particular function or principle works is more effective in producing learning that can be transferred beyond the immediate task. More metacognitive.
© Mike Timms, 2005 3

4. Activity Selection Process
Conceptual diagram of the system
FOSS Self-Assessment System
Activity Selection Process
Presentation Process
Computer-based
Learner
Administrator
Task Specification
Repository
The assessment cycle is produced by the interaction of the following four processes:
Activity selection process: The Activity Selection Process is the system responsible for selecting and sequencing items from the Self Assessment System. In this case items are selected using item shells and “realistic values” that are randomly selected for each new problem. Students move through 10 levels of items that relate to progressions in the curriculum unit.
Presentation process: The Presentation Process presents the tasks (items) to the student via computer. It takes details about the task from the task library. It presents the problem statement, realistic values, graphics that make up the task. At certain points, when the learner performs the task, the Presentation Process will capture responses (Work Products) from the learner. These are periodically delivered to Response Processing for evaluation. The presentation of the task is governed by the Task Model, which describes the kinds of material that must be presented, as well as the kinds of Work Products that are expected to be produced.
Response processing: Response Processing performs the first step in the scoring process: It identifies the essential features of the response that provide evidence about the participant's current knowledge, skills, and abilities. These are recorded as a series of Observations that are passed to the next process. Based upon prior evidence of performance, the system provides task level feedback in the form of hints that are tailored to a learner’s ability level.
Summary Scoring Process: The Summary Scoring Process performs the second, or summary, stage in the scoring process: It updates our beliefs about the learner’s knowledge, skills, and abilities based on this evidence. Through a series of “quick check” items at each of 10 levels, the student ability level is monitored and the student and teacher can see a summary feedback report.
Task Level Feedback
Summary
Feedback
Hints
Summary scoring
process
Response processing
Adapted from Mislevy, Almond & Steinberg, 2001
© Mike Timms, 2005 4

5. Presentation interface
Problems were developed according to a common shell. Parts of the shell remain constant across types of problem, but values of variables that are used in the equation are drawn randomly from a database of plausible values.
Problem statement: This is the item shell into which plausible values are plugged to dynamically generate a problem.
Student selects equation to use from a drop down menu of variations of the speed equation. We see only the selected equation here.
Workspace: When student hits the “select equation” button, the equation boxes appear in the work space. Student fills in the values and makes their calculation of the answer.
© Mike Timms, 2005 5

6. Presentation interface
When a student hits “check my work” button, they receive feedback on the problem. It systematically searches for errors and gives feedback that is tailored to their ability level.
This example shows concrete feedback for a lower ability student.
Hints appear as text and also highlighting of the problem areas (in red) combined with pointing out information that needs to be used in the solution (in green).
© Mike Timms, 2005 6

7. FOSS Self-Assessment System
How PADI supports the system
Design Phase
Implementation Phase
Copy of FOSS Task Specs
Realistic values
Hints
PADI Database
FOSS Self-Assessment System
PADI Scoring Engine
PADI
Design System
This shows how the PADI system supported the design of the interactive assessment system and its implementation.
Design Phase:
The PADI Design System was used for the development of task specifications, or shells.
Task Specifications get stored in the PADI database
Implementation Phase:
A copy of the task specifications is stored in the Self-Assessment application's local database, and tasks are filled in dynamically at the time they are delivered to a student using the plausible values and hints as needed.
The self assessment system maintains a database of the student (the student model)
Student scores are sent to the PADI scoring engine after each practice level. When the student completes a set of 3 quick-check items.
The PADI scoring engine supplies ability estimates that the system uses to select appropriate hints for each student. This process repeats as students advance through ten levels of problem solving.
Student Database
© Mike Timms, 2005 7

8. FOSS: Using PADI to Develop an Online Self-Assessment System that Supports Student Science Learning
Mike Timms
Cathleen Kennedy
AERA April 2005