1. Meaningful Assessment for Meaningful Learning
Rose M. Marra – Associate Professor
5/26/2018

2. What is Assessment?
Process of gathering data to determine the extent to which a person’s performance or a product or program has met its intended objectives.
Focus on objectives.
Focus on data that provide information about achievement of objectives.
Issues:
Validity, reliability
Implementing meaningful assessments

3. If we must assess  make it meaningful!
Why Assess? Benefits
Assessment as learning improvement tool.
Assessment as providing formative improvement data and information.
We are required to assess.
If we must assess  make it meaningful!
“High Stakes” testing

4. Meaningful Assessment
Can only occur with meaningful learning.
Is task specific.
Should occur in multiple forms.
Provides feedback to learners.

5. Meaningful Assessment
Can only occur with meaningful learning
Is writing a research paper a meaningful learning or assessment activity?
Is changing a spark plug meaningful?
Meaningful learning will depend on the context
What kind of students?
What do we want them to do?

6. Meaningful Assessment
Can only occur with meaningful learning
If the learning is rote or at a lower cognitive level, it makes no sense to assess knowledge at a higher level.
Align assessment with learning objectives.

7. Meaningful Assessment
What task do we want learners to be able to do?
Type of thinking (e.g. analysis)
Skill (e.g. operate a loom)
Chose strategies that provide supported opportunities to complete that task.
Assess that task – not something else.
Is Task Specific

8. Meaningful Assessment
Provides Feedback to Learners
Feedback is a primary purpose for assessment.
It allows learners to improve their performance.
Yet we often assess without providing feedback
For example
A numerical score with no explanation or annotations
Standardized tests where results are never returned

9. Rubrics
A rubric is a code, or a set of codes, designed to govern action.
A tool represented as a set of scales used to be used for assessing a complex performance.
Often used as a part of performance assessment activities.
Provides a means for systemically assessing the degree that certain criteria are demonstrated in the product or process.
In essence, a rubric helps the scorer consistently apply a set of valid criteria to the product or the process.

10. Characteristic of Effective Rubrics*
All important elements are included.
Each element is unidimensional – it can’t be sub-divided further.
Voice quality versus volume and intonation
Ratings are distinct, comprehensive and descriptive.
Using a similar scale for all elements is problematic
Communicates well to learners, parents, etc.
Avoids over-summarizing into a single “score”.
* Jonassen (2000) Mindtools for Schools, 2nd. Edition.

11. Meaningful Assessment
Provides Feedback to Learners
A well-designed rubric provides learner feedback.
Beginning
Developing
Accomplished
WebQuest -- Overall visual appeal
There are few or no graphic elements. No variation in layout or typography.
Graphic elements sometimes, but not always, contribute to the understanding of concepts, ideas and relationships. There is some variation in type size, color, and layout.
Appropriate and thematic graphic elements are used to make visual connections that contribute to the understanding of concepts, ideas and relationships. Differences in type size and/or color are used well and consistently.

12. Rubric Example Element Excellent Unsatisfactory
Concepts are descriptive of content domain (breadth of net)
Map includes all important concepts; describes domain on multiple levels
Map includes most important concepts; describes domain on limited number of levels
Map includes numerous concepts; important concepts missing; describes domain on only one level
Map includes minimum concepts with many important concepts missing
Embeddedness and interconnectedness
All concepts interlinked with several other concepts
Most concepts interlinked with 3 or more other concepts
Several concepts linked to only one or two other concepts
Most concepts linked to only one or two other concepts
Links are descriptive
Links succinctly, accurately describe all relationships
Links are descriptive and valid for most relationships
Some links unclear or vague; some invalid or unclear
Links are vague; show inconsistent relationships
Links are efficient (parsimonious, no more nor less than necessary)
Each link type is distinct from all others, clearly describes relationship; used consistently
Most links are distinct from others; discriminate concepts; present variety of relationships; used fairly consistently
Several links are synonymous; don't discriminate concepts well; don't show a variety of relationships; used inconsistently
Most links synonymous or vaguely describe relationships and aren't distinct from other links

13. Assessing Concept Maps -- Rubrics
Breadth
Number of nodes in the network.
Embeddedness
Ratio of instances to concepts; shows how well integrated the concepts in the domain are.
Centrality of nodes
Centrality is a measure of the importance of concepts in a domain. Number of direct links (concepts linked directly to it) and indirect links (concepts linked to other concepts directly linked to it). Are the most “central” concepts the most important to the map?
Depth (hierarchicalness)
Measured by the levels of nodes represented. Further, is each subordinate concept more specific than concept above?
Concept map rubric criteria (adapted from Jonassen (2000)

14. Assessing Concept Maps -- Rubrics
Link direction
Does the direction of links with arrows convey hierarchical or causal relationship between nodes in propositions (McClure & Bell, 1990)?
Number of links
Links should be used economically. If six different links will describe all of the relationships in the net, then do not use more than six (i.e., don't use three different links that mean the same thing, e.g., "attribute of," "property of," and "characteristic of").
Salience
Number of valid links in map divided by total number of links in map (Hoz, Tomer, & Tamir, 1990)
Deadend links
The number of "dead-end" nodes (linked to only one other concept)

15. Assessing Concept Maps -- Rubrics
Accuracy
Are learners making meaningful connections? Is the text in nodes correct? That is, is the information in the net correct?
This part of rubric will vary based on content domain.

16. Meaningful Assessment
Should Occur in Multiple Forms
No one form of assessment is completely valid
No form is a perfect measure
Multiple measures can help to assess different aspects of an outcome, task or performance

17. Science experiment on the water cycle
Multiple Assessments
Science experiment on the water cycle
Complete “quiz”
Can be multiple choice and measure different types of knowledge
Recall / recognition items
What is evaporation?
process where liquid changes from its liquid state to a gaseous state
Process where gas changes to a solid
Process where solid changes to a liquid
None of the above

18. Multiple Assessments
Quiz – prediction or inference items assess higher order thinking
“Predict” what will happen under these circumstances
Requires students to look at the causal relationships in the situation (one that they have not encountered before) and figure out what will happen based on principles, theories, rules etc. already learned.
Inference - Backwards reasoning: Given an effect, determine (infer) what the cause(s) were

19. Higher order – Prediction- Example
What would happen to the freezing temperature of water if we add NaCl to the water?
The freezing temperature would remain the same.
The freezing temperature would be lower.
The freezing temperature would be higher.
We don’t have enough information to answer this question.

20. Higher Order - Inference Example
Trial
Time (s) 1 5 2 7 3 4
The table shows times required for the same toy car to travel 10 m across an identical section of a floor after it is pushed. The difference in times was probably caused by differences in —
force exerted
surface friction
air resistance
car mass

21. Higher Order - Inference Example
Which conclusion is best supported by these data?
Earth’s surface is composed mostly of silicon and calcium.
Oxygen is closer to Earth’s surface than aluminum.
There is more aluminum than calcium at Earth’s surface.
Silicon is much heavier than iron at Earth’s surface.
Most Abundant Elements at Earth’s Surface
Element
Percent by Mass
Oxygen
46.6
Silicon
27.7
Aluminum
8.13
Iron
5.00
Calcium
3.63

22. Multiple Assessments
Create a concept map to show the relationships between the concepts used in the experiment.

23. Using Concept Maps as Assessment
Assessing Domain knowledge
Graded task
Assign points, etc.
Again, can be individual or group task
Create concept map as reflection tool
Before and after an experiment
Before - predict how concepts are related
After – verify relationships; update

24. Concept Mapping Assessment Methods
Built in statistics provided by some tools -- Semantica
Not provided by Inspiration

26. Concept Mapping Assessment Methods
Built in statistics provided by some tools -- Semantica
Embeddedness
Depth
All provided automatically
Facilitates marking with rubric

27. Assessing Learners’ Concept Maps
Relevant questions:
Are all important concepts represented in the map as nodes? (completeness).
Are the relationships between nodes (links) descriptive? For instance, XX “is related to” YY is a meaningless link. Links should be descriptive and accurate.
How well connected is the map? Try to eliminate orphans or dead-ends.
What are the most important concepts (most interrelated) in the map? Does this align with how the knowledge domain is actually structured?
Also - Student self – assessment

28. Multiple Assessments Define a new experiment to test ________.
Generate an analogy for wiring a circuit.
Task that requires learners to create and design – these are among the highest order thinking activities.

29. Concluding Points Meaningful assessment Grounded in meaningful tasks
Makes good sense for vocational educational students
Can be used to provide evidence of higher level learning
Might take form of traditional or non-traditional assessment methods
Is designed to support and improve student learning
Might require a shift in how one views and implements assessment!

31. Send comments / questions to rmarra@missouri.edu
Thank you!
Send comments / questions to

32. Checklist Example- Keyboarding
Always
Often
Sometimes
Seldom
Never
Keeps feet flat on floor
Centers body behind keyboard (between G & H keys).
Keeps back straight (lower back touching back of chair).
Sits a comfortable distance from the keyboard (hand span from the waist).
Relaxes arms with elbows close to body.
“Hovers” hands over home row keys (no wrists resting on keyboard).
Keeps eyes on copy (screen, text, etc.)
Strikes keys with correct fingers.
After striking a key, returns fingers to home row.
Strikes space bar with thumb of right hand.
Strikes Enter key with little finger of right hand.
Shifts with opposite little finger.

33. “Show Me State”
Missouri –

34. Where is the University of Missouri?

35. Main Message
Use technology-based assessments if they can support higher order thinking ..
Not technology for technology's sake ..

36. Evaluation
Process of determining the worth or value of an instructional, or other type of social program.
It is focused on a particular target.
It involves a judgment.
Based in part, on data collected concerning the target.
Emphasis on determining value or worth.

37. Higher order – Inference - Example
You have just received a shipment of three boxes, each containing one of the isotope sources for the three nuclear thickness gauges described above. Each box has a radioactive material label affixed. The sources all weigh the same amount. The boxes are the same size but have different weights. What is the likely reason for the different box weights?
The sources each emit radiation of different energy, so they each weigh differently because of the different shielding needed.
The sources each emit radiation of different penetrating ability, so they each weigh differently because of the different shielding needed to attenuate the radiation from each source.
The sources each have a different amount of radioactivity, so they each need a different amount/weight of shielding depending on the amount of radioactive material.
The sources each have a different half-life, so they each need different shielding depending on the half-life.

38. Concept Maps Concept mapping is a process of
identifying important concepts,
arranging those concepts spatially,
identifying relationships among those concepts,
and labeling the nature of the relationships among those concepts.
Allow visual, external representation of internal models of knowledge.

39. Technology Based Assessment
Using technology for technology’s sake isn’t enough ..
What are meaningful applications of technology to assessment?
Any assessment – technology based or otherwise – must focus on the ultimate educational goal – student learning.

40. Technology Based Assessment - Overview
Definitions and purposes
Why technology applied to assessment?
Sample technology based assessments:
Rubric generators and banks
Assessing online discussion forums
Assessing with concept maps / semantic networks

41. Technology-Based Assessment of Online Discussion Boards
5/26/2018

42. Focus: The Discussion Board
One type of CMC (Computer Mediated Communication).
The online discussion board is a key instructional strategy in most web-based courses.
It enables many types of learning outcomes.

43. Online Discussion Boards
Provide interactivity and a means of community building to promote inquiry (Garrison et al, 2001).
Analysis of subject content
Questioning
Reflection
Unpacking and challenging assumptions
Note that these are higher order learning activities!
The project-based nature of many of our courses make these interactions essential.

44. Why Focus on Assessment of DBs?
Provide a means for conducting many course activities.
Many course learning outcomes are realized through DBs
Can provide “externalized” display of thinking and learning activities
Similar to a “think aloud” protocol
Feasibility – DB content is easily captured and ready to assess / research!
However, there are needs ….
instructors need a way to assess this method of reaching many course learning outcomes.
Researchers/assessors need to know effectiveness of different types of DBs and DB activities.

45. E Portfolio Rubric Sample
Criteria
Performance Ranges
Selection of artifacts
Artifacts meet defined Artifacts do not
portfolio purpose meet defined purpose
(or artifacts are missing)
X X X X X
Student reflections
Illustrate ability to Reflections are missing, do
Self-evaluate artifacts not demonstrate ability to
Self-evaluate artifacts.
Navigation
Portfolio is easy to navigate;……………………………………………Site navigation is confusing
Finding artifacts is easy or does not work.
X X X X X
Text design
(white space, etc.)
Text layout is clear, easy to read Text layout is sloppy or
& enhances portfolio’s goals Misleading; detracts
from Portfolio purpose.
Writing mechanics
Writing is correct grammatically; Writing contains many grammatical
Clearly communicates to reader Errors, misspellings or incorrect
no spelling error punctuation.
X X X X X

46. Technology-Based DB Assessment Methods
Quantitative analysis of constrained discussion forums
Mapping / Sequential Analysis
These methods sit on the boundary of assessment and research – they can serve both purposes!

47. Limitations / Cautions ..
Using technology for technology’s sake..
Disadvantaging learners who have limited technology access or skills.
The “digital divide”.

48. There’s more ..
Any of the performance assessments embedded in Jonassen-esque learning environments.
E-portfolios
Classroom response systems (AKA “clickers”)
Assessment facilitated by PDAs
Managing and reporting assessment and evaluation data.

49. How can assessing with online test tools impact learning?
Facilitates data gathering, summarizing and thus providing student feedback.
Item feedback to students
Support of forced response items, graphics etc. allows for assessment of higher order thinking… if we generate well-written items.

50. Technology Based Assessment -Concept Mapping

51. Technology Based Assessment
Technology is being applied to many aspects of learning.

52. Technology Based Assessment
And sure, computers are great at counting things, aren’t they?
So they ought to be great for assessment? Right?

53. Supporting Objectives
The Common Theme
Alignment
Alignment
Alignment
Broad Goal
Alignment
Supporting Objectives
BARBARA
Align is the word for today.
To create successful programming--and to maintain focus in implementation, align, align, align.
Evidence
Measures
ALWAYS align outcomes with assessments

54. Meaningful Assessment
May be instantiated as
Authentic assessment
Performance assessment
Traditional assessment

55. Related Terms … Authentic assessment Performance assessments
Occurs in a meaningful context that
goes beyond simple recall and recognition knowledge
Higher order learning outcomes – analysis, synthesis
Performance assessments
learners to demonstrate an approach and strategies for solving a posed problem; usually embedded in a meaningful context.