1. Dissertation Defense Machine Assisted Grading of Rare Collectibles through the COINS framework By: Rick Bassett July 24, 2003

2. Grading is… Determining the condition (or grade) of an item. This is important as the grade is a major contributing factor in the value of a collectible item. That many expert graders argue that grading is an art and not an exact science….While others take the opposite position. The problem is…

3. General Research Framework This research takes the position that effective grading needs to be both an art and a science. Science (objective) - represents the undistorted views of a grader, which are free of emotion or personal bias that are based on observable phenomena, appraisal and evidence without the distortion of personal feelings or interpretation. Art (subjective) – is the judgment exercised by a grader that may be based on that individual’s personal impressions, visual observations, cognitive abilities, past experiences, feelings and opinions rather than objective external facts

4. Research Goals 1.To develop an automated machine-based technical grading system that produces results which are consistent and reliable enough to be used as a baseline grade for subjective grading by expert graders. 2.To determine if having humans applying their subjective grading opinions to the technical machine-based grade enhances the grading experience above that of human only or machine only.

5. COINS Model Computer-based Objective Interactive Numismatic System The the first component is a machine-based system that performs objective grading and yields a technical grade as a result. The second component of the COINS model utilizes the output from the machine-based component as the baseline grade for subjective human/machine evaluation process.

6. COINS Model Purpose To provide grading experts with a reliable baseline machine generated technical grade as a starting point for grading so that they can then layer on their own subjective interpretations and ultimately arrive at a market grade. COINS is intended to enhance the ability of graders.

7. COINS Model – Part I Objective (Science) – based on hard facts To explore the objective aspects of grading a machine-based grader was developed. This automated grading allows us to submit a scanned GIF image for comparison against a trained database of image representations.

8. COINS Model – Part I Objective (Science) – based on hard facts Using an algorithm known as Histogram Distance Measurement the system returns a result of the image in the database that most closely matches that of the image being evaluated. Technical Grade: The machine-based system is objective consistent as it can focus on the hard facts when evaluating collectibles.

9. COINS Model – Part II Subjective (Art) – based on interpretation plus hard facts To explore the subjective aspects of grading a browser-based grader was developed. Market Grade: This browser-based grading tool allows expert human grading consultants to provide their subjective opinion of what the grade is.

10. Experimentation Two series of experiments were developed to explore the research questions 1.Objective: Machine-based experiments 2.Subjective: Human-Machine based experiments

11. Machine Based Experiments Hypothesis: The larger the trained database that the machine was working from the better the machine would perform. Perform 3 experiments (A,B & C) to test this hypothesis

12. Machine Based Experiments As the number of images in the database increased in these 3 experiments the error rate decreased

13. Machine Based Experiments The hypothesis was tested further in tests D & E Test D – is the most exhaustive & most rigorous of the 5 tests Compared to experiments A, B & C

14. Machine Based Experiments The hypothesis was tested further in tests D & E Test E – Also exhaustive, it compared the coins graded by the 3 rd party grading services Compared to experiments A, B & C

15. Human-Machine Experiments Experts Acceptance Testing #1 Hypothesis: The correlation will be lower than seen between the machine grades and those of the services on the twenty selected coins. Compared to experiments A, B & C The substantive reason is that subjective factors are now brought into active consideration. The statistical reason is that with a greater number of human graders, there is bound to be those with idiosyncratic standards.

16. Human-Machine Experiments Advance Grade Knowledge Testing #2 Hypothesis: Online grading experts who believe that they are provided with a reliable machine-based grade will do better at grading than those graders that are not provided an accurate grade. The substantive reason is that the expert graders should be able to use the grade provided as a reasonable starting baseline or starting point. Lowest Highest

17. Human-Machine Experiments The Importance of Subjective Qualities Testing #3 Hypothesis: Identifying which subjective qualities of importance that are used most by the expert graders will yield less than impressive statistical significance. Expert graders often claim that grading is an art and not a science by referring to certain subjective qualities that are difficult to repeat and to measure on a repeatable basis. The vast majority of graders failed to identify the incorporation of subjective qualities into their decision process

18. Human-Machine Experiments Grader Consensus Testing #4 Hypothesis: Expert grading consultants will have widely diverse grading opinions on the images that they are grading. Informal experience that humans grade inconsistently is supported by the Stujoe grading tests, Coin World tests, Kevin Foley tests on human grading.

19. Human-Machine Experiments Duplicate Grade Testing #5 Hypothesis: Given that it was previously hypothesized that humans provide widely divergent grading opinions it is expected that graders will offer differing grading opinions on coins which they have previously graded. The expectation was that there would be a certain amount of change that occurred in the grading of the duplicates but the hope was that the change, or variance, would not be significant. Lower grade coins generally provide more challenges to graders

20. Human-Machine Experiments Perception of Internet based Grading Test #6 Hypothesis: Expert graders would see the value in grading images over the Internet although they would rather grade from the actual coins. The graders that participated in this test are predisposed to the use of technology as they are on the Internet.