1. The Model of Hierarchical Complexity and the Testing and Individualization of Instruction Michael Lamport Commons, Ph.D. Assistant Clinical Professor Department of Psychiatry Beth Israel Deaconess Medical Center Harvard Medical School 234 Huron Avenue Cambridge, MA 02138-1328 Telephone (617) 497-5270 Facsimile (617) 491-5270 Cellular (617) 320–0896 Commons@tiac.net http://dareassociation.org/ Presented at The University of Kiel Monday, September 27, 2010, 2pm 1

2. Four Reasons To Study Stages Of Development Using The Model Of Hierarchical Complexity Understand stages of development – Do they develop linearly? Are they equally spaced? – Are there gaps in measured difficulty between stages Can these gaps be quantified Understand to what extent development is related in different domains – Apply here to scientific, mathematical and logical problem solving Understand how to construct a developmental curriculum Assess development accurately and validly – Use to find out what kinds of education work Individualize instruction 2

3. 3 Combined Results Stage scores were regressed on the order of hierarchical complexity of all the items  Algebra  Balance Beam  Infinity  Laundry Stage scores are derived from the Rasch scores r(250) =.956**

4. Four Categories of Change are Possible Developmental Stages o These stages show qualitative differences at different developmental times Periods or Seasons of Life o These refer to socially-designated changes in roles and behavior that are most often age-related Skills and knowledge o These are learned or acquired within a given stage Maturation o Further physical maturation that is driven by a biological clock o Here we will only discuss stages 4

5. Plan for This Presentation General description of the Model of Hierarchical Complexity Present some empirical data Discussion of how to use the Model to study and bring about change in development and learning 5

6. Some Problems With Earlier Stage Proposals The theories were mentalistic and phenomenological – Stages had representations as mental structures These were a logical structure of thought or actions They posited that performance develops evenly across tasks and domains – Except for decaláge Tasks and actions observed in performance were confounded – Because action was how tasks were made understandable (Genetic Epistemology) For this, as well as other reasons, Commons and colleagues proposed the Model of Hierarchical Complexity 6

7. The Model of Hierarchical Complexity (MHC) The MHC is – An enhancement of the model of Bärbel Inhelder and Jean Piaget (1958) – And a simplification The model classifies tasks as to their Order of Hierarchical Complexity Stages are a description of successful performance on tasks of increasingly higher Orders of Hierarchical Complexity 7

8. Model of Hierarchical Complexity The Model is a case of representational theory of measurement – It deals with complexity of tasks and not inferences of states of mind The model classifies tasks as to their hierarchical complexity – This allows for the separation of the independent variables within tasks and the resulting performance on tasks It specifies more stages than Inhelder and Piaget, allowing for finer grained analyses The model is without content or context – As a result it can be applied to all tasks that fall in a sequence 8

9. Model of Hierarchical Complexity The Model of Hierarchical Complexity (MHC) suggests an objective explanation for why stage-like performances are observed A fundamental assumption is that development proceeds across a large number of general sequences of behavior These sequences exist in every domain including – Mathematical – Logical – Scientific – Social Moral Ethical Interpersonal Development on each task series may proceed more or less independently 9

10. The Model of Hierarchical Complexity The Model of Hierarchical Complexity differentiates – Tasks demands from Mental structures Mental operations If one knows the order of hierarchical complexity of a task correctly done – One then knows the stage of performance One needs only to learn one scoring system to score any – Task action – Task demand In any domain In any form – Problems – Narratives 10

11. A task sequence starts with a task made up of simple or elemental behaviors Developmental sequences progress from simpler to more complex tasks Tasks form a hierarchy from simple to more complex When tasks are done correctly, – The behavior of people and animals appears stage-like The Model of Hierarchical Complexity deconstructs tasks into the actions that must be done to successfully complete a task By doing so, it classifies each task by its order of hierarchical complexity 11

12. One Task is More Hierarchically Complex Than Another Task If: It is defined in terms of two or more lower order task actions – This is the same as a set being formed out of elements – This creates the hierarchy A = {a, b} a, b are “lower” than A and compose set A A ≠ {A,...}, A set cannot contain itself It organizes lower order task actions – In simplest terms, this is a relation on actions – The relations are order relations A = (a, b) = {a, {b}} -- an ordered pair This organization is non-arbitrary – This means that there is a match between the model-designated orders and the real world orders Not P(a,b) -- not all permutations are allowed 12

13. Example: Higher Order Action Coordinating Lower Order Actions Order 8: 2 x (3 + 4) = 14 Concrete  Order 7 : (2 x 3) + (2 x 4) = 14 Primary   Order 7: 6 + 8 = 14 Primary The order 8 action coordinates adding and multiplying by non-arbitrarily ordering those actions The distributive action is, therefore, more complex than the acts of adding and multiplying alone

14. Order n + 2 Actions: Defined In Terms Of Order n +1 Actions & Non-arbitrarily Organize Them Order n + 2 Action 1 Order n + 1 Action 1 Order n Action 1 Order n Action 2 Order n + 1 Action 2 Order n Action 3 Order n Action 4 14

15. 15 Orders of Hierarchical Complexity OrderName Complexity 0Calculatory 1Sensory & Motor 2Circular Sensory-motor 3Sensory-motor 4Nominal 5Sentential 6Preoperational 7Primary 8Concrete 9Abstract 10Formal 11Systematic 12Metasystematic 13Paradigmatic 14Crossparadigmatic

16. The Model Of Hierarchical Complexity Explains Stages Of Development The term Order of Complexity characterizes the underlying task – A task analysis allows for specification of this Order, to be used as an independent variable The term Stage refers to the performance observed on that task – Studying the relationship of this dependent measure to the Order of Complexity allows for a new and more powerful account of stage- like behaviors

17. General Stage of Performance Required StageStudents Grade ActionTeaching Level Primary 7Grade 1-3Follow instructions and imitate modeled behavior Teacher’s Aides Concrete 8Grade 4-6Follow a manual and effectively carry out procedures Teachers in early grades of Elementary School K-4 Abstract 9Grade 7-10Carry out the normative teacher behaviors Late Elementary School Grades; and Junior High 5-9 Formal 10Grade 11-16Graph student performance and adjust tasks to fit student performance High School teachers 10-12 Systematic 11Graduate School See multivariate determinants of student performance Four and five-year college professors Metasys- tematic 12 Design an entire educational enterprise that works well such as computer-aided instruction Professors at research universities 17

18. MHC Has Been Used To Study Development In A Variety Of Domains Science and Logical/Mathematical tasks, including: o Balance beam and pendulum (Inhelder & Piaget, 1958; Commons, Goodheart, & Bresette, 1995) o Algebra (Commons et al.) o Combustion (Bernholt, 2010) o Infinity (Commons et al.) o A large number of social/moral and other tasks that may be briefly referred to later in the talk 18

19. Relationship Between Piaget and Commons Notions There are some common elements between Piaget and Commons notions of stage and many more that are different In both one finds: – Higher order actions are defined in terms of lower order actions This forces the hierarchical nature of the relations and makes the higher order tasks include the lower ones – Higher order of complexity actions organize those lower order actions This makes them more powerful than the lower order actions 19

20. What Commons et al. Have Added Higher order of hierarchical complexity actions organize those lower order actions in an non-arbitrary way – This makes it possible for the organization to meet real world requirements, including the empirical and analytic Task demands and task performance are separated All tasks have an order of hierarchical complexity There is only one sequence of orders of hierarchical complexity – Hence, there is structure of the whole for ideal task actions 20

21. What Results From the Model There are always gaps between the orders of hierarchical complexity – It is an ordinal scale The stage of performance is the most hierarchically complex task solved Gaps in Rasch-Scaled Stage of Performance are predicted when – All other sources of difficulty are controlled for Performance stage may be different task area to task area – A structure of the whole is not necessary – Horizontal decaláge for performance – It is not inconsistency in thinking within a developmental stage. – Decaláge is the normal or modal state of affairs 21

22. What Has Been Taken Out Of Piaget, Kohlberg, Etc The theories are mentalistic and phenomenological Tasks and actions are the same because action is how tasks are made understandable (Genetic Epistemology) Stages are only about human thought and action Stages have representations as mental structures There are logical structures of thought or actions Performance is at the same stage across domains and tasks except for decaláge 22

23. Empirical Studies A series of empirical studies will now be introduced These will be in the mathematical and/or scientific domains All studies follow the same general procedure – Devise a related series of tasks that differ only in hierarchical complexity – Administer tasks – Rasch analyze to see if tasks fall on one dimension – See how well the order of hierarchical complexity predicts Rasch Scaled Stage of difficulty How well does the order of hierarchical complexity of a task predict the stage of performance? The Rasch performance scores on items from each series were converted into Stage scores The Stage scores were then regressed against the order of hierarchical complexity of each of the item from a series 23

24. Algebra Task 24

25. Algebra Instrument Sequences Primary: There are behaviors act on natural numbers we call simple arithmetic operations  Concrete: Behaviors that order the simple arithmetic behaviors when multiplying a sum by a number. Such distributive behaviors require the simple arithmetic behavior as a prerequisite, not just a precursor  Abstract: Forming classes based on abstract feature. In this case the abstract feature is a variable 25

26. Algebra Instrument Sequences  Formal: Just one relationship between variables is to be found. One equation, one unknown. Non-related variables may be present  Systematic: Two or more relationships between variables or two or more variable leading to a solution – Two equations in two unknowns  Metasystematic: Two or more relationships between systems which forms a metasystem 26

27. Rasch Analysis of the Data Uses logistic regression to jointly o Minimize the errors in person scores o Minimize the errors in items’ scores Converts raw scores into equal-interval, linear scales o Producing an objective, scale The Rasch model produces additive, one dimensional linear measures that are – Item-free (item-distribution-free) – Person-free (person-distribution-free) This means that the measures are statistically equivalent – For the items regardless of which persons (from the same population) are analyzed, and – For the people regardless of which items (from the same set) are analyzed The item score represents how difficult the item was The person scores represent how good a person was at dealing with the item difficulty 27

28. Order of Hierarchical Complexity & Rasch Scaled Scores The measure of hierarchical complexity of a task item is analytically represented by its Order of Hierarchical Complexity The measure of stage of performance on that task item is empirically represented by the Rasch Scaled Score 28

29. Algebra Rasch Map Participants: 223 (79.5%) men 58 (20.6%) women Ages from 14 to 81 (M = 24.06, S.D. = 8.60) Education varied from high school to graduate school (M = 3.85, S.D. = 1.01) Higher Stage Lower Stage Rasch person scores on the left- hand side Rasch items scores on the right- hand side The primary items at the bottom The metasystematic items at the top All of the other stage are seen to be in the correct order One can see gaps between the stages 29

30. 30 The Analysis The higher a person's performance relative to the difficulty of an item, the higher the probability of a correct response on that item When a person's location on the latent trait is equal to the difficulty of the item, by definition there is a 0.5 probability of a correct response The Rasch scores for items and participants were converted into stage scores by using the formula:  Stage scores represent the stage of a person performance or item performance according to the Model of Hierarchical Complexity

31. Algebra Regression r(39) =.953** 31

32. Isolation of Variables Problems 32

33. Some History of the Isolation of Variables Problems Inhelder and Piaget (1958) developed the pendulum and chemicals tasks – Participants had to perform an experiment by manipulating a single variable while holding all other variables constant – They had to figure out which variable controlled the rate that a pendulum weight would cross the low point The analogous plant problem was created by Kuhn and Brannock (1977) – They felt it offered greater external validity – They argued that individuals were more likely to encounter isolation of variables problems as already constructed “natural experiments” In the plant problem, they were presented with observations that differed in terms of – Which values of variables were present e.g. Amount of water – What outcomes resulted e.g. Healthy plant Participants were required to make inferences based on the information presented 33

34. There was no separation of the independent variable (task characteristics) and performance – Therefore, it was difficult to explain differences in performance We undertook a series of studies – Based on the basic isolation of variables problem – But instead created a series of subtasks – Each subtask required a given order of hierarchical complexity – They represented a predicted sequence of development in this area 34

35. In this set of studies, we examined three properties that may influence item difficulty: – Order of hierarchical complexity of the items in a task – Task content – Language And the associated country of the participants Which of the three was most important? It is hypothesized that the most important predictor of task difficulty is the order of hierarchical complexity of the items Plan For This Part Of The Presentation 35

36. Method Participants The overall study is made up of data collected from 5 separate convenience samples – Each was tested on one of five similarly constructed isolation of variables problems There were a total of 1263 participants across all studies Here, we will briefly describe – The characteristics of each separate sample – The characteristics of the instrument used with each sample. There were two quasi-independent variables that were not independent of one another – Content – Language 36

37. Subsample Language, Version and Content 1) IraqiArabic Laundry450Paper & Pencil 2) EnglishFirst Revised Laundry 215Various Listservs 3) EnglishShortened Laundry78Various Listservs 4) GermanGerman Combustion459Paper & Pencil 5) EnglishAtheism and Belief61Various Listservs 37

38. Laundry Problem Example: Abstract Order 9 Participants were required to see what the operative variable is Lower order problems gave the solution in the example Higher order problems added ingredients to the mixture – Also make deduction more difficult by including many more possible operative variables – They also included multiple possible relationships among the variables such as And/or scenarios 38

39. Procedure One gets better performance in going from easy to hard (e.g. Aamodt & Mcshane, 1992; Hodson, 2006) Fischer found that such support improved measurement quality Hence, the subtasks were presented in a sequence from easy to hard – Low to high order of hierarchical complexity Successfully completing the easy problems – Served as support for the harder problems – Most of the participants completed lower order problems correctly, if they also completed higher order problems correctly 39

40. Rasch Model The person and item total raw scores used as dependent variables – Sum the item raw scores separately for each order of complexity and then that could be used to correlate with hierarchical complexity The Rasch person reliability of the combined data was.94 The Rasch item reliability was 1.0 – In the context of a Rasch analysis, this means that there is a high probability that items estimated with higher measures do in fact have higher measures than those estimated with lower measures – There is no equivalent traditional measure 40

41. Rasch Analysis of Data from All Studies How well did the order of hierarchical complexity predict performance stage on the items? The results are illustrated using a Rasch variable map If performance on the items were in perfect order, there would be no item reversals – No cases in which a higher order item appears below a lower order item The closer to the top the items are, the more difficult they were to answer Participants had a 50% chance of correctly answering items that are located directly across from them 41

42. Person Rasch scores on the left hand side Items Rasch scores on the right hand side The scaling showed – The primary order items at the bottom – The metasystematic order items at the top All of the other stage are seen to be in the correct order – The exception is the abstract and concrete orders, which are intermixed – One can see gaps between the stages – The mixing of concrete and abstract was due to the concrete tasks having too many ingredients (variables) – These were removed in subsequent versions of the instruments 42

43. Item Stage Score Versus Item Order of Hierarchical Complexity The item stage score shows how the items performed – Items with an order of hierarchical complexity of 10 is expected to have a stage score of 10 This plot shows that this trend is followed r =.898 43

44. Shortened Problems 44

45. Univariate ANOVA An ANOVA showed that the largest contribution was the order of hierarchical complexity of the items – F(5, 398) = 326.83, p =.0005, η (eta) =.804 – A large effect size The effect of content (laundry, atheism & belief, combustion) of the problem was also significant – F(1,398) = 33.405, p <.0005, η (eta) =.077 – A very small effect size There was no main effect of language (country) There were also significant interactions between – Hierarchical complexity of the items and content (laundry, atheism & belief, combustion), F(1,398) = 19.324, p =.0005, η (eta) =.195, a small effect size; – Hierarchical complexity and language, F(1,398) = 4.524, p =.001, η (eta) =.054, an even smaller effect-size) – There was no significant three-way interaction 45

46. ** Relationship Between Content and Item Order of Hierarchical Complexity Items at the primary order 7 should have an item stage score of close to 7 – If the mean item stage score is higher than 7, than the item was more difficult for the participants then predicted 14 out of 17 of the cells contain numbers that are exactly at the predicted value or less than.30 away, 3 cells show more divergence Most strikingly, the metasystematic order 12 task with Atheism and Belief content was exceedingly difficult for participants (Order 12 item stage mean = 16.38) For the concrete and abstract laundry problem items, the concrete items were somewhat more difficult than predicted and the abstract items were easier than predicted (closer to 8 than to 9). There were too many ingredients in the concrete HC789101112 Atheism and Belief7.008.019.0010.0011.0016.38 Combustion7.008.009.0010.0511.00N/A Laundry7.188.348.359.9810.9112.26 Mean Content Item Stage Scores and the Item Order of Hierarchical Complexity of Items 46

47. ** Interaction Between Hierarchical Complexity and Language Again, 14 out of 17 of the possible cells show numbers at exactly the predicted value or less than.30 away One divergence that contributed to the interaction effect was the elevated difficulty of metasystematic order 12 items from instruments given in English (Order 12 item stage M = 13.19). This was due to the Atheism and Belief content There were a few more minor interactions, particularly at orders 8 and 9 for instruments written in both Arabic and English HC789101112 Arabic7.008.568.0010.0011.0012.00 English7.248.168.719.9810.9113.19 German7.008.009.0010.0511.00N/A Mean Language Stage Item Stage Scores and the Item Order of Hierarchical Complexity 47

48. Comparisons Among the Five Tasks The regression showed very strong prediction of performance by the order of hierarchical complexity – The prediction was equally good within each of the five tasks – There were shown to be some variations in the item stage scores obtained, depending upon task type (content) How well did order of hierarchical complexity (OHC) of the items explain difference in the predictability of Rasch item stage scores for each individual instrument – Significant Difference calculator (2009) was used 48

49. Differences Between r Values of Item Rasch Scores and Item Order of Hierarchical Complexity This table shows the difference between each instrument’s r value and whether or not that difference was statistically significant. The only instruments that produced any significant differences in their regression score were t – Atheism and Belief – Combustion instrument – Even though both of these instruments were significantly different from all of the other instruments, the r values were quite low Note. *p <.05 Regression Scores OriginalArabicInitial ShortFinal ShortAtheism and BeliefCombustion OriginalNA0.04.0170.006.159*.149* ArabicNA.013.01.163*.153* Initial ShortNA.023.176*.166* Final ShortNA.153*.143* Atheism and BeliefNA1 CombustionNA 49

50. Discussion Order of hierarchical complexity of the items was an excellent predictor of performance with r(419) =.898 This strongly supports the notion that OHC was by far the strongest component of difficulty The stage-like performances are much better explained by OHC of items – This was previously explained by changes in mental structures – Clearly there has to be learning and maturation that underlie these changes in performances Some of the other variables and some of their interactions did make very small contributions – The mean stage scores of the content. – The interaction between order of hierarchical complexity of the item and the content also was significant but the eta was very small. 50

51. Discussion Language, and therefore country did not have an effect on the overall performance of participants – Because all participants received modern education, the mean stage of development was not as sensitive to culture as previously suggested (Cole, Cole & Lightfoot, 2004) This means that what gives rise to differences in stage of performances on tasks is due to the order of hierarchical complexity of the tasks Then people performing the different tasks exhibit different proclivity in doing so correctly Gaps were found in between all the stages but the abstract and concrete and – Gaps were found in these stages in the combustion and atheism and belief instruments Concrete tasks were modified for these instruments It is not clear to us how to show that these gaps were statistically significant 51

52. What else contributes to task difficulty – Horizontal complexity – Familiarity with the language and symbols – The organization of the information Placed in a given array As there is always some error variance – What could be viewed as a flaw in Piagetian theory is used as a basis for estimating relatively how far apart the stages are on the latent variable Provided there is a useful level of construct relevant noise in the observations, which we observed in our data, Rasch scaling is possible. 52

53. 53 How Related are Performances Within the Mathematics and Science Domains? There were a number of steps to finding the relatedness – First, how well did hierarchical complexity predict stage of performance in each task sequence? – How well did hierarchical complexity predict the Rasch stage of performance of all the items from all the sequences? – How many factors were found to underlie participant stage?

54. Instrument Descriptions Algebra (Mathematics): Arithmetic to comparing multivariate algebraic problems Balance Beam (Physics): Higher and lower order extensions of Inhelder and Piaget (1958) balance beam task Infinity (Mathematics): Problems dealing with concepts of infinity Laundry (Chemistry and Logic): Extension of Inhelder & Piaget (1958) pendulum task with Laundry Content. Detection of causal relationships and properties of causal systems 54

55. Summary of Results In the algebra problem, as well as in related science and math problems, the order of hierarchical complexity strongly predicted the stage of performance There were very large regression coefficients o. 982 – Balance Beam o.964 – Laundry o.953 – Algebra o.912 – Infinity Most of the variance in difficulty of tasks is due to The order of hierarchical complexity of that task 55

56. 56 Overall Rasch Map  This Rasch map illustrates the order of complexity for each item in as pairs of letters on the right for all four instruments  Algebra  Balance Beam  Infinity  Laundry  p primary  c concrete  a abstract  f formal  s systematic  m metasystematic  There are a total of 250 items represented. Higher Stage/order Lower Stage/Order

57. 57 Combined Results Stage scores were regressed on the order of hierarchical complexity of all the items  Balance Beam  Algebra  Infinity  Laundry The Rasch scores have been converted into stage scores r(250) =.956**

58. 58 A Principal Component Analysis on Stage Scores of Person A principal component analysis was performed on all of the person stage scores The analysis showed only one component for the stage scores That component explained 90.513% of the variance

59. 59 Discussion These mathematics and science instruments behaved as though they belonged to a single domain This may explain why Inhelder & Piaget (1958) thought that development was synchronous across tasks and domains – This explains their positing of development of the whole The linear regressions of Stage Scores on the item’s orders of hierarchical complexity had an extremely large correlation

60. 60 Discussion Commons et al. believe that most unevenness in development is due to participant’s – Problems of coding of the information in the problem by the participants The participants may lack Knowledge required by the task A way of seeing what is operative The task sequences in the mathematics and physical science study came precoded What the variables were was clear Vignettes that were given in other domains, such as the social or moral had to be first coded by the participants

61. Implications for Instruction 61

62. Getting The Order Of The Hierarchical Complexity Of Items Right Individualization when combined with reinforcement is by far and way the most effective educational pedagogy MHC promotes designing the correct sequences of tasks This makes possible accurate and valid assessments of progress through the curriculum Such assessment make it easier to individualize instruction – This alone has the largest effect in promoting learning The second largest effect is provided by reinforcement This is especially true of group reinforcement in teams Individual’s points are added together together against other teams 62

63. Environments Where Interventions Could Be Done Education is most useful, particularly if one is interested in developing higher stage behaviors in a domain o Currently, education is highly studied, but mostly without developmental measures o When there are developmental measures, they are gross, such as the Defining Issues Test or the Sentence Completion Test Corporate training is another important arena where interventions are carried out o Currently a lot of the emphasis is on group processes o Attending to the underlying tasks and how they could be better addressed would be helpful as well Therapy is another area of applications o Especially the kinds of therapy that include coaching 63

64. Discussion and Conclusions These and other related data support the idea that task hierarchical complexity explains stage-like performances 64

65. All of these instruments and scoring are available for free at http://dareassociation.org/ We will also work with other researchers and both students and faculty to develop instruments in areas of interest to them – So far we have had over 15 international collaborators – Commons@tiac.net 65

66. Social and Behavioral Tasks 66

67. Kohlberg’s moral interviews (as scored by Armon & Dawson, 1997; Dawson, 2000) Stages of religion and atheism (Commons-Miller, 2010) Views of the “good life” (Danaher, 1994; Dawson, 2000; Lam, 1995) Loevinger’s Sentence Completion task (Cook-Greuter, 1990) Workplace culture (Commons, Krause, Fayer, & Meaney, 1993) Workplace organization (Bowman, 1996a; 1996b) Political development (Sonnert & Commons, 1994) Therapists’ decisions to report patient’s prior crimes (Commons, Lee, Gutheil, Goldman, Rubin, & Appelbaum, 1995) The relationships between more and less powerful persons such as doctors and patients (Commons & Rodriguez, 1990, 1993) & counselors and patients (Commons, et al, 2006) Social and Behavioral Tasks 67

68. 68 Helper-Person Problem Figure 2a. Method Figure 2b. InformFigure 2c. Guide  Participants received 5 vignettes  A set of vignette represented each order of Hierarchical Complexity  Participants were asked to answer questions about how well informed consent was attained in each

69. 69 Politician-Voter Problem Figure 3a. Method Figure 3b. Inform Figure 3c. Vote *** Each data point represents an average of all 3 items at each stage***  Participants received 5 vignettes  A set of vignette represented each order of Hierarchical Complexity  They rated three issues about the interaction between politician and voter  the method  how well each politician informed the voter  how likely the participant would be to vote for the politicians

70. Item Order of Hierarchical Complexity Predicted Item Rasch Scaled Scores Helper-Person r(3) =.967, r(3) =.978, r(3) =.973 Politician-Voter r(3) =.920, r(3) =.895, r(3) =.900 Anti-Incest Reportingr(3) =.898, r(3) =.850, r(3) =.834 Anti Death Penalty r(3) =.854, r(3) =.801 Pro Death Penalty r(3) =.849, r(3) =.758, r(3) =.875 Jesus Stoning Dilemma r(3) =.539, r(3) =.564 The measured difficulty reflects the theoretical difficulty (Order of hierarchical complexity) 70

71. 71 **Overall Best Next Best Figure 9a. Overall Figure 9b. Helper-Person Inform Figure 9c. Politician-Voter Inform

72. Flora, S. R., & Flora, D. B. (1999). Effects of extrinsic reinforcement for reading during childhood on reported reading habits of college of students. The Psychological Report, 49: 3B14. 72