1 Optimizing Subjective Results Julia C. O'Neill Karl R. Wursthorn Rohm and Haas Company ASA/ASQ Fall Technical Conference October 18, 2002

2 Outline Problem: Topaz Appearance Solution Techniques: –Factorial Design –Paired Comparisons –Multidimensional Scaling References and Software

3 The Topaz Finish: Objective To develop a formula which will consistently provide the customer the appearance they want; and investigate the effects of formulation and application variables on Topaz appearance. To become 100% supplier of powder coatings to this customer.

4 Problem: Topaz Appearance Does our coating “look right”? –What matters is what the customer sees. The Topaz coating appearance depends on many variables. –Formulation –Application conditions Measurable attributes do not give the whole picture for multi-color finishes. –Gloss and color are “averages”.

5 The Solution Factorial experiments allow the estimation and comparison of many effects. Paired comparisons simplify the task of judging appearance. Multidimensional scaling reveals the underlying dimensions of appearance.

6 Experiment Design Variables: –Grinding equipment (3 types) –Particle size (3 distributions) –Additive (A, B, or none) –Spray gun tip (conical or slot) –Powder charge (60 or 90 kV) –Gun feed (cup or fluid bed) Initial screening design had 16 runs 110 panels sprayed Optimal selection of 20 panels

7 20 Panels Evaluated

8 Appearance Data Collection Discover, rather than impose, dimensions of appearance Do not specify attributes Trained, experienced “eyes” are critical

9 Paired Comparisons When objects to be compared can be judged only subjectively. When differences between objects are small. When comparison of two objects at one time is simpler than comparing many objects. When the comparison between 2 objects should not be influenced by other objects. When we want to uncover “hidden structure” of objects.

10 Paired Comparisons vs. Ranking vs. Rating Ranking is preferred when: –Several objects can easily be compared simultaneously. –Differences between objects are fairly apparent. Rating is preferred when: –Several grades can be distinguished with consensus among judges. –Ratings can be treated like measurements.

11 Instructions to Evaluators Cluster based on similarity: Looking at all 20 panels, form groups of any panels which are indistinguishable –Panels in same group have dissimilarity 0 Select one panel to represent each group Rate each pair of distinct panels 1 to 4: –1 means hardly any difference –4 means very different Rate based on “appearance”, whatever that means to the judge.

12 Dissimilarity Data Collection Number of pairs is: For 20 objects, 190 pairs. Similar to a Balanced Incomplete Block design with block size 2 Modification of all possible pairs of panels –Sort first into indistinguishable groups Can be optimized for order of presentation Can be fractionated

13 Dissimilarity Ratings – averages from 3 evaluators

14 Mapping Example of Multidimensional Scaling Start with distances between objects (cities). Given only the distances, produce the map.

15 U. S. Map Reconstruction

16 U. S. Map Reconstruction

17 Distances among Objects No meaning when i=j Same distance from i to j as from j to i.

18 Dissimilarities among Objects No meaning when i=j No effective difference for ij versus ji

19 Motivating Concept of MDS Dissimilarities behave like distances.  ij should correspond to d ij. Configuration should place similar objects near each other.

20 Advantages of MDS Uncover hidden structure of data Reduce to a few dimensions –Compare objects at opposite ends –Examine physical configuration of objects

21 Common applications: Psychologists –Perception of speech and musical tones –Perception of colors and faces Anthropologists –Comparing different cultural groups Marketing researchers –Consumer reactions to products

22 Multidimensional Scaling for Topaz Dissimilarity is analogous to distance. Given only the dissimilarities, produce the configuration. If configuration is good, unlike objects will be far apart. More complicated than distances: –Noise in the data –Number of dimensions is unknown

23 Types of MDS: d = f(  ) Metric –d = b  –d = a + b  Nonmetric –ordinal relationship, increasing or decreasing Choice of function has little effect on configuration

24 Computational Approach Define an objective function (badness-of- fit). –Stress is sum of squared discrepancies divided by a scaling factor. Specify f. Find the best configuration X to minimize the objective function.

25 Determining Dimensionality Useful –People can focus on only a few attributes –Difficult to interpret or display more than 3 Rule of thumb –R  (I-1)/4 Statistical –range of reasonable dimensions

26 Three-Way MDS Can use average dissimilarities in two-way MDS. ALSCAL/PROXSCAL in SAS, SPSS, make full use of information from multiple judges For Topaz data, judges were very consistent, not much difference with 3-way MDS.

27 The Configuration

28 Shepard Diagram

29 Interpretation Examine the Configuration! Dimensional Approach –Regress other variables on coordinates Attends most to large distances Neighborhood Approach –Shared characteristics in the same region Focus is on small distances Eclectic Approach –Use any means at your disposal to understand as much as possible about the configuration

30 Examining the Configuration Glossy, Fine texture Lighter, More metallic Darker, Less metallic Finer Coarser

31 Relating Effects to Dimensions Lighter, More metallic Darker, Less metallic

32 Relating Effects to Dimensions Finer Coarser

33 Hierarchical Clustering – 2D

34 Hierarchical Clustering – 3D

35 Gun Tip Effect

36 Charge Effect

37 Additive Effect

38 Equipment Effect

39 Particle Size Effect

40 Gun Type Effect

41 Topaz Results Created a formulation the customer accepted. Discovered that gun tip is critical to the appearance of this formula. Identified variables to consider when trouble-shooting complaints.

42 Summary Factorial experiments allow the estimation and comparison of many effects. Paired comparisons simplify the task of judging appearance. Multidimensional scaling reveals underlying dimensions of appearance.

43 References DAVID, HERBERT A. (1988) “The Method of Paired Comparisons.” London: Charles Griffin & Company Limited. KRUSKAL, JOSEPH B. and WISH, MYRON (1978) “Multidimensional Scaling.” Sage University Paper series on Quantitative Applications in the Social Sciences, Newbury Park and London: Sage Pubns. VENABLES, W. N. and RIPLEY, B. D. (1994) “Modern Applied Statistics with S-Plus.” New York: Springer.

44 Software SPSS SAS S-Plus –MASS library from Venables & Ripley