Systems Realization Laboratory Criteria for evaluating uncertainty representations ASME DETC/CIE 2006 Philadelphia, PA Workshop on Uncertainty Representation in Robust and Reliability- Based Design September 10, 2006 Jason Matthew Aughenbaugh

2 Outline: basic questions discussed What is an “uncertainty representation”? What requirements must an uncertainty representation meet to be valid? How can uncertainty representations be compared?

3 Basic definition “Uncertainty representation” is used here as an all encompassing term ▪Underlying model of uncertainty ▪Definition of mathematical operations ▪Computational implementations ▪Associated decision rules Conjecture ▪Uncertainty representations are merely models of a decision-maker’s information state ▪As such, there is no absolute validation of a single “true” representation, only relative validation

4 Basic criteria for comparing uncertainty representation in engineering design How easy to implement in practice? Can it be applied consistently? Does it help lead to better designs? ▪Better = ??? More robust? More reliable? Better optima? ▪How measure quality external to the formalism? e.g.: Robustness = f(x,y), then f*(x,y) clearly the most robust

5 Fundamental requirements for theories of uncertainty A mathematical representation of that uncertainty ▪e.g., the axioms of probability A calculus for manipulating that uncertainty ▪e.g., P(A or B) = P(A)+P(B)-P(A and B) A meaningful way of measuring the uncertainty ▪e.g., P(A)=(time A occurred)/(total events) Mature methodological aspects of the theory ▪including procedures for making the various uncertainty principles operational within the theory In design, also need a method for decision making Klir, G. J. and R. M. Smith (2001). "On measuring uncertainty and uncertainty-based information: recent developments." Annals of Mathematics and Artificial Intelligence : ? ? Adapted from (Klir and Smith 2001)

6 Measurement example: a meter What is a “meter”? Is there ambiguity? What are the formal definitions? ▪the length of pendulum with period of 2 seconds ▪one ten-millionth of the length of the earth’s meridian along one-fourth the polar circumference ▪a particular platinum metre bar placed in the National Archives in France ▪the length of the path traveled by a particular wavelength of light in vacuum during a time interval of 1/299,792,458 of a second So what is a meter? …It depends on the def’n

7 Operational definitions in science How can we define quantities? Bridgeman writes the following: ▪“We evidently know what we mean by length if we can tell what the length of any and every object is....” ▪“To find the length of an object, we have to perform certain physical operations.” ▪The concept of length is therefore fixed when the operations by which length is measured are fixed… Bridgeman, P. W. (1927). Logic of modern physics. New York: The Macmillan Company.

8 Operational definitions in uncertainty modeling Example (adapted from Cooke) ▪I ask Dr. Paredis, “What is the fadizzle that it rains tomorrow?” ▪He answers: “First tell me what a fadizzle is.” ▪Right answer. But instead of telling him, I said: “Well, just use your own idea of what you think a fadizzle is, and tell what the fadizzle that it rains tomorrow is.” An uncertainty representation must include a clearly defined procedure for measuring uncertainty Can the outcome of this procedure be dependent on the individual? Cooke, R. (2004). "The anatomy of the squizzel - the role of operational definitions in representing uncertainty." Reliability Engineering & System Safety : 313.

9 The role of the observer in measurement Example: relativity ▪Observations of length and time depend on state of the observer Two people observing the same object can measure different lengths However, two observers in the same state will measure the same length Can measurements of uncertainty be based on the state of the observer? e.g., on his or her preferences and beliefs? I believe “yes”, but there is still some debate ▫(e.g., frequentist versus subjective/personalist probabilities) Still requires a defined procedure for eliciting uncertainty measurements

10 Comparing uncertainty representations Why compare them? ▪To see which one is the one to use in engineering design How compare them? ▪Start with theoretical criteria ▪Proceed to practical criteria

11 Theoretical comparisons Does each meet the main requirements? ▪Do any meet all of the main requirements? What is uncertainty? ▪Do we even know what we are trying to represent? an inherent property of the natural universe? a psychological state in the brain? a purely philosophical construct? ▪Is there only one true uncertainty? Some theoretical considerations may be unanswerable

12 Practical comparisons How are they used in practice? Which one works better? ▪Easier to measure? ▪Faster to compute with? ▪Lead to better design decisions? Practical analysis ▪Empirical results ▪Theoretical performance bounds Cost Benefit

13 Summary: Questions to ask when evaluating uncertainty formalisms What is X? How does one represent uncertainty in X? How does X support decision making in design? ▪How does one measure/elicit uncertain information in X? ▪What is the inference formalism in X? ▪How does one implement such inference numerically? ▪How does one make decisions based on uncertain information represented in X? What are the advantages and limitations of X? For which design scenarios is X most appropriate? For formalism “X” (a model and its associated methods)

14 Thank you for your attention Acknowledgements ▪Office of Naval Research Contract No. N G ▪NSF Graduate Research Fellowship Questions?