Numeric Accuracy and Precision Jerry Fitzpatrick

Introduction Floating-point data is used extensively in most computer programs The limited accuracy and precision of floating-point data is a source of several problems – Computation errors Comparison errors Misleading user interfaces Many developers are unaware of these problems Even though they know that floating-point values are not exact Precision, in particular, is counter-intuitive to many people Copyright 2009 Software Renovation Corporation. All rights reserved.

Accuracy & Precision Accuracy is the closeness of a value to a standard or known-correct value Precision is the variance or relative exactness of a value Can also be thought of as “resolution” Accuracy and precision often apply to direct measurement But also apply to computations (derived measurements) Copyright 2009 Software Renovation Corporation. All rights reserved.

Accuracy Versus Precision Accuracy and precision are very different quantities But often used the same way in casual conversation High accuracy Low precision High precision Low accuracy Copyright 2009 Software Renovation Corporation. All rights reserved.

Quick Review Floating-point data types permit fractional values to be used In contrast to integer types that only allow whole numbers Floating-point types also support very large and very small values to be represented Many values cannot be represented exactly using floating-point variables. For example: 1/3 = 0.3333333… PI = 3.1415926… Copyright 2009 Software Renovation Corporation. All rights reserved.

Floating-Point Types Most CPUs (and therefore programming languages) use the IEEE 754 floating-point standard – Defines formats of various precisions The most common are single precision and double precision Single precision: C++ or C# float data type 7 significant digits of precision (~0.00001%) Double precision: C++ or C# double data type 15 significant digits of precision (~0.0000000000001%) Significant *decimal* digits. Copyright 2009 Software Renovation Corporation. All rights reserved.

Significant Digits “Significant digits” are the digits of a written value that contribute to its precision Some people call this “significant figures” Strict rules govern the relationship between precision and significant digits Additional rules govern the precision of computations Important – in most cases, the number of significant digits is not the number of digits after the decimal point! Copyright 2009 Software Renovation Corporation. All rights reserved.

Significant Digit Examples 1347 4 significant digits 0.1347 4 significant digits 1347.0 5 significant digits 134700 cannot tell (4, 5 or 6 significant digits) Your turn: 53 25.208 0.00659 53 2 significant digits 25.208 5 significant digits 0.00659 3 significant digits Copyright 2009 Software Renovation Corporation. All rights reserved.

User Interface Issues Scientific notation show the number of significant digits directly. For example: 7.231E+13 has four significant digits Unfortunately, many users dislike scientific notation Instead, they prefer fixed-point numbers that “line up” – Digits and decimal points in columns The same number of digits after the decimal point Numbers arranged in fixed columns create “false precision” The numbers seem more precise than they really are Remember the triceratops story at the beginning of the presentation? It’s an amusing example of false precision. Copyright 2009 Software Renovation Corporation. All rights reserved.

False Precision Consider the following “lined up” (tabular) values: Sig. Digits. 0.00 2? (ambiguous) 10.25 4 55.78 4 112.74 5 466.12 5 2319.55 6 Almost every program I’ve seen introduces false precision in order to make things “look nice”. This is a poor choice if users accept the data at face value, or if they take precision seriously. The precision of the device these values came from is fixed change, but this format makes it seem variable. The ‘2’ in 466.12 and the ‘55’ in 2319.55 are probably meaningless “random” digits. Copyright 2009 Software Renovation Corporation. All rights reserved.

Floating-Point Calculations Arithmetic operations reduce accuracy and precision due to truncation and rounding – A computed result cannot be more accurate than the least accurate input A computed result cannot be more precise than the least precise input Using a higher-precision data type (e.g. double) helps preserve accuracy and precision, but cannot increase it For example, copying a ‘float’ value to a ‘double’ variable does not increase its precision However, it may help preserve the original precision when calculations are performed using it Copyright 2009 Software Renovation Corporation. All rights reserved.

Floating-Point Comparisons Most programming languages user relational operators to compare numbers e.g. C++ or C# operators: == != < <= > >= These operators use the full precision of the data type for comparisons For example, for two values to be equal, the bit pattern of both values must be identical The strictness of these comparisons can create unexpected results Copyright 2009 Software Renovation Corporation. All rights reserved.

Comparison Example What is displayed from this C# code? Hint: 60.1 times 25 is exactly 1502.5 float x = 25.0f, y = 60.1f; float z = x * y; if (z == 1502.5f) Console.WriteLine("equal"); else Console.WriteLine("not equal"); if (x * y == 1502.5f) Answer: equal not equal Changing the data type from float to double would not necessarily change the result. Copyright 2009 Software Renovation Corporation. All rights reserved.

Comparing Values Consistently In most programs, floating-point values should be compared using special functions or methods Precision-aware Used consistently throughout the application These code snippets should not be taken too literally. There are special cases that need to be handled (e.g. infinities). Copyright 2009 Software Renovation Corporation. All rights reserved.

Comparison Class Example class FloatCompare { FloatCompare(ushort sigDigits); bool IsZero(float x); bool IsEqual(float x, float y); bool IsLessThan(float x, float y); bool IsLessOrEqual(float x, float y); bool IsGreaterThan(float x, float y); bool IsGreaterOrEqual(float x, float y); float Minimum(float x, float y); float Maximum(float x, float y); int Compare(float x, float y); } There are, of course, several variations of such a class and its implementation. Copyright 2009 Software Renovation Corporation. All rights reserved.

Comparing Values Correctly Values should be compared using relative, not absolute, differences. For example – abs((x – y ) / x) < 0.001 NOT (x – y) < 0.001 In practice, comparisons must be more complicated To account for special cases such as zero and infinity Unfortunately, programs often use absolute comparisons Due to tradition, the absolute value is often called ‘epsilon’ This approach is convenient, but approximate at best Copyright 2009 Software Renovation Corporation. All rights reserved.

What’s Wrong with Absolute Comparison? Copyright 2009 Software Renovation Corporation. All rights reserved.

What’s Wrong with Absolute Comparison? The fundamental reason is that precision – by definition – is a relative, not absolute, measurement Copyright 2009 Software Renovation Corporation. All rights reserved.

What’s Wrong with Absolute Comparison? The fundamental reason is that precision – by definition – is a relative, not absolute, measurement Example: Compare 1 and 2 using an epsilon of 0.1 Abs(2 – 1) > 0.1, therefore 2 > 1, right? In most cases, 2 is actually greater than 1, but not due to the comparison shown here. Copyright 2009 Software Renovation Corporation. All rights reserved.

What’s Wrong with Absolute Comparison? The fundamental reason is that precision – by definition – is a relative, not absolute, measurement Example: Compare 1 and 2 using an epsilon of 0.1 Abs(2 – 1) > 0.1, therefore 1.5 > 1, right? This seems right intuitively, but – The epsilon value of 0.1 is 10% of 1, yet 5% of 2 5% is twice the precision of 10%! Copyright 2009 Software Renovation Corporation. All rights reserved.

What’s Wrong with Absolute Comparison? The fundamental reason is that precision – by definition – is a relative, not absolute, measurement Example: Compare 1 and 2 using an epsilon of 0.1 Abs(2 – 1) > 0.1, therefore 1.5 > 1, right? This seems right intuitively, but – An epsilon of 0.1 is 10% of 1, yet 5% of 2 5% is twice the precision of 10%! You can’t have two precisions at the same time! Values within 10% of 1 are 0.9 – 1.1 Values with 5% of 1 are 0.95 – 1.05 Copyright 2009 Software Renovation Corporation. All rights reserved.

Conclusions Floating-point values pose problems that many developers are unaware of, so – Avoid floating-point comparisons (or types) when feasible Especially avoid floating-point types in financial applications Good user interface design can reduce or eliminate false precision A thorough understanding of accuracy and precision can help prevent coding errors As a rule, floating-point values should be compared using custom functions, not the built-in relational operators Using floating-point types for financial applications almost always results in needless disaster! Copyright 2009 Software Renovation Corporation. All rights reserved.

Q&A Copyright 2009 Software Renovation Corporation. All rights reserved.