1. Numeric Accuracy and Precision
Jerry Fitzpatrick

2. Introduction
Floating-point data is used extensively in most computer programs
The limited accuracy and precision of floating-point data is a source of several problems –
Computation errors
Comparison errors
Misleading user interfaces
Many developers are unaware of these problems
Even though they know that floating-point values are not exact
Precision, in particular, is counter-intuitive to many people
Copyright 2009 Software Renovation Corporation. All rights reserved.

3. Accuracy & Precision
Accuracy is the closeness of a value to a standard or known-correct value
Precision is the variance or relative exactness of a value
Can also be thought of as “resolution”
Accuracy and precision often apply to direct measurement
But also apply to computations (derived measurements)
Copyright 2009 Software Renovation Corporation. All rights reserved.

4. Accuracy Versus Precision
Accuracy and precision are very different quantities
But often used the same way in casual conversation
High accuracy
Low precision
High precision
Low accuracy
Copyright 2009 Software Renovation Corporation. All rights reserved.

5. Quick Review
Floating-point data types permit fractional values to be used
In contrast to integer types that only allow whole numbers
Floating-point types also support very large and very small values to be represented
Many values cannot be represented exactly using floating-point variables. For example:
1/3 = …
PI = …
Copyright 2009 Software Renovation Corporation. All rights reserved.

6. Floating-Point Types
Most CPUs (and therefore programming languages) use the IEEE 754 floating-point standard –
Defines formats of various precisions
The most common are single precision and double precision
Single precision: C++ or C# float data type
7 significant digits of precision (~ %)
Double precision: C++ or C# double data type
15 significant digits of precision (~ %)
Significant *decimal* digits.
Copyright 2009 Software Renovation Corporation. All rights reserved.

7. Significant Digits
“Significant digits” are the digits of a written value that contribute to its precision
Some people call this “significant figures”
Strict rules govern the relationship between precision and significant digits
Additional rules govern the precision of computations
Important – in most cases, the number of significant digits is not the number of digits after the decimal point!
Copyright 2009 Software Renovation Corporation. All rights reserved.

8. Significant Digit Examples
significant digits
significant digits
significant digits
cannot tell (4, 5 or 6 significant digits)
Your turn: 53
25.208
53 2 significant digits
significant digits
significant digits
Copyright 2009 Software Renovation Corporation. All rights reserved.

9. User Interface Issues
Scientific notation show the number of significant digits directly. For example:
7.231E+13 has four significant digits
Unfortunately, many users dislike scientific notation
Instead, they prefer fixed-point numbers that “line up” –
Digits and decimal points in columns
The same number of digits after the decimal point
Numbers arranged in fixed columns create “false precision”
The numbers seem more precise than they really are
Remember the triceratops story at the beginning of the presentation? It’s an amusing example of false precision.
Copyright 2009 Software Renovation Corporation. All rights reserved.

10. False Precision Consider the following “lined up” (tabular) values:
Sig. Digits.
? (ambiguous)
Almost every program I’ve seen introduces false precision in order to make things “look nice”. This is a poor choice if users accept the data at face value, or if they take precision seriously.
The precision of the device these values came from is fixed change, but this format makes it seem variable. The ‘2’ in and the ‘55’ in are probably meaningless “random” digits.
Copyright 2009 Software Renovation Corporation. All rights reserved.

11. Floating-Point Calculations
Arithmetic operations reduce accuracy and precision due to truncation and rounding –
A computed result cannot be more accurate than the least accurate input
A computed result cannot be more precise than the least precise input
Using a higher-precision data type (e.g. double) helps preserve accuracy and precision, but cannot increase it
For example, copying a ‘float’ value to a ‘double’ variable does not increase its precision
However, it may help preserve the original precision when calculations are performed using it
Copyright 2009 Software Renovation Corporation. All rights reserved.

12. Floating-Point Comparisons
Most programming languages user relational operators to compare numbers
e.g. C++ or C# operators: == != < <= > >=
These operators use the full precision of the data type for comparisons
For example, for two values to be equal, the bit pattern of both values must be identical
The strictness of these comparisons can create unexpected results
Copyright 2009 Software Renovation Corporation. All rights reserved.

13. Comparison Example What is displayed from this C# code?
Hint: times 25 is exactly
float x = 25.0f, y = 60.1f;
float z = x * y;
if (z == f)
Console.WriteLine("equal");
else
Console.WriteLine("not equal");
if (x * y == f)
Answer:
equal
not equal
Changing the data type from float to double would not necessarily change the result.
Copyright 2009 Software Renovation Corporation. All rights reserved.

14. Comparing Values Consistently
In most programs, floating-point values should be compared using special functions or methods
Precision-aware
Used consistently throughout the application
These code snippets should not be taken too literally. There are special cases that need to be handled (e.g. infinities).
Copyright 2009 Software Renovation Corporation. All rights reserved.

15. Comparison Class Example
class FloatCompare {
FloatCompare(ushort sigDigits);
bool IsZero(float x);
bool IsEqual(float x, float y);
bool IsLessThan(float x, float y);
bool IsLessOrEqual(float x, float y);
bool IsGreaterThan(float x, float y);
bool IsGreaterOrEqual(float x, float y);
float Minimum(float x, float y);
float Maximum(float x, float y);
int Compare(float x, float y); }
There are, of course, several variations of such a class and its implementation.
Copyright 2009 Software Renovation Corporation. All rights reserved.

16. Comparing Values Correctly
Values should be compared using relative, not absolute, differences. For example –
abs((x – y ) / x) < 0.001
NOT (x – y) < 0.001
In practice, comparisons must be more complicated
To account for special cases such as zero and infinity
Unfortunately, programs often use absolute comparisons
Due to tradition, the absolute value is often called ‘epsilon’
This approach is convenient, but approximate at best
Copyright 2009 Software Renovation Corporation. All rights reserved.

17. What’s Wrong with Absolute Comparison?
Copyright 2009 Software Renovation Corporation. All rights reserved.

18. What’s Wrong with Absolute Comparison?
The fundamental reason is that precision – by definition – is a relative, not absolute, measurement
Copyright 2009 Software Renovation Corporation. All rights reserved.

19. What’s Wrong with Absolute Comparison?
The fundamental reason is that precision – by definition – is a relative, not absolute, measurement
Example: Compare 1 and 2 using an epsilon of 0.1
Abs(2 – 1) > 0.1, therefore 2 > 1, right?
In most cases, 2 is actually greater than 1, but not due to the comparison shown here.
Copyright 2009 Software Renovation Corporation. All rights reserved.

20. What’s Wrong with Absolute Comparison?
The fundamental reason is that precision – by definition – is a relative, not absolute, measurement
Example: Compare 1 and 2 using an epsilon of 0.1
Abs(2 – 1) > 0.1, therefore 1.5 > 1, right?
This seems right intuitively, but –
The epsilon value of 0.1 is 10% of 1, yet 5% of 2
5% is twice the precision of 10%!
Copyright 2009 Software Renovation Corporation. All rights reserved.

21. What’s Wrong with Absolute Comparison?
The fundamental reason is that precision – by definition – is a relative, not absolute, measurement
Example: Compare 1 and 2 using an epsilon of 0.1
Abs(2 – 1) > 0.1, therefore 1.5 > 1, right?
This seems right intuitively, but –
An epsilon of 0.1 is 10% of 1, yet 5% of 2
5% is twice the precision of 10%!
You can’t have two precisions at the same time!
Values within 10% of 1 are 0.9 – 1.1
Values with 5% of 1 are 0.95 – 1.05
Copyright 2009 Software Renovation Corporation. All rights reserved.

22. Conclusions
Floating-point values pose problems that many developers are unaware of, so –
Avoid floating-point comparisons (or types) when feasible
Especially avoid floating-point types in financial applications
Good user interface design can reduce or eliminate false precision
A thorough understanding of accuracy and precision can help prevent coding errors
As a rule, floating-point values should be compared using custom functions, not the built-in relational operators
Using floating-point types for financial applications almost always results in needless disaster!
Copyright 2009 Software Renovation Corporation. All rights reserved.

23. Q&A
Copyright 2009 Software Renovation Corporation. All rights reserved.