Section 3: Description of the Flaw
The flaw as it manifests itself in the CPU is now described. The following characterization statements can be made:
1. On certain input data, the Floating Point Divide Instructions on the Pentium® processor produce inaccurate results.
2. The problem can occur in all three operating precisions (single, double, extended) for the divide instruction. Empirical studies involving over 1 trillion data cases indicate that far fewer failures are found in single precision than in double or extended precision. The remainder function, and those transcendental functions which rely on the divide instruction, also exhibit reduced precision. For the remainder and transcendental instructions, which operate only in extended precision, the problem can only occur in extended precision.
3. The incidence of the problem is independent of the processor rounding modes.
4. Encountering the problem is highly dependent upon the input data. Only certain input data will trigger the problem. It is not straightforward to describe the exact set of input operands on which the problem can get triggered. Hence it is necessary to describe the incidence of occurrence in terms of a probability distribution statistic. Characterization based on two independent methods consistently yields a probability that 1 in 9 billion randomly fed divide or remainder instructions will produce inaccurate results. The fraction of the total input number space that is prone to failure is 1.14 x 10 to the -10th power.
[The first characterization method is analytical and mathematical, and is based upon a Markov
chain analysis of the iterative implementation algorithm. The second method is empirical,
relying upon running billions of random input samples through the instructions under test. Over
1 trillion data points were run for the second method. Both methods correlate well.]
5. The degree of the inaccuracy of the result delivered depends upon the input data and upon the instruction involved.
On the divide instruction, the worst case inaccuracy occurs in the 12th bit position to the right of the binary point of the significand of the result, or in the 4th significant decimal digit. Statistical measurements using over a trillion test points indicate that the inaccuracy is equally likely to manifest itself in bit positions 12 through 52 to the right of the binary point. The likelihood of encountering an inaccuracy in any one bit position is then 1 in every 360 billion randomly fed divides.
6. The problem does not occur on the specific use of the divide instruction to compute the reciprocal of the input operand in single precision.
The cause of the problem traces itself to a few missing entries in a lookup table used in the hardware implementation algorithm for the divide operation. Since this divide operation is used by the Divide,
Remaindering, and certain Transcendental Instructions, an inaccuracy introduced in the operation manifests itself as an inaccuracy in the results generated by these instructions.
This applies to: