[PATCH] Fix rounding bug in emulation for double floatoperating

[Liu Yu]

> -----Original Message-----
> From: Kumar Gala [mailto:galak at kernel.crashing.org] 
> Sent: Monday, December 10, 2007 11:01 PM
> To: Liu Yu
> Cc: David Gibson; linuxppc-dev at ozlabs.org
> Subject: Re: [PATCH] Fix rounding bug in emulation for double 
> floatoperating
> 
> 
> On Dec 9, 2007, at 11:25 PM, Liu Yu wrote:
> >> -----Original Message-----
> >> From: David Gibson [mailto:david at gibson.dropbear.id.au]
> >> Sent: Monday, December 10, 2007 12:56 PM
> >> To: Liu Yu
> >> Cc: linuxppc-dev at ozlabs.org
> >> Subject: Re: [PATCH] Fix rounding bug in emulation for double 
> >> floatoperating
> >>
> >>
> >> On Mon, Dec 10, 2007 at 01:00:52PM +0800, Liu Yu wrote:
> >>>
> >>> This patch fixes rounding bug in emulation for double float
> >> operating on PowerPC platform.
> >>>
> >>> When pack double float operand, it need to truncate the
> >> tail due to the limited precision.
> >>> If the truncated part is not zero, the last bit of work bit
> >> (totally 3 bits) need to '|' 1.
> >>>
> >>> This patch is completed in _FP_FRAC_SRS_2(X,N,sz)
> >> (arch/powerpc/math-emu/op-2.h).
> >>> Originally the code leftwards rotates the operand to just 
> keep the 
> >>> truncated part, then check whether it is zero. However, the
> >> number it
> >>> rotates is not correct when N is not smaller than
> >> _FP_W_TYPE_SIZE, and it will cause the work bit '|' 1 in 
> the improper 
> >> case.
> >>>
> >>> This patch fixes this issue.
> >>>
> >>> Signed-off-by: Liu Yu <b13201 at freescale.com>
> >>
> >> Wow someone deciphered the hideous macro hell of the math 
> emulation 
> >> code enough to fix a bug.  I don't suppose you'd care to fix the 
> >> millions of warnings that the math-emu code generates...?
> >
> > Oh, I don't like macro define either. But it's really a bug...
> 
> 
> how did you find this?

Well, when divide the min positive number by 2, you will get the work
bit "101" which is supposed to be "100".
And the wrong result will influent the succedent rounding operation.

In fact, the similar macro _FP_FRAC_SRS_1 and _FP_FRAC_SRS_4 are
correct. This bug just exists in _FP_FRAC_SRS_2.

> 
> - k
>