Re: ai record support has an error

[Marty Kraimer <mrk@aps.anl.gov>]

Benjamin Franksen wrote:
> 
> Hi Folks,
> 
> it's been quite a few weeks ago when one of my colleagues (Goetz
> Pfeiffer) had a problem with the ai record and another one (Jens Bergl)
> proposed a _very_ simple solution. The problem concerns the record's
> behavior when it does linear conversion from a 32 bit raw value to an
> analogue (double) value.
> 
> The formula
> 
>         val = (rval + roff) * eslo + egul
> 
> is implemented in the ai record support inside the function convert().
> Here, the first thing that happens is
> 
>     val = pai->rval + pai->roff;
> 
> which leads to an overflow (and thus to an incorrect pai->val) if the
> raw value uses all 32 bits and the egul/eguf values are (for example)
> min(long) and max(long). This is because the calculation uses long
> arithmetic. A simple cast _before_ the calculation like
> 
>     val = (double)pai->rval + (double)pai->roff;
> 
> solves the problem (one of the casts would do).
> 
>         Ben


This change will appear in the next release.

Marty Kraimer