[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: Avoiding roundoff errors in calculations?

• To: "Roger L. Costello" <costello@m...>
• Subject: Re: Avoiding roundoff errors in calculations?
• From: Jonathan Borden <jonathan@o...>
• Date: Fri, 29 Aug 2003 17:18:18 -0400
• Cc: xml-dev@l...
• In-reply-to: <3F4FA206.5E2367A2@m...>
• References: <3F4FA206.5E2367A2@m...>
• User-agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.5b) Gecko/20030723 Thunderbird/0.1

There are many techniques and which to use depends on the application. 
Lookup:

bigint, bignum, arbitrary precision arithmetic

to be practical, if your measurement error is much greater than the 
possible roundoff error you are ok. if you need to completely avoid 
roundoff error, then you generally convert the problem to integer (e.g. 
bigint) and deal with the radii (plural of radix?) and moduli. then 
there's the lispish way of dealing with this problem, which is to return 
a formulaic expression that can be evaluated by the client to get the 
desired result :-)

Jonathan

Roger L. Costello wrote:

>Hi Folks,
>
>Suppose that I would like to convert this Length to meters (there are 6
>picas per inch):
>
><Length>
>    <value>13</value>
>    <unit rdf:resource="#Pica"/>
></Length>
>
>Doing the arithmetic:
>
>13 picas * in/6 picas * 2.54 cm/in * m/100 cm 
>
>    = 33.02/600 m
> 
>    = 0.055033333333333333333333333333 m (the "3" repeats infinitely)
>
>The problem is that everyone has different precision requirements.  Some
>people would be happy with this as the result:
>
><Length>
>    <value>0.055</value>
>    <unit rdf:resource="#Meter"/>
></Length>
>
>For other people such a roundoff could be catastrophic.  
>
>Even if I were to return a result that is as precise as my machine can
>produce, 
>
><Length>
>    <value>0.055033333333333333333333333333</value>
>    <unit rdf:resource="#Meter"/>
></Length>
>
>there is still a roundoff that may be unacceptable to some.
>
>I am considering returning the result in numerator/denominator form, and
>leaving it up to the recipient to do the division (and decide what
>precision is suitable), i.e.,
>
><Length>
>    <value>
>        <numerator>33.02</numerator>
>        <denominator>600</denominator>
>    </value>
>    <unit rdf:resource="#Meter"/>
></Length>
>
>What are your thoughts on this?  What techniques do people use to avoid
>introducing roundoff errors?  /Roger
>  
>

• Follow-Ups:
  • Re: Avoiding roundoff errors in calculations?
    • From: John Cowan <jcowan@r...>

• References:
  • Avoiding roundoff errors in calculations?
    • From: "Roger L. Costello" <costello@m...>

• Prev by Date: Re: Avoiding roundoff errors in calculations?
• Next by Date: Re: Avoiding roundoff errors in calculations?
• Previous by thread: Re: Avoiding roundoff errors in calculations?
• Next by thread: Re: Avoiding roundoff errors in calculations?
• Index(es):
  • Date
  • Thread