[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Avoiding roundoff errors in calculations?

• To: xml-dev@l...
• Subject: Avoiding roundoff errors in calculations?
• From: "Roger L. Costello" <costello@m...>
• Date: Fri, 29 Aug 2003 14:57:10 -0400
• Organization: The MITRE Corporation

Hi Folks,

Suppose that I would like to convert this Length to meters (there are 6
picas per inch):

<Length>
    <value>13</value>
    <unit rdf:resource="#Pica"/>
</Length>

Doing the arithmetic:

13 picas * in/6 picas * 2.54 cm/in * m/100 cm 

    = 33.02/600 m
 
    = 0.055033333333333333333333333333 m (the "3" repeats infinitely)

The problem is that everyone has different precision requirements.  Some
people would be happy with this as the result:

<Length>
    <value>0.055</value>
    <unit rdf:resource="#Meter"/>
</Length>

For other people such a roundoff could be catastrophic.  

Even if I were to return a result that is as precise as my machine can
produce, 

<Length>
    <value>0.055033333333333333333333333333</value>
    <unit rdf:resource="#Meter"/>
</Length>

there is still a roundoff that may be unacceptable to some.

I am considering returning the result in numerator/denominator form, and
leaving it up to the recipient to do the division (and decide what
precision is suitable), i.e.,

<Length>
    <value>
        <numerator>33.02</numerator>
        <denominator>600</denominator>
    </value>
    <unit rdf:resource="#Meter"/>
</Length>

What are your thoughts on this?  What techniques do people use to avoid
introducing roundoff errors?  /Roger

• Follow-Ups:
  • Re: Avoiding roundoff errors in calculations?
    • From: Jonathan Borden <jonathan@o...>
  • Re: Avoiding roundoff errors in calculations?
    • From: Tim Bray <tbray@t...>

• Prev by Date: RE: Linking xml files together
• Next by Date: Mixed Content
• Previous by thread: RE: Linking xml files together
• Next by thread: Re: Avoiding roundoff errors in calculations?
• Index(es):
  • Date
  • Thread