[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message] Avoiding roundoff errors in calculations?
Hi Folks, Suppose that I would like to convert this Length to meters (there are 6 picas per inch): <Length> <value>13</value> <unit rdf:resource="#Pica"/> </Length> Doing the arithmetic: 13 picas * in/6 picas * 2.54 cm/in * m/100 cm = 33.02/600 m = 0.055033333333333333333333333333 m (the "3" repeats infinitely) The problem is that everyone has different precision requirements. Some people would be happy with this as the result: <Length> <value>0.055</value> <unit rdf:resource="#Meter"/> </Length> For other people such a roundoff could be catastrophic. Even if I were to return a result that is as precise as my machine can produce, <Length> <value>0.055033333333333333333333333333</value> <unit rdf:resource="#Meter"/> </Length> there is still a roundoff that may be unacceptable to some. I am considering returning the result in numerator/denominator form, and leaving it up to the recipient to do the division (and decide what precision is suitable), i.e., <Length> <value> <numerator>33.02</numerator> <denominator>600</denominator> </value> <unit rdf:resource="#Meter"/> </Length> What are your thoughts on this? What techniques do people use to avoid introducing roundoff errors? /Roger
|
PURCHASE STYLUS STUDIO ONLINE TODAY!Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced! Download The World's Best XML IDE!Accelerate XML development with our award-winning XML IDE - Download a free trial today! Subscribe in XML format
|