[XSL-LIST Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message] RE: Comparing numbers with different precisions
Hi David, I'm comparing data loaded into a system with the values extracted from that system. Not all the numbers are the same precision, but it would seem that the system is rounding slightly less than XSL treats as double precision. I'm already casting the two values to xs:double to carry out the comparison, but that is still regarding the values as different. The comparision I'd like to make ideally is "Is the value A the same as B if A is rounded to the same precision that B is stated.". I've considered comparing as float, but I think that may not really have sufficient precision. I also note that the xpath function "round" doesn't have any precision associated with it, so I can't just count the decimal places of the shorter value and round the longer to that precision. Yours, Clint. -----Original message----- To: xsl-list@xxxxxxxxxxxxxxxxxxxxxx; CC: Clint Redwood <clint@xxxxxxxxxxxxxxx>; From: David Carlisle <davidc@xxxxxxxxx> Sent: Thu 09-02-2012 09:40 Subject: Re: Comparing numbers with different precisions > On 09/02/2012 09:24, Clint Redwood wrote: > > Hi! > > > > I've a slightly awkward comparison, where I have two numbers of > > differing precisions, and I want to regard them as the same if the > > less preicse is the more precise rounded down to the same precision. > > > > Examples are: 17.166666666666668 ~= 17.1666666666667 and > > 8.333333333333334 ~= 8.33333333333333 > > > > Unfortunately, the precision is not always the same for all the > > numbers I'm trying to compare. > > > > I just wondered if anyone on this list had done anything similar, and > > had a good way of doing it, while I try and figure out a suitable way > > to do it. > > > > Yours, > > > > Clint Redwood. > > How big are your precisions? in particular do you want xpath to treat > then as doubles (which will mean that anything after 16 or so places > will be lost anyway) and you can just use double precision numeric > functions such as round to deal with the rounding, or do you need > arbitrary precision decimals or (equivalently srings0 and expect to do > teh rounding "by hand". > > David > > > > ________________________________________________________________________ > The Numerical Algorithms Group Ltd is a company registered in England > and Wales with company number 1249803. The registered office is: > Wilkinson House, Jordan Hill Road, Oxford OX2 8DR, United Kingdom. > > This e-mail has been scanned for all viruses by Star. The service is > powered by MessageLabs. > ________________________________________________________________________
|
PURCHASE STYLUS STUDIO ONLINE TODAY!Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced! Download The World's Best XML IDE!Accelerate XML development with our award-winning XML IDE - Download a free trial today! Subscribe in XML format
|