[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: hashing

• To: XML Developers List <xml-dev@l...>
• Subject: Re: hashing
• From: David Megginson <dmeggin@a...>
• Date: Thu, 29 Apr 2004 16:26:43 -0400
• In-reply-to: <20040429195817.E62427@a...>
• References: <20040429195817.E62427@a...>
• User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.6) Gecko/20040413 Debian/1.6-5

Eric Hanson wrote:

> I have a large collection of XML documents, and want to find and
> group any duplicates.  The obvious but slow way of doing this is
> to just compare them all to each other.  Is there a better
> approach?

You figured it out in the next paragraph.

> Particularly, is there any APIs or standards for "hashing" a
> document so that duplicates could be identified in a similar way
> to what you'd do with a hash table?

The quick-and-dirty approach would be to run all of them through an XML 
normalizer (such as any SAX parser hooked up to XML-Writer).  You could add 
some SAX filters in-between the parser and the writer to tweak the 
normalization for any business rules, such as ignoring certain attributes, 
supplying default values, case insensitivity, numerical precision, etc.  If 
you don't want to play with SAX, you can do something similar with an XSLT 
transformation.

Next, create a list of all the normalized copies, sorted by file size or 
hash, and run Unix cmp or the Windows equivalent on any files with the same 
byte size or hash code.

If this isn't a one-off, then it would not be too hard to write your own 
hash generator based on SAX events, again, applying local business rules as 
appropriate.


All the best,


David

• Follow-Ups:
  • Re: hashing
    • From: Rich Salz <rsalz@d...>
  • Re: hashing
    • From: Tim Bray <tbray@t...>

• References:
  • hashing
    • From: Eric Hanson <eric@a...>

• Prev by Date: WAY OFFTOPIC: ( RE: ISO and the Standards Golden Hammer (was Re: [xml-d ev] You call that a standard?))
• Next by Date: Re: hashing
• Previous by thread: hashing
• Next by thread: Re: hashing
• Index(es):
  • Date
  • Thread