[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

RE: More on Vector Models

• To: 'David Lyon' <david.lyon@c...>
• Subject: RE: More on Vector Models
• From: "Bullard, Claude L (Len)" <len.bullard@i...>
• Date: Wed, 4 May 2005 10:23:03 -0500
• Cc: xml-dev@l...

David, you seemed to be determined to play Bacchus come to 
demand a seat at the table of the gods by disruption.  
It's an immature strategy.  

Yes, VSM is a form of document indexing and classification. 
It uses term frequency to create similarity metrics, typically 
a cosine for the angle between terms which normalizes the 
distance.  There are a LOT of papers you can read freely 
available simply by entering "vector space model" into 
that ever-loving simple box that does such a good job for 
cases where SQL falls on its bum.  No structure == no SQL.

So one last time as clear as I can:

1.  The problem of weakly structured (think RSS) or 
unstructured (think notepad files) data is classification. 

2.  The problem of XML is it requires apriori classification 
that may result in weak structuring or high costs.

3.  The problem of the publish/subscribe model is that it 
invokes problems one and two automatically if a human does 
not intervene.  Notification based systems rely on triggers 
because humans know where to put those.  Humans are expensive 
and make mistakes.  Wyman is right: analyze the query. 

This is the classic pattern identification problem.  Regardless 
of the database system you use, query analysis is required to 
enable matching.  In an unstructured or weakly structured world, 
the information of interest is in the text nodes.  It is like 
having a message system that only contains two fields: call and response.

Vector Space Models and others like use term frequency to establish 
similarity metrics.   These metrics can be used to cluster documents 
with similar content even in the face of polysemy and synonymy.  These 
are relatively old techniques and do require preprocessing but HTML 
was an even older technique as was markup before they were recognized 
by the database community.  

Again, the problem of XML is apriori classification.  Just as HTML was a
leap backwards 
to make forward progress, the publish/subscribe methods, particularly 
where based on weakly tagged message formats such as RSS require another
look 
to the past to bring forward the worst/best of the IR technologies because
the  
formats and models create exactly the same problems.  The database gurus of
fifteen 
years ago did not believe markup was a solution for database integration
issues. 
The markup gurus of today don't believe that geometry is a solution for 
pattern analysis.  The past is not always informative if the environment 
has changed; on the other hand, a proven technique in a new environment 
can work better.  HTML and XML are the proof that for the most part, 
the SGMLers were right and the database experts were wrong.

A day in the library is worth a month in the lab.

len




From: David Lyon [mailto:david.lyon@c...]

ok, well I'm lost. Vectors are a simple mathematic
paradigm. How do they apply to xml? or is it just
a new type of marketing speek?

• Follow-Ups:
  • Re: More on Vector Models
    • From: David Lyon <david.lyon@c...>

• Prev by Date: Re: Jim Gray article on Next Generation Databases
• Next by Date: "Late-Breaking News" Presentations Announced for IDEAlliance XTech 2005
• Previous by thread: Re: More on Vector Models
• Next by thread: Re: More on Vector Models
• Index(es):
  • Date
  • Thread