Re: indexing and querying XML (not XQuery)
On Aug 23, 2005, at 12:41 PM, Robert Koberg wrote: > Wolfgang Hoschek wrote: > >> A starting point may be MemoryIndex (http://dsd.lbl.gov/nux/api/ >> index.html), which can be used for Lucene fulltext search over >> comparatively small *transient main memory* XML documents, for >> example as in Nux XQuery. Note, however, that it is not >> straightforward to extend XML fulltext quering over transient >> main memory to XML fulltext quering over huge persistent XML >> document collections; the underlying technology is bound to be >> vastly different wrt. data management, indexing and >> transactional properties, though the high level search API may >> indeed remain indentical or similar. >> > > Cool. (I saw some of your posts on the lucene list). > > Does this require Saxon 8x? The XQuery integration does, the org.apache.lucene.index.memoryindex.* package does not. Actually, I've become a lucene committer recently, so you'll fine org.apache.lucene.index.memoryindex.* in the lucene SVN trunk as well. > > I have done some RAMDirectory searches, but for things like forum > posts that need to deny postings that contain 'bad' words (for a > young girls portal). But I need to search large collections of XML > config and content. > > The worst thing about this is updating the index, but it is not a > showstopper. > > Can you expand more on: > > "the underlying > > technology is bound to be vastly different wrt. data management, > > indexing and transactional properties" > > it would seem similar to me. That is, if the instance XML changes > you have to open the reader, delete the Lucene document, reindex > the new document, add it, optimize and close the reader. > If all you need is to index the document's flat text instead of XML documents with their structure, that's straighforward, yes. But presumably you'll want to combine structured search (e.g. XPath navigation and predicates) with unstructured fulltext search, and now you're in database terrain, wrt. choosing cost effective persistent index data structures and execution plans for a mix of the main expected queries/data types/access patterns/read&write frequency, plus static and/or dynamic query optimization, including materialized view maintenance, transactional updates, etc. All well known problems, now in the context of XML and fulltext search, but without easy solutions nonetheless. Wolfgang.
PURCHASE STYLUS STUDIO ONLINE TODAY!
Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced!
Download The World's Best XML IDE!
Accelerate XML development with our award-winning XML IDE - Download a free trial today!
Subscribe in XML format