Re: XML and databases
On 4/26/07, Arthur S Bridges <Arthur_S_Bridges@p...> wrote: > > I am working on a large data classification dictionary in XML and I am > wondering at what point do I need to use a database as a back end. > > Project Profile: > From a 'table' point of view, I have about a dozen fields and 25,000 > records which we project to grow to 250,000+ > Each record contains resource profiles and descriptions as well as > ownership information which needs to be kept up to date with personnel > changes. > We plan to use XQuery/XSLT as well as C# for access/update > programming > > Has anyone out there had to deal with a project of this size? > > We are planning to run a web service fro Query and update functions. > From your description it's sort of hard to tell what you're doing. We have a large metadata driven system for collecting medical research data which may match up well with your "data classification dictionary"? We use XSLT to customize the data for about 70 different medical protocols each of which needs different presentation and different fields, but using web serviced query and update would have a CRUD profile similar in nature. The back end is Java with EJB. Currently we track what would be about 16,000 colums in a conventional schema and have the equivalent of what would be about 5,000,000 active rows of data in a conventional schema. We expect to row another order of magnitude in size over the next two years and may be starting to manage gene and phenom data in which case we expect perhaps another 2 orders of magnitude growth. We actually use about 30 tables in a highly normalized proprietary EAV type pattern for the actual relational store behind the scenes. We dynamically assemble the presentation based on authorizations and the metadata at run time. For most of the conventional access response times are subsecond though some complex hierarchical searches can run into the 20 second range. Our actual system is highly resource intensive but is seems necessary given that the presentation and schema requirements change daily (new protocols, protocol revisions, etc.) . Bottom, line; I can't see why you would have any problems managing the volumes you're talking about if you design the system properly. -- Peter Hunsberger
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index]
PURCHASE STYLUS STUDIO ONLINE TODAY!
Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced!
Download The World's Best XML IDE!
Accelerate XML development with our award-winning XML IDE - Download a free trial today!
Subscribe in XML format