[XSL-LIST Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message] RE: Regular expression /s whitespace : Which whitespac
> I am using > count(tokenize(lower-case(.),'(\s|[,.!:;])+')[string(.)]) -a > technique I retrieved from the list for counting words. I > have been questioned about the regular expression that is > being used to find white spaces. The content can contain many > kinds of whitespaces and i am being asked to defend using > this expression to find words. Does the saxon 8b > interpretation of this regular expression covers as whitespaces Splitting text into words is potentially much more complicated than this, depending on what you are trying to achieve and what natural languages you are trying to process. (For example, in English, how should hypenated words be handled? What if the hyphen appears just before a newline?) Only you can judge whether this crude approach is good enough to meet your particular requirements. The definition of \s is found in http://www.w3.org/TR/xmlschema-2/#regexs (which XPath 2.0 references normatively): it matches any one of the four characters x09, x0A, x0D, x20. You could consider replacing the second argument of tokenize with '\W+' which matches any sequence of characters classified in Unicode as separators, punctuation, or "other" - loosely, non-word characters. Michael Kay http://www.saxonica.com/
|
PURCHASE STYLUS STUDIO ONLINE TODAY!Purchasing Stylus Studio from our online shop is Easy, Secure and Value Priced! Download The World's Best XML IDE!Accelerate XML development with our award-winning XML IDE - Download a free trial today! Subscribe in XML format
|