RE: Regular expression /s whitespace : Which whitespac

> I am using 
> count(tokenize(lower-case(.),'(\s|[,.!:;])+')[string(.)]) -a 
> technique I retrieved from the list for counting words. I 
> have been questioned about the regular expression that is 
> being used to find white spaces. The content can contain many 
> kinds of whitespaces and i am being asked to defend using 
> this expression to find words. Does the saxon 8b 
> interpretation of this regular expression covers as whitespaces

Splitting text into words is potentially much more complicated than this,
depending on what you are trying to achieve and what natural languages you
are trying to process. (For example, in English, how should hypenated words
be handled? What if the hyphen appears just before a newline?) Only you can
judge whether this crude approach is good enough to meet your particular
requirements.

The definition of \s is found in http://www.w3.org/TR/xmlschema-2/#regexs
(which XPath 2.0 references normatively): it matches any one of the four
characters x09, x0A, x0D, x20.

You could consider replacing the second argument of tokenize with 

'\W+' 

which matches any sequence of characters classified in Unicode as
separators, punctuation, or "other" - loosely, non-word characters.

Michael Kay
http://www.saxonica.com/