XML Editor
Sign up for a WebBoard account Sign Up Keyword Search Search More Options... Options
Chat Rooms Chat Help Help News News Log in to WebBoard Log in Not Logged in
Show tree view Topic
Go to previous topicPrev TopicGo to next topicNext Topic
Postnext
Neal  WaltersSubject: XQuery on all XML Files in a Directory?
Author: Neal Walters
Date: 11 Mar 2005 03:23 PM
Does XQuery have any support to read all files in a directory
(or a mask such as ABC*.xml)
and "peel off" one or two XML elements from each file?

So for example, I have a directory with 300 XML orders in it.
I want to create a unique list of credit card types, or create
a single XML file that contains for example order number
and order amount (for all orders in that directory).

Thanks in advance,
Neal Walters
http://XML-Online-Training.com
http://Biztalk-Training.com
http://VBScript-Training.com

Postnext
Ivan PedruzziSubject: XQuery on all XML Files in a Directory?
Author: Ivan Pedruzzi
Date: 14 Mar 2005 11:25 AM
Neal



XQuery doesn't expose direct access to file system.

There are differen approach to solve the problem:


1) define an external variable and let caller fill it with
a sequence of strings that represent the file paths

2) Assuming the file have some name + postfix index like
books001.xml, books002.xml, etc you could write

for $row in 1 to 4
return doc(concat('books00', xs:string($row), '.xml'))

3) use an extension function; here a quick example that concatenates
all .xml files in the simpleMappings directory

---------------------------------------------
FileUtils.java

import java.net.*;
import java.io.*;

public class FileUtils
{
public static String listFiles(String root) throws URISyntaxException
{
File f = new File(new URI(root));
String[] list = f.list();
StringBuffer sb = new StringBuffer();
for(int i = 0; i < list.length ; i++){
if(sb.length()>0)
sb.append(",");
sb.append(list[i]);
}
return sb.toString();
}
}

---------------------------------
FileDir.xquery

declare namespace ext = "FileUtils";
declare function ext:listFiles($path as xs:string) as xs:string* external;
declare variable $root as xs:string := "file:///c:/Program%20Files/Stylus%20Studio%20XML%20Professional%20Edition/examples/simpleMappings";
<root>
{
for $file in fn:tokenize(ext:listFiles($root), ",")[fn:matches(., ".xml")]
return
<file>{doc(fn:concat($root, "/", $file)) } </file>
}
</root>

Hope this helps
Ivan

Postnext
Neal  WaltersSubject: XQuery on all XML Files in a Directory?
Author: Neal Walters
Date: 14 Mar 2005 12:25 PM
Very cool!

Do external functions only work with Java?
We are more of a Microsoft shop.
What about VB/Script and/or .NET?

Thanks,
Neal Walters

Postnext
Ivan PedruzziSubject: XQuery on all XML Files in a Directory?
Author: Ivan Pedruzzi
Date: 14 Mar 2005 10:37 PM

Neal,

Unfortunately Microsoft is not going to ship XQuery API (client side) in the upcoming .NET Framework :(
So I can't suggest an XQuery solution but...

If you are willing to use XSLT (and the .NET processor) you could try this


<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:msxsl="urn:schemas-microsoft-com:xslt"
xmlns:u="urn:my-scripts"
exclude-result-prefixes="u msxsl">

<xsl:output method="text"/>

<xsl:param name="root" select="'c:\Program Files\Stylus Studio XML Professional Edition\examples\simpleMappings'"/>

<msxsl:script language="C#" implements-prefix="u"><![CDATA[
public string Dir(string path, string pattern)
{
StringBuilder sb = new StringBuilder();
foreach (String s in System.IO.Directory.GetFiles(path, pattern)){
if( sb.Length>0)
sb.Append(",");
sb.Append(s);
}
return sb.ToString();
}

public int FilesCount(String files)
{
return files.Split(',').Length;
}

public string GetFile(String files, int pos)
{
return files.Split(',')[pos];
}
]]>
</msxsl:script>

<xsl:template match="/">
<xsl:variable name="files" select="u:Dir($root, '*.xml')"/>
<xsl:call-template name="listFiles">
<xsl:with-param name="files" select="$files"/>
<xsl:with-param name="index" select="u:FilesCount($files)-1"/>
</xsl:call-template>
</xsl:template>

<xsl:template name="listFiles">
<xsl:param name="files"/>
<xsl:param name="index"/>

<xsl:value-of select="u:GetFile($files, $index)"/><xsl:text>
</xsl:text>
<xsl:if test="$index > 0">
<xsl:call-template name="listFiles">
<xsl:with-param name="files" select="$files"/>
<xsl:with-param name="index" select="$index - 1"/>
</xsl:call-template>
</xsl:if>
</xsl:template>

</xsl:stylesheet>


Hope this helps
Ivan

Postnext
Neal  WaltersSubject: XQuery on all XML Files in a Directory?
Author: Neal Walters
Date: 15 Mar 2005 02:24 PM
I really want to use Xquery, not XSLT for it's join features.

Although I'm a SCJP (Sun Cert. Java Programmer),
I haven't used Java much in about 2 years.
I just spent an hour or two reinstalling the SDK on my laptop which
I have had for almost a year without touching Java.
I would like to share this XQuery with other people, but I the Java
install might be a "kill-joy".

I kind of like your "external variable" idea.
I hope to play with that later. I could run a VBScript
which could build the value of the external variable, then
copy/paste that value. The only problem is that this would be
a manual step that would have to be done any time new files are
added to the directory (and if the directory had 2 or 3 thousand
files in it, that would be a rather large variable value).

Here's some variations of your code that I are working for me:

declare namespace ext = "FileUtils";
declare function ext:listFiles($path as xs:string) as xs:string* external;
declare variable $dirname as xs:string := "file:///c:/Documents%20and%20Settings/nwalters/My%20Documents/Altura/TestDir/";
<root>
{
for $filename in fn:tokenize(ext:listFiles($dirname), ",")
[fn:matches(., ".xml")]
return
<filename>{$filename}</filename>
}
</root>

The above works fine, and displays the filenames.
Your code also worked which displayed the entire doc.
I did change some of your names to make more sense to me.

Below is a fancier version that I then created to
pick off the XML elements/attributes that I was interested
in showing.

declare namespace ext = "FileUtils";
declare function ext:listFiles($path as xs:string) as xs:string* external;
declare variable $dirname as xs:string := "file:///c:/Documents%20and%20Settings/nwalters/My%20Documents/Altura/TestDir/";
<OrderReport>
{
for $filename in fn:tokenize(ext:listFiles($dirname), ",")
[fn:matches(., ".xml")]
, $rootel in doc(fn:concat($dirname, "/", $filename))/CC_TRANSMISSION
return
<filename name='{$filename}'>
<orderNo>{$rootel/CC_ORDER/@ORDER_NO/text()}</orderNo>
<orderAmt>{$rootel/CC_ORDER/TOTALS/TL_TOTAL/text()}</orderAmt>
</filename>
}
</OrderReport>

One of the "gotchas" was that the dirname value must have the %20
instead of blanks. I copied a filename from another place on
Windows, and started getting strange errors.

Thanks so much for getting me started with this!

Neal Walters
http://Biztalk-Training.com
http://XML-Online-Training.com
http://VBScript-Training.com




Postnext
Neal WaltersSubject: XQuery on all XML Files in a Directory?
Author: Neal Walters
Date: 01 Aug 2006 05:27 PM
Hi Ivan,

It's over a year later - and I'm trying this type of code again at another client.

I have found that if the filenames contains braces such as this:
{003D81D1-39CA-4A85-9C63-F5570E9EAA29}.xml
then I get the following error:
Invalid argument to fn:doc function [err: FODC0005]

So is there a replace or something I can do in the Xquery code that will switch left curly to it's hex equivalent? Would that work?

Thanks,
Neal

Postnext
Ivan PedruzziSubject: XQuery on all XML Files in a Directory?
Author: Ivan Pedruzzi
Date: 01 Aug 2006 06:01 PM

Hello Neal,

On which XQuery processor are you planning to deploy?

If you need to execute the query in the IDE you can use Saxon.

Hope this helps
Ivan Pedruzzi
Stylus Studio Team

Postnext
Neal WaltersSubject: XQuery on all XML Files in a Directory?
Author: Neal Walters
Date: 01 Aug 2006 06:21 PM
I'm running in the IDE - just a one-time job or a model for future jobs that developers will run (will not go to production).

I was using Built-In, but just now tried switching to Saxon, but then it has trouble with the Java FileUtils.
Error on line 2 column 66 of file:///c:/myquery.xquery
XPST0003:XQuery syntax error in #...tring) as xs:string* exteranl;#;
Saxon does not allow external functions to be declared.

Also, where can one quickly find a list of all the available functions in Xquery (which are the same as XSLT and Xpath, correct?). Is there a function section in Stylus Studio help, or do I need to look elsewhere.
I have gone to Stylus Studio Doc, Search, fn:Concat for example, and do not find anything.






Postnext
Ivan PedruzziSubject: XQuery on all XML Files in a Directory?
Author: Ivan Pedruzzi
Date: 01 Aug 2006 06:53 PM

Saxon uses a different syntax to bind extension functions

declare namespace fu = "java:FileUtils";
fu:listFiles("file:///c:/temp")


XQuery 1.0 and XPath 2.0 Functions and Operators can be found here
http://www.w3.org/TR/xpath-functions/

if you type in the XQuery editor the letter c you should see the suggestion list, once you select "concat" you get tool-tip prototype

Because concat is also XPath 1.0 is documented in the Stylus Studio online help. See "XPath Functions Quick Reference" or search concat.


Ivan Pedruzzi
Stylus Studio Team

Postnext
Tony LavinioSubject: XQuery on all XML Files in a Directory?
Author: Tony Lavinio
Date: 02 Aug 2006 11:10 AM
If the question is 'how do I escape the special-purpose { and }
characters in XQuery?', then try doubling them.

See the second-to-last paragraph under section 3.7.1
http://www.w3.org/TR/xquery/#id-element-constructor

Postnext
Ivan PedruzziSubject: XQuery on all XML Files in a Directory?
Author: Ivan Pedruzzi
Date: 02 Aug 2006 12:29 PM
Hi Neal,

The doc function takes as argument a URL. If you want to be sure to run on any XQuery implementetations use fn:encode-for-uri to generate a valid URL.

http://www.w3.org/TR/xpath-functions/#func-encode-for-uri

doc(fn:encode-for-uri("{003D81D1-39CA-4A85-9C63-F5570E9EAA29}.xml"))

Hope this helps
Ivan Pedruzzi
Stylus Studio Team

Posttop
Neal WaltersSubject: XQuery on all XML Files in a Directory?
Author: Neal Walters
Date: 02 Aug 2006 03:29 PM
Thanks again for the tip. Here's how I used it:

declare namespace ext = "FileUtils";
declare function ext:listFiles($path as xs:string) as xs:string* external;
declare variable $dirname as xs:string := "file:///c:/xmltest";

<Sample>
{
for $filename in fn:tokenize(ext:listFiles($dirname), ",")
[fn:matches(., ".xml")]
, $rootel in doc(fn:concat($dirname, "/", fn:encode-for-uri($filename)))
return
<filename name='{$filename}'>
<root>{$rootel//*[local-name()='AdminError']/*[local-name()='CorrelationId'][1]}</root>
</filename>
}
</Sample>

 
Go to previous topicPrev TopicGo to next topicNext Topic
Download A Free Trial of Stylus Studio 6 XML Professional Edition Today! Powered by Stylus Studio, the world's leading XML IDE for XML, XSLT, XQuery, XML Schema, DTD, XPath, WSDL, XHTML, SQL/XML, and XML Mapping!  
go

Log In Options

Site Map | Privacy Policy | Terms of Use | Trademarks
Stylus Scoop XML Newsletter:
W3C Member
Stylus Studio® and DataDirect XQuery ™are from DataDirect Technologies, is a registered trademark of Progress Software Corporation, in the U.S. and other countries. © 2004-2016 All Rights Reserved.