[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: Got a huge XML document?

• From: Hermann Stamm-Wilbrandt <STAMMW@de.ibm.com>
• To: "Costello, Roger L." <costello@mitre.org>
• Date: Fri, 13 Sep 2013 16:01:06 +0200

Hello,

DataPower appliances provide an XSLT 1.0+EXSLT processor.
With new v6.0 firmware XQuery 1.0 and JSONiq processors are available.

The 9004 appliance models provide 4GB of main memory, the 9005 models 96GB.

DataPower provides XSLT streaming processing since more than 8 years.
Of course there are restrictions on streaming processing, see [1] and
slides 7-10 of [2], but no limits on what can be processed.

From posting [3], even 25GB XML files can be processed non-streaming:
...
If service is not streaming, the 500MB on 9004 devices should be good,
sometimes 1GB is possible, on 9005 XI52 I have processed a XML message of
size 25GB with a non-streaming service successfully.
...

[1]
http://pic.dhe.ibm.com/infocenter/wsdatap/v5r0m0/index.jsp?topic=%2Fcom.ibm.dp.xm.doc%2Foptimizingthroughstreaming.htm
[2] http://www-01.ibm.com/support/docview.wss?uid=swg27019118
[3]
https://www.ibm.com/developerworks/community/forums/html/topic?id=50c29113-9de3-4de1-b447-5caf4a390000#fbb7c129-5aeb-43a2-b139-3810adcee7ca


Mit besten Gruessen / Best wishes,

Hermann Stamm-Wilbrandt
Level 3 support for XML Compiler team and Fixpack team lead
WebSphere DataPower SOA Appliances
https://www.ibm.com/developerworks/mydeveloperworks/blogs/HermannSW/
https://twitter.com/HermannSW/     http://www.stamm-wilbrandt.de/ce/
----------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschaeftsfuehrung: Dirk Wittkopp
Sitz der Gesellschaft: Boeblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294


|------------>
| From:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"Costello, Roger L." <costello@mitre.org>                                                                                                         |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To:        |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |"xml-dev@lists.xml.org" <xml-dev@lists.xml.org>,                                                                                                  |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date:      |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  |09/12/2013 06:15 PM                                                                                                                               |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject:   |
|------------>
  >--------------------------------------------------------------------------------------------------------------------------------------------------|
  | Got a huge XML document?                                                                                                                |
  >--------------------------------------------------------------------------------------------------------------------------------------------------|





Hi Folks,

In the past it was difficult to process huge XML documents using XSLT
because the entire XML document had to fit in memory.

Now, with XSLT 3.0 it is possible to process XML documents of unlimited
size, using a technique called streaming.

I am putting together a tutorial on streaming and would like to show an
example of stream processing a huge XML document.

Got a huge XML document that you can share? The bigger, the better. A
gigabyte sized XML document would be awesome.

/Roger

_______________________________________________________________________

XML-DEV is a publicly archived, unmoderated list hosted by OASIS
to support XML implementation and development. To minimize
spam in the archives, you must subscribe before posting.

[Un]Subscribe/change address: http://www.oasis-open.org/mlmanage/
Or unsubscribe: xml-dev-unsubscribe@lists.xml.org
subscribe: xml-dev-subscribe@lists.xml.org
List archive: http://lists.xml.org/archives/xml-dev/
List Guidelines: http://www.oasis-open.org/maillists/guidelines.php

• References:
  • Got a huge XML document?
    • From: "Costello, Roger L." <costello@mitre.org>