[XML-DEV Mailing List Archive Home] [By Thread] [By Date] [Recent Entries] [Reply To This Message]

Re: Out of memory error with XML and Java ResultSet

• To: xml-dev@l...
• Subject: Re: Out of memory error with XML and Java ResultSet
• From: Rick Jelliffe <ricko@a...>
• Date: Tue, 09 Sep 2003 21:28:00 +1000
• In-reply-to: <D0FBDF2C29301849A1F44D2B4944C8A94F3872@m...>
• References: <D0FBDF2C29301849A1F44D2B4944C8A94F3872@m...>
• User-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.3.1) Gecko/20030428

Brown, Jason B. wrote:

>One row of XML text is approximately 230 characters, including tag name
>and attribute specifiers. 
>  
>
So you have about 230 characters x 300,000 rows. 70,000,000 characters. 
In Java, a character
takes two bytes, so that is 140,000,000 bytes. If you are appending to a 
StringBuffer, it
doubles the size of the underlying array whenever it needs more; at 
worse case, if your
StringBuffer had a full array with 70,000,000 characters and you 
appended one more character,
then StringBuffer would allocate a new underlying array of 140,000,000 
characters.  So that
takes up 280,000,000 bytes.   *Plus* not forgetting that the previous 
array was still needed at the
same time as the new one, so at the time of expanding the capacity of 
the underlying array,
you needed 420,000,000 bytes!

And that apart from anything else going on in your program.

I suggest you:--
  1) Allocate the StringBuffer with the correct amount of size (e.g. 70 
million characters) 
to prevent the worst-case blowout and reallocation requirements; If you 
regularly have
to handle large StringBuffers, you should add it to your QA checklist to 
check that
all large StringBuffers that are appended to should have preallocated 
capacities on
initialization;
  2) Make sure your JVM has been allocated with enough heap (-Xms);
  3) Double check your code so that you make sure the StringBuffer is 
explicitly set to null
as soon as it is not needed (which may help garbage collection), and 
that any external
references to the StringBuffer free it as soon as needed.

Cheers
Rick Jelliffe

• References:
  • RE: Out of memory error with XML and Java ResultSet
    • From: "Brown, Jason B." <Jason.B.Brown@i...>

• Prev by Date: Re: searching sql dtd
• Next by Date: Announce: 1060 NetKernel
• Previous by thread: Re: Out of memory error with XML and Java ResultSet
• Next by thread: RE: Out of memory error with XML and Java ResultSet
• Index(es):
  • Date
  • Thread