RE-POST: extension functions - help needed

I need some help to figure out how extension functions work:
I'm using Xalan Java extension.  All set up is OK as I can use simple
functions.

The question is: I'm converting a string to a NodeList
[processTextNode(string(.)) returns a NodeList]. The content of the NodeList
is a mix of text nodes and img tags.
i.e. This is a special char that will not display: <img
src="specialchar.gif" />. More text....

The function returns the right NodeList when tested in Java.  

Here is the XSLT:
<!-- text() node, do HTML string conversion -->
  <xsl:template match="text()">
       <xsl:copy-of select="cops:processTextNode(string(.))"/>
  </xsl:template>

Will this create a infinite loop (the returned nodes are processed again)?
I got out of memory error when I ran the conversion. The XML document is not
big. How can I fix it? Or is this the right way to do it?

Note: the function may return null. I wonder if that's a problem.

********************
The real question is:

I have some strange characters in my docs that cannot be displayed properly
using HTML. I have to use images for them. How can I do the substitution in
XSLT?

I thought of 
(1) scanning the converted HTML to do the replacement, but most of the time
I don't have the special chars in my docs
(2) using SAX to convert the XML docs before doing XSLT, but need to modify
the DTD to include <img> that is not desirable

Is it feasible to scan the doc before written to a file?

If you are doing similar conversion, please help!!!

********************

Thanks.

Peiyun Jiang