extension functions - help needed

I need some help to figure out how extension functions work:
I'm using Xalan Java extension.  All set up is OK as I can use simple
functions.

The question is this: I'm converting a string to a NodeList
(processTextNode(string(.)) returns NodeList). The content of the NodeList
is a mix of text nodes and img tags.
i.e. This is a special char that will not display: <img
src="specialchar.gif" />. More text....

The function returns the right NodeList when tested in Java.  

Here is the XSLT:
<!-- text() node, do HTML string conversion -->
  <xsl:template match="text()">
       <xsl:copy-of select="cops:processTextNode(string(.))"/>
  </xsl:template>

Will this create a infinite loop (the returned nodes are processed again)?
Because I got out of memory error when I ran the conversion. How can I fix
it?

Note: the function may return null. I wonder if that's a problem.

********************
The real question is:

I have some strange characters in my docs that cannot be displayed properly
using HTML. I have to use images for them. How can I do the substitution in
XSLT?

I thought of 
(1) scanning the converted HTML to do the replacement, but most of the time
I don't have the special chars in my docs
(2) using SAX to convert the XML docs before doing XSLT, but need to modify
the DTD to include <img> that is not desirable

If you are doing similar conversion, please help!!!

********************

Thanks.

Peiyun Jiang