Something like this perhaps:

<xsl:function name="my:int-to-hex" as="xs:string">
  <xsl:param name="in" as="xs:integer"/>
  <xsl:sequence select="if ($in eq 0) then '0'
                        else concat(if ($in gt 16 then my:int-to-hex($in
idiv 16) else '',
                               substring('0123456789ABCDEF', ($in mod 16) +
1, 1)))"/>
</xsl:function>

Thank you very much for this solution, and thanks to Michael M.-H. for
his one which looks just as neat. Both work fine, but in the code above
I first had to re-arrange the parentheses:

<xsl:function name="my:int-to-hex" as="xs:string">
  <xsl:param name="in" as="xs:integer"/>
  <xsl:sequence
    select="if ($in eq 0)
            then '0'
            else
              concat(if ($in gt 16)
                     then my:int-to-hex($in idiv 16)
                     else '',
                     substring('0123456789ABCDEF',
                               ($in mod 16) + 1, 1))"/>
</xsl:function>

But what's wrong with having two functions anyway?

Well, not too much. I just don't like function definitions cluttering my
stylesheet, their sole raison d'jtre being my limited understanding of
XSLT and XPath... :-)

Incidentally, do try to get into the habit of declaring your parameter and
return types. It helps people reading your code, it helps the compiler, and
it helps you.

You are absolutely right. I normally do declare all of my parameters and
return types, but in this case copied the code from somewhere and forgot
to "augment" it.