If I understand you just want to know which SubSection the reference and
the figure are in and  test if they are the same.

So first key on the figures

<xsl:key name="fig" match="Figure" use="@id"/>

Then 

<xsl:template match="a[@type="figure"]"
  <xsl:variable name="thissec" select="ancestor::SubSection[1]/@id"/>
  <xsl:variable name="figsec"
          select="key('fig',@href)/ancestor::SubSection[1]/@id"/>
 <xsl:choose>
  <xsl:when test="$thissec=$figsec">
   <a href="#{@href}">
    <xsl:apply-templates/>
   </a>
  </xsl:when>
  <xsl:otherwise>
   Could link to other doc
   <a href="{$figsec}.html#{@href}">
    <xsl:apply-templates/>
   </a>
  or just make text
    <xsl:apply-templates/>
 </xsl:otherwise>
 </xsl:choose>
</xsl:template>

David