I used XSLT streaming to generate a training corpus forEnglish-Korean la

Hi Folks,

The Open Street Map XML file for South Korea

http://downloads.cloudmade.com/asia/eastern_asia/south_korea/south_korea.osm.bz2

is quite interesting. Each <node> element contains data about a thing (airport, university, office, bus stop, etc.) in South Korea. Within each <node> element is a <tag> element that shows the name of the thing in English and another <tag> element that shows its name in Korean. For example, this <node> element contains the name of an airport in English and Korean:

<node lat="37.5582" lon="126.7906">
    <tag k="name:en" v="Gimpo International Airport"/>
    <tag k="name:ko" v="김포국제공항"/>
</node>

The English name is identified by @k="name:en" and the Korean name is identified by @k="name:ko" (@k means ‘key’ and @v means ‘value’).

These pairs of values may be collected and then used to train an English-Korean language translator tool.

The Open Street Map XML file is quite large -- 464 MB -- so I elected to extract all the English-Korean pairs using XSLT streaming. I wrote an XSLT streaming program (see below) and ran it. It generated over 30,000 English-Korean pairs. Here is a sample of the output:

<English-Korean>
    <translation>
        <English>Gimpo International Airport</English>
        <Korean>김포국제공항</Korean>
    </translation>
    <translation>
        <English>Incheon International Airport</English>
        <Korean>인천국제공항</Korean>
    </translation>
    <translation>
        <English>South Korea</English>
        <Korean>대한민국</Korean>
    </translation>
    <translation>
        <English>Jeju-si</English>
        <Korean>제주시</Korean>
    </translation>
    <translation>
        <English>Munui</English>
        <Korean>문의</Korean>
    </translation>
    <translation>
        <English>Bukcheon Junction</English>
        <Korean>북천교차로</Korean>
    </translation>

    …
    <translation>
        <English>Odong Islet</English>
        <Korean>오동도</Korean>
    </translation>
    <translation>
        <English>To Sinwon, Hapcheon, Chunjeon</English>
        <Korean>신원, 합천, 춘전방면</Korean>
    </translation>
</English-Korean>

Here is the streaming XSLT program

-------------------------------------------------------

   generate-training-corpus.xsl

-------------------------------------------------------

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:xs="http://www.w3.org/2001/XMLSchema"
                exclude-result-prefixes="#all"
                version="3.0">
   
    <xsl:output method="xml" />
   
    <xsl:template match="/">
        <xsl:stream href="../huge-file-Korea/south_korea.xml">
            <English-Korean>
                <xsl:for-each select="osm">
                    <xsl:iterate select="node">
                        <xsl:variable name="thisNode" select="copy-of(.)"/>
                        <xsl:if test="$thisNode[tag[@k eq 'name:en'] and tag[@k eq 'name:ko']]">
                            <translation>
                                <English><xsl:value-of select="$thisNode/tag[@k eq 'name:en']/@v" /></English>
                                <Korean><xsl:value-of select="$thisNode/tag[@k eq 'name:ko']/@v" /></Korean>
                            </translation>
                            <xsl:next-iteration />
                        </xsl:if>
                    </xsl:iterate>
                </xsl:for-each>
            </English-Korean>
        </xsl:stream>
    </xsl:template>
   
</xsl:stylesheet>

/Roger