I tried without specifying any encoding and in that
case even the single quotes are appearing as hash
symbol.

This is a FAQ. What you think of as "single quote"
are most likely typographical single quotes, and the
defaul fonts don't have glyphs for them. See
 http://xmlgraphics.apache.org/fop/faq.html#pdf-characters
for mor information.