[mapserver-users] Confirmation of status of UTF8 support, and where transcoding to Latin-1 may be happening.

[Russell McOrmond]

   This will likely be my final update on this specific, given I believe 
I've found all our issue.

   The problem came down to two unrelated problems, eithor of which meant 
that nationally encoded strings from ArcSDE weren't being properly handled 
in mapserver.

a) gd 2.0.35

   This version of GD was incorrectly detecting the iconv library due to a 
missing macro.

configure.ac:64: error: possibly undefined macro: AM_ICONV

   This meant that it was missing the '#define HAVE_ICONV 1' in the 
config.h.

   If this is not defined, this activates a set of stub functions in 
gdkanji.c .  This was the source of iconv_open() always returning -1.

   This problem is fixed in gd 2.0.36RC1 .  There is likely a way to add 
the right macro with 2.0.35 and use it as well.

b) endianness setting with NSTRING data from SDE

   See http://trac.osgeo.org/mapserver/ticket/2878
http://lists.gnu.org/archive/html/bug-gnu-libiconv/2009-02/msg00001.html

   It turns out that we need to specify the endianness for the NSTRING data 
(UTF-16 encoded) coming from libsde. The glibc version of the iconv() 
functions seem to assume little-endian for "UTF-16", while the libiconv 
version seems to assume network order or big-endian.

   If we specify "UTF-16LE", it will always work regardless of which 
iconv() version we are using.


   With this problem now solved, I just wish the problem with zlib in 
libsde was resolved.
http://forums.esri.com/Thread.asp?c=2&f=1718&t=212867

   If libsde from ArcSDE 9.3 is still using a known insecure version of 
zlib from 1998, it is quite likely that other aspects of ESRI tools are 
using this as well.  I am surprised to learn that this was not considered 
a high priority for ESRI, given the most recent version of zlib is from 
2005 and there has been plenty of opportunity for distributing a fix.

-- 
  Russell McOrmond, Internet Consultant: <http://www.flora.ca/>
  Please help us tell the Canadian Parliament to protect our property
  rights as owners of Information Technology. Sign the petition!
  http://digital-copyright.ca/petition/ict/     http://KillBillC61.ca

  "The government, lobbied by legacy copyright holders and hardware
   manufacturers, can pry control over my camcorder, computer,
   home theatre, or portable media player from my cold dead hands!"