[mapserver-dev] MapServer vs character encodings

[Daniel Morissette]

Howard Butler wrote on mapserver-users:
> 
> Yes, I was asking if this was an NSTRING column or a STRING column.  The 
> MapServer ArcSDE code makes an attempt to transcode the data if it is an 
> nstring column, but it is highly likely this is broken.  I didn't have 
> very extensive data to test this with, and all I looked to ensure was 
> that a few pictures looked right.  The problem might be as simple as the 
> function msConvertWideStringToUTF8 being broken.  Here's where MapServer 
> tries to convert it: 
> http://trac.osgeo.org/mapserver/browser/trunk/mapserver/mapsde.c#L750
> 
> The reason that this is such a problem is that MapServer doesn't really 
> have its unicode/wide character strategy well defined. ArcSDE forces the 
> issue on us now because by default they're using nstring columns for 
> loading data by default, so the ArcSDE users are on the front lines of 
> this, but I think it will be more of a problem as time marches on.  
> Questions the MapServer devs need to answer are:
> 
> - how are strings to be internally represented in MapServer (lots of 
> things to balance by answering this question)?
> - who is responsible for transforming the data?  The driver for both the 
> in and out?  Have a smarter string object that carries its own encoding 
> around?
> - Unicode is a conspiracy :)
> 

I copied this to the -dev list to see if anyone had ideas. It's an ugly 
problem that none of us wants to touch and seems to hit SDE users mostly 
at this point.

Honestly I am not sure I understand the issue with SDE that Howard 
describes. Would it not be enough to require that data access drivers 
that read data in unicode transform all attribute values to UTF-8? I 
thought that was what the SDE driver was doing and honestly don't 
understand why that doesn't work.

Daniel
-- 
Daniel Morissette
http://www.mapgears.com/