[mapserver-users] ArcSDE and UNICODE

[Neil Fincham]

Hi All,

	Sorry to bring this up again after 6 months of so but the problem
has once again landed on my desk.  Howard below hit the nail on the head,
the text is being stored in nvarchar (or SE_NSTRING as sdetable -o describe
calls it) so mapserver is having a lot of problems with the UNICODE.

	My question is is what is the best way around this?  I am sure
people have had this problem before and can save me a lot of
experimentation.  Howard below mentions loading the data as string, how
would I do that?  SDE does not seem to give me the option.  I have tried a
sql "alter table alter column Stand varchar(30)" and that does not seem to
work.  Perhaps I should be using "sdetable -o alter_column" does anyone have
the correct syntax for that?

Thanks heaps for your time,

Neil

-----Original Message-----
From: UMN MapServer Users List [mailto:MAPSERVER-USERS at LISTS.UMN.EDU] On
Behalf Of Howard Butler
Sent: Monday, 7 January 2008 5:47 p.m.
To: MAPSERVER-USERS at LISTS.UMN.EDU
Subject: Re: [UMN_MAPSERVER-USERS] Unusual Fonts (resend, image too big)

Neil,

Looking through your post, and by virtue of using the SDE 9.2 plugin,  
I'll assume you are connecting to ArcSDE 9.2.

First, what does the output of sdetable -o describe say for your  
SpeciesLabel column about the column data type?  Is it NSTRING?

Here's the story on unicode characters and MapServer+ArcSDE 9.2...  As  
long as your characters aren't wide format, ie UTF-8'ish, they should  
be printable and usable with the iconv support in GD/AGG.  If they are  
wide, however, they are garishly converted down to UTF8.  My rational  
for this approach is that many of the Arc tools emit unicode columns  
for string data in 9.2 SDE layers, but much of the time, people don't  
want/need that, and they are storing ASCII or LATIN-type data in them  
anyway.  MapServer internally does not support wide characters,  
however, so there are some significant limitations in this approach,  
and if MapServer is converting your wide characters down to UTF8, you  
are pretty much screwed.  Another huge limitation of this approach is  
that it is really only usable for labeling things.  You cannot use any  
expressions on NSTRING columns in ArcSDE with MapServer and expect to  
do anything.

You shouldn't have to specify an encoding for the SDE layer because  
any NSTRING columns (which SDE stores as UTF-16) are smashed down to  
UTF8, which iconv/GD/AGG can render and deal with.

All of this blather says that the unicode support for MapServer is  
very immature, and if you can reload your ArcSDE layer to use regular  
string columns in your ArcSDE layers, things should work much smoother  
and as expected.

Howard

On Jan 6, 2008, at 4:12 PM, Neil Fincham wrote:

> Hi Ed,
>
> 	Thanks for the quick reply, a picture says 1000 words so I have  
> done a
> screen dump out of ArcVIEW for you.
>
> 	I have tried a few different fonts and a few different string fields
> from the SDE and get the same result with the string's, the integers  
> and
> <Null>'s render fine and using a shape file exported directly out of  
> the
> SDE also works fine.
>
> Very puzzling
>
> Neil
>
> Ed McNierney wrote:
>> Neil -
>>
>> Could you give an example of the text in one of the SpeciesLabel  
>> fields?  What alphabet, character set, and language is the  
>> SpeciesLabel field stored in?  If you switch to a different FONT  
>> does the output change?
>>
>> At first glance you appear to have code points in the label that  
>> are not representable in the selected font.  That might not be the  
>> problem, but it's worth eliminating - hence my questions above.
>>
>> 	- Ed
>>
>> Ed McNierney
>> Chief Mapmaker
>> Demand Media / TopoZone.com
>> 73 Princeton Street, Suite 305
>> North Chelmsford, MA  01863
>> Phone: 978-251-4242, Fax: 978-251-1396
>> ed at topozone.com
>>
>> -----Original Message-----
>> From: UMN MapServer Users List [mailto:MAPSERVER- 
>> USERS at LISTS.UMN.EDU] On Behalf Of Neil Fincham
>> Sent: Sunday, January 06, 2008 3:37 PM
>> To: MAPSERVER-USERS at LISTS.UMN.EDU
>> Subject: [UMN_MAPSERVER-USERS] Unusual Fonts
>>
>> Hi all,
>>
>> 	I am really enjoying my first delve into mapping and like to say  
>> thanks for the great software.
>>
>> 	I am havering a bit of a problem and I am sure it is something  
>> easy, I am getting unusual fonts rendering using SDE.  It works  
>> fine with the shape files but with SDE the text parts come out  
>> wrong.  I have attached a sample.
>>
>> Here are some bits from my mapfile;-
>>
>> OUTPUTFORMAT
>> NAME 'AGG'
>> DRIVER AGG/PNG
>> IMAGEMODE RGB
>> END
>>
>> LAYER
>>  NAME LandUnit
>>  METADATA
>>    "DESCRIPTION"   "Land Unit"
>>    "RESULT_FIELDS" "ForestCode Description"
>>  END
>>  TYPE POLYGON
>>  STATUS ON
>>  CONNECTIONTYPE PLUGIN
>>  PLUGIN "/ms4w/Apache/specialplugins/msplugin_sde_92.dll"
>>  CONNECTION "palustris,port:5155,d,ms_user,p4ss\/\/ord"
>>  DATA "d.DBO.LandUnit,SHAPE,SDE.DEFAULT"
>>  TOLERANCE 5
>>  TEMPLATE "ttt_query.html"
>>  CLASS
>>    SYMBOL 0
>>    COLOR 255 255 255
>>    OUTLINECOLOR 0 0 0
>>    TEXT ([SpeciesLabel] [YOE] [AreaHA]ha)
>>    LABEL
>>      FONT "vera"
>>      TYPE truetype
>>      SIZE 8
>>      BUFFER 1
>>      COLOR 0 0 0
>>      BACKGROUNDCOLOR 255 255 255
>>      WRAP " "
>>      ENCODING UTF-16
>>    END
>>  END
>> END # Layer
>>
>> 	The [SpeciesLabel] is the one with the problem, the "[YOE]  
>> [AreaHA]ha"
>> are numbers and render correctly.  I have tried various ENCODING  
>> types to no avail.
>>
>> Thanks for your time
>>
>> Neil
> <fonts.jpg>