<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-AU link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>HI,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have found the cause of the problem. The field of interest in Oracle is defined as NVARCHAR2. It seems that GDAL cannot deal with this data type properly. See <a href="http://docs.oracle.com/cd/B19306_01/server.102/b14220/datatype.htm#i14946">http://docs.oracle.com/cd/B19306_01/server.102/b14220/datatype.htm#i14946</a> from the Oracle documentation regarding the NVARCHAR2 data type.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I took the SDE Server extract of the shapefile (the correct one) and imported it into the same database using a different table name. The newly created table had the corresponding field defined using VARCHAR2. An extract using GDAL had no problem with it.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I would recommend that either a warning be placed in documentation regarding NVARCHAR2 fields or fix the real problem. One solution is easy, the other hard.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I’ll be recommending to our client to change the data type of NVARCHAR2 fields to VARCHAR2.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Brett<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-AU'>From:</span></b><span lang=EN-US style='font-size:10.0pt;font-family:"Tahoma","sans-serif";mso-fareast-language:EN-AU'> gdal-dev-bounces@lists.osgeo.org [mailto:gdal-dev-bounces@lists.osgeo.org] <b>On Behalf Of </b>Brett Walker<br><b>Sent:</b> Thursday, 29 November 2012 11:30 AM<br><b>To:</b> gdal-dev@lists.osgeo.org<br><b>Subject:</b> [gdal-dev] Character Encoding Problem<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hi All,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have a problem with a character encoding problem. It could be a UTF-16 to UTF-8 problem or some other encoding problem.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have an SDE Server with the DB being Oracle from which I am extracting an ESRI Shapefile.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Using ogr2ogr from GDAL to extract the shapefile I am losing every second character from strings. As an example the original string of ‘Murray channel’ is being extracted as ‘Mra hne’.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Using sde2shp from the SDE Server to extract the shapefile I am not losing any characters from strings. But during the extraction I receive the warning:<o:p></o:p></p><p class=MsoNormal style='text-indent:36.0pt'>WARNING: Possible loss of data when converting SE_NSTRING to ANSI string.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Could this problem be that GDAL is not detecting a string of SE_NSTRING and not dealing with it appropriately?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I doubt it would be related to platform, but the SDE Server is running on Windows Server 2003 32-bit and GDAL is running on GNU/Linux 2.6.18-194.el5 64-bit<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks or any help,<o:p></o:p></p><p class=MsoNormal>Brett<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>Brett Walker</b> <<a href="mailto:brett.walker@geometryit.com">brett.walker@geometryit.com</a>><o:p></o:p></p><p class=MsoNormal>Software Developer / Analyst<o:p></o:p></p><p class=MsoNormal>Geometry Pty Ltd<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:9.0pt'>Telephone 03 6223 1999<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt'>Mobile 0458 498 386<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt'>Fax 03 6223 1988<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt'>Web <a href="http://www.geometryit.com/">www.geometryit.com</a><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:9.0pt'>Address 31 Salamanca Square, Battery Point, TAS 7004, Australia<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>