[gdal-dev] Character Encoding Problem

[Brett Walker]

HI,

I have found the cause of the problem. The field of interest in Oracle is defined as NVARCHAR2. It seems that GDAL cannot deal with this data type properly. See http://docs.oracle.com/cd/B19306_01/server.102/b14220/datatype.htm#i14946 from the Oracle documentation regarding the NVARCHAR2 data type.

I took the SDE Server extract of the shapefile (the correct one) and imported it into the same database using a different table name. The newly created table had the corresponding field defined using VARCHAR2.  An extract using GDAL had no problem with it.

I would recommend that either a warning be placed in documentation regarding NVARCHAR2 fields or fix the real problem. One solution is easy, the other hard.

I'll be recommending to our client to change the data type of NVARCHAR2 fields to VARCHAR2.

Brett

From: gdal-dev-bounces at lists.osgeo.org [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Brett Walker
Sent: Thursday, 29 November 2012 11:30 AM
To: gdal-dev at lists.osgeo.org
Subject: [gdal-dev] Character Encoding Problem

Hi All,

I have a problem with a character encoding problem. It could be a UTF-16 to UTF-8 problem or some other encoding problem.

I have an SDE Server with the DB being Oracle from which I am extracting an ESRI Shapefile.

Using ogr2ogr from GDAL to extract the shapefile I am losing every second character from strings. As an example the original string of 'Murray channel' is being extracted as 'Mra hne'.

Using sde2shp from the SDE Server to extract the shapefile I am not losing any characters from strings. But during the extraction I receive the warning:
WARNING: Possible loss of data when converting SE_NSTRING to ANSI string.

Could this problem be that GDAL is not detecting a string of SE_NSTRING and not dealing with it appropriately?

I doubt it would be related to platform, but the SDE Server is running on Windows Server 2003 32-bit and GDAL is running on GNU/Linux 2.6.18-194.el5 64-bit

Thanks or any help,
Brett

Brett Walker <brett.walker at geometryit.com<mailto:brett.walker at geometryit.com>>
Software Developer / Analyst
Geometry Pty Ltd

Telephone               03 6223 1999
Mobile                     0458 498 386
Fax                          03 6223 1988
Web                        www.geometryit.com<http://www.geometryit.com/>
Address                   31 Salamanca Square, Battery Point, TAS 7004, Australia

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20121129/7cd120fe/attachment-0001.html>