[GRASS-dev] [GRASS GIS] #3220: WinGRASS not recognizing accented utf-8 (nor cp1252) attribute values

[GRASS GIS]

#3220: WinGRASS not recognizing accented utf-8 (nor cp1252) attribute values
-------------------------+---------------------------------
 Reporter:  hellik       |      Owner:  grass-dev@…
     Type:  defect       |     Status:  new
 Priority:  normal       |  Milestone:  7.2.0
Component:  Default      |    Version:  svn-releasebranch72
 Keywords:               |        CPU:  Unspecified
 Platform:  MSWindows 8  |
-------------------------+---------------------------------
 taken from the user ML:

 https://lists.osgeo.org/pipermail/grass-user/2016-December/075682.html

 {{{
 I've got shape files with Swedish accented letters (ÄÖÅ) in the some of
 the attribute
 values. The Attributes are shwon as they should in the GUI. SQL
 statements, however,
 are not recognizing them. They're also messed up in the command output if
 another
 (not accented) values are queried.

 I sat GRASS_DB_ENCODING to cp1252 firstly and it didn't work. Then I
 converted the
 dbf file into utf-8 and sat it as the value of the variable, to no avail.
 I also
 tried using the 'encoding' parameter in v.in.ogr in both cases, didn't
 work.

 I tried it on windows 8.1 and windows 10. The same is happening in both,
 stable GRASS
 7.0.5 and GRASS 7.2.0RC1.

 The problem is only happening on Windows. Fedora and Mac OsX don't have
 this issue
 with the same shape files.
 }}}

 https://lists.osgeo.org/pipermail/grass-user/2016-December/075688.html

 {{{
 confirmed with

 GRASS version: 7.3.svn
 GRASS SVN revision: r70001
 Build date: 2016-12-06
 Build platform: x86_64-w64-mingw32
 GDAL: 2.1.2
 PROJ.4: 4.9.3
 GEOS: 3.5.0
 SQLite: 3.14.1
 Python: 2.7.5
 wxPython: 2.8.12.1
 Platform: Windows-8-6.2.9200 (OSGeo4W)
 }}}

 {{{
 and a test vector with following attributes

 v.db.select map=test_points at data file=D:\temp\test_point.txt

 cat|id|names
 1|1|ÄÖÅ
 2||Æ
 3||Ø
 4||Å,å,Æ,æ,Ø,ø
 5||ø, Ø
 6||Þ
 7||Ð
 8||Å
 9||æ
 }}}

 {{{
 d.vect map=test_points2 at data where="names = 'Å,å,Æ,æ,Ø,ø'" width=1
 icon=basic/point size=10

 doesn't show the selected point in the map display.
 }}}

 {{{
 v.report map=test_points at data option=coor
 cat|id|names|x|y|z
 1|1|ÄÖÅ|1.37409120951759|47.039352838731|0.0
 2||Æ|2.62326503635168|28.5515802015863|0.0
 3||Ø|44.095836087244|57.2825782187707|0.0
 4||Å,å,Æ,æ,Ø,ø|30.8545935228025|49.787535257766|0.0
 5||ø, Ø|10.1183079973563|51.0367090846001|0.0
 6||Þ|20.361533377396|52.0360481460674|0.0
 8||Ã…|15.1491119517375|60.3621017805262|0.0
 9||æ|-1.26290587954035|52.5879880709736|0.0

 Traceback (most recent call last):
   File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
 i_core\goutput.py", line 473, in OnCmdOutput

 self.cmdOutput.AddStyledMessage(message, type)
   File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
 i_core\goutput.py", line 772, in AddStyledMessage

 self.AddTextWrapped(message, wrap=None)
   File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
 i_core\goutput.py", line 721, in AddTextWrapped

 txt = EncodeString(txt)
   File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\co
 re\gcmd.py", line 97, in EncodeString

 return string.encode(_enc)
   File "C:\OSGEO4~1\apps\Python27\lib\encodings\cp1252.py",
 line 12, in encode

 return codecs.charmap_encode(input,errors,encoding_table)
 UnicodeDecodeError
 :
 'ascii' codec can't decode byte 0xc3 in position 3: ordinal
 not in range(128)
 }}}

--
Ticket URL: <https://trac.osgeo.org/grass/ticket/3220>
GRASS GIS <https://grass.osgeo.org>