[GRASS-dev] [GRASS GIS] #3925: winGRASS 7.8.1dev: 'charmap' codec can't decode byte 0x9d - issue in vector attribute data handling (e.g. opening attribute table, v.report, etc)

[GRASS GIS]

#3925: winGRASS 7.8.1dev: 'charmap' codec can't decode byte 0x9d - issue in vector
attribute data handling (e.g. opening attribute table, v.report, etc)
----------------------+------------------------------------
  Reporter:  hellik   |      Owner:  grass-dev@…
      Type:  defect   |     Status:  new
  Priority:  blocker  |  Milestone:  7.8.1
 Component:  Vector   |    Version:  git-releasebranch78
Resolution:           |   Keywords:  python3, py3, wingrass
       CPU:  x86-64   |   Platform:  MSWindows
----------------------+------------------------------------

Comment (by hellik):

 now tested in a windows 10 box with a german locale:

 {{{
 GRASS Version: 7.8.1dev
 Code revision: f5bfe545c
 Build date: 2019-10-27
 Build platform: x86_64-w64-mingw32
 GDAL: 2.4.1
 PROJ: 5.2.0
 GEOS: 3.8.0
 SQLite: 3.29.0
 Python: 3.7.0
 wxPython: 4.0.3
 Platform: Windows-10-10.0.18362-SP0 (OSGeo4W)
 }}}

 download geonames data of an
 [http://download.geonames.org/export/dump/AT.zip AT dump]

 {{{
 v.in.geonames input=D:\temp\geonames\AT\AT.txt output=at_geonames
 Converting 51999 place names...
 Scanne die Eingabe zur Ermittelung der Spaltentypen...
 Number of columns: 19
 Number of rows: 51999
 WARNING: Spalte Nummer 13 <admin3code> ist als string definiert und hat
 nur Integer-Werte.
 WARNING: Spalte Nummer 14 <admin4code> ist als string definiert und hat
 nur Integer-Werte.
 Importiere Punkte...
 Fülle Tabelle...
 Erstelle Topologie für die Vektorkarte <at_geonames at data>...
 Registriere Primitive...
 GRASS_INFO_PROGRESS: 10000
 GRASS_INFO_PROGRESS: 20000
 GRASS_INFO_PROGRESS: 30000
 GRASS_INFO_PROGRESS: 40000
 GRASS_INFO_PROGRESS: 50000
 }}}

 clicking on a point with a german umlaut

 {{{
 east, north: 11.31700474110463, 47.23295282435024
 at_geonames at data:
   Type: Point
   Id: 10136
   Layer: 1
   Category: 10136
   Driver: sqlite
   Database: D:\grassdata\loc_test_vingeonames\data\sqlite\sqlite.db
   Table: at_geonames
   Key_column: cat
   Attributes:
     cat: 10136
     geonameid: 2762446
     name: Vellenberg
     asciiname: Vellenberg
     latitude: 47.23333
     longitude: 11.31667
     featureclass: S
     featurecode: FRM
     countrycode: AT
     admin1code: 07
     admin2code: 703
     admin3code: 70312
     population: 0
     gtopo30: 865
     timezone: Europe/Vienna
     modification: 2014-05-03
 at_geonames at data:
   Type: Point
   Id: 25781
   Layer: 1
   Category: 25781
   Driver: sqlite
   Database: D:\grassdata\loc_test_vingeonames\data\sqlite\sqlite.db
   Table: at_geonames
   Key_column: cat
   Attributes:
     cat: 25781
     geonameid: 2778215
     name: Götznerberg
     asciiname: Goetznerberg
     latitude: 47.23333
     longitude: 11.31667
     featureclass: S
     featurecode: FRM
     countrycode: AT
     admin1code: 07
     admin2code: 703
     admin3code: 70312
     population: 0
     gtopo30: 865
     timezone: Europe/Vienna
     modification: 2014-05-03
 }}}

 trying v.report

 {{{
 v.report map=at_geonames at data option=coor
 Traceback (most recent call last):
   File "C:\OSGEO4~1\apps\grass\grass78/scripts/v.report.py",
 line 226, in <module>
     main()
   File "C:\OSGEO4~1\apps\grass\grass78/scripts/v.report.py",
 line 108, in main
     cols = decode(line).rstrip('\r\n').split('|')
   File "C:\OSGEO4~1\apps\grass\grass78\etc\python\grass\scri
 pt\utils.py", line 193, in decode
     return bytes_.decode(enc)
   File "C:\OSGEO4~1\apps\Python37\lib\encodings\cp1252.py",
 line 15, in decode
     return
 codecs.charmap_decode(input,errors,decoding_table)
 UnicodeDecodeError: 'charmap' codec can't decode byte 0x9d
 in position 195: character maps to <undefined>
 }}}

 or trying to open the vector attribute table

 {{{
 Traceback (most recent call last):
   File
 "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\lmgr\frame.py",
 line 2060, in OnShowAttributeTable

 selection=selection)
   File "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\dbmgr\ma
 nager.py", line 112, in __init__

 self.CreateDbMgrPage(parent=self, pageName='browse')
   File
 "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\dbmgr\base.py",
 line 811, in CreateDbMgrPage

 parent=parent, parentDbMgrBase=self, onlyLayer=onlyLayer)
   File
 "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\dbmgr\base.py",
 line 1095, in __init__

 self.AddLayer(layer)
   File
 "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\dbmgr\base.py",
 line 1138, in AddLayer

 self.dbMgrData, layer, self.pages)
   File
 "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\dbmgr\base.py",
 line 113, in __init__

 keyColumn = self.LoadData(layer)
   File
 "C:\OSGEO4~1\apps\grass\grass78\gui\wxpython\dbmgr\base.py",
 line 278, in LoadData

 record = decode(outFile.readline().strip()).replace('\n',
 '')
   File "C:\OSGEO4~1\apps\grass\grass78\etc\python\grass\scri
 pt\utils.py", line 193, in decode

 return bytes_.decode(enc)
   File "C:\OSGEO4~1\apps\Python37\lib\encodings\cp1252.py",
 line 15, in decode

 return codecs.charmap_decode(input,errors,decoding_table)
 UnicodeDecodeError
 :
 'charmap' codec can't decode byte 0x9d in position 219:
 character maps to <undefined>
 }}}

 opening the table freezes the attribute table window

 it seems to be an encoding issue of attribute data handling

 {{{
 'charmap' codec can't decode byte 0x9d
 }}}

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/3925#comment:2>
GRASS GIS <https://grass.osgeo.org>