[GRASS-dev] [GRASS GIS] #2120: wxgui: encoding errors

[GRASS GIS]

#2120: wxgui: encoding errors
-----------------------------+----------------------------------------------
 Reporter:  mlennert         |       Owner:  grass-dev@…              
     Type:  defect           |      Status:  new                      
 Priority:  normal           |   Milestone:  6.4.4                    
Component:  wxGUI            |     Version:  svn-releasebranch64      
 Keywords:  locale encoding  |    Platform:  Unspecified              
      Cpu:  Unspecified      |  
-----------------------------+----------------------------------------------

Comment(by glynn):

 Replying to [comment:7 hcho]:

 > I'm not sure if it's a good idea to include this file in GRASS SVN

 It isn't.

 It's working around a bug, namely that Python's default encoding is being
 used. Fix the bug, not the symptoms.

 > I don't know why the encoding setting in Preferences doesn't work. It
 seems like we cannot change the default encoding outside sitecustomize.py.

 That's correct. The reason is related to the use of hashtables to
 implement dictionaries (including those which underlie Python objects).

 When a Unicode object is hashed, if it can be converted to a byte string
 using the default encoding then it is, and the resulting string's hash is
 used. This allows a dictionary key which is a string to be matched using
 the string's Unicode equivalent, and vice-versa.

 However, if the default encoding is changed after any non-ASCII keys have
 been added to dictionaries, any lookups for those keys would use the wrong
 hash and fail. To protect against this possibility, the site module
 deletes the setdefaultencoding() function from the sys module after
 loading the sitecustomize and/or usercustomize modules. Basically, once
 the interpreter is fully "up and running", the default encoding is fixed.

 Note that Python's default encoding (as per sys.getdefaultencoding()) is
 only used for implicit conversions between str and unicode objects, e.g.
 via the str() and unicode() constructors, or via the C API (e.g.
 `PyString_AsString`). But not for filenames (see below).

 It is not the same thing as the locale's default encoding as per
 locale.getpreferredencoding() or locale.getdefaultlocale(). Nor is it the
 same thing as the filesystem encoding as per sys.getfilesystemencoding(),
 which is used for converting filenames (from unicode to str on Unix, Mac
 and Windows 9x, from str to Unicode on Windows NT). These are what
 application code should be using.

 So, the implicit conversions should never be used by wxGUI. Conversions
 between Unicode strings (which is what wxPython normally uses) and byte
 strings should be explicit, with a specified encoding (e.g. that from the
 locale, unless information is available to indicate a different encoding).

-- 
Ticket URL: <https://trac.osgeo.org/grass/ticket/2120#comment:10>
GRASS GIS <http://grass.osgeo.org>