[GRASS-dev] wxgui encoding issues

[Moritz Lennert]

Hi all,

I'm sending this to the mailing list as I'm not sure where we're at in 
terms of the discussions concerning encoding issues in the wxgui (I know 
Maris has been active on the windows front). If my problems are real 
bugs, then I'll post relevant tickets.

Using any version, I have the following encoding issues in the GUI 
(knowing that my machine is configured with locale fr_BE.UTF-8) :

- r.reclass interactive input with accents gives me

"Traceback (most recent call last):
   File 
"/home/mlennert/SRC/GRASS/grass_trunk/dist.i686-pc-linux-gnu/etc/gui/wxpython/gui_core/forms.py", 
line 1738, in OnFileText
     f.write(text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in 
position 7: ordinal not in range(128)"

The same data in a file works without issues.

Here are example reclass rules for the NC landclass96 raster to reproduce:

1 = 1 développé
2 = 2 agricole
3 = 3 herbacé
4 = 4 broussaille
5 = 5 forêt
6 = 6 eau
7 = 7 sédiment

- Trying to save any model from the graphical modeller, even a 
completely empty one gives me:

Traceback (most recent call last):
   File 
"/home/mlennert/SRC/GRASS/grass_trunk/dist.i686-pc-linux-gnu/etc/gui/wxpython/gmodeler/frame.py", 
line 868, in WriteModelFile
     WriteModelFile(fd = tmpfile, model = self.model)
   File 
"/home/mlennert/SRC/GRASS/grass_trunk/dist.i686-pc-linux-gnu/etc/gui/wxpython/gmodeler/model.py", 
line 1758, in __init__
     self._properties()
   File 
"/home/mlennert/SRC/GRASS/grass_trunk/dist.i686-pc-linux-gnu/etc/gui/wxpython/gmodeler/model.py", 
line 1807, in _properties
     self.fd.write('%s<name>%s</name>\n' % (' ' * self.indent, 
self.properties['name']))
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe8' in 
position 17: ordinal not in range(128)

This second issue seems to come from line 58 in 
gui/wxpython/gmodele/model.py :

self.properties = { 'name'        : _("model"),

Does this property really have to be translatable ? If I change line 
1807 mentioned in the traceback to

             self.fd.write('%s<name>%s</name>\n' % (' ' * self.indent, 
self.properties['name'].encode('UTF-8')))

it works. What I don't understand is that I don't have to do this for 
the description in the next line which also contains an accent in French.

- In all versions but 6.4.2 trying to import a file which resides in a 
directory with accent in the name ("données"), I see this in the file 
text field of the import wizard after browing to the file

/home/mlennert/Desktop/données/boundary_county.shp


Exploring this a bit further, I see the following behaviour:

- My own locale is defined as follows

LANG=fr_BE.UTF-8
LANGUAGE=
LC_CTYPE="fr_BE.UTF-8"
LC_NUMERIC="fr_BE.UTF-8"
LC_TIME="fr_BE.UTF-8"
LC_COLLATE="fr_BE.UTF-8"
LC_MONETARY="fr_BE.UTF-8"
LC_MESSAGES="fr_BE.UTF-8"
LC_PAPER="fr_BE.UTF-8"
LC_NAME="fr_BE.UTF-8"
LC_ADDRESS="fr_BE.UTF-8"
LC_TELEPHONE="fr_BE.UTF-8"
LC_MEASUREMENT="fr_BE.UTF-8"
LC_IDENTIFICATION="fr_BE.UTF-8"
LC_ALL=

- When I start 6.4.2 and then type 'locale' at the command prompt I get 
exactly the same.

- But when I start 6.4.3RC1 I see

LANG=
LANGUAGE=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

and the gui starts in English.

When I set LC_ALL=fr_BE.UTF-8 before starting grass6.4.3 RC1, and the 
type 'locale' at the command prompt, I get


LANG=
LANGUAGE=
LC_CTYPE="fr_BE.UTF-8"
LC_NUMERIC="fr_BE.UTF-8"
LC_TIME="fr_BE.UTF-8"
LC_COLLATE="fr_BE.UTF-8"
LC_MONETARY="fr_BE.UTF-8"
LC_MESSAGES="fr_BE.UTF-8"
LC_PAPER="fr_BE.UTF-8"
LC_NAME="fr_BE.UTF-8"
LC_ADDRESS="fr_BE.UTF-8"
LC_TELEPHONE="fr_BE.UTF-8"
LC_MEASUREMENT="fr_BE.UTF-8"
LC_IDENTIFICATION="fr_BE.UTF-8"
LC_ALL=fr_BE.UTF-8

and the gui in French. I can then import the file from the directory 
with accents, but the problems with saving the model and with using 
reclass rules with accents interactively remain.

I see some differences in the way LANG and LANGUAGE are handled in 
Init.sh, but I don't really understand where that comes to play for the GUI.

In any case, these encoding errors are quite an issue for my students...

So, if anyone can point me to existing discussions or bug tickets that I 
have failed to find, I'm more than happy to document these issues there. 
Or I can file a new ticket. Unless I'm just doing something wrong here...

Moritz