[Mapbender-dev] Encoding issues

[Christoph Baudson]

Hello,

today I fixed a WMC related bug in Mapbender that occured when the 
CHARSET constant is set to "ISO-8859-1".

As Mapbender uses UTF-8 as internal encoding, all data coming in must be 
converted to UTF-8, and all outgoing data must be converted back to 
CHARSET. Having a single internal encoding is a must, because Mapbender 
mixes data from various sources (for example, metadata is gathered from 
various capabilities documents). It is also essential, because all the 
source code files are in UTF-8!

Currently, this is not done very strict in most classes or modules; we 
should improve this. Here's what I found out:

1) in class_administration.php, there is a function parseXml. I changed 
the target encoding to UTF-8 (before, it was set to CHARSET), so all 
data will be converted to UTF-8, even if the XML is in ISO-8859-1.

2) when interacting with a ISO-8859-1 database, all strings must be 
converted as well when before inserting and after selecting data.

3) In PHP, the saveXML method of the DOM library seems to convert all 
data to the character set given in the constructor

        $this->doc = new DOMDocument("1.0", CHARSET);

There seems to be no need to manually convert data manually!

4) jQuery seems to send all data as UTF-8, so all JSON data coming from 
the client seems to be UTF-8 as well, no conversion is necessary.

5) When echoing data to the client, it must be converted back to CHARSET.

For cases (2) and (5) I have added new static functions to 
class_administration.

"convertIncomingString" converts a string from CHARSET to UTF-8.
"convertOutgoingString" converts a string from UTF-8 to CHARSET.

We should also think about adding another constant for database 
character set. Currently, if you are using ISO-8859-1 HTML output, 
Mapbender also assumes you are using an ISO-8859-1 database, and vice versa.

Dealing with both character sets is not a problem, if dealt with 
properly. Either we abandon ISO-8859-1, or take some time for proper 
testing and bug fixing, as bug fixing is painful.

Thanks,

Christoph



-- 

----------------------------------

 FOSSGIS Konferenz 2009
 17.-19. März 2009 in Hannover
 http://www.fossgis.de/konferenz

---------------------------------- 

_______________________________________

W h e r e G r o u p GmbH & Co. KG

Siemensstraße 8
53121 Bonn
Germany

Christoph Baudson
Anwendungsentwickler

Fon: +49 (0)228 / 90 90 38 - 15
Fax: +49 (0)228 / 90 90 38 - 11
christoph.baudson at wheregroup.com
www.wheregroup.com
Amtsgericht Bonn, HRA 6788
_______________________________________

Komplementärin:
WhereGroup Verwaltungs GmbH
vertreten durch:
Olaf Knopp, Peter Stamm
_______________________________________