[mapserver-dev] Re: [FriBidi] Incorrect Arabic text rendering in maperver

[Stephen Woodbridge]

Hi Mapserver Devs,

I have been helping track down an issue with fibidi and mapserver and 
have an answer from Behdad the fribidi developer. From the response I 
seems that we can improve our output if we filter out the character 
FRIBIDI_CHAR_FILL out of the output character stream. See his response 
below.

Anyone know how to do this? I can write this thread up as a bug if 
needed. Any change this can make it into 6.0?

Hi Behdad,

Many thanks for the quick response! and your very excellent fribidi package.

-Steve W

On 1/17/2011 10:49 AM, Ian Walberg wrote:
> Behdad,
>
> Many thanks for this, however I am not sure I know enough about the
> text rendering to understand what this means.
>
> Hopefully Steve has more experience.
>
> Regards
>
> Ian
>
> -----Original Message----- From: Behdad Esfahbod
> [mailto:behdad.esfahbod at gmail.com] On Behalf Of Behdad Esfahbod Sent:
> Monday, January 17, 2011 6:56 AM To: Stephen Woodbridge Cc: Ian
> Walberg; fribidi at lists.freedesktop.org Subject: Re: [FriBidi]
> Incorrect Arabic text rendering in maperver
>
> Ok, I actually have an answer!
>
> Just filter out the character FRIBIDI_CHAR_FILL out of the output
> character stream.  When FriBidi forms a ligature, it stuffs in
> FRIBIDI_CHAR_FILL characters in instead.  That character is actually
> the Unicode ZeroWidth NoBreak Space, so if your font has a glyph for
> it, there's no problem at all, otherwise you get that box.  It's safe
> to not render that character.
>
> Hope that helps.
>
> behdad
>
> PS.  Have you considered using Pango in mapserver?
>
> On 01/17/11 09:40, Stephen Woodbridge wrote:
>> Hi Behdad,
>>
>> In mapserver, mapserver.org, all strings are run through the
>> following function:
>>
>> On my linux box, assume: #define USE_FRIBIDI #undef
>> FRIBIDI_NO_CHARSETS
>>
>> I am assuming on windows builds: #define USE_FRIBIDI #define
>> FRIBIDI_NO_CHARSETS
>>
>> Ian, are you using Windows or Linux?
>>
>> After the string is encoded, it is rendered using freetype and
>> hopefully an appropriate font. I have had clients on Linux use this
>> successfully except that most fonts do not seem to have all the
>> joining characters. Missing characters typically showed up as a
>> square box or just unjoined characters. They have been able to get
>> good results by finding a font with more of the joining characters
>> included and in some cases by using a font editor to add the
>> missing glyphs. I'm not sure if this is Ian's problem or not.
>>
>> Also I would be interested in knowing of any good truetype fonts
>> that work well with Fribidi for Arabic, especially any free ones.
>>
>> Best regards, -Steve
>>
>> #ifdef USE_FRIBIDI char *msGetFriBidiEncodedString(const char
>> *string, const char *encoding) { FriBidiChar logical[MAX_STR_LEN];
>> FriBidiCharType base = FRIBIDI_TYPE_ON; size_t len;
>>
>> #ifdef FRIBIDI_NO_CHARSETS iconv_t to_ucs4, from_ucs4; #else int
>> to_char_set_num; int from_char_set_num; #endif
>>
>> len = strlen(string);
>>
>> #ifdef FRIBIDI_NO_CHARSETS to_ucs4 = iconv_open ("WCHAR_T",
>> encoding); from_ucs4 = iconv_open ("UTF-8", "WCHAR_T"); #else
>> to_char_set_num = fribidi_parse_charset ((char*)encoding);
>> from_char_set_num = fribidi_parse_charset ("UTF-8"); #endif
>>
>> #ifdef FRIBIDI_NO_CHARSETS if (to_ucs4 == (iconv_t) (-1) ||
>> from_ucs4 == (iconv_t) (-1)) #else if (!to_char_set_num ||
>> !from_char_set_num) #endif { msSetError(MS_IDENTERR, "Encoding not
>> supported (%s).", "msGetFriBidiEncodedString()", encoding); return
>> NULL; }
>>
>> #ifdef FRIBIDI_NO_CHARSETS { char *st = string, *ust = (char *)
>> logical; int in_len = (int) len; len = sizeof logical; iconv
>> (to_ucs4,&st,&in_len,&ust, (int *)&len); len = (FriBidiChar *) ust
>> - logical; } #else len = fribidi_charset_to_unicode
>> (to_char_set_num, (char*)string, len, logical); #endif
>>
>> { FriBidiChar *visual; char outstring[MAX_STR_LEN]; FriBidiStrIndex
>> *ltov, *vtol; FriBidiLevel *levels; FriBidiStrIndex new_len;
>> fribidi_boolean log2vis;
>>
>> visual = (FriBidiChar *) malloc (sizeof (FriBidiChar) * (len +
>> 1)); ltov = NULL; vtol = NULL; levels = NULL;
>>
>> /* Create a bidi string. */ log2vis = fribidi_log2vis (logical,
>> len,&base, /* output */ visual, ltov, vtol, levels);
>>
>> if (!log2vis) { msSetError(MS_IDENTERR, "Failed to create bidi
>> string.", "msGetFriBidiEncodedString()"); return NULL; }
>>
>> new_len = len;
>>
>> /* Convert it to utf-8 for display. */ #ifdef FRIBIDI_NO_CHARSETS
>> { char *str = outstring, *ust = (char *) visual; int in_len = len *
>> sizeof visual[0]; new_len = sizeof outstring; iconv
>> (from_ucs4,&ust,&in_len,&str, (int *)&new_len); *str = '\0';
>> new_len = str - outstring; } #else new_len =
>> fribidi_unicode_to_charset (from_char_set_num, visual, len,
>> outstring); #endif return strdup(outstring); } } #endif
>>
>>
>> On 1/17/2011 8:48 AM, Behdad Esfahbod wrote:
>>> And how are you calling FriBidi exactly?
>>>
>>> On 01/15/11 11:53, Ian Walberg wrote:
>>>> As requested here are the images of the problem we are seeing
>>>> and the text string that is not working.
>>>>
>>>>
>>>>
>>>> This is UniversOTOArabic:- http://i52.tinypic.com/156s5ea.jpg
>>>>
>>>>
>>>>
>>>> This is ArialUni :- http://i54.tinypic.com/25ouses.jpg
>>>>
>>>>
>>>>
>>>> This is andale and as far as we can tell is correct
>>>> :-http://i53.tinypic.com/hx56kn.jpg
>>>>
>>>>
>>>>
>>>> And in English :- http://i54.tinypic.com/i1jkmb.jpg
>>>>
>>>>
>>>>
>>>> arabic.htm has the Arabic text for Alexandria in it.
>>>>
>>>>
>>>>
>>>> Arabic Issues 01.xls has the English and Arabic for
>>>> Alexandria.
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>>
>>>>
>>>> Ian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *From:*Ian Walberg *Sent:* Wednesday, January 05, 2011 8:02 PM
>>>> *To:* fribidi at lists.freedesktop.org *Subject:* Incorrect Arabic
>>>> text rendering in maperver
>>>>
>>>>
>>>>
>>>> Hello list and Happy New Year,
>>>>
>>>>
>>>>
>>>> I have posted this on the mapserver list and our current
>>>> assumption is this is an issue related to fribidi and the font
>>>> we are using.
>>>>
>>>>
>>>>
>>>> We are seeing incorrectly rendered Arabic text, typically a
>>>> 'square' displayed instead of one of the characters or
>>>> sometimes one or more characters missing.
>>>>
>>>>
>>>>
>>>> The data is coming from shape files and most the names appear
>>>> to be displayed correctly.
>>>>
>>>>
>>>>
>>>> This is being seen on both ms4w and the target Linux
>>>> installation.
>>>>
>>>>
>>>>
>>>> Any idea where to look would be greatly appreciated. Is the
>>>> 'square' character what Freetype displays if it cannot find a
>>>> character in the font.
>>>>
>>>>
>>>>
>>>> Specifically we get the issue with the Arabic text for the city
>>>> Alexandria (in Egypt) with the font UniversOTSArabic.ttf and
>>>> ArialUni.ttf. Andaltewt.ttf works ok.
>>>>
>>>>
>>>>
>>>> Many thanks
>>>>
>>>>
>>>>
>>>> Ian
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________ fribidi mailing
>>>> list fribidi at lists.freedesktop.org
>>>> http://lists.freedesktop.org/mailman/listinfo/fribidi
>>> _______________________________________________ fribidi mailing
>>> list fribidi at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/fribidi
>>
>>