[mapserver-dev] [mapserver-users] Rendering Hindi Characters

Fri Apr 5 00:36:59 PDT 2013

Ian,
Steve's explanation of the rendering pipeline is spot on, and your issue is
clearly showing a bug somewhere in the rendering pipeline. You can open an
issue at https://github.com/mapserver/mapserver/issues , however I suspect
that the resolution implies switching from fribidi to another shaping
library (icu, harfbuzz), i.e. is not going to happen without funding.

regards,
thomas

On 4 April 2013 17:12, Ian Walberg <ian.walberg at airborne.aero> wrote:

> Folks,
>
> Added the dev list, how do I raise a ticket on this?
>
> Thanks
>
> Ian
>
> -----Original Message-----
> From: mapserver-users-bounces at lists.osgeo.org [mailto:
> mapserver-users-bounces at lists.osgeo.org] On Behalf Of Ian Walberg
> Sent: Tuesday, April 02, 2013 2:34 PM
> To: Stephen Woodbridge; mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] Rendering Hindi Characters
>
> Hello again list,
>
> The Hindi rendering problem has come to the top of our priorities again
> and we have been investigating the cause of the issue again.
>
> To recap the problem we are seeing can be reproduced easily, paste Mexico
> City into google translate and then use the Hindi characters with any
> Unicode font.
>
> The top (incorrect)  line of text is from mapserver and lower (correct)
> line is from Excel in this image http://tinyurl.com/cs7pax5
>
> From the research we have done freetype does not know how to render all of
> these characters together correctly. It does appear that pango can be
> configured to render them correctly.
>
> I have not yet dived into the details of the rendering pipeline and I hope
> that someone can jump in and confirm we have a real issue and there is not
> something stupid we have done.
>
> Many thanks
>
> Ian
>
> -----Original Message-----
> From: mapserver-users-bounces at lists.osgeo.org [mailto:
> mapserver-users-bounces at lists.osgeo.org] On Behalf Of Stephen Woodbridge
> Sent: Wednesday, January 23, 2013 5:06 PM
> To: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-users] Rendering Hindi Characters
>
> FYI, Mapserver does not do anything for rendering text except:
>
> 1. pass the text to fribidi which converts a utf8 sequence of characters
> into a "displayable" sequence of characters.
>
> 2. mapserver then passes that to the text rendering function of whatever
> renderer you are using.
>
> Mapserver does not try to understand it, it totally relies on the
> underlying rendering code to do the right thing whatever that is.
>
> Thomas Bonfort might have some more insight into this as he re-engineer a
> lot of the rendering pipeline and it might be helpful to know what code is
> doing the actual text rendering. It used to be freetype and may still be.
>
> I'm also not convinced the fribidi is not the right place for this to
> happen regardless as it is responsible for taking a sequence of characters
> and generating an "displayable" sequence that can be rendered correctly
> given the correct font.
>
> -Steve
>
> On 1/23/2013 7:17 PM, Mr. Puneet Kishor wrote:
> > Peter,
> >
> > My guru in all things Hindi fonts tells me that the problem lies at the
> level where encodings are converted into font renderings. Apparently, र् ट
> is a perfectly correct alternate rendering of the unicode, just not Correct
> (with a capital C). I have no idea where this happens with MapServer.
> Thankfully I've never had to create a map with Hindi labels (I would be
> curious to learn, perhaps off-list, who wants to see maps with Hindi
> labels), but I too would like to see this resolved.
> >
> > Hope someone knowledgable chimes in.
> >
> >
> > On Jan 23, 2013, at 3:58 PM, "Mr. Puneet Kishor" <punk.kish at gmail.com>
> wrote:
> >
> >> (adding the list back to cc so brains better than mine may help)
> >>
> >>
> >> On Jan 23, 2013, at 3:47 PM, "Peter Mallen" <peter.mallen at airborne.aero>
> wrote:
> >>
> >>> Hi Puneet,
> >>>
> >>> I have passed the codes you have provided into MapServer  (U+0930
> U+094D U+091F ) and it displays the characters separately, not as one
> character.
> >>>
> >>> This is how Map Server displays it: र् ट
> >>
> >>
> >>
> >> hmmm... well, then this is definitely above my pay grade. Seems like
> you are doing the right thing, but my guess is MapServer's Unicode
> implementation is not working properly.
> >>
> >> I don't see any mention of Unicode on mapserver docs page. The only
> >> potentially relevant bit I see is
> >> (http://www.mapserver.org/installation/unix.html#obtaining-the-necess
> >> ary-software) that libgd should be compiled against the iconv library
> >> which provides full support for Unicode. On my MBA I get
> >>
> >>     $ ~ > gdlib-config --libs
> >>     -lXpm -lX11 -ljpeg -lfontconfig -lfreetype -lpng -lz -liconv
> >>
> >>
> >>
> >>>
> >>> Thank You,
> >>>
> >>> Peter Mallen
> >>> -----Original Message-----
> >>> From: Mr. Puneet Kishor [mailto:punk.kish at gmail.com]
> >>> Sent: Wednesday, January 23, 2013 3:36 PM
> >>> To: Peter Mallen
> >>> Subject: Re: [mapserver-users] Rendering Hindi Characters
> >>>
> >>>
> >>> On Jan 23, 2013, at 3:00 PM, "Peter Mallen" <
> peter.mallen at airborne.aero> wrote:
> >>>
> >>>> Hi Puneet,
> >>>>
> >>>> I did try feeding MapServer the correct Unicode characters, and it
> did fix the characters using what I believe is called the "Reph" symbol (ि)
> but I could not get the conjunct consonants to display correctly.
> >>>
> >>>
> >>> I have no idea what the above means ;-), but, did you try passing
> >>> U+0930 U+094D U+091F to MapServer? See
> >>> http://www.unicode.org/charts/PDF/U0900.pdf
> >>>
> >>>
> >>> U+0930 -> र -> r
> >>> U+094D ->  ् -> "halves" the previously typed r 091F -> ट -> thick t
> >>> U+sound
> >>>
> >>>
> >>>
> >>>>
> >>>> Also, I have posted on the fribidi list and I believe it does only
> help with bidirectional languages, but I could be wrong.
> >>>>
> >>>> Thank You,
> >>>>
> >>>> Peter Mallen
> >>>>
> >>>> -----Original Message-----
> >>>> From: mapserver-users-bounces at lists.osgeo.org
> >>>> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Mr.
> >>>> Puneet Kishor
> >>>> Sent: Wednesday, January 23, 2013 12:27 PM
> >>>> To: mapserver-users at lists.osgeo.org
> >>>> Subject: Re: [mapserver-users] Rendering Hindi Characters
> >>>>
> >>>>
> >>>> On Jan 23, 2013, at 12:17 PM, Stephen Woodbridge <
> woodbri at swoodbridge.com> wrote:
> >>>>
> >>>>> On 1/23/2013 2:20 PM, Peter Mallen wrote:
> >>>>>> Hello Everyone,
> >>>>>>
> >>>>>> I am having issues rendering certain Hindi characters. I am using
> >>>>>> UTF-8 encoding.
> >>>>>>
> >>>>>> When attempting to display the following Hindi character in
> >>>>>> MapServer: र्ट
> >>>>>>
> >>>>>> It is rendered in MapServer as the following two characters: र् ट
> >>>>>>
> >>>>>> I believe these characters form what is called a conjunct
> >>>>>> consonant, of which there appear to be thousands of and most of
> >>>>>> them do not render using MapServer.
> >>>>>>
> >>>>>> Also, It appears that MapServer is having an issue displaying
> >>>>>> these conjunct consonants and wants to display these characters
> >>>>>> separately rather than to form them as one combined character.
> >>>>>>
> >>>>>> Here is a link describing more about the Devanagari alphabet:
> >>>>>> http://www.omniglot.com/writing/devanagari.htm
> >>>>>>
> >>>>>> Can anyone please help with this?
> >>>>>>
> >>>>>> Thank You,
> >>>>>>
> >>>>>> Peter Mallen
> >>>>>
> >>>>> Hi Peter,
> >>>>>
> >>>>> I can not speak to this specific problem but I can give you and
> overview. Mapserver uses fribidi library for rending this data and I know
> from working with Arabic that a lot depends on the font you are using, in
> additional to the capabilities and version of fribidi library.
> >>>>
> >>>>
> >>>>
> >>>> Isn't fribidi specifically for bidirectional languages? Hindi is
> strictly LTR.
> >>>>
> >>>>
> >>>>>
> >>>>> This kind of question is probably best asked on the fribidi list as
> they will know more about what is supported or not for any given language
> and they might be able to point you to a better font.
> >>>>>
> >>>>> http://www.fribidi.org/
> >>>>>
> >>>>> You should probably make sure you are using a recent version of the
> library like 0.19.2 or better and be aware the 0.19.5 was released in Dec
> 2012.
> >>>>>
> >>>>
> >>>>
> >>>> Peter, I forget where you and I left our off-list conversation re.
> the issues you were having with rendering Devnagari. Did you try feeding
> MapServer the correct Unicode characters?
> >>>>
> >>>> Hope you figure this out.
> >>>>
> >>>>
> >>>> --
> >>>> Puneet Kishor
> >>>
> >>
> >
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20130405/de420edb/attachment.html>