[mapserver-dev] [mapserver-users] Rendering Hindi Characters

Fri Apr 5 06:50:53 PDT 2013

http://behdad.org/text/ is a good read if you haven't already stumbled upon
it.

--
thomas

On 5 April 2013 15:45, Ian Walberg <ian.walberg at airborne.aero> wrote:

> Steve,
>
> I will raise an issue to track this and we will be looking into it in more
> detail.
>
> From the research we have done so far I am not sure that it is an issue
> with fribidi as my understanding that this just addresses the l->r and r->l
> ordering issues.
>
> What we believe is that the mapserver text rendering pipeline cannot
> support the more complex character layouts required in some cases for
> Hinidi and other languages.
>
> Regards
>
> Ian
>
> -----Original Message-----
> From: mapserver-dev-bounces at lists.osgeo.org [mailto:
> mapserver-dev-bounces at lists.osgeo.org] On Behalf Of Stephen Woodbridge
> Sent: Friday, April 05, 2013 5:56 AM
> To: mapserver-dev
> Cc: mapserver-users at lists.osgeo.org
> Subject: Re: [mapserver-dev] [mapserver-users] Rendering Hindi Characters
>
> Ian,
>
> As I mentioned before, I think this is an issue with fribidi or the font
> that you are using. I have seen nothing that indicates that this is a
> mapserver problem other than the implied suggestion that maybe mapserver
> should find a different library to solve this issue.
>
> Did you post this issue on the fribidi list?
> Did you post the response back here?
>
> I saw that you tried the text in other windows products and it works. I
> can appreciate your frustration with this fact, but this does not in itself
> mean the mapserver is broken. We rely on fribidi and the selected font to
> generate the correct glyphs for rendering. If the gylphs are not rendered
> correctly there are a few possibilities:
>
> 1. a bug in mapserver - not likely as we have plenty of working examples
> using multiple other languages 2. a bug/limitation in fribidi 3. a font
> file with missing or broken glyphs.
>
> In most cases of rendering problems that we have seen to date have been
> problems with fonts. The fribidi list has been helpful identifying good
> fonts for specific languages.
>
> If you want to file a bug on the mapserver site, goto
> http://mapserver.org/ and follow the link on the top right of the page to
> the issue tracker. You will need to supply a test case that demonstrates
> the problem and some graphics so non hindi readers can see what is bad and
> what is expected.
>
> Without some more clear evidence that this is a mapserver problem, I'm not
> sure this will get much action.
>
> I am sympathetic, as I ran into a similar issue with Arabic a few years
> ago and had to do this whole dance with fribidi to understand the issues
> and identify a font that resolved the issues.
>
> -Steve W
>
> On 4/4/2013 11:12 AM, Ian Walberg wrote:
> > Folks,
> >
> > Added the dev list, how do I raise a ticket on this?
> >
> > Thanks
> >
> > Ian
> >
> > -----Original Message-----
> > From: mapserver-users-bounces at lists.osgeo.org
> > [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Ian
> > Walberg
> > Sent: Tuesday, April 02, 2013 2:34 PM
> > To: Stephen Woodbridge; mapserver-users at lists.osgeo.org
> > Subject: Re: [mapserver-users] Rendering Hindi Characters
> >
> > Hello again list,
> >
> > The Hindi rendering problem has come to the top of our priorities again
> and we have been investigating the cause of the issue again.
> >
> > To recap the problem we are seeing can be reproduced easily, paste
> Mexico City into google translate and then use the Hindi characters with
> any Unicode font.
> >
> > The top (incorrect)  line of text is from mapserver and lower
> > (correct) line is from Excel in this image http://tinyurl.com/cs7pax5
> >
> >  From the research we have done freetype does not know how to render all
> of these characters together correctly. It does appear that pango can be
> configured to render them correctly.
> >
> > I have not yet dived into the details of the rendering pipeline and I
> hope that someone can jump in and confirm we have a real issue and there is
> not something stupid we have done.
> >
> > Many thanks
> >
> > Ian
> >
> > -----Original Message-----
> > From: mapserver-users-bounces at lists.osgeo.org
> > [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Stephen
> > Woodbridge
> > Sent: Wednesday, January 23, 2013 5:06 PM
> > To: mapserver-users at lists.osgeo.org
> > Subject: Re: [mapserver-users] Rendering Hindi Characters
> >
> > FYI, Mapserver does not do anything for rendering text except:
> >
> > 1. pass the text to fribidi which converts a utf8 sequence of characters
> into a "displayable" sequence of characters.
> >
> > 2. mapserver then passes that to the text rendering function of whatever
> renderer you are using.
> >
> > Mapserver does not try to understand it, it totally relies on the
> underlying rendering code to do the right thing whatever that is.
> >
> > Thomas Bonfort might have some more insight into this as he re-engineer
> a lot of the rendering pipeline and it might be helpful to know what code
> is doing the actual text rendering. It used to be freetype and may still be.
> >
> > I'm also not convinced the fribidi is not the right place for this to
> happen regardless as it is responsible for taking a sequence of characters
> and generating an "displayable" sequence that can be rendered correctly
> given the correct font.
> >
> > -Steve
> >
> > On 1/23/2013 7:17 PM, Mr. Puneet Kishor wrote:
> >> Peter,
> >>
> >> My guru in all things Hindi fonts tells me that the problem lies at the
> level where encodings are converted into font renderings. Apparently, र् ट
> is a perfectly correct alternate rendering of the unicode, just not Correct
> (with a capital C). I have no idea where this happens with MapServer.
> Thankfully I've never had to create a map with Hindi labels (I would be
> curious to learn, perhaps off-list, who wants to see maps with Hindi
> labels), but I too would like to see this resolved.
> >>
> >> Hope someone knowledgable chimes in.
> >>
> >>
> >> On Jan 23, 2013, at 3:58 PM, "Mr. Puneet Kishor" <punk.kish at gmail.com>
> wrote:
> >>
> >>> (adding the list back to cc so brains better than mine may help)
> >>>
> >>>
> >>> On Jan 23, 2013, at 3:47 PM, "Peter Mallen" <
> peter.mallen at airborne.aero> wrote:
> >>>
> >>>> Hi Puneet,
> >>>>
> >>>> I have passed the codes you have provided into MapServer  (U+0930
> U+094D U+091F ) and it displays the characters separately, not as one
> character.
> >>>>
> >>>> This is how Map Server displays it: र् ट
> >>>
> >>>
> >>> hmmm... well, then this is definitely above my pay grade. Seems like
> you are doing the right thing, but my guess is MapServer's Unicode
> implementation is not working properly.
> >>>
> >>> I don't see any mention of Unicode on mapserver docs page. The only
> >>> potentially relevant bit I see is
> >>> (http://www.mapserver.org/installation/unix.html#obtaining-the-neces
> >>> s
> >>> ary-software) that libgd should be compiled against the iconv
> >>> library which provides full support for Unicode. On my MBA I get
> >>>
> >>>      $ ~ > gdlib-config --libs
> >>>      -lXpm -lX11 -ljpeg -lfontconfig -lfreetype -lpng -lz -liconv
> >>>
> >>>
> >>>
> >>>> Thank You,
> >>>>
> >>>> Peter Mallen
> >>>> -----Original Message-----
> >>>> From: Mr. Puneet Kishor [mailto:punk.kish at gmail.com]
> >>>> Sent: Wednesday, January 23, 2013 3:36 PM
> >>>> To: Peter Mallen
> >>>> Subject: Re: [mapserver-users] Rendering Hindi Characters
> >>>>
> >>>>
> >>>> On Jan 23, 2013, at 3:00 PM, "Peter Mallen" <
> peter.mallen at airborne.aero> wrote:
> >>>>
> >>>>> Hi Puneet,
> >>>>>
> >>>>> I did try feeding MapServer the correct Unicode characters, and it
> did fix the characters using what I believe is called the "Reph" symbol (ि)
> but I could not get the conjunct consonants to display correctly.
> >>>>
> >>>> I have no idea what the above means ;-), but, did you try passing
> >>>> U+0930 U+094D U+091F to MapServer? See
> >>>> http://www.unicode.org/charts/PDF/U0900.pdf
> >>>>
> >>>>
> >>>> U+0930 -> र -> r
> >>>> U+094D ->  ् -> "halves" the previously typed r 091F -> ट -> thick
> >>>> U+t sound
> >>>>
> >>>>
> >>>>
> >>>>> Also, I have posted on the fribidi list and I believe it does only
> help with bidirectional languages, but I could be wrong.
> >>>>>
> >>>>> Thank You,
> >>>>>
> >>>>> Peter Mallen
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: mapserver-users-bounces at lists.osgeo.org
> >>>>> [mailto:mapserver-users-bounces at lists.osgeo.org] On Behalf Of Mr.
> >>>>> Puneet Kishor
> >>>>> Sent: Wednesday, January 23, 2013 12:27 PM
> >>>>> To: mapserver-users at lists.osgeo.org
> >>>>> Subject: Re: [mapserver-users] Rendering Hindi Characters
> >>>>>
> >>>>>
> >>>>> On Jan 23, 2013, at 12:17 PM, Stephen Woodbridge <
> woodbri at swoodbridge.com> wrote:
> >>>>>
> >>>>>> On 1/23/2013 2:20 PM, Peter Mallen wrote:
> >>>>>>> Hello Everyone,
> >>>>>>>
> >>>>>>> I am having issues rendering certain Hindi characters. I am
> >>>>>>> using
> >>>>>>> UTF-8 encoding.
> >>>>>>>
> >>>>>>> When attempting to display the following Hindi character in
> >>>>>>> MapServer: र्ट
> >>>>>>>
> >>>>>>> It is rendered in MapServer as the following two characters: र्
> >>>>>>> ट
> >>>>>>>
> >>>>>>> I believe these characters form what is called a conjunct
> >>>>>>> consonant, of which there appear to be thousands of and most of
> >>>>>>> them do not render using MapServer.
> >>>>>>>
> >>>>>>> Also, It appears that MapServer is having an issue displaying
> >>>>>>> these conjunct consonants and wants to display these characters
> >>>>>>> separately rather than to form them as one combined character.
> >>>>>>>
> >>>>>>> Here is a link describing more about the Devanagari alphabet:
> >>>>>>> http://www.omniglot.com/writing/devanagari.htm
> >>>>>>>
> >>>>>>> Can anyone please help with this?
> >>>>>>>
> >>>>>>> Thank You,
> >>>>>>>
> >>>>>>> Peter Mallen
> >>>>>> Hi Peter,
> >>>>>>
> >>>>>> I can not speak to this specific problem but I can give you and
> overview. Mapserver uses fribidi library for rending this data and I know
> from working with Arabic that a lot depends on the font you are using, in
> additional to the capabilities and version of fribidi library.
> >>>>>
> >>>>>
> >>>>> Isn't fribidi specifically for bidirectional languages? Hindi is
> strictly LTR.
> >>>>>
> >>>>>
> >>>>>> This kind of question is probably best asked on the fribidi list as
> they will know more about what is supported or not for any given language
> and they might be able to point you to a better font.
> >>>>>>
> >>>>>> http://www.fribidi.org/
> >>>>>>
> >>>>>> You should probably make sure you are using a recent version of the
> library like 0.19.2 or better and be aware the 0.19.5 was released in Dec
> 2012.
> >>>>>>
> >>>>>
> >>>>> Peter, I forget where you and I left our off-list conversation re.
> the issues you were having with rendering Devnagari. Did you try feeding
> MapServer the correct Unicode characters?
> >>>>>
> >>>>> Hope you figure this out.
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Puneet Kishor
> >> _______________________________________________
> >> mapserver-users mailing list
> >> mapserver-users at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/mapserver-users
> >>
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
> > _______________________________________________
> > mapserver-users mailing list
> > mapserver-users at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
> _______________________________________________
> mapserver-dev mailing list
> mapserver-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-dev
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20130405/f1e810c3/attachment-0001.html>