[OSGeo-Discuss] How to assess rendering quality?

[Traian Stanev]

Antialiasing, when done wrong is just "making the edges intermediate colors".

When done right it also involves subpixel positioning which improves not only the visual appearance but also the relative accuracy of lines -- i.e. visual weight is most correctly distributed over exact position of the line given the discrete sampling frequency (pixels). There is actually solid scientific basis in that (Niquist-Shannon theorem). "Sharpness of detail" in non-AA lines is fake accuracy.

Unfortunately antialiasing is also hard to get right, because it depends on gamma correction. For example compare the antialiased output here (http://www.realtimerendering.com/gamma10.png with gamma = 1.0) and here with gamma 2.2 (http://www.realtimerendering.com/gamma22.png). They are both antialiased using the same algorithm, yet one looks better than the other.

Asking people to compare smooth versus sharp will be sensitive not only to the subjective appeal of the picture, but also by the ambient lighting conditions, the gamma used by the AA algorithm versus the gamma of the monitor used for the test, and by whether or not the AA algorithm used subpixel positioning. Also, it depends on whether the image reading application honors the gamma value stored in the image. In addition, you will get "fanboy bias" where people will prefer the output they have seen before. This comes up with things like font glyph rasterization surveys, where people who are more familiar with Macs prefer blurry but correct glyphs while people who use predominantly Windows prefer sharp but deformed glyphs, only because that's what they are used to seeing. So you will need to have lots of questions that ask for the same thing using screenshots that do not say which product they are from. You should also include output from a third party app like Adobe Acrobat. I would also include test questions where the two images being compared are identical.



Traian






> -----Original Message-----
> From: discuss-bounces at lists.osgeo.org [mailto:discuss-
> bounces at lists.osgeo.org] On Behalf Of Ed McNierney
> Sent: Wednesday, October 10, 2007 9:02 AM
> To: OSGeo Discussions
> Subject: RE: [OSGeo-Discuss] How to assess rendering quality?
> 
> Gilles -
> 
> Just keep in mind that subjective metrics are, after all, subjective,
> and some of your metrics are mutually exclusive.  "Smoothness of lines"
> is normally accomplished by antialiasing those lines, making the edges
> intermediate colors and a little "soft".  This is a good thing, but is
> incompatible with "sharpness of details", which is best accomplished
> without antialiasing but with more jagged artifacts on curves and
> diagonal lines.
> 
> 	- Ed
> 
> Ed McNierney
> Chief Mapmaker
> Demand Media / TopoZone.com
> 73 Princeton Street, Suite 305
> North Chelmsford, MA  01863
> Phone: 978-251-4242, Fax: 978-251-1396
> ed at topozone.com
> 
> 
> 
> 
> -----Original Message-----
> From: discuss-bounces at lists.osgeo.org [mailto:discuss-
> bounces at lists.osgeo.org] On Behalf Of Gilles Bassière
> Sent: Wednesday, October 10, 2007 8:38 AM
> To: OSGeo Discussions
> Subject: Re: [OSGeo-Discuss] How to assess rendering quality?
> 
> Hi Michael,
> 
> I'm more concerned with subjective metrics, I actually plan to survey
> some users. The questionnaire will include some map samples and gather
> user preference for each criteria. My problem is to identify what
> questions/criteria should I ask to a user.
> 
> Gilles
> 
> 
> Michael P. Gerlek wrote:
> > Gilles-
> >
> > Is your idea to measure the quality by having a human look at outputs
> (subjective metrics) or automatically via some analysis routine
> (objective metrics)?
> >
> > -mpg
> >
> >
> >
> >
> >> -----Original Message-----
> >> From: discuss-bounces at lists.osgeo.org
> >> [mailto:discuss-bounces at lists.osgeo.org] On Behalf Of Gilles
> Bassière
> >> Sent: Wednesday, October 10, 2007 3:01 AM
> >> To: discuss at lists.osgeo.org
> >> Subject: [OSGeo-Discuss] How to assess rendering quality?
> >>
> >> Hi list,
> >>
> >> I'm doing a comparative study of OpenSource cartographic servers
> >> (Mapserver, Geoserver and Mapnik). Beside raw performance and
> >> features,
> >> I'd like to assess the rendering quality, say how pretty
> >> produced maps
> >> are. Precisely, I'm interested in the quality of the drawing work,
> my
> >> point is not about symbology, nor styling of maps.
> >>
> >> I have some problems to find a set of objective criteria I could
> >> benchmark my servers against. So far, I have already identified the
> >> following:
> >> - sharpness of details
> >> - smoothness of lines
> >> - uniformity of colors
> >>
> >> I'm open to any comments. Do you think these criteria are consistent
> >> regarding the purpose of my study? Does anyone have other criteria
> to
> >> suggest?
> >>
> >> Regards
> >>
> >> --
> >> Gilles Bassiere
> >> MAKINA CORPUS
> >> 30 rue des Jeuneurs
> >> FR-75011 PARIS
> >> http://www.makina-corpus.com
> >>
> >> _______________________________________________
> >> Discuss mailing list
> >> Discuss at lists.osgeo.org
> >> http://lists.osgeo.org/mailman/listinfo/discuss
> >>
> >>
> > _______________________________________________
> > Discuss mailing list
> > Discuss at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/discuss
> >
> 
> 
> --
> Gilles Bassiere
> MAKINA CORPUS
> 30 rue des Jeuneurs
> FR-75011 PARIS
> http://www.makina-corpus.com
> 
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss