[mapserver-users] processing scale=auto with float multibands rasters

Wed Aug 5 01:57:27 PDT 2015

Hi Jim
thx for explain correct formula.
I try it.
The result is surely better, but again it seem a too much smoothing.
But these is the right direction.
I try also with lower values of userSelectedScaleFactor rather than 2.

Meanwhile I try also using directly the min/max retrieved from
statistics and see the the result is absolutelly not usefule.
The rasters have some spikes values (due to noise) that produce an
image with a color dynamic too compress.

Thx,

A.

2015-08-05 4:47 GMT+02:00 Jim Klassen <klassen.js at gmail.com>:
> Generally the formula for this is:
> Mean +/- (userSelectedScaleFactor * StdDev)
>
> So Band 4 (with userSelectedScaleFactor = 2) would be:
> min = 0.082 - (2 * 0.042) = -0.002
> max = 0.082 + (2 * 0.042) = 0.166
>
> On 08/04/2015 04:43 PM, Andrea Peri wrote:
>> Hi Jukka,
>>
>> The value I use for test was retrieve using qgis defaults so I don't
>> know if your formula was the same.
>>
>> So I do a quick test
>> these are the values for my test raster:
>>
>> Band 1  Minimum=0.038, Maximum=1.584, Mean=0.135, StdDev=0.026
>> Band 2  Minimum=0.004, Maximum=1.567, Mean=0.115, StdDev=0.027
>> Band 3  Minimum=0.003, Maximum=1.636, Mean=0.096, StdDev=0.031
>> Band 4  Minimum=-0.016, Maximum=1.796, Mean=0.082, StdDev=0.042
>> Band 5  Minimum=-0.311, Maximum=4.486, Mean=0.256, StdDev=0.138
>> Band 6  Minimum=-0.197, Maximum=3.277, Mean=0.161, StdDev=0.098
>> Band 7  Minimum=-0.118, Maximum=2.295, Mean=0.088, StdDev=0.063
>> Band 8  Minimum=0.039, Maximum=1.366, Mean=0.090, StdDev=0.035
>> Band 9  Minimum=-0.039, Maximum=0.413, Mean=0.003, StdDev=0.010
>> Band 10 Minimum=240.580, Maximum=524.602, Mean=299.351, StdDev=6.342
>> Band 11 Minimum=249.037, Maximum=514.462, Mean=297.375, StdDev=5.855
>>
>> Using you formula:
>> (I hope to understand it correctly)
>>
>> PROCESSING "BANDS=4,3,2"  (the red,green,blue channels)
>>
>> the values should be:
>>
>> Band 4:
>> 0.082 - (-0.016 * 0.042) = 0.082672
>> 0.082 + (1.796 * 0.042) = 0.157432
>>
>> Band 3:
>> 0.096 - (0.003 * 0.031) = 0.095907
>> 0.096 + (1.636 * 0.031) = 0.146716
>>
>> Band 2:
>> 0.115 - (0.004 * 0.027) = 0.114892
>> 0.115 + (1.567 * 0.027) = 0.157309
>>
>> so the setting should be:
>>
>>  PROCESSING "SCALE_1= 0.082672 , 0.157432"
>>  PROCESSING "SCALE_2= 0.095907 , 0.146716"
>>  PROCESSING "SCALE_3= 0.114892 , 0.157309"
>>
>> With this values the image is not correct .
>>
>> I add a link to a sample images of what the mapserver will produce
>> with these values.
>>
>> https://www.dropbox.com/s/izleegjbp18j81f/sample-mapserver-with-values.gif?dl=0
>>
>>
>> A.
>>
>> 2015-08-04 22:52 GMT+02:00 Rahkonen Jukka (MML)
>> <jukka.rahkonen at maanmittauslaitos.fi>:
>>> Hi,
>>>
>>> Do we talk about the same thing? If gdalinfo -stats shows
>>> Minimum=2.000, Maximum=255.000, Mean=96.297, StdDev=37.015
>>> you should put min at 96.297-(2*37.015)
>>> You can test the effect with QGIS as described here http://gis.stackexchange.com/questions/87914/how-to-perform-a-standard-deviation-stretch-on-a-raster-in-qgis
>>> However, I have not really used 64-bit images and don't know what kind of histograms they usually have.
>>>
>>> In Mapserver 2.0 the PROCESSING stuff should be possible to set via variable substitution and your users could make the adjustment on-the-fly. Perhaps they could accept that?
>>>
>>> -Jukka-
>>>
>>> Andrea Peri wrote:
>>>
>>>> Hi Jukka, I could open a feature request for this, but before I need to understand if the results with this strategy are affordable for us.
>>>> Infact I do some test with a domain min/max lesser than 2x and it results unaccepatable for our users. The raster are too poor contrasted as reported me from our users.
>>>> So I'm not sure that an AUTO with a STD stretch and a multiplicator 2x could be instead an acceptable solution.
>>> I should do more tests...
>>>
>>> Thx.
>>>
>>>
>>>
>>> 2015-08-04 20:29 GMT+02:00 Rahkonen Jukka (MML)
>>> <jukka.rahkonen at maanmittauslaitos.fi>:
>>>> Hi Andrea,
>>>>
>>>> For viewing purposes it is rather common to do LUT stretch by standard deviation which is not so sensible as min-max stretch. Consider making a feature request for a new enhanced PROCESSING AUTO directive which would do the STD stretch, possibly with an optional multiplication. 2 x STD might be a good default value.
>>>>
>>>> -Jukka Rahkonen-
>>>>
>>>> Andrea Peri wrote:
>>>>
>>>>> HI Jukka,
>>>>> unfortunatelly, I cannot use the LUT because the raster I use is update weekly and every week it could change the min/max values for every bands (it has 11 bands).
>>>>> Hovewer if it work using the min/max from get map , probably it could be ok form me.
>>>>> The question s that even when the getmap request the entire bbox the image reported seem to be empty without no error log.
>>>>
>>>>
>>>> 2015-08-04 19:50 GMT+02:00 Rahkonen Jukka (MML)
>>>> <jukka.rahkonen at maanmittauslaitos.fi>:
>>>>> Hi,
>>>>>
>>>>> I suppose that Even means that with AUTO setting Mapserver is analyzing the min/max pixel values from inside the BBOX of each GetMap and then scales pixel values to 0-256 for the output.  One totally black or white pixel within the BBOX can make the result from 64bit source very bad.
>>>>>
>>>>> I would try to use LUT instead of SCALE http://www.mapserver.org/input/raster.html#special-processing-directives.
>>>>>
>>>>> -Jukka Rahkonen-
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> -----Alkuperäinen viesti-----l
>>>>> Lähettäjä: mapserver-users-bounces at lists.osgeo.org
>>>>> [mailto:mapserver-users-bounces at lists.osgeo.org] Puolesta Andrea Peri
>>>>> Lähetetty: 4. elokuuta 2015 20:07
>>>>> Vastaanottaja: Even Rouault
>>>>> Kopio: mapserver-users at lists.osgeo.org
>>>>> Aihe: Re: [mapserver-users] processing scale=auto with float
>>>>> multibands rasters
>>>>>
>>>>> Hi Even,
>>>>>
>>>>> thx for response.
>>>>>
>>>>> Please can you help me to better understand the question ?
>>>>>
>>>>> My raster is effectively a 18000 x 24000 px image with tiles.
>>>>>
>>>>> it have a tile size of 1024 x 1024 produced from gdalwarp:
>>>>>
>>>>> gdalwarp -s_srs EPSG:32632 -t_srs EPSG:32632 -overwrite --config
>>>>> GDAL_CACHEMAX 1000 -wm 1000 -co TILED=YES -co COMPRESS=NONE -co
>>>>> BLOCKXSIZE=1024 -co BLOCKYSIZE=1024 -co BIGTIFF=YES -co SPARSE_OK=TRUE .......
>>>>>
>>>>> and also have overviews from
>>>>>
>>>>> gdaladdo --config GDAL_CACHEMAX 1500 -ro -r average ..... 2 4 8 16 32
>>>>> 64 128 256 512 1024
>>>>>
>>>>> If I understand correctly your info.
>>>>> The mapserver could search the min/max using only the first tile ?
>>>>>
>>>>> This mean that it more probably found always only 0 values or near 0 values beacuse these could be the values in the first tile.
>>>>>
>>>>> And so it return a white image because the min/max is always min=0/
>>>>> max=0 (or something quite like this)
>>>>> I understand correct ?
>>>>>
>>>>> thx,
>>>>>
>>>>> A.
>>>>>
>>>>>
>>>>> 2015-08-04 16:01 GMT+02:00 Even Rouault <even.rouault at spatialys.com>:
>>>>>> On Monday 03 August 2015 11:02:43 Andrea Peri wrote:
>>>>>>> Hi,
>>>>>>> I have a geotiff multibands with float64 values to serve with a wms
>>>>>>> mapserver service.
>>>>>>>
>>>>>>> With this kind of rasters,
>>>>>>> is affordable the directive
>>>>>>>
>>>>>>>     PROCESSING "BANDS=4,3,2"
>>>>>>>     PROCESSING "SCALE=AUTO"
>>>>>>>
>>>>>>> (the 4,3,2 are the red,green,blue channels)
>>>>>>>
>>>>>>> Infact it seem do nothing.
>>>>>>>
>>>>>>> Instead all work well if I set:
>>>>>>>
>>>>>>>     PROCESSING "BANDS=4,3,2"
>>>>>>>     PROCESSING "SCALE_1=0.01,0.2"
>>>>>>>     PROCESSING "SCALE_2=0.01,0.2"
>>>>>>>     PROCESSING "SCALE_3=0.01,0.2"
>>>>>>>
>>>>>>> where ii use the min=0.01 and max=0.2 But I like to have the exact
>>>>>>> values of min/max for every raster.
>>>>>> Andrea,
>>>>>>
>>>>>> SCALE=AUTO should work but looking at the actual implementation I
>>>>>> can see it computes the min/max on the pixel values intersecting the
>>>>>> requested area, and not on the whole raster as I'd have expected. So
>>>>>> in a tiling context, it will likely not produce the expected result.
>>>>>>
>>>>>> Even
>>>>>>
>>>>>> --
>>>>>> Spatialys - Geospatial professional services
>>>>>> http://www.spatialys.com
>>>>>
>>>>>
>>>>> --
>>>>> -----------------
>>>>> Andrea Peri
>>>>> . . . . . . . . .
>>>>> qwerty àèìòù
>>>>> -----------------
>>>>> _______________________________________________
>>>>> mapserver-users mailing list
>>>>> mapserver-users at lists.osgeo.org
>>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>>
>>>>
>>>> --
>>>> -----------------
>>>> Andrea Peri
>>>> . . . . . . . . .
>>>> qwerty àèìòù
>>>> -----------------
>>>
>>>
>>> --
>>> -----------------
>>> Andrea Peri
>>> . . . . . . . . .
>>> qwerty àèìòù
>>> -----------------
>>
>>
>
> _______________________________________________
> mapserver-users mailing list
> mapserver-users at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/mapserver-users

-- 
-----------------
Andrea Peri
. . . . . . . . .
qwerty àèìòù
-----------------