[mapserver-users] processing scale=auto with float multibands rasters

Tue Aug 4 19:47:53 PDT 2015

Generally the formula for this is:
Mean +/- (userSelectedScaleFactor * StdDev)

So Band 4 (with userSelectedScaleFactor = 2) would be:
min = 0.082 - (2 * 0.042) = -0.002
max = 0.082 + (2 * 0.042) = 0.166

On 08/04/2015 04:43 PM, Andrea Peri wrote:
> Hi Jukka,
>
> The value I use for test was retrieve using qgis defaults so I don't
> know if your formula was the same.
>
> So I do a quick test
> these are the values for my test raster:
>
> Band 1  Minimum=0.038, Maximum=1.584, Mean=0.135, StdDev=0.026
> Band 2  Minimum=0.004, Maximum=1.567, Mean=0.115, StdDev=0.027
> Band 3  Minimum=0.003, Maximum=1.636, Mean=0.096, StdDev=0.031
> Band 4  Minimum=-0.016, Maximum=1.796, Mean=0.082, StdDev=0.042
> Band 5  Minimum=-0.311, Maximum=4.486, Mean=0.256, StdDev=0.138
> Band 6  Minimum=-0.197, Maximum=3.277, Mean=0.161, StdDev=0.098
> Band 7  Minimum=-0.118, Maximum=2.295, Mean=0.088, StdDev=0.063
> Band 8  Minimum=0.039, Maximum=1.366, Mean=0.090, StdDev=0.035
> Band 9  Minimum=-0.039, Maximum=0.413, Mean=0.003, StdDev=0.010
> Band 10 Minimum=240.580, Maximum=524.602, Mean=299.351, StdDev=6.342
> Band 11 Minimum=249.037, Maximum=514.462, Mean=297.375, StdDev=5.855
>
> Using you formula:
> (I hope to understand it correctly)
>
> PROCESSING "BANDS=4,3,2"  (the red,green,blue channels)
>
> the values should be:
>
> Band 4:
> 0.082 - (-0.016 * 0.042) = 0.082672
> 0.082 + (1.796 * 0.042) = 0.157432
>
> Band 3:
> 0.096 - (0.003 * 0.031) = 0.095907
> 0.096 + (1.636 * 0.031) = 0.146716
>
> Band 2:
> 0.115 - (0.004 * 0.027) = 0.114892
> 0.115 + (1.567 * 0.027) = 0.157309
>
> so the setting should be:
>
>  PROCESSING "SCALE_1= 0.082672 , 0.157432"
>  PROCESSING "SCALE_2= 0.095907 , 0.146716"
>  PROCESSING "SCALE_3= 0.114892 , 0.157309"
>
> With this values the image is not correct .
>
> I add a link to a sample images of what the mapserver will produce
> with these values.
>
> https://www.dropbox.com/s/izleegjbp18j81f/sample-mapserver-with-values.gif?dl=0
>
>
> A.
>
> 2015-08-04 22:52 GMT+02:00 Rahkonen Jukka (MML)
> <jukka.rahkonen at maanmittauslaitos.fi>:
>> Hi,
>>
>> Do we talk about the same thing? If gdalinfo -stats shows
>> Minimum=2.000, Maximum=255.000, Mean=96.297, StdDev=37.015
>> you should put min at 96.297-(2*37.015)
>> You can test the effect with QGIS as described here http://gis.stackexchange.com/questions/87914/how-to-perform-a-standard-deviation-stretch-on-a-raster-in-qgis
>> However, I have not really used 64-bit images and don't know what kind of histograms they usually have.
>>
>> In Mapserver 2.0 the PROCESSING stuff should be possible to set via variable substitution and your users could make the adjustment on-the-fly. Perhaps they could accept that?
>>
>> -Jukka-
>>
>> Andrea Peri wrote:
>>
>>> Hi Jukka, I could open a feature request for this, but before I need to understand if the results with this strategy are affordable for us.
>>> Infact I do some test with a domain min/max lesser than 2x and it results unaccepatable for our users. The raster are too poor contrasted as reported me from our users.
>>> So I'm not sure that an AUTO with a STD stretch and a multiplicator 2x could be instead an acceptable solution.
>> I should do more tests...
>>
>> Thx.
>>
>>
>>
>> 2015-08-04 20:29 GMT+02:00 Rahkonen Jukka (MML)
>> <jukka.rahkonen at maanmittauslaitos.fi>:
>>> Hi Andrea,
>>>
>>> For viewing purposes it is rather common to do LUT stretch by standard deviation which is not so sensible as min-max stretch. Consider making a feature request for a new enhanced PROCESSING AUTO directive which would do the STD stretch, possibly with an optional multiplication. 2 x STD might be a good default value.
>>>
>>> -Jukka Rahkonen-
>>>
>>> Andrea Peri wrote:
>>>
>>>> HI Jukka,
>>>> unfortunatelly, I cannot use the LUT because the raster I use is update weekly and every week it could change the min/max values for every bands (it has 11 bands).
>>>> Hovewer if it work using the min/max from get map , probably it could be ok form me.
>>>> The question s that even when the getmap request the entire bbox the image reported seem to be empty without no error log.
>>>
>>>
>>> 2015-08-04 19:50 GMT+02:00 Rahkonen Jukka (MML)
>>> <jukka.rahkonen at maanmittauslaitos.fi>:
>>>> Hi,
>>>>
>>>> I suppose that Even means that with AUTO setting Mapserver is analyzing the min/max pixel values from inside the BBOX of each GetMap and then scales pixel values to 0-256 for the output.  One totally black or white pixel within the BBOX can make the result from 64bit source very bad.
>>>>
>>>> I would try to use LUT instead of SCALE http://www.mapserver.org/input/raster.html#special-processing-directives.
>>>>
>>>> -Jukka Rahkonen-
>>>>
>>>>
>>>>
>>>>
>>>> -----Alkuperäinen viesti-----l
>>>> Lähettäjä: mapserver-users-bounces at lists.osgeo.org
>>>> [mailto:mapserver-users-bounces at lists.osgeo.org] Puolesta Andrea Peri
>>>> Lähetetty: 4. elokuuta 2015 20:07
>>>> Vastaanottaja: Even Rouault
>>>> Kopio: mapserver-users at lists.osgeo.org
>>>> Aihe: Re: [mapserver-users] processing scale=auto with float
>>>> multibands rasters
>>>>
>>>> Hi Even,
>>>>
>>>> thx for response.
>>>>
>>>> Please can you help me to better understand the question ?
>>>>
>>>> My raster is effectively a 18000 x 24000 px image with tiles.
>>>>
>>>> it have a tile size of 1024 x 1024 produced from gdalwarp:
>>>>
>>>> gdalwarp -s_srs EPSG:32632 -t_srs EPSG:32632 -overwrite --config
>>>> GDAL_CACHEMAX 1000 -wm 1000 -co TILED=YES -co COMPRESS=NONE -co
>>>> BLOCKXSIZE=1024 -co BLOCKYSIZE=1024 -co BIGTIFF=YES -co SPARSE_OK=TRUE .......
>>>>
>>>> and also have overviews from
>>>>
>>>> gdaladdo --config GDAL_CACHEMAX 1500 -ro -r average ..... 2 4 8 16 32
>>>> 64 128 256 512 1024
>>>>
>>>> If I understand correctly your info.
>>>> The mapserver could search the min/max using only the first tile ?
>>>>
>>>> This mean that it more probably found always only 0 values or near 0 values beacuse these could be the values in the first tile.
>>>>
>>>> And so it return a white image because the min/max is always min=0/
>>>> max=0 (or something quite like this)
>>>> I understand correct ?
>>>>
>>>> thx,
>>>>
>>>> A.
>>>>
>>>>
>>>> 2015-08-04 16:01 GMT+02:00 Even Rouault <even.rouault at spatialys.com>:
>>>>> On Monday 03 August 2015 11:02:43 Andrea Peri wrote:
>>>>>> Hi,
>>>>>> I have a geotiff multibands with float64 values to serve with a wms
>>>>>> mapserver service.
>>>>>>
>>>>>> With this kind of rasters,
>>>>>> is affordable the directive
>>>>>>
>>>>>>     PROCESSING "BANDS=4,3,2"
>>>>>>     PROCESSING "SCALE=AUTO"
>>>>>>
>>>>>> (the 4,3,2 are the red,green,blue channels)
>>>>>>
>>>>>> Infact it seem do nothing.
>>>>>>
>>>>>> Instead all work well if I set:
>>>>>>
>>>>>>     PROCESSING "BANDS=4,3,2"
>>>>>>     PROCESSING "SCALE_1=0.01,0.2"
>>>>>>     PROCESSING "SCALE_2=0.01,0.2"
>>>>>>     PROCESSING "SCALE_3=0.01,0.2"
>>>>>>
>>>>>> where ii use the min=0.01 and max=0.2 But I like to have the exact
>>>>>> values of min/max for every raster.
>>>>> Andrea,
>>>>>
>>>>> SCALE=AUTO should work but looking at the actual implementation I
>>>>> can see it computes the min/max on the pixel values intersecting the
>>>>> requested area, and not on the whole raster as I'd have expected. So
>>>>> in a tiling context, it will likely not produce the expected result.
>>>>>
>>>>> Even
>>>>>
>>>>> --
>>>>> Spatialys - Geospatial professional services
>>>>> http://www.spatialys.com
>>>>
>>>>
>>>> --
>>>> -----------------
>>>> Andrea Peri
>>>> . . . . . . . . .
>>>> qwerty àèìòù
>>>> -----------------
>>>> _______________________________________________
>>>> mapserver-users mailing list
>>>> mapserver-users at lists.osgeo.org
>>>> http://lists.osgeo.org/mailman/listinfo/mapserver-users
>>>
>>>
>>> --
>>> -----------------
>>> Andrea Peri
>>> . . . . . . . . .
>>> qwerty àèìòù
>>> -----------------
>>
>>
>> --
>> -----------------
>> Andrea Peri
>> . . . . . . . . .
>> qwerty àèìòù
>> -----------------
>
>