[MapServer-users] Varying use of overviews across Mapserver versions
Scott
public at postholer.com
Fri Jan 10 12:04:46 PST 2025
This may be a clue.
I gdalwarp'd your typic_xerofluvents.tif to a proper -f COG format and
did stats on both. The block sizes are wildly different. Try making a
proper COG out of your .tif and see if you get the same results.
Block=1766x4, as opposed to Block=512x512
Here's what I'm looking at:
gdalinfo -stats typic_xerofluvents.tif
Band 1 Block=1766x4 Type=Byte, ColorInterp=Gray
Minimum=1.000, Maximum=95.000, Mean=10.684, StdDev=12.230
NoData Value=0
Overviews: 883x1262, 442x631, 221x316, 111x158
Metadata:
STATISTICS_MINIMUM=1
STATISTICS_MAXIMUM=95
STATISTICS_MEAN=10.684186169509
STATISTICS_STDDEV=12.229580166735
STATISTICS_VALID_PERCENT=0.3393
####
gdalwarp -f COG typic_xerofluvents.tif tst.tif
gdalinfo -stats tst.tif
Band 1 Block=512x512 Type=Byte, ColorInterp=Gray
Minimum=1.000, Maximum=95.000, Mean=10.684, StdDev=12.230
NoData Value=0
Overviews: 883x1262, 441x631, 220x315, 110x157
Metadata:
STATISTICS_MINIMUM=1
STATISTICS_MAXIMUM=95
STATISTICS_MEAN=10.684186169509
STATISTICS_STDDEV=12.229580166735
STATISTICS_VALID_PERCENT=0.3393
On 1/10/25 11:30, Dylan Beaudette wrote:
> Hi Rich,
>
> Now that is a good question! Perhaps an implied use of those overviews
> has changed across versions, or a previously-default mapfile parameter
> has changed. We didn't see anything in the changelog, but to be honest,
> a lot has changed between versions and maybe I missed something.\
>
> Here is a direct link to the file, in case that is helpful:
> https://soilmap4-1.lawr.ucdavis.edu/taxa-grid-tms/taxsubgrp/
> typic_xerofluvents.tif <https://soilmap4-1.lawr.ucdavis.edu/taxa-grid-
> tms/taxsubgrp/typic_xerofluvents.tif>
>
> Here is a snippet from the mapfile, the same on both machines. Note that
> we are using a combination of pattern matching and expressions to select
> files and apply a color styling. Perhaps there was an implied change to
> "processing" or "use of overviews" directives in the more recent
> mapserver version?
>
> MAP
> OUTPUTFORMAT
> NAME "png"
> IMAGEMODE RGBA
> DRIVER "AGG/PNG"
> MIMETYPE "image/png"
> EXTENSION "png"
> FORMATOPTION "GAMMA=0.75"
> END
>
> STATUS ON
>
> # set the output projection of map images to 'spherical
> mercator' used by leaflet maps
> PROJECTION
> "init=epsg:3857"
> END
>
> ##### blue 9 from dylan
> LAYER
> NAME blue9dylan
> STATUS ON
> DATA "%type%/%name%.tif"
> TYPE RASTER
> # set the input projection of the source data
> PROJECTION
> "init=epsg:3857"
> END
> VALIDATION
> "type" "^taxorder|taxsuborder|taxgrtgroup|
> taxsubgrp|fm\/gg|fm\/sg$"
> "name" "^[a-zA-z_\-]+$"
> "minvalue" "^[0-9]{1,3}$"
> END
> CLASS # NoData Values
> EXPRESSION ([pixel] = 0)
> STYLE
> COLOR "#00000000"
> END
> END
> CLASS
> EXPRESSION ([pixel] >= %minvalue% AND [pixel] <= 2)
> STYLE
> COLOR "#E0F7E1"
> END
> END
> [...]
>
> Thanks,
> Dylan
>
>
> On Fri, Jan 10, 2025 at 11:10 AM Richard Greenwood
> <richard.greenwood at gmail.com <mailto:richard.greenwood at gmail.com>> wrote:
>
> Hi Dylan,
>
> Welcome back. Why do you think the issue is related to the use of
> overviews and not something else?
>
> Rich
>
> On Fri, Jan 10, 2025 at 10:50 AM Dylan Beaudette via MapServer-users
> <mapserver-users at lists.osgeo.org <mailto:mapserver-
> users at lists.osgeo.org>> wrote:
>
> Thanks Scott,
>
> I tried that: md5sums on the GeoTiffs and diff on the .map files
> confirms that they are identical.
>
> The only difference between these two systems is the version of
> Mapserver.
>
> MapServer version 7.2.2 OUTPUT=PNG OUTPUT=JPEG OUTPUT=KML
> SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO
> SUPPORTS=SVG_SYMBOLS SUPPORTS=RSVG SUPPORTS=ICONV
> SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT
> SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER
> SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS
> SUPPORTS=GEOS SUPPORTS=PBF INPUT=JPEG INPUT=POSTGIS INPUT=OGR
> INPUT=GDAL INPUT=SHAPEFILE
>
> MapServer version 7.6.4 OUTPUT=PNG OUTPUT=JPEG OUTPUT=KML
> SUPPORTS=PROJ SUPPORTS=AGG SUPPORTS=FREETYPE SUPPORTS=CAIRO
> SUPPORTS=SVG_SYMBOLS SUPPORTS=RSVG SUPPORTS=ICONV
> SUPPORTS=FRIBIDI SUPPORTS=WMS_SERVER SUPPORTS=WMS_CLIENT
> SUPPORTS=WFS_SERVER SUPPORTS=WFS_CLIENT SUPPORTS=WCS_SERVER
> SUPPORTS=SOS_SERVER SUPPORTS=FASTCGI SUPPORTS=THREADS
> SUPPORTS=GEOS SUPPORTS=POINT_Z_M SUPPORTS=PBF INPUT=JPEG
> INPUT=POSTGIS INPUT=OGR INPUT=GDAL INPUT=SHAPEFILE
>
> The newer version is provided via apt:
> cgi-mapserver 7.6.4-2build2
> amd64 CGI executable for MapServer
>
> Any other leads?
> Thanks!
> Dylan
>
> On Thu, Jan 9, 2025 at 2:45 PM Scott via MapServer-users
> <mapserver-users at lists.osgeo.org <mailto:mapserver-
> users at lists.osgeo.org>> wrote:
>
> I would do a sanity check first, just to make sure the 2
> raster files
> are truly the same. Try something like:
>
> gdalinfo -stats -hist file1.tif
> gdalinfo -stats -hist file2.tif
>
> Then, I would continue with the .map files, to make sure
> they are the
> same. Something like:
>
> diff serv1.map serv2.map
>
> See what falls out.
>
> On 1/9/25 13:10, Dylan Beaudette via MapServer-users wrote:
> >
> > Hi Everyone,
> >
> > I've got identical GeoTiffs and .map files on two servers
> with varying
> > versions of Mapserver that is being used as a TMS.
> Mapserver 7.2.2 is (I
> > think) correctly using the overviews while Mapserver
> 7.6.4 is not.
> >
> > Here is an example of the same file, tile, and styling
> applied across
> > both versions of Mapserver.
> >
> > # Mapserver 7.2.2:
> > https://soilmap2-1.lawr.ucdavis.edu/cgi-bin/mapserv?map=/
> data1/website/ <https://soilmap2-1.lawr.ucdavis.edu/cgi-bin/
> mapserv?map=/data1/website/>
> > ste/
> >
> taxa.map&mode=tile&tilemode=gmap&tile=5+12+5&layer=blue9dylan&type=taxsubgrp&name=typic_xerofluvents&minvalue=5 <https://soilmap2-1.lawr.ucdavis.edu/cgi-bin/mapserv?map=/data1/website/ste/taxa.map&mode=tile&tilemode=gmap&tile=5+12+5&layer=blue9dylan&type=taxsubgrp&name=typic_xerofluvents&minvalue=5 <https://soilmap2-1.lawr.ucdavis.edu/cgi-bin/mapserv?map=/data1/website/ste/taxa.map&mode=tile&tilemode=gmap&tile=5+12+5&layer=blue9dylan&type=taxsubgrp&name=typic_xerofluvents&minvalue=5>>
> >
> > Mapserver 7.6.4:
> > https://soilmap4-1.lawr.ucdavis.edu/cgi-bin/mapserv?map=/
> data1/website/ <https://soilmap4-1.lawr.ucdavis.edu/cgi-bin/
> mapserv?map=/data1/website/>
> > ste/
> >
> taxa.map&mode=tile&tilemode=gmap&tile=5+12+5&layer=blue9dylan&type=taxsubgrp&name=typic_xerofluvents&minvalue=5 <https://soilmap4-1.lawr.ucdavis.edu/cgi-bin/mapserv?map=/data1/website/ste/taxa.map&mode=tile&tilemode=gmap&tile=5+12+5&layer=blue9dylan&type=taxsubgrp&name=typic_xerofluvents&minvalue=5 <https://soilmap4-1.lawr.ucdavis.edu/cgi-bin/mapserv?map=/data1/website/ste/taxa.map&mode=tile&tilemode=gmap&tile=5+12+5&layer=blue9dylan&type=taxsubgrp&name=typic_xerofluvents&minvalue=5>>
> >
> > It appears to me that the newer version is either
> ignoring or
> > interpreting the overviews differently. Each file has an
> internal tiling
> > and overview scheme like this:
> >
> >
> ----------------------------------------------------------------------------------
> > Band 1 Block=1766x4 Type=Byte, ColorInterp=Gray
> > NoData Value=0
> > Overviews: 883x1262, 442x631, 221x316, 111x158
> >
> ----------------------------------------------------------------------------------
> >
> > Does anyone have a clue about the differences? Is either
> version in error?
> >
> > Thanks, and good to be back on the mailing list after a
> nearly 15 year
> > hiatus!
> > Dylan
> >
> >
> >
> >
> > _______________________________________________
> > MapServer-users mailing list
> > MapServer-users at lists.osgeo.org <mailto:MapServer-
> users at lists.osgeo.org>
> > https://lists.osgeo.org/mailman/listinfo/mapserver-users
> <https://lists.osgeo.org/mailman/listinfo/mapserver-users>
>
> _______________________________________________
> MapServer-users mailing list
> MapServer-users at lists.osgeo.org <mailto:MapServer-
> users at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
> <https://lists.osgeo.org/mailman/listinfo/mapserver-users>
>
> _______________________________________________
> MapServer-users mailing list
> MapServer-users at lists.osgeo.org <mailto:MapServer-
> users at lists.osgeo.org>
> https://lists.osgeo.org/mailman/listinfo/mapserver-users
> <https://lists.osgeo.org/mailman/listinfo/mapserver-users>
>
>
>
> --
> Richard W. Greenwood
> www.greenwoodmap.com <http://www.greenwoodmap.com>
>
More information about the MapServer-users
mailing list