[mapserver-users] MapServer Tile Indexes

[Schepers, Benjamin]

Hi all,

another try for troubleshooting could be building a VRT (use gdalbuildvrt - it produces some kind of "advanced raster-catalogue"). If you have faulty data, there would be a chance, that you get some error-messages, which dataset produces problems...

This VRT - if everything went fine - could also be the source to build a large TIFF-Dataset, similar to what Jukka recommended.
Some good compression-parameters for elevation-data in large GeoTIFF would be to use gdal_translate with "-of GTiff -ot Float32 -co BIGTIFF=YES -co COMPRESS=DEFLATE -co TILED=YES" (if you use GDAL 2.1 "-co NUM_THREADS=ALL_CPUS" will speed up this process, as it uses all cpu-cores).

You should also think about overviews for small scales. Interesting post from Paul Ramsey:
http://blog.cleverelephant.ca/2015/02/geotiff-compression-for-dummies.html - This is about compressing Aerial Images, but it points you into the right direction (remind my compression parameters above for DEM-data).

Such compressed dataset will be presented by mapserver amazingly fast. I've got a dataset with similar dimensions but 10m res which is about 100MB --> plain upscaling to your resolution would produce a dataset about 10-15GB, but this depends on your data...


Cheers
Benjamin

Luftbild und Geoinformationssysteme
Regionalverband Ruhr
Kronprinzenstraße 35
45128 Essen
Germany
luftbilder.geoportal.ruhr


-----Ursprüngliche Nachricht-----
Von: mapserver-users [mailto:mapserver-users-bounces at lists.osgeo.org] Im Auftrag von Jeff McKenna
Gesendet: Dienstag, 21. Juni 2016 13:13
An: mapserver-users at lists.osgeo.org
Betreff: Re: [mapserver-users] MapServer Tile Indexes

Hello,

MapServer should definitely be able to handle your dataset, it is not large.

If I was in your shoes, here is what I would do:

- try to see more debug information (follow the steps and examples in 
http://www.mapserver.org/optimization/debugging.html)
   -- set in your mapfile: CONFIG "CPL_DEBUG" "ON"
   -- shp2img -m my.map -o test.png -all_debug 5
   -- mapserv -nh "QUERY_STRING=map=c:/ms4w/apps/my.map&mode=map"

- it does sound like you have a faulty GRID file.  Start from your 
single working tileindex, then add in maybe 5 more rasters into your 
index and retry; if it works, add in another 5 etc.

Sometimes the best way to solve this is the hard way.

-jeff


-- 
Jeff McKenna
MapServer Consulting and Training Services
http://www.gatewaygeomatics.com/



On 2016-06-21 6:12 AM, Wright, Simon M. wrote:
> Dear All
>
>
>
> I’m trying to use the MapServer Tile Indexes to create a map service,
> and ultimately a WMS, of some DSM data but am getting an Internal Server
> Error.
>
>
>
> The DSM data are organised as 466 5 × 5 km rasters with a resolution of
> 1 × 1 m with a total size of 69.2 GB.  The data format is Esri ASCII GRID.
>
>
>
> To test the use of Tile Indexes in MapServer I used the gdaltindex to
> create an index shapefile for just twenty of the 5 × 5 km rasters and
> created a basic map file:
>
>
>
> MAP
>
>        IMAGETYPE            PNG
>
>        EXTENT               240000 100000 250000 150000
>
>        SIZE                 400 2000
>
>        PROJECTION
>
>               "init=epsg:27700"
>
>        END
>
>        LAYER
>
>               NAME          "tiled-rasters"
>
>               STATUS        ON
>
>               TILEINDEX
> "/datastore/b81071f2-85b3-4e31-8506-cabe899f989a/b81071f2-85b3-4e31-8506-cabe899f989a_copy/index_file"
>
>               TILEITEM      "location"
>
>               TYPE          RASTER
>
>               PROJECTION
>
>                      "init=epsg:27700"
>
>               END
>
>        END
>
> END
>
>
>
>
>
> When I request an image of the map with the URL
> http://localhost:8080/cgi-bin/mapserv?map=/vagrant/maps/tiled-rasters.map&layer=tiled-rasters&mode=map
> I get an “Internal Server Error”.
>
>
>
> Therefore I created a second index shapefile using the gdaltindex tool
> that only contained one of the 5 × 5 km rasters and updated the map file
> to point to the second index shapefile.  When I request an image of the
> map I can see an image of the DSM data as expected.
>
>
>
> Therefore, am I asking too much of the tile indexing approach in
> MapServer to work against a massive dataset – 69.2 GB in 466 raster
> datasets – or have MapServer users been able to get tile indexing too
> work in MapServer against large data volumes and can provide some guidance.
>
>
>
> Many thanks in advance of any help that MapServer users can offer.
>
>
>
>
>
>
>
> Best wishes, Simon.
>
>

_______________________________________________
mapserver-users mailing list
mapserver-users at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users