VS: [UMN_MAPSERVER-USERS] Problem with reprojected images

[Rahkonen Jukka]

Hello Dirk,
 
No, I don't have experience on very large datasets. I have only created once a GDAL/VRT dataset from a site with about 120 aerial photos. Each image was 10000 x 10000 pixels and in lossless JPEG 2000 format.  Creation of the VRT file was fast with the modified gdal_merge.py script, but building pyramid layers was very slow.  Frank Warmerdam informed that vrt system was not made for large data sets and was not surprised about the poor performance in overview creation.  But once the process was done my earial photo layer behaved very well with Mapserver.
 
However, I found it much more convenient for me just to make a tileindex of my JPEG 2000 images and use that in Mapserver without any overviews.  I guess I won't do any more large VRT datasets to be used with Mapserver.  However, I will need those to be used with gdal_transform when reprojecting aerial photos from Finnish national reference system to EUREF. The gdal_transform seems to work as fast with VRT virtual layer as with individual images and the final product is just perfect without any seems at the place of original image boundaries.
 
I think that the tileindes system and not GDAL/VRT is the way to handle your 250000 tiles once the lines-at-the-boundary problem is solved. Geotiffs with overviews gives propably the best performance, but using JPEG 2000 images compressed to some 1:15 ratio (3-channel images, mayby some 1:7 with greyscale) is not bad alternative either and saves some disk space.  Of course your servers processor will be more busy because of the need to decompress JPEG 2000 imagery and that limits the load as requests/sec it can stand.  My server has so little load that speed is not any problem for me and I cannot say anything about how it would behave with high load.
 
Regards,
 
Jukka Rahkonen
 
 
 
 


________________________________

Lähettäjä: UMN MapServer Users List puolesta: Dirk Jesko
Lähetetty: ke 15.3.2006 12:31
Vastaanottaja: MAPSERVER-USERS at LISTS.UMN.EDU
Aihe: Re: [UMN_MAPSERVER-USERS] Problem with reprojected images



Dear Jukka,

thanks for the tip. I will try it. However, since there are some 250.000
tiles, I expect problems with the performance even with pyramid layers.
Do you have any experience with very large datasets and GDAL/VRT?
<http://dict.leo.org/se?lp=ende&p=/Mn4k.&search=experiences>

Kind regards,
Dirk Jesko

>Hi,
>
>I think it is exactly as Ed says.  I got similar kind of problems once when I started to reproject images with "gdal_translate", and mosaic and cut the reprojected images to suit mapsheet system of the destination projection.
>
>With gdal_translate I can get rid of those white (in my case black) pixel lines by using a GDAL virtual format (.vrt)as a source data set instead of individual files. I have been thinking that this means in practise building a mosaic before reprojection, thus there is no place in the process where those lines would be formed. Judged by the result you get I guess that maybe the tileindex system in Mapserver is handling each tile separately and not in the same way than GDAL is handling VRT. If getting rid of those lines is important for you it might be worth trying to use GDAL virtual data set as source data for your Mapserver instead of tileindex. There has been a link to a modified "gdal_merge.py" script in the gdal_dev mailing list possible in May 2005, but I could not find the link right now. With that script you can make a virtual data set from a bunch of original image files, and after creating pyramid layers for the vrt-file (this will be slow if your VRT is large) it is quite usable for Mapserver.
>
>Regards,
>
>Jukka Rahkonen
>
><http://xserve.flids.com/pipermail/gdal-dev/2005-May/008605.html> 
>
>________________________________
>
>Lähettäjä: UMN MapServer Users List puolesta: Ed McNierney
>Lähetetty: ti 14.3.2006 16:38
>Vastaanottaja: MAPSERVER-USERS at LISTS.UMN.EDU
>Aihe: Re: [UMN_MAPSERVER-USERS] Problem with reprojected images
>
> 
>