[gdal-dev] Quick question about gdal2tiles
Jorge Arévalo
jorge at cartodb.com
Thu Feb 5 09:59:19 PST 2015
Hi,
Thanks for the tips, Klokan. Really useful. We already have a maptiler
license, and we use it, but we can't use that tool this time.
Best regards,
Jorge
> Klokan Petr Pridal <mailto:klokan at klokan.cz>
> February 3, 2015 at 2:18 AM
> Hi Jorge,
>
> rendering of your sample file down to zoomlevel 19 would generate over
> 2 billion tiles - probably tens of terabytes of data. I guess it is
> not what you want to do... Typically the overzooming is done on client
> side in a JavaScript viewer or on the tileserver hosting.
>
> The tiling utilities suggests you the native zoom level automatically
> - based on the raster resolution of the input.
>
> In case you want to render large datasets composed of multiple files
> efficiently I would recommend to try our MapTiler Pro command line
> utility.
> It is significantly faster then GDAL2Tiles and produces optimised and
> correct tiles, while handling merging of multiple files, solving
> issues with dateline crossing, reprojection and merging of data in
> different srs, direct output to MBTiles, OGC WMTS compatible export,
> and many more features.
>
> The usage on command line is quite easy:
>
> $ maptiler -o merged_output_dir input1.tif input2.tif input3.tif
>
> Input files can be in the original srs - every extra reprojection
> degrades visual quality of the output.
> Merging with VRT is not recommend - it slow down the tile rendering,
> as there are artificial blocks introduced - it is better for
> performance to pass the original files directly.
>
> MapTiler is a complete C/C++ reimplementation of my GDAL2Tiles
> utility. There are several significant improvements in the merging and
> internal tiling process including multi-thread CPU parallelization and
> automatic file size optimisation (PNG color quantization and JPEG tweaks).
> Installation is possible with the .deb or .rpm packages or binaries
> for Mac OS X or Windows or Docker.
>
> The small file, like your sample mentioned above, can be rendered
> directly with the MapTiler Free in the graphical user interface which
> is available for download at: http://www.maptiler.com/download/
>
> If you drop us an email we send you also the Pro version with command
> line for your free testing.
>
> Best regards,
>
> Petr Pridal, PhD
> Klokan Technologies GmbH
>
>
>
>
> --
> http://blog.klokan.cz/
> http://www.maptiler.org/
> http://www.oldmapsonline.org/
> Jorge Arévalo <mailto:jorge at cartodb.com>
> January 29, 2015 at 1:13 AM
>
>
> El jueves, 29 de enero de 2015, Even Rouault
> <even.rouault at spatialys.com <mailto:even.rouault at spatialys.com>> escribió:
>
> Le jeudi 29 janvier 2015 00:05:32, Jorge Arévalo a écrit :
> > > Even Rouault <mailto:even.rouault at spatialys.com <javascript:;>>
> > > January 28, 2015 at 9:28 PM
> > >
> > > Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
> > >> Hi,
> > >>
> > >> I'm working with a patched version of gdal2tiles, which makes
> use of
> > >> parallelization:
> > >>
> http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti
> > >> le- creation-processes/74446#74446
> > >>
> > >> I want to create a complete TMS cache from raster imagery. No
> > >> assumptions about SRS of data type for input data.
> > >>
> > >> I want the tiling process to be as fast as possible
> (gdal2tiles is
> > >> really heavy process), do you have any recomendations about
> the data or
> > >> the parameters used?
> > >>
> > >> Right now, I'm doing this
> > >>
> > >> Step 1: build vrt from input images
> > >>
> > >> gdal_vrtmerge.py -o merged.vrt<list of input tif files>
> > >>
> > >> Step 2: build tiles from vrt file
> > >>
> > >> gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt
> tms_dir
> > >>
> > >> Even with parallelization, process still feels really slow.
> Would it be
> > >> faster if, for example, I convert all my input files to
> epsg:3857? Or if
> > >> I scale them to 8-bit? Or if I use near resampling method
> instead of
> > >> cubic? (I'm using cubic because I'm working with continuous data:
> > >> satellite images, am I doing it right?).
> > >>
> > > From a quick look at the source, it seems that there's an
> optimization
> > > if the
> > >
> > > input SRS == output SRS that avoids going through the warped
> VRT path.
> > >
> > > That said, we definitely need one or several maintainers for
> gdal2tiles.
> > > There are quite a faw patches floating around in Trac that
> would need
> > > someone to review, test, fix, apply them, as well as writing
> tests (no
> > > autotest for gdal2tiles yet), etc...
> >
> > Ok. But the applications is taking hours to generate a complete tile
> > cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4
> cores
> > machine with 8GB of RAM. The file is this one
> >
> >
> https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif
> >
> > Taking so much time for a 3MB file sounds ridiculous. I'm
> probably doing
> > something wrong. This is the line
> >
> > gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif
> tiles_dir
> >
> > Do you see something wrong in this approach?
>
> The performance seems to be deeply impacted by "-r cubic", but
> even without
> it, it still sucks. The reason is the zoom level 19. Your dataset
> has a
> resolution of 3469 m. zoom 19 corresponds to a resolution of ~ 0.3
> m. So you
> basically try to generate a TMS that is rougly 12000 x 12000
> larger than your
> source dataset...
>
> Wow, thanks for the enlightment. So, with that resolution, I could
> easily create TMS cache until levels 5-6, according to this
> https://msdn.microsoft.com/en-us/library/aa940990.aspx
>
> If I'd want higher zoom levels... That could mean
> high processing cost. Worse if cúbic resampling method is used.
>
> Conclussion: better guessing the highest zoom level for
> that resolution using the table on the previous link, and building the
> TMS cache with that constraint in mind.
>
> Thanks again, Even!
>
> >
> > Anyway, if this is a problem of gdal2tiles and it needs fine
> tunning or
> > maintenance, we could talk. I don't know if there's any other
> method to
> > generate a complete TMS cache using GDAL.
> >
> > >> Any other tips?
> > >>
> > >> Many thanks in advance
> > >>
> > >> Best regards
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>
>
>
> --
> Jorge Arévalo
>
> http://cartodb.com/
> Skype ID: jorgecartodb
>
> Even Rouault <mailto:even.rouault at spatialys.com>
> January 29, 2015 at 12:27 AM
> Le jeudi 29 janvier 2015 00:05:32, Jorge Arévalo a écrit :
>>> Even Rouault<mailto:even.rouault at spatialys.com>
>>> January 28, 2015 at 9:28 PM
>>>
>>> Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
>>>> Hi,
>>>>
>>>> I'm working with a patched version of gdal2tiles, which makes use of
>>>> parallelization:
>>>> http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti
>>>> le- creation-processes/74446#74446
>>>>
>>>> I want to create a complete TMS cache from raster imagery. No
>>>> assumptions about SRS of data type for input data.
>>>>
>>>> I want the tiling process to be as fast as possible (gdal2tiles is
>>>> really heavy process), do you have any recomendations about the data or
>>>> the parameters used?
>>>>
>>>> Right now, I'm doing this
>>>>
>>>> Step 1: build vrt from input images
>>>>
>>>> gdal_vrtmerge.py -o merged.vrt<list of input tif files>
>>>>
>>>> Step 2: build tiles from vrt file
>>>>
>>>> gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
>>>>
>>>> Even with parallelization, process still feels really slow. Would it be
>>>> faster if, for example, I convert all my input files to epsg:3857? Or if
>>>> I scale them to 8-bit? Or if I use near resampling method instead of
>>>> cubic? (I'm using cubic because I'm working with continuous data:
>>>> satellite images, am I doing it right?).
>>>>
>>> From a quick look at the source, it seems that there's an optimization
>>> if the
>>>
>>> input SRS == output SRS that avoids going through the warped VRT path.
>>>
>>> That said, we definitely need one or several maintainers for gdal2tiles.
>>> There are quite a faw patches floating around in Trac that would need
>>> someone to review, test, fix, apply them, as well as writing tests (no
>>> autotest for gdal2tiles yet), etc...
>> Ok. But the applications is taking hours to generate a complete tile
>> cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4 cores
>> machine with 8GB of RAM. The file is this one
>>
>> https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif
>>
>> Taking so much time for a 3MB file sounds ridiculous. I'm probably doing
>> something wrong. This is the line
>>
>> gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif tiles_dir
>>
>> Do you see something wrong in this approach?
>
> The performance seems to be deeply impacted by "-r cubic", but even without
> it, it still sucks. The reason is the zoom level 19. Your dataset has a
> resolution of 3469 m. zoom 19 corresponds to a resolution of ~ 0.3 m. So you
> basically try to generate a TMS that is rougly 12000 x 12000 larger than your
> source dataset...
>
>> Anyway, if this is a problem of gdal2tiles and it needs fine tunning or
>> maintenance, we could talk. I don't know if there's any other method to
>> generate a complete TMS cache using GDAL.
>>
>>>> Any other tips?
>>>>
>>>> Many thanks in advance
>>>>
>>>> Best regards
>
> Jorge Arévalo <mailto:jorge at cartodb.com>
> January 29, 2015 at 12:05 AM
>
>
>> Even Rouault <mailto:even.rouault at spatialys.com>
>> January 28, 2015 at 9:28 PM
>> Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
>>> Hi,
>>>
>>> I'm working with a patched version of gdal2tiles, which makes use of
>>> parallelization:
>>> http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile-
>>> creation-processes/74446#74446
>>>
>>> I want to create a complete TMS cache from raster imagery. No
>>> assumptions about SRS of data type for input data.
>>>
>>> I want the tiling process to be as fast as possible (gdal2tiles is
>>> really heavy process), do you have any recomendations about the data or
>>> the parameters used?
>>>
>>> Right now, I'm doing this
>>>
>>> Step 1: build vrt from input images
>>>
>>> gdal_vrtmerge.py -o merged.vrt<list of input tif files>
>>>
>>> Step 2: build tiles from vrt file
>>>
>>> gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
>>>
>>> Even with parallelization, process still feels really slow. Would it be
>>> faster if, for example, I convert all my input files to epsg:3857? Or if
>>> I scale them to 8-bit? Or if I use near resampling method instead of
>>> cubic? (I'm using cubic because I'm working with continuous data:
>>> satellite images, am I doing it right?).
>>
>> > From a quick look at the source, it seems that there's an optimization if the
>> input SRS == output SRS that avoids going through the warped VRT path.
>>
>> That said, we definitely need one or several maintainers for gdal2tiles. There
>> are quite a faw patches floating around in Trac that would need someone to
>> review, test, fix, apply them, as well as writing tests (no autotest for
>> gdal2tiles yet), etc...
>
> Ok. But the applications is taking hours to generate a complete tile
> cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4 cores
> machine with 8GB of RAM. The file is this one
>
> https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif
>
> Taking so much time for a 3MB file sounds ridiculous. I'm probably
> doing something wrong. This is the line
>
> gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif tiles_dir
>
> Do you see something wrong in this approach?
>
> Anyway, if this is a problem of gdal2tiles and it needs fine tunning
> or maintenance, we could talk. I don't know if there's any other
> method to generate a complete TMS cache using GDAL.
>>> Any other tips?
>>>
>>> Many thanks in advance
>>>
>>> Best regards
>>
>
> Even Rouault <mailto:even.rouault at spatialys.com>
> January 28, 2015 at 9:28 PM
> Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
>> Hi,
>>
>> I'm working with a patched version of gdal2tiles, which makes use of
>> parallelization:
>> http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile-
>> creation-processes/74446#74446
>>
>> I want to create a complete TMS cache from raster imagery. No
>> assumptions about SRS of data type for input data.
>>
>> I want the tiling process to be as fast as possible (gdal2tiles is
>> really heavy process), do you have any recomendations about the data or
>> the parameters used?
>>
>> Right now, I'm doing this
>>
>> Step 1: build vrt from input images
>>
>> gdal_vrtmerge.py -o merged.vrt<list of input tif files>
>>
>> Step 2: build tiles from vrt file
>>
>> gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
>>
>> Even with parallelization, process still feels really slow. Would it be
>> faster if, for example, I convert all my input files to epsg:3857? Or if
>> I scale them to 8-bit? Or if I use near resampling method instead of
>> cubic? (I'm using cubic because I'm working with continuous data:
>> satellite images, am I doing it right?).
>
> From a quick look at the source, it seems that there's an optimization if the
> input SRS == output SRS that avoids going through the warped VRT path.
>
> That said, we definitely need one or several maintainers for gdal2tiles. There
> are quite a faw patches floating around in Trac that would need someone to
> review, test, fix, apply them, as well as writing tests (no autotest for
> gdal2tiles yet), etc...
>
>> Any other tips?
>>
>> Many thanks in advance
>>
>> Best regards
>
--
Sent with Postbox <http://www.getpostbox.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150205/004d6339/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1217 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150205/004d6339/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: postbox-contact.jpg
Type: image/jpeg
Size: 1187 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150205/004d6339/attachment-0004.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compose-unknown-contact.jpg
Type: image/jpeg
Size: 770 bytes
Desc: not available
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150205/004d6339/attachment-0005.jpg>
More information about the gdal-dev
mailing list