<html><head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head><body bgcolor="#FFFFFF" text="#000000">Hi,<br>
<br>
Thanks for the tips, Klokan. Really useful. We already have a maptiler
license, and we use it, but we can't use that tool this time.<br>
<br>
Best regards,<br>
<br>
Jorge<br>
<blockquote style="border: 0px none;"
cite="mid:CAH-RUCQ5VqVWPa-9x6kbaOrcFduP4rYjZCjdtMUTP8yJs++AwA@mail.gmail.com"
type="cite">
<div style="margin:30px 25px 10px 25px;" class="__pbConvHr"><div
style="display:table;width:100%;border-top:1px solid
#EDEEF0;padding-top:5px"> <div
style="display:table-cell;vertical-align:middle;padding-right:6px;"><img
photoaddress="klokan@klokan.cz" photoname="Klokan Petr Pridal"
src="cid:part1.03080603.01000708@cartodb.com" name="postbox-contact.jpg"
height="25px" width="25px"></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;width:100%">
<a moz-do-not-send="true" href="mailto:klokan@klokan.cz"
style="color:#737F92
!important;padding-right:6px;font-weight:bold;text-decoration:none
!important;">Klokan Petr Pridal</a></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;">
<font color="#9FA2A5"><span style="padding-left:6px">February 3, 2015
at 2:18 AM</span></font></div></div></div>
<div style="color:#888888;margin-left:24px;margin-right:24px;"
__pbrmquotes="true" class="__pbConvBody"><div dir="ltr">Hi Jorge,<div><br></div><div>rendering
of your sample file down to zoomlevel 19 would generate over 2 billion
tiles - probably tens of terabytes of data. I guess it is not what you
want to do... Typically the overzooming is done on client side in a
JavaScript viewer or on the tileserver hosting.</div><div><br></div><div>The
tiling utilities suggests you the native zoom level automatically -
based on the raster resolution of the input.</div><div><br></div><div>In
case you want to render large datasets composed of multiple files
efficiently I would recommend to try our MapTiler Pro command line
utility.</div><div>It is significantly faster then GDAL2Tiles and
produces optimised and correct tiles, while handling merging of multiple
files, solving issues with dateline crossing, reprojection and merging
of data in different srs, direct output to MBTiles, OGC WMTS compatible
export, and many more features.<br></div><div><br></div><div>The usage
on command line is quite easy:</div><div><br></div><div>$ maptiler -o
merged_output_dir input1.tif input2.tif input3.tif<br></div><div><br></div><div>Input
files can be in the original srs - every extra reprojection degrades
visual quality of the output.</div><div>Merging with VRT is not
recommend - it slow down the tile rendering, as there are artificial
blocks introduced - it is better for performance to pass the original
files directly.</div><div><br></div><div>MapTiler is a complete C/C++
reimplementation of my GDAL2Tiles utility. There are several significant
improvements in the merging and internal tiling process including
multi-thread CPU parallelization and automatic file size optimisation
(PNG color quantization and JPEG tweaks).</div><div>Installation is
possible with the .deb or .rpm packages or binaries for Mac OS X or
Windows or Docker.</div><div><br></div><div>The small file, like your
sample mentioned above, can be rendered directly with the MapTiler Free
in the graphical user interface which is available for download at: <a
moz-do-not-send="true" target="_blank"
href="http://www.maptiler.com/download/">http://www.maptiler.com/download/</a></div><div><br></div><div><div>If
you drop us an email we send you also the Pro version with command line
for your free testing.</div></div><div><br></div><div>Best regards,</div><div><br></div><div>Petr
Pridal, PhD</div><div>Klokan Technologies GmbH</div><div><br></div><div
class="gmail_extra"><br><br clear="all"><div><br></div>-- <br><div><a
moz-do-not-send="true" target="_blank" href="http://blog.klokan.cz/">http://blog.klokan.cz/</a><br><a
moz-do-not-send="true" target="_blank" href="http://www.maptiler.org/">http://www.maptiler.org/</a><br><a
moz-do-not-send="true" target="_blank"
href="http://www.oldmapsonline.org/">http://www.oldmapsonline.org/</a></div>
</div></div>
</div>
<div style="margin:30px 25px 10px 25px;" class="__pbConvHr"><div
style="display:table;width:100%;border-top:1px solid
#EDEEF0;padding-top:5px"> <div
style="display:table-cell;vertical-align:middle;padding-right:6px;"><img
photoaddress="jorge@cartodb.com" photoname="Jorge Arévalo"
src="cid:part2.03090502.00080104@cartodb.com" name="postbox-contact.jpg"
height="25px" width="25px"></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;width:100%">
<a moz-do-not-send="true" href="mailto:jorge@cartodb.com"
style="color:#737F92
!important;padding-right:6px;font-weight:bold;text-decoration:none
!important;">Jorge Arévalo</a></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;">
<font color="#9FA2A5"><span style="padding-left:6px">January 29, 2015
at 1:13 AM</span></font></div></div></div>
<div style="color:#888888;margin-left:24px;margin-right:24px;"
__pbrmquotes="true" class="__pbConvBody"><br><br>El jueves, 29 de enero
de 2015, Even Rouault <<a moz-do-not-send="true"
href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>>
escribió:<br><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex" class="gmail_quote">Le jeudi 29 janvier 2015
00:05:32, Jorge Arévalo a écrit :<br>
> > Even Rouault <mailto:<a moz-do-not-send="true"
onclick="_e(event, 'cvml', 'even.rouault@spatialys.com')"
href="javascript:;">even.rouault@spatialys.com</a>><br>
> > January 28, 2015 at 9:28 PM<br>
> ><br>
> > Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :<br>
> >> Hi,<br>
> >><br>
> >> I'm working with a patched version of gdal2tiles, which
makes use of<br>
> >> parallelization:<br>
> >> <a moz-do-not-send="true" target="_blank"
href="http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti">http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti</a><br>
> >> le- creation-processes/74446#74446<br>
> >><br>
> >> I want to create a complete TMS cache from raster imagery.
No<br>
> >> assumptions about SRS of data type for input data.<br>
> >><br>
> >> I want the tiling process to be as fast as possible
(gdal2tiles is<br>
> >> really heavy process), do you have any recomendations
about the data or<br>
> >> the parameters used?<br>
> >><br>
> >> Right now, I'm doing this<br>
> >><br>
> >> Step 1: build vrt from input images<br>
> >><br>
> >> gdal_vrtmerge.py -o merged.vrt<list of input tif
files><br>
> >><br>
> >> Step 2: build tiles from vrt file<br>
> >><br>
> >> gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all
merged.vrt tms_dir<br>
> >><br>
> >> Even with parallelization, process still feels really
slow. Would it be<br>
> >> faster if, for example, I convert all my input files to
epsg:3857? Or if<br>
> >> I scale them to 8-bit? Or if I use near resampling method
instead of<br>
> >> cubic? (I'm using cubic because I'm working with
continuous data:<br>
> >> satellite images, am I doing it right?).<br>
> >><br>
> > From a quick look at the source, it seems that there's an
optimization<br>
> > if the<br>
> ><br>
> > input SRS == output SRS that avoids going through the warped
VRT path.<br>
> ><br>
> > That said, we definitely need one or several maintainers for
gdal2tiles.<br>
> > There are quite a faw patches floating around in Trac that
would need<br>
> > someone to review, test, fix, apply them, as well as writing
tests (no<br>
> > autotest for gdal2tiles yet), etc...<br>
><br>
> Ok. But the applications is taking hours to generate a complete
tile<br>
> cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4
cores<br>
> machine with 8GB of RAM. The file is this one<br>
><br>
> <a moz-do-not-send="true" target="_blank"
href="https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif">https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif</a><br>
><br>
> Taking so much time for a 3MB file sounds ridiculous. I'm probably
doing<br>
> something wrong. This is the line<br>
><br>
> gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif
tiles_dir<br>
><br>
> Do you see something wrong in this approach?<br>
<br>
The performance seems to be deeply impacted by "-r cubic", but even
without<br>
it, it still sucks. The reason is the zoom level 19. Your dataset has a<br>
resolution of 3469 m. zoom 19 corresponds to a resolution of ~ 0.3 m. So
you<br>
basically try to generate a TMS that is rougly 12000 x 12000 larger than
your<br>
source dataset...<br>
<br></blockquote><div>Wow, thanks for the enlightment. So, with that
resolution, I could easily create TMS cache until levels 5-6, according
to this <a moz-do-not-send="true"
href="https://msdn.microsoft.com/en-us/library/aa940990.aspx">https://msdn.microsoft.com/en-us/library/aa940990.aspx</a></div><div><br></div><div>If
I'd want higher zoom levels... That could mean high processing cost.
Worse if cúbic resampling method is used.</div><div><br></div><div>Conclussion:
better guessing the highest zoom level for that resolution using the
table on the previous link, and building the TMS cache with that
constraint in mind.</div><div><br></div><div>Thanks again, Even!</div><div><br></div><blockquote
style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"
class="gmail_quote">
><br>
> Anyway, if this is a problem of gdal2tiles and it needs fine
tunning or<br>
> maintenance, we could talk. I don't know if there's any other
method to<br>
> generate a complete TMS cache using GDAL.<br>
><br>
> >> Any other tips?<br>
> >><br>
> >> Many thanks in advance<br>
> >><br>
> >> Best regards<br>
<br>
--<br>
Spatialys - Geospatial professional services<br>
<a moz-do-not-send="true" target="_blank"
href="http://www.spatialys.com">http://www.spatialys.com</a><br>
</blockquote><br><br>-- <br><div dir="ltr"><font
face="arial,helvetica,sans-serif">Jorge Arévalo<br><br><a
moz-do-not-send="true" target="_blank" href="http://cartodb.com/">http://cartodb.com/</a><br></font><div>Skype
ID: jorgecartodb</div></div><br>
</div>
<div style="margin:30px 25px 10px 25px;" class="__pbConvHr"><div
style="display:table;width:100%;border-top:1px solid
#EDEEF0;padding-top:5px"> <div
style="display:table-cell;vertical-align:middle;padding-right:6px;"><img
photoaddress="even.rouault@spatialys.com" photoname="Even Rouault"
src="cid:part3.02010608.06010708@cartodb.com"
name="compose-unknown-contact.jpg" height="25px" width="25px"></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;width:100%">
<a moz-do-not-send="true" href="mailto:even.rouault@spatialys.com"
style="color:#737F92
!important;padding-right:6px;font-weight:bold;text-decoration:none
!important;">Even Rouault</a></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;">
<font color="#9FA2A5"><span style="padding-left:6px">January 29, 2015
at 12:27 AM</span></font></div></div></div>
<div style="color:#888888;margin-left:24px;margin-right:24px;"
__pbrmquotes="true" class="__pbConvBody"><pre wrap="">Le jeudi 29 janvier 2015 00:05:32, Jorge Arévalo a écrit :
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">Even Rouault <a class="moz-txt-link-rfc2396E" href="mailto:even.rouault@spatialys.com"><mailto:even.rouault@spatialys.com></a>
January 28, 2015 at 9:28 PM
Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
</pre><blockquote type="cite"><pre wrap="">Hi,
I'm working with a patched version of gdal2tiles, which makes use of
parallelization:
<a class="moz-txt-link-freetext" href="http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti">http://gis.stackexchange.com/questions/7743/performance-of-google-map-ti</a>
le- creation-processes/74446#74446
I want to create a complete TMS cache from raster imagery. No
assumptions about SRS of data type for input data.
I want the tiling process to be as fast as possible (gdal2tiles is
really heavy process), do you have any recomendations about the data or
the parameters used?
Right now, I'm doing this
Step 1: build vrt from input images
gdal_vrtmerge.py -o merged.vrt<list of input tif files>
Step 2: build tiles from vrt file
gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
Even with parallelization, process still feels really slow. Would it be
faster if, for example, I convert all my input files to epsg:3857? Or if
I scale them to 8-bit? Or if I use near resampling method instead of
cubic? (I'm using cubic because I'm working with continuous data:
satellite images, am I doing it right?).
</pre></blockquote><pre wrap=""> From a quick look at the source, it seems that there's an optimization
if the
input SRS == output SRS that avoids going through the warped VRT path.
That said, we definitely need one or several maintainers for gdal2tiles.
There are quite a faw patches floating around in Trac that would need
someone to review, test, fix, apply them, as well as writing tests (no
autotest for gdal2tiles yet), etc...
</pre></blockquote><pre wrap="">Ok. But the applications is taking hours to generate a complete tile
cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4 cores
machine with 8GB of RAM. The file is this one
<a class="moz-txt-link-freetext" href="https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif">https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif</a>
Taking so much time for a 3MB file sounds ridiculous. I'm probably doing
something wrong. This is the line
gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif tiles_dir
Do you see something wrong in this approach?
</pre></blockquote><pre wrap=""><!---->
The performance seems to be deeply impacted by "-r cubic", but even without
it, it still sucks. The reason is the zoom level 19. Your dataset has a
resolution of 3469 m. zoom 19 corresponds to a resolution of ~ 0.3 m. So you
basically try to generate a TMS that is rougly 12000 x 12000 larger than your
source dataset...
</pre><blockquote type="cite"><pre wrap="">Anyway, if this is a problem of gdal2tiles and it needs fine tunning or
maintenance, we could talk. I don't know if there's any other method to
generate a complete TMS cache using GDAL.
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">Any other tips?
Many thanks in advance
Best regards
</pre></blockquote></blockquote></blockquote><pre wrap=""><!---->
</pre></div>
<div style="margin:30px 25px 10px 25px;" class="__pbConvHr"><div
style="display:table;width:100%;border-top:1px solid
#EDEEF0;padding-top:5px"> <div
style="display:table-cell;vertical-align:middle;padding-right:6px;"><img
photoaddress="jorge@cartodb.com" photoname="Jorge Arévalo"
src="cid:part2.03090502.00080104@cartodb.com" name="postbox-contact.jpg"
height="25px" width="25px"></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;width:100%">
<a moz-do-not-send="true" href="mailto:jorge@cartodb.com"
style="color:#737F92
!important;padding-right:6px;font-weight:bold;text-decoration:none
!important;">Jorge Arévalo</a></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;">
<font color="#9FA2A5"><span style="padding-left:6px">January 29, 2015
at 12:05 AM</span></font></div></div></div>
<div style="color:#888888;margin-left:24px;margin-right:24px;"
__pbrmquotes="true" class="__pbConvBody">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<br>
<br>
<blockquote type="cite"
cite="mid:201501282128.46180.even.rouault@spatialys.com" style="border:
0px none;">
<div class="__pbConvHr" style="margin:30px 25px 10px 25px;"><div
style="display:table;width:100%;border-top:1px solid
#EDEEF0;padding-top:5px"> <div
style="display:table-cell;vertical-align:middle;padding-right:6px;"><img
moz-do-not-send="true" name="compose-unknown-contact.jpg"
src="cid:part3.02010608.06010708@cartodb.com" photoname="Even Rouault"
photoaddress="even.rouault@spatialys.com" height="25px" width="25px"></div>
<div
style="display:table-cell;white-space:nowrap;vertical-align:middle;width:100%">
<a style="color:#737F92
!important;padding-right:6px;font-weight:bold;text-decoration:none
!important;" href="mailto:even.rouault@spatialys.com"
moz-do-not-send="true">Even Rouault</a></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;">
<font color="#9FA2A5"><span style="padding-left:6px">January 28, 2015
at 9:28 PM</span></font></div></div></div>
<div class="__pbConvBody" __pbrmquotes="true" style="color: rgb(136,
136, 136); margin-left: 24px;
margin-right: 24px;"><pre wrap="">Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
</pre><blockquote type="cite"><pre wrap="">Hi,
I'm working with a patched version of gdal2tiles, which makes use of
parallelization:
<a moz-do-not-send="true" href="http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile" class="moz-txt-link-freetext">http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile</a>-
creation-processes/74446#74446
I want to create a complete TMS cache from raster imagery. No
assumptions about SRS of data type for input data.
I want the tiling process to be as fast as possible (gdal2tiles is
really heavy process), do you have any recomendations about the data or
the parameters used?
Right now, I'm doing this
Step 1: build vrt from input images
gdal_vrtmerge.py -o merged.vrt <list of input tif files>
Step 2: build tiles from vrt file
gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
Even with parallelization, process still feels really slow. Would it be
faster if, for example, I convert all my input files to epsg:3857? Or if
I scale them to 8-bit? Or if I use near resampling method instead of
cubic? (I'm using cubic because I'm working with continuous data:
satellite images, am I doing it right?).
</pre></blockquote><pre wrap=""><!---->
>From a quick look at the source, it seems that there's an optimization if the
input SRS == output SRS that avoids going through the warped VRT path.
That said, we definitely need one or several maintainers for gdal2tiles. There
are quite a faw patches floating around in Trac that would need someone to
review, test, fix, apply them, as well as writing tests (no autotest for
gdal2tiles yet), etc...</pre></div></blockquote>
<br>
Ok. But the applications is taking hours to generate a complete tile
cache (zoom levels 0-19) for a 3MB tiff file, in epsg:3857. A 4 cores
machine with 8GB of RAM. The file is this one<br>
<br>
<a moz-do-not-send="true"
href="https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif"
class="moz-txt-link-freetext">https://dl.dropboxusercontent.com/u/6599273/gis_data/katrina-3857.tif</a><br>
<br>
Taking so much time for a 3MB file sounds ridiculous. I'm probably doing
something wrong. This is the line<br>
<br>
gdal2tiles.py -s epsg:3857 -z 0-19 -r cubic katrina-3857.tif tiles_dir<br>
<br>
Do you see something wrong in this approach?<br>
<br>
Anyway, if this is a problem of gdal2tiles and it needs fine tunning or
maintenance, we could talk. I don't know if there's any other method to
generate a complete TMS cache using GDAL.<br>
<blockquote type="cite"
cite="mid:201501282128.46180.even.rouault@spatialys.com" style="border:
0px none;">
<div class="__pbConvBody" __pbrmquotes="true"
style="color:#888888;margin-left:24px;margin-right:24px;">
<blockquote type="cite"><pre wrap="">Any other tips?
Many thanks in advance
Best regards
</pre></blockquote><pre wrap=""><!---->
</pre></div></blockquote>
<br>
</div>
<div style="margin:30px 25px 10px 25px;" class="__pbConvHr"><div
style="display:table;width:100%;border-top:1px solid
#EDEEF0;padding-top:5px"> <div
style="display:table-cell;vertical-align:middle;padding-right:6px;"><img
photoaddress="even.rouault@spatialys.com" photoname="Even Rouault"
src="cid:part3.02010608.06010708@cartodb.com"
name="compose-unknown-contact.jpg" height="25px" width="25px"></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;width:100%">
<a moz-do-not-send="true" href="mailto:even.rouault@spatialys.com"
style="color:#737F92
!important;padding-right:6px;font-weight:bold;text-decoration:none
!important;">Even Rouault</a></div> <div
style="display:table-cell;white-space:nowrap;vertical-align:middle;">
<font color="#9FA2A5"><span style="padding-left:6px">January 28, 2015
at 9:28 PM</span></font></div></div></div>
<div style="color:#888888;margin-left:24px;margin-right:24px;"
__pbrmquotes="true" class="__pbConvBody"><pre wrap="">Le mercredi 28 janvier 2015 20:17:08, Jorge Arévalo a écrit :
</pre><blockquote type="cite"><pre wrap="">Hi,
I'm working with a patched version of gdal2tiles, which makes use of
parallelization:
<a class="moz-txt-link-freetext" href="http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile">http://gis.stackexchange.com/questions/7743/performance-of-google-map-tile</a>-
creation-processes/74446#74446
I want to create a complete TMS cache from raster imagery. No
assumptions about SRS of data type for input data.
I want the tiling process to be as fast as possible (gdal2tiles is
really heavy process), do you have any recomendations about the data or
the parameters used?
Right now, I'm doing this
Step 1: build vrt from input images
gdal_vrtmerge.py -o merged.vrt <list of input tif files>
Step 2: build tiles from vrt file
gdal2tiles.py -r cubic -s epsg:XXXX -z 0-19 -w all merged.vrt tms_dir
Even with parallelization, process still feels really slow. Would it be
faster if, for example, I convert all my input files to epsg:3857? Or if
I scale them to 8-bit? Or if I use near resampling method instead of
cubic? (I'm using cubic because I'm working with continuous data:
satellite images, am I doing it right?).
</pre></blockquote><pre wrap=""><!---->
>From a quick look at the source, it seems that there's an optimization if the
input SRS == output SRS that avoids going through the warped VRT path.
That said, we definitely need one or several maintainers for gdal2tiles. There
are quite a faw patches floating around in Trac that would need someone to
review, test, fix, apply them, as well as writing tests (no autotest for
gdal2tiles yet), etc...
</pre><blockquote type="cite"><pre wrap="">Any other tips?
Many thanks in advance
Best regards
</pre></blockquote><pre wrap=""><!---->
</pre></div>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<div>Sent with <a href="http://www.getpostbox.com"><span style="color:
rgb(51, 102, 153);">Postbox</span></a></div></div>
</body></html>