[gdal-dev] Gdal2tiles gains parallel processing features

[Even Rouault]

Jeremy,

From a quick look, both should be achievable:

- GeoPackage output: I guess you ask because of multiprocessing ? Otherwise a plain 
gdal_translate will do ;-) Multiprocessing brings the interesting question of write 
concurrency with sqlite3. So I think "manual" writing with sqlite3 low level API would be 
needed to add tiles (the GPKG driver could be used to create the structure of the DB). Much 
likely the resampling/reprojection/creation of PNG/JPEG blob overhead is much larger than 
writing to the DB, so locking shouldn't hopefully be too much of a bottleneck

- custom tiling schemes. should be doable too. If you need a tiling scheme with a 
discontinuity at the antimeridian ;-), that'd probably a bit more difficult (but that's an issue 
that must already exists with the existing GlobalMercator or GlobalGeodetic profiles)


Grégory,

GeoPackage is a SQLite3 based container for vector and raster layers. For rasters, it is tiles 
organized in a pyramid structure, with tiles in PNG or JPEG. So basically it is ""just"" a matter 
of writing gdal2tiles output files as records in a database.
See http://www.geopackage.org/spec/ and
http://gdal.org/drv_geopackage_raster.html


Even

> Hey Jeremy,
> 
> I would say I don't know anything about GeoPackage or tile grids profile.
> What I can say:
> - the new version only brings parallelism to the script
> - because of the rewrite, I would hope anything to be slightly easier.
> 
> Cheers
> 
> 
> ---
> Gregory Bataille
> 
> On Fri, Sep 29, 2017 at 9:24 PM, Jeremy Palmer <palmerjnz at gmail.com> wrote:
> > Thank you for these great improvements Grégory.
> > 
> > I'm wondering does gdal2tiles now support output to GeoPackage to save
> > storage of tiles on disk? Also how hard would it be to support tile grids
> > profiles with custom origins and resolutions in the new architecture?
> > 
> > Cheers
> > Jeremy
> > 
> > On Sat, Sep 30, 2017 at 4:50 AM, Grégory Bataille <
> > 
> > gregory.bataille at gmail.com> wrote:
> >> Hi all,
> >> 
> >> I just wanted to announce that after a few months of work (took long, I
> >> got lazy), *gdal2tiles has gained parallel computing abilities*
> >> 
> >> It is now *on trunk*.
> >> 
> >> *A few things to know:*
> >> - I took upon me to rewrite the script almost entirely to make it more
> >> modular, testable, ...
> >> - Because the rewrite + the pararellization are a big and risky work
> >> (since there were no tests really), there is a new gdal2tiles_old.py
> >> script
> >> on trunk to provide an easy "back-out" for people who would get into
> >> trouble in their production
> >> - The script continues to work in a single thread/process mode by default
> >> - Actually, if you chose the default (or explicitely ask for 1 single
> >> process), the script will not use any python multiprocessing library
> >> (which
> >> have been known to be flaky). That should ensure that default behavior is
> >> not disturbed.
> >> - To activate parallel processing, you need to pass a new flag
> >> *--processes=n*


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20170930/5d2628d1/attachment.html>