[gdal-dev] Processing huge DEM dataset

Peter Baumann p.baumann at jacobs-university.de
Thu Apr 23 06:07:29 PDT 2015 

...or rasdaman, www.rasdaman.org. It allows to establish a single seamless map
on which you can run algorithms in a high-level language. The system takes care
of borders etc. Prefab VM is available on request (production not yet automated).
my 2 cents,
Peter


On 04/23/15 15:03, Rémi Cura wrote:
> Hey,
> gdal may not be the right tool for this.
>
> You may use Orfeo Tool Box (https://www.orfeo-toolbox.org/),
> which has been designed to process huge images with limited cpu and memory.
>
> Cheers,
> Rémi-C
>
>
> 2015-04-23 14:36 GMT+02:00 Even Rouault <even.rouault at spatialys.com
> <mailto:even.rouault at spatialys.com>>:
>
>     Le jeudi 23 avril 2015 14:22:30, Paul Ramsey a écrit :
>     > I have in the past, with other tool sets, not GDAL, approached this by
>     > building out padded tiles as the first step. So for each tile, merge it
>     > with it’s neighbors, then clip out the middle so you get a somewhat larger
>     > tile. Give it a nice thick buffer.
>     >
>     > Now all your tiles overlap. Process them all individually. Thanks to the
>     > generous overlap they should, on the boundaries, come up w/ the same
>     > answers. Once you have final results (hillshades, contours, etc) clip them
>     > to the real tile boundary. In the case of contours you may need to finally
>     > snap the ends together, but it should not be a huge snap, just a tiny one.
>     >
>
>     +1 for Paul's above workflow. Some answers to your questions below
>
>     > --
>     > http://postgis.net
>     > http://cleverelephant.ca
>     >
>     > On April 23, 2015 at 3:01:45 AM, Marcos Dione (mdione at grulic.org.ar
>     <mailto:mdione at grulic.org.ar>) wrote:
>     > > I have SRTM's DEM 1x1 degree 30m resolution tiles for the whole Europe
>     > > and I'm trying to generate several raster images based on
>     > > that (elevation coloring, slopeshade and hillshade), but I'm not sure
>     > > about the right approach to do it for that amount of data.
>     > >
>     > > The simplest approach is to stitch the DEMs and then process, but that
>     > > takes ages, specially if I try to use uncompressed, tiled
>     > > GeoTIFFs as output. The stitching can even be done using a virtual file,
>     > > which saves space.
>     > >
>     > > If I process each tile individually, and then build a virtual file on
>     > > top, I get shades on the edges of tiles. This shade is due
>     > > to the tile ending and the shading algorithm assuming there's a 0
>     > > elevation point right to it. So, question A) is that so?
>
>     Most gdaldem algorithms (except color-relief) need to compute some form of
>     gradient (a 3x3 window around the pixel being computed), so you have edge
>     effects. By default, they put a nodata value on the edges.
>     If you specify -compute_edges, then they will interpolate extra values from
>     the ones available so that the edge pixels can be computed. You could still
>     see some discontinuity if the prediction isn't that great.
>
>     > >
>     > > I think that getting the shade in the output is due to the algorithm for
>     > > finding a pixel uses the first tile that has it.
>     > > Question B) is that so?
>     > >
>     > > If so, C) could I simply avoid this by generating another vrt file that
>     > > lists each tile as having a bbox of only the 1x1 degree
>     > > instead of the 1x1 degree plus an extra pixel border? If I get the time,
>     > > I'll try this this afternoon (I just thought of it).
>
>     For each tile, you could have a VRT of 3x3 tiles. Let's say that a tile if NxN
>     pixel, then the VRT would be (N+2)*(N+2). And you would extract NxN pixels
>     from the output of gdaldem on that VRT, keeping NxN pixels only. Well, this is
>     basically Paul's approach using VRT to do the buffer.
>
>     > >
>     > > All this I can do more or less with the gdal command line tools, without
>     > > much programming. Then comes a more programmatic way:
>     > > either use gdaldem or use the GDAL API to process each tile, then cut the
>     > > 1x1 degree image and save that; then stitch them/build a
>     > > vrt file on top. As you can see, this is what I've been avoiding to do :)
>     > >
>     > > Finally, I would also like to generate contour lines for this. So far I
>     > > managed to generate them for 5x5 tiles with 90m
>     > > resolution, then I import them in postgis. When I render them, on the
>     > > edges of such tiles I see the lines from one crossing the
>     > > others, looking ugly. For instance:
>     > >
>     > > http://grulicueva.homenet.org/~mdione/Elevation/#14/45.0000/15.0000
>     <http://grulicueva.homenet.org/%7Emdione/Elevation/#14/45.0000/15.0000>
>     > >
>     > > I tried used a stitched file for the whole region but I ran out of memory
>     > > with gdal_contour. Again, this was with 90m resolution
>     > > tiles; now I have 30m, which means 9 times as much data. D) How could I
>     > > properly process all that?
>     > >
>     > > Thanks in advance for any ideas,
>     > >
>     > > -- Marcos.
>     > > _______________________________________________
>     > > gdal-dev mailing list
>     > > gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>     > > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>     >
>     > _______________________________________________
>     > gdal-dev mailing list
>     > gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>     > http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>     --
>     Spatialys - Geospatial professional services
>     http://www.spatialys.com
>     _______________________________________________
>     gdal-dev mailing list
>     gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>     http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>
>
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
Dr. Peter Baumann
 - Professor of Computer Science, Jacobs University Bremen
   www.faculty.jacobs-university.de/pbaumann
   mail: p.baumann at jacobs-university.de
   tel: +49-421-200-3178, fax: +49-421-200-493178
 - Executive Director, rasdaman GmbH Bremen (HRB 26793)
   www.rasdaman.com, mail: baumann at rasdaman.com
   tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20150423/6c8dc2bd/attachment.html>

More information about the gdal-dev mailing list