[gdal-dev] Speeding up gdalwarp process

Grigory Pomadchin gr.pomadchin at gmail.com
Fri Jan 24 16:43:53 PST 2020

Hey Simon,

We had a successful enough experience with working with GDAL & Spark.

Initially there was a try to use GDAL JNI bindings for these purposes, but
it turned out that GDAL, and especially WARP (using its VRT driver), is not
completely thread safe.

You can have a look into an alternative version of JNI bindings (very
limited and covering warp only)

However, even though bindings are thread safe, it was achieved by dataset
locks. Usage of GDAL in a highly multithreaded environment can still be
challenging in terms of a desired performance.

To resolve your need in Spark you can in general try GeoTrellis (and there
is a GDAL support to perform all warp steps).

On Fri, Jan 24, 2020 at 7:04 PM Marius Jigmond <mariusjigmond at hotmail.com>

> We've been using Geotrellis (https://geotrellis.io/) for Spark jobs.
> -marius
> ------------------------------
> *From:* gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Simon <
> twinbirds at protonmail.com>
> *Sent:* Thursday, January 23, 2020 6:30 PM
> *To:* gdal-dev at lists.osgeo.org <gdal-dev at lists.osgeo.org>
> *Subject:* [gdal-dev] Speeding up gdalwarp process
> Hi gdal-devs,
> I have a question, if there is some way to use gdalwarp in a clustering
> system (e.g., Sparks) or with GPU to speed up the process? My task involves
> re-projection and re-sampling of hundreds of high-resolution images. Any
> ideas to make use of Sparks or GPU is welcomed. Thank you.
> Simon
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

Grigory Pomadchin
LinkedIn: http://www.linkedin.com/in/grpomadchin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20200124/55ab2d06/attachment.html>

More information about the gdal-dev mailing list