[gdal-dev] Reading the same OSM data in multiple threads

Damian Dixon damian.dixon at gmail.com
Wed Jun 1 09:17:16 PDT 2016


Hi,

I'm trying to speed up processing of OSM data by opening an OSM file into
multiple datasets in multiple threads. One dataset per thread. Each thread
is processing a separate section of data, basically tiling the data.

I've however run into a scaling issue with the amount of memory allocated
per dataset.

The Open in the OSM driver seems to allocate a lot of memory for buffers
for processing regardless of the size of the data loaded.

So I have a couple of questions:

1. is there away of reducing the memory load when reading OSM in multiple
threads?

2. Could I convert the OSM data into a different format that can be read
efficiently from multiple threads? and what would that format be?

My thought for (2) would be to load the data into a database and read from
the database using ogr. If this is the correct way forward which database
would be recommended (PostGIS, SpatialLite,...) ?

Thanks
Damian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20160601/0185f139/attachment.html>


More information about the gdal-dev mailing list