[gdal-dev] Reading the same OSM data in multiple threads

[Damian Dixon]

Hi,

I'm trying to speed up processing of OSM data by opening an OSM file into
multiple datasets in multiple threads. One dataset per thread. Each thread
is processing a separate section of data, basically tiling the data.

I've however run into a scaling issue with the amount of memory allocated
per dataset.

The Open in the OSM driver seems to allocate a lot of memory for buffers
for processing regardless of the size of the data loaded.

So I have a couple of questions:

1. is there away of reducing the memory load when reading OSM in multiple
threads?

2. Could I convert the OSM data into a different format that can be read
efficiently from multiple threads? and what would that format be?

My thought for (2) would be to load the data into a database and read from
the database using ogr. If this is the correct way forward which database
would be recommended (PostGIS, SpatialLite,...) ?

Thanks
Damian
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20160601/0185f139/attachment.html>