[gdal-dev] ogr2ogr for downloading extracts from overturemaps

[Rahkonen Jukka]

Hi,

Your command took 110 seconds to run from Finland with "GDAL 3.10.0dev-b6edb5191e, released 2024/10/07". There are 827 features in the "type=water" table.

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta Even Rouault via gdal-dev
Lähetetty: torstai 24. lokakuuta 2024 23.39
Vastaanottaja: Varun Sharma <vsharma.next at gmail.com>; gdal-dev at lists.osgeo.org
Aihe: Re: [gdal-dev] ogr2ogr for downloading extracts from overturemaps



Le 24/10/2024 à 22:26, Varun Sharma via gdal-dev a écrit :
Thanks Even for your prompt reply!

1. Just to clarify, with GDAL v3.10.0, the command

ogr2ogr -f GPKG ogr_water.gpkg -spat 7.5 46.5 7.7 46.7 /vsis3/overturemaps-us-west-2/release/2024-08-20.0/theme=base/type=water/
is fine and I should see a ( significant ) speed up .. yes ?

Normally. but far from being instantaneous on my currently not super fast Internet connection

You may try the gdal-master conda-forge channel to test right now: https://gdal.org/en/latest/download.html#gdal-master-conda-builds

2. the apache arrow project libs itself have many knobs to tweak like threadpools, i/o threads, memory pools etc - are these exposed in GDAL configuration options ?

some are. See https://github.com/OSGeo/gdal/blob/master/ogr/ogrsf_frmts/parquet/ogrparquetdatasetlayer.cpp#L505-L542

They should probably be mentioned in https://gdal.org/en/latest/drivers/vector/parquet.html but I haven't been brave enough to do that, not always fully understanding what they actually do in libarrow/libparquet

Even

--

http://www.spatialys.com<http://www.spatialys.com/>

My software is free, but my time generally not.

Butcher of all kinds of standards, open or closed formats. At the end, this is just about bytes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20241024/4428d0f8/attachment.htm>