[gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

[Rahkonen Jukka]

Hi,

I tested with this environmental variable and command

set OGR_MVT_REMOVE_TEMP_FILE=NO
ogr2ogr -f mvt -dsco temporary_db=c:\data\mtk\temp\temp.db out.mvt mtkmaasto.gpkg rakennus --debug on

Once ogr2ogr stopped I had the temp.db file left in the directory. It seems to be a simple SQLite database. Without setting the environmental variable the temporary database was immediately deleted. During the process the file explorer only shows that the file exists but with file size of 0 bytes. I believe that this is normal behavior with SQLite.

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org> Puolesta Leo Fuhrmann via gdal-dev
Lähetetty: perjantai 6. lokakuuta 2023 16.08
Vastaanottaja: gdal-dev at lists.osgeo.org
Aihe: [gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

Hi,

I'm creating vector tiles from a PostGIS database using ogr2ogr with the MVT driver. The region is quite large and I'm creating tiles for levels 0-15, so the process takes quite some time (hours). The tiles are written directly to a S3 storage using the /vsis3/ virtual file system (see: https://gdal.org/user/virtual_file_systems.html#vsis3-aws-s3-files).

Looking at the logs, I can see that, firstly, a "temporary database" file (*.temp.db) is created. Some things are notable:

- this step seems to be CPU intensive compared to the creation of the actual pbf files
- the temporary database file is not being written to the output file directory (S3 storage in this case) as stated in the documentation
- the dataset creation option `TEMPORARY_DB` (see: https://gdal.org/drivers/vector/mvt.html#dataset-creation-options) seems to be ignored (I tried writing the file to a local directory but no such file shows up)
- my guess is that the file is only kept in memory (which would be /vsimem/ if I'm correct)

Is it possible to skip the step of first creating the temp db and directly create tiles from PostGIS or any other data source? Where/how exactly is the database file stored? Any ideas on optimizing the process?

I'm looking forward to some discussion on this topic. I will happily try to contribute where I can.

Regards,

Leo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231006/cdedb476/attachment-0001.htm>