[gdal-dev] Questions regarding temporary database file (ogr2ogr with MVT driver)

[Even Rouault]

Hi,,
>
> I’m creating vector tiles from a PostGIS database using ogr2ogr with 
> the MVT driver. The region is quite large and I’m creating tiles for 
> levels 0-15, so the process takes quite some time (hours). The tiles 
> are written directly to a S3 storage using the /vsis3/ virtual file 
> system (see: 
> https://gdal.org/user/virtual_file_systems.html#vsis3-aws-s3-files).
>
> Looking at the logs, I can see that, firstly, a “temporary database” 
> file (*.temp.db) is created. Some things are notable:
>
> - this step seems to be CPU intensive compared to the creation of the 
> actual pbf files
> - the temporary database file is not being written to the output file 
> directory (S3 storage in this case) as stated in the documentation

I don't observe this:

$ ogr2ogr /vsis3/XXXXX/foo.mvt poly.shp
ERROR 1: wb+ not supported for /vsis3/, unless 
CPL_VSIL_USE_TEMP_FILE_FOR_RANDOM_WRITE is set to YES

vs

$ ogr2ogr /vsis3/spatialys/foo.mvt poly.shp -dsco 
TEMPORARY_DB=/tmp/my.temp.db

> - the dataset creation option `TEMPORARY_DB` (see: 
> https://gdal.org/drivers/vector/mvt.html#dataset-creation-options) 
> seems to be ignored (I tried writing the file to a local directory but 
> no such file shows up)
On Unix, the file is immediately deleted after being opened, so that if 
you kill the process there's no remaining temporary file. You can set 
the OGR_MVT_REMOVE_TEMP_FILE env variable to NO to disable this
> - my guess is that the file is only kept in memory (which would be 
> /vsimem/ if I’m correct)
>
> Is it possible to skip the step of first creating the temp db and 
> directly create tiles from PostGIS or any other data source?
No
> Where/how exactly is the database file stored? Any ideas on optimizing 
> the process?

You could possibly paralle

Also, the MVT generator is intended to be basic. You might perhaps have 
more functionality with https://github.com/felt/tippecanoe

>
> I’m looking forward to some discussion on this topic. I will happily 
> try to contribute where I can.
>
> Regards,
>
> Leo
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.