[gdal-dev] GDAL 3.8.0beta1 available for testing

[Even Rouault]

Hi Mike,

thanks for the report. Issue fixed in master. I've refreshed the Docker 
image

Note however that this Overture dataset contains 1,390,499,721 features, 
and with our estimate of 83 bytes / feature to build the FlatGeobuf 
RTree, that means you need at least 115 GB RAM on your machine 
(presumably more due to fragmentation etc)

You could potentially save a few minutes by not adding -progress since 
GetFeatureCount() on such dataset requires getting some bits in each of 
the ~1600 files, but I guess that for a process that's going to take a 
few hours, having some display progress justifies a few extra minutes :-)

Even

Le 01/11/2023 à 11:18, Michael Smith a écrit :
>
> Using the ghcr.io/osgeo/gdal:ubuntu-full-latest docker image, I get a 
> segfault when trying to convert overture maps to flatgeobuf
>
> ogr2ogr -f flatgeobuf /data/overturemaps_2023_10_19.fgb 
> /vsis3/overturemaps-us-west-2/release/2023-10-19-alpha.0/theme=buildings/ 
> theme=buildings -progress
>
> …
>
> FlatGeobuf: Spatial index requested will write to temp file and do 
> second pass on close
>
> Segmentation fault (core dumped)
>
> But no core file.
>
> -- 
>
> Michael Smith
>
> Remote Sensing/GIS Center
>
> US Army Corps of Engineers
>
> *From: *gdal-dev <gdal-dev-bounces at lists.osgeo.org> on behalf of Even 
> Rouault via gdal-dev <gdal-dev at lists.osgeo.org>
> *Reply-To: *Even Rouault <even.rouault at spatialys.com>
> *Date: *Tuesday, October 31, 2023 at 1:14 PM
> *To: *Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>, 
> "gdal-dev at lists.osgeo.org" <gdal-dev at lists.osgeo.org>
> *Subject: *Re: [gdal-dev] GDAL 3.8.0beta1 available for testing
>
> Le 31/10/2023 à 17:36, Rahkonen Jukka a écrit :
>
>     Hi,
>
>     I made a simple test with ogr2ogr and geopackage to geopackage on Windows. With GDAL 3.8.0dev-3e4dc710a2 (no arrow, old R-Tree) the timing was 36 minutes, with GDAL 3.8.0dev-6bbd2c080a the same conversion took 21 minutes. The gpkg file is 75 GB, with 120 layers.
>
>     Could it be that the new R-Tree implementation does not print anything into debug?
>
> Indeed. Debug traces added back in https://github.com/OSGeo/gdal/pull/8640
>
> Even
>
> -- 
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> _______________________________________________ gdal-dev mailing list 
> gdal-dev at lists.osgeo.org 
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20231101/89063869/attachment-0001.htm>