[QGIS-Developer] Flatgeobuf for processing

[Even Rouault]

On samedi 9 mai 2020 08:54:06 CEST Matthias Kuhn wrote:
> Hi Richard,
> 
> On 5/8/20 5:24 PM, Richard Duivenvoorde wrote:
> > About Tobias' flatgeobuf: instead of a shp/gpkg file alternative, would
> > this not be a very good candidate to store our intermediate processing
> > steps in (which was shp, not shure what it is now?)?
> 
> Good shout!
> 
> I would love to see someone systematically test it with processing.

A few insights of the GDAL implementation (CC'ing Björn):
- the default behaviour is to generate a flatgeobuf file with the spatial index
- to do so, a temporary file is created with the features serialized as soon as they are passed 
to GDAL
- the bounding box of each feature is added in a in-memory vector (40 bytes / feature)
- this vector is sorted with the Packed Hilbert R-Tree sorting order
- the final FlatGeobuf file is written with the features also sorted accordingly (doesn't require 
bring them all into memory, so not an issue)

One potential issue could be for huge files (> tens of millions of features), where the size of 
the vector storing the bounding box could become significant (10 millions features = 400 
MB), especially on 32 bit processes.

So probably not going to be a limitation for common usage (and someone really eager to 
limit the RAM usage could probably use a merge sort, with the array split into sub ones, 
stored onto several files on disk, and bringing into RAM only chunks for final sorting. That 
would be fun to write !!!)

If not writing a spatial index, the writing is done in a streaming way in the final file, without 
storing in RAM the bounding box of each feature.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/qgis-developer/attachments/20200509/cbc603de/attachment.html>