[postgis-users] [postgis-devel] PSC Vote: Keep or drop Flatgeobuf in PostGIS 3.2.0

Thu Nov 25 03:32:42 PST 2021

Just a FWIW,

FlatGeobuf is actually not prioritizing space optimization, instead
designed for simplicity and speed. Even so, it can be smaller than both
shapefile and geojson but it depends on data. But so, it can make sense to
compress if space optimization is required.

The primary reason for why FlatGeobuf is fast is the use of Flatbuffers
which is not far from C structs (i.e direct access, no decode, no extra
allocations needed) but has a stable platform independent wire format. If
created without index FlatGeobuf can be both written and read as stream,
which makes it very fast as a system to system transfer format. If written
with an index a FlatGeobuf can be accessed with a bbox filter very
efficiently because it has a guaranteed balanced Hilbert R-tree, so can be
traversed with forward seek and found data locality is good because data is
ordered by the spatial index. Unfortunately bytea input/output does not
allow full advantage of these properties - something like COPY would be
better (since it can be streamed).

/Björn

Den tors 25 nov. 2021 kl 00:10 skrev Regina Obe <lr at pcorp.us>:

> As mentioned on IRC just reiterated here.
>
>
>
> I would like to see ST_AsFlatGeoBuf in there. Here are my reasons.
>
>
>
> 1) To Bruce’s point that sure ogr_fdw can read it, I’m more interested in
> the writing of it (which at least I can’t do from scratch with ogr_fdw) and
> getting out of the database onto the command line is not one of my great
> ambitions in life.
>
>
>
> 2) The fact it is a lighter weight format than ST_AsGeoJSON which has the
> same 1GB limitation.
>
> Or as mentioned https://flatgeobuf.org/ a better shapefile format (for <
> 1GB)
>
> And the fact that FlatGeobuf is about 50% lighter than GeoJSON
>
> From site:
>
>   (shapefile 1, FlatGeoBuf 0.77,  GeoJSON  1.2)
>
>
>
> Means I can stuff much more data in the 1GB limit than I can in GeoJSON.
>
> Though not sure about how that places with the other attribute data if
> they are compacted as efficiently.
>
>
>
> 3) Given it is  demonstrated it can work fine with leaflet and openlayers
> means it is possible to hook it into something like
>
> https://github.com/CrunchyData/pg_featureserv
>
>
>
> as an alternative format to geojson (not saying you should Martin Davis –
> just throwing it out there J )
>
>
>
> 4) One more reason to fight that 1GB limit – as postgis raster is impacted
> by that too.
>
>
>
>
>
> Thanks,
>
> Regina
>
>
>
> *From:* postgis-users [mailto:postgis-users-bounces at lists.osgeo.org] *On
> Behalf Of *Björn Harrtell
> *Sent:* Wednesday, November 24, 2021 5:26 PM
> *To:* PostGIS Development Discussion <postgis-devel at lists.osgeo.org>
> *Cc:* PostGIS Users Discussion <postgis-users at lists.osgeo.org>
> *Subject:* Re: [postgis-users] [postgis-devel] PSC Vote: Keep or drop
> Flatgeobuf in PostGIS 3.2.0
>
>
>
> Even though I spent quite a bit of effort on implementing this stuff and
> I'm sure I can fix the crashers I agree with the arguments to remove it.
> That is, 1GB limit is really bad and better to use GDAL which has a well
> maintained impl of it.
>
>
>
> If there was a way to stream in and out binary with custom encoding and no
> size limit (i.e COPY with custom/ext binary format) it could make sense but
> I don't think that is going to happen any time soon.
>
>
>
> Oh well, it was fun. Some of it. 😂
>
>
>
> PS. ST_AsGeobuf should be deprecated/removed too - it's even less useful
> IMHO.
>
>
>
> PS2. I do still believe in FlatGeobuf and it is used in production. ;)
>
>
>
> /Björn
>
> Den ons 24 nov. 2021 22:26Bruce Rindahl <bruce.rindahl at gmail.com> skrev:
>
> FWIW I say remove it and seriously think about not including it at all.
> Looks like you can use the format right now via ogr_fdw using GDAL.
>
>
>
> On Wed, Nov 24, 2021 at 12:51 PM Regina Obe <lr at pcorp.us> wrote:
>
> FWIW it’s already in GDAL since 3.1 and yah GDAL is a better home since it
> doesn’t have the  1GB PostgreSQL limitation
>
>
>
> https://gdal.org/drivers/vector/flatgeobuf.html
>
>
>
> Also here are OpenLayers and Leaflet examples for those not familiar with
> the format
>
>
>
> OpenLayers: https://flatgeobuf.org/examples/openlayers/
>
>
>
>
>
> Leaflet: https://flatgeobuf.org/examples/leaflet/
>
>
>
> Thanks,
>
> Regina
>
>
>
> *From:* postgis-users [mailto:postgis-users-bounces at lists.osgeo.org] *On
> Behalf Of *Darafei "Kom?pa" Praliaskouski
> *Sent:* Wednesday, November 24, 2021 3:27 PM
> *To:* PostGIS Development Discussion <postgis-devel at lists.osgeo.org>
> *Cc:* PostGIS Users Discussion <postgis-users at lists.osgeo.org>
> *Subject:* Re: [postgis-users] [postgis-devel] PSC Vote: Keep or drop
> Flatgeobuf in PostGIS 3.2.0
>
>
>
> Hi,
>
> I have not seen flatgeobuf in the wild, and I believe it can be safely
> removed.
>
>
>
> The current implementation is impaired by Postgres' life choices of 1GB
> limit and thus not usable for any data, just size-limited subset. ogr2ogr
> seems like a better suited place for it to reside.
>
>
>
> I'm -0 on adding flatgeobuf to core, and -1 on releasing with known
> crashers. This would converge to "remove if nobody can fix crashers".
>
>
>
>
>
> On Wed, Nov 24, 2021 at 11:10 PM Regina Obe <lr at pcorp.us> wrote:
>
> This is a PSC vote, but we would like some feedback on this from packagers
> and users as such comments will sway our vote.
>
> We have two blockers that center around the new FlatGeoBuf format.
>
> https://trac.osgeo.org/postgis/ticket/5005  (this one is easily
> replicatable)
>
> https://trac.osgeo.org/postgis/ticket/5014 (this one I can only replicate
> with the cowbuilder setup Bas Cowenberg provided)
>
> both I think are manifestations of the same problem how the header is
> derived and what it's doing with numeric and geometry fields.
>
> I've taken a stab at troubleshooting and fixing, but did not have much
> luck.
> That said, if anyone is willing to help fix that would be great and fix
> within a 1 to 2 week time period.
>
> If not I feel that we really need to take it out of our PostGIS 3.2.0
> release (which will be going on to 3.2.0beta2).
>
> I'd like to release PostGIS 3.2.0beta2 in about a week or so with
> flatgeobuf
> fixed or removed.  If removed, we'll  push flatgeobuf to PostGIS 3.3.0
> cycle.
>
> Thanks,
> Regina
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20211125/28cc8541/attachment.html>