[postgis-users] [postgis-devel] PSC Vote: Keep or drop Flatgeobuf in PostGIS 3.2.0 - MOTION keep in
Regina Obe
lr at pcorp.us
Fri Nov 26 10:22:57 PST 2021
I do too. Special thanks for getting the 5005 bug fixed Björn.
I think that is good enough to keep Flatgeobuf in.
I'm planning to release a PostGIS 3.2.0beta2 later today.
The only outstanding issue is:
https://trac.osgeo.org/postgis/ticket/5014
But I've been unable to replicate that issue (even compiling debian's beta1 package ) except when using:
dpkg-buildpackage -uc -uc 2>&1 | tee ../postgis.build
all details in the ticket. So it's unclear what is causing that - might be something the use of fakeroot build system is catching (or preventing from including) that we can't replicate in our standard setups. All detailed on the ticket.
Thanks,
Regina
> -----Original Message-----
> From: postgis-devel [mailto:postgis-devel-bounces at lists.osgeo.org] On Behalf
> Of Jeff McKenna
> Sent: Thursday, November 25, 2021 12:08 PM
> To: postgis-devel at lists.osgeo.org
> Subject: Re: [postgis-devel] [postgis-users] PSC Vote: Keep or drop Flatgeobuf
> in PostGIS 3.2.0
>
> As an aside, I appreciate this explanation on speed and benefits, Bj rn, of
> FlatGeobuf. Thanks,
>
> -jeff
>
>
>
>
> --
> Jeff McKenna
> GatewayGeo: Developers of MS4W, MapServer Consulting and Training co-
> founder of FOSS4G http://gatewaygeo.com/
>
>
>
> On 2021-11-25 7:32 a.m., Bj rn Harrtell wrote:
> > Just a FWIW,
> >
> > FlatGeobuf is actually not prioritizing space optimization, instead
> > designed for simplicity and speed. Even so, it can be smaller than both
> > shapefile and geojson but it depends on data. But so, it can make sense
> > to compress if space optimization is required.
> >
> > The primary reason for why FlatGeobuf is fast is the use of Flatbuffers
> > which is not far from C structs (i.e direct access, no decode, no extra
> > allocations needed) but has a stable platform independent wire
> > format. If created without index FlatGeobuf can be both written and read
> > as stream, which makes it very fast as a system to system transfer
> > format. If written with an index a FlatGeobuf can be accessed with a
> > bbox filter very efficiently because it has a guaranteed balanced
> > Hilbert R-tree, so can be traversed with forward seek and found data
> > locality is good because data is ordered by the spatial index.
> > Unfortunately bytea input/output does not allow full advantage of these
> > properties - something like COPY would be better (since it can be streamed).
> >
> > /Bj rn
> >
> > Den tors 25 nov. 2021 kl 00:10 skrev Regina Obe <lr at pcorp.us
> > <mailto:lr at pcorp.us>>:
> >
> > As mentioned on IRC just reiterated here. ____
> >
> > __ __
> >
> > I would like to see ST_AsFlatGeoBuf in there. Here are my reasons.____
> >
> > __ __
> >
> > 1) To Bruce s point that sure ogr_fdw can read it, I m more
> > interested in the writing of it (which at least I can t do from
> > scratch with ogr_fdw) and getting out of the database onto the
> > command line is not one of my great ambitions in life.____
> >
> > __ __
> >
> > 2) The fact it is a lighter weight format than ST_AsGeoJSON which
> > has the same 1GB limitation.____
> >
> > Or as mentioned https://flatgeobuf.org/ <https://flatgeobuf.org/> a
> > better shapefile format (for < 1GB)____
> >
> > And the fact that FlatGeobuf is about 50% lighter than GeoJSON ____
> >
> > From site:____
> >
> > (shapefile 1, FlatGeoBuf 0.77, GeoJSON 1.2)____
> >
> > __ __
> >
> > Means I can stuff much more data in the 1GB limit than I can in
> > GeoJSON.____
> >
> > Though not sure about how that places with the other attribute data
> > if they are compacted as efficiently.____
> >
> > __ __
> >
> > 3) Given it is demonstrated it can work fine with leaflet and
> > openlayers means it is possible to hook it into something like ____
> >
> > https://github.com/CrunchyData/pg_featureserv
> > <https://github.com/CrunchyData/pg_featureserv> ____
> >
> > __ __
> >
> > as an alternative format to geojson (not saying you should Martin
> > Davis just throwing it out there J)____
> >
> > __ __
> >
> > 4) One more reason to fight that 1GB limit as postgis raster is
> > impacted by that too.____
> >
> > __ __
> >
> > __ __
> >
> > Thanks,____
> >
> > Regina____
> >
> > __ __
> >
> > *From:*postgis-users [mailto:postgis-users-bounces at lists.osgeo.org
> > <mailto:postgis-users-bounces at lists.osgeo.org>] *On Behalf Of *Bj rn
> > Harrtell
> > *Sent:* Wednesday, November 24, 2021 5:26 PM
> > *To:* PostGIS Development Discussion <postgis-devel at lists.osgeo.org
> > <mailto:postgis-devel at lists.osgeo.org>>
> > *Cc:* PostGIS Users Discussion <postgis-users at lists.osgeo.org
> > <mailto:postgis-users at lists.osgeo.org>>
> > *Subject:* Re: [postgis-users] [postgis-devel] PSC Vote: Keep or
> > drop Flatgeobuf in PostGIS 3.2.0____
> >
> > __ __
> >
> > Even though I spent quite a bit of effort on implementing this stuff
> > and I'm sure I can fix the crashers I agree with the arguments to
> > remove it. That is, 1GB limit is really bad and better to use GDAL
> > which has a well maintained impl of it.____
> >
> > __ __
> >
> > If there was a way to stream in and out binary with custom encoding
> > and no size limit (i.e COPY with custom/ext binary format) it could
> > make sense but I don't think that is going to happen any time soon.____
> >
> > __ __
> >
> > Oh well, it was fun. Some of it. ??____
> >
> > __ __
> >
> > PS. ST_AsGeobuf should be deprecated/removed too - it's even less
> > useful IMHO.____
> >
> > __ __
> >
> > PS2. I do still believe in FlatGeobuf and it is used in production.
> > ;)____
> >
> > __ __
> >
> > /Bj rn____
> >
> > Den ons 24 nov. 2021 22:26Bruce Rindahl <bruce.rindahl at gmail.com
> > <mailto:bruce.rindahl at gmail.com>> skrev:____
> >
> > FWIW I say remove it and seriously think about not including it
> > at all. Looks like you can use the format right now via ogr_fdw
> > using GDAL. ____
> >
> > __ __
> >
> > On Wed, Nov 24, 2021 at 12:51 PM Regina Obe <lr at pcorp.us
> > <mailto:lr at pcorp.us>> wrote:____
> >
> > FWIW it s already in GDAL since 3.1 and yah GDAL is a better
> > home since it doesn t have the 1GB PostgreSQL limitation____
> >
> > ____
> >
> > https://gdal.org/drivers/vector/flatgeobuf.html
> > <https://gdal.org/drivers/vector/flatgeobuf.html>____
> >
> > ____
> >
> > Also here are OpenLayers and Leaflet examples for those not
> > familiar with the format____
> >
> > ____
> >
> > OpenLayers: https://flatgeobuf.org/examples/openlayers/
> > <https://flatgeobuf.org/examples/openlayers/>____
> >
> > ____
> >
> > ____
> >
> > Leaflet: https://flatgeobuf.org/examples/leaflet/
> > <https://flatgeobuf.org/examples/leaflet/>____
> >
> > ____
> >
> > Thanks,____
> >
> > Regina____
> >
> > ____
> >
> > *From:*postgis-users
> > [mailto:postgis-users-bounces at lists.osgeo.org
> > <mailto:postgis-users-bounces at lists.osgeo.org>] *On Behalf
> > Of *Darafei "Kom?pa" Praliaskouski
> > *Sent:* Wednesday, November 24, 2021 3:27 PM
> > *To:* PostGIS Development Discussion
> > <postgis-devel at lists.osgeo.org
> > <mailto:postgis-devel at lists.osgeo.org>>
> > *Cc:* PostGIS Users Discussion
> > <postgis-users at lists.osgeo.org
> > <mailto:postgis-users at lists.osgeo.org>>
> > *Subject:* Re: [postgis-users] [postgis-devel] PSC Vote:
> > Keep or drop Flatgeobuf in PostGIS 3.2.0____
> >
> > ____
> >
> > Hi,
> >
> > I have not seen flatgeobuf in the wild, and I believe it can
> > be safely removed. ____
> >
> > ____
> >
> > The current implementation is impaired by Postgres' life
> > choices of 1GB limit and thus not usable for any data, just
> > size-limited subset. ogr2ogr seems like a better suited
> > place for it to reside.____
> >
> > ____
> >
> > I'm -0 on adding flatgeobuf to core, and -1 on releasing
> > with known crashers. This would converge to "remove if
> > nobody can fix crashers".____
> >
> > ____
> >
> > ____
> >
> > On Wed, Nov 24, 2021 at 11:10 PM Regina Obe <lr at pcorp.us
> > <mailto:lr at pcorp.us>> wrote:____
> >
> > This is a PSC vote, but we would like some feedback on
> > this from packagers
> > and users as such comments will sway our vote.
> >
> > We have two blockers that center around the new
> > FlatGeoBuf format.
> >
> > https://trac.osgeo.org/postgis/ticket/5005
> > <https://trac.osgeo.org/postgis/ticket/5005> (this one
> > is easily
> > replicatable)
> >
> > https://trac.osgeo.org/postgis/ticket/5014
> > <https://trac.osgeo.org/postgis/ticket/5014> (this one I
> > can only replicate
> > with the cowbuilder setup Bas Cowenberg provided)
> >
> > both I think are manifestations of the same problem how
> > the header is
> > derived and what it's doing with numeric and geometry
> > fields.
> >
> > I've taken a stab at troubleshooting and fixing, but did
> > not have much luck.
> > That said, if anyone is willing to help fix that would
> > be great and fix
> > within a 1 to 2 week time period.
> >
> > If not I feel that we really need to take it out of our
> > PostGIS 3.2.0
> > release (which will be going on to 3.2.0beta2).
> >
> > I'd like to release PostGIS 3.2.0beta2 in about a week
> > or so with flatgeobuf
> > fixed or removed. If removed, we'll push flatgeobuf to
> > PostGIS 3.3.0
> > cycle.
> >
> > Thanks,
> > Regina
> >
> >
> > _______________________________________________
> > postgis-devel mailing list
> > postgis-devel at lists.osgeo.org
> > <mailto:postgis-devel at lists.osgeo.org>
> > https://lists.osgeo.org/mailman/listinfo/postgis-devel
> > <https://lists.osgeo.org/mailman/listinfo/postgis-devel>____
> >
> > _______________________________________________
> > postgis-devel mailing list
> > postgis-devel at lists.osgeo.org
> > <mailto:postgis-devel at lists.osgeo.org>
> > https://lists.osgeo.org/mailman/listinfo/postgis-devel
> > <https://lists.osgeo.org/mailman/listinfo/postgis-devel>____
> >
> > _______________________________________________
> > postgis-devel mailing list
> > postgis-devel at lists.osgeo.org <mailto:postgis-devel at lists.osgeo.org>
> > https://lists.osgeo.org/mailman/listinfo/postgis-devel
> > <https://lists.osgeo.org/mailman/listinfo/postgis-devel>____
> >
> > _______________________________________________
> > postgis-devel mailing list
> > postgis-devel at lists.osgeo.org <mailto:postgis-devel at lists.osgeo.org>
> > https://lists.osgeo.org/mailman/listinfo/postgis-devel
> > <https://lists.osgeo.org/mailman/listinfo/postgis-devel>
> >
> >
> > _______________________________________________
> > postgis-devel mailing list
> > postgis-devel at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/postgis-devel
> >
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/postgis-devel
More information about the postgis-users
mailing list