[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format
Lars Schylberg
lars.schylberg at blixtmail.se
Wed May 18 12:07:46 PDT 2022
Thanks, Björn and Jeff for the effort to do this.I am eager to start
testing soon.As a user I feel that Santa is coming early this year.
I also have one question.Is the native FlatGeobuf driver doing the
reading without checks, like "VERIFY_BUFFERS" "NO" that you can set in
the OGR driver?
My inofficial +1
Lars Schylberg
Den 2022-05-18 kl. 17:09, skrev Steve Lime:
> Nice work all - esp. Björn. Evan has a good point that sounds like it
> could be addressed. Couple of comments:
>
> 1. I'd like to see a security impact section added just so it's clear
> that it has been thought about. For example, what is behavior if a
> corrupt, invalid or truncated file is accessed.
> 2. Does the flatgeobuf directory contain only code specific to
> MapServer or is there a mix of custom code and 3rd party code? If
> there is 3rd party code could there be a discussion about how that
> should be maintained over time?
> 3. Are things pinned to a particular version of flatbuf - I assume
> yes and that should be noted.
> 4. I'm assuming there are no MapScript issues - might be nice to
> explicitly state that.
>
> --Steve
>
> On Wed, May 18, 2022 at 6:38 AM Even Rouault
> <even.rouault at spatialys.com> wrote:
>
> I'm 0 on this. While this is always nice to see a perf
> improvement, I'm
> somewhat concerned by the duplication of code between MapServer
> and GDAL.
>
> And from a purely technical point of view, one of my worry is that
> symbol clashes might occur between GDAL and MapServer on the
> common code
> they share in the FlatGeobuf and flatbuffers C++ namespaces. If the
> implementations are not kept in sync (and that will necessarily
> happen
> at some point because we cannot control which GDAL version is used by
> MapServer), weird things (crashes) could possibly happen at
> runtime (or
> at build time if doing static builds). That worry could be solved by
> possibly having a mapserver:: prefix in front of those namespace
> names
> (can be done with some #define trickery. See
> https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104
>
> for potential inspiration)
>
> The interesting take-away from my perspective is that the OGR
> overhead
> for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one
> interpretation would be that there's some inefficiency in the OGR
> Shapefile driver vs the native MapServer provider.
>
> Even
>
> Le 17/05/2022 à 15:19, Jeff McKenna a écrit :
> > Update: Björn has completed the effort (which now includes several
> > msautotests) and we've created an RFC (137) for the native
> FlatGeobuf
> > support: https://mapserver.org/development/rfc/ms-rfc-137.html
> >
> > I therefore motion to include FlatGeobuf as a native MapServer
> driver,
> > per RFC 137, and maintained by Björn Harrtell.
> >
> > starting with my +1
> >
> > Some other interesting updates:
> >
> > - great to see changes here made through testing benefiting so many
> > other projects
> >
> > - here is an updated benchmark result (that is mentioned in the
> RFC) :
> >
> >
> > FlatGeobuf (native) 0.008s
> > Shapefile (native) 0.010s
> > FlatGeobuf (OGR) 0.013s
> > Shapefile (OGR) 0.023s
> > GeoPackage (OGR) 0.042s
> > SpatiaLite (OGR) 0.045s
> > PostGIS (native) 0.053s
> > GeoJSON (OGR) 0.089s
> >
> > (that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)
> >
> > - with all this testing done, and the existing FlatGeobuf
> > documentation effort, it will be very easy to add this to the main
> > documentation now...
> >
> > Thanks,
> >
> >
> > -Björn & Jeff
> >
> >
> >
> >
> >
> > On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
> >> Great point Even, here is an updated result:
> >>
> >> Shapefile 0.011s
> >> FlatGeobuf 0.014s
> >> Shapefile (OGR) 0.024s
> >> GeoPackage 0.042s
> >> SpatiaLite 0.045s
> >> PostGIS 0.053s
> >> GeoJSON 0.089s
> >>
> >>
> >> -jeff
> >>
> >>
> >>
> >>
> >> On 2022-04-27 2:09 p.m., Even Rouault wrote:
> >>> Jeff,
> >>>
> >>> as a data point, perhaps you could enhance your bench with
> >>> shapefiles through OGR ? It would be interesting to have a
> sense of
> >>> the OGR overhead (that will be the same for any OGR supported
> >>> datasource) to have an idea if it is really worth the effort
> to have
> >>> a native FlatGeobuf provider for MapServer w.r.t going through
> OGR
> >>> (it is hard to beat shapefile, and the current difference between
> >>> shapefile and flatgeobuf is small)
> >>>
> >>> Even
> >>>
> >>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
> >>>> (to followup from our chat on IRC yesterday)
> >>>>
> >>>> To clarify: this would be a new native format in MapServer,
> such as
> >>>> Shapefile or PostGIS, and the goal would be to not connect to
> >>>> FlatGeobuf through OGR, but instead directly, such as:
> >>>>
> >>>> DATA countries.fgb
> >>>>
> >>>> instead of:
> >>>>
> >>>> CONNECTIONTYPE OGR
> >>>> CONNECTION "countries.fgb"
> >>>> DATA "countries"
> >>>>
> >>>> This should improve the performance even more. And it will
> be easy
> >>>> to update the documentation, as the existing files just need
> to be
> >>>> updated (see the new page at
> >>>> https://mapserver.org/input/vector/flatgeobuf.html )
> >>>>
> >>>> Great plan Björn! I look forward to your Pull Request, ha, and
> >>>> will tackle the documentation changes for you.
> >>>>
> >>>> This is also great news for MapServer, and will give us some
> nice
> >>>> publicity.
> >>>>
> >>>> Thanks so much!
> >>>>
> >>>> -jeff
> >>>>
> >>>>
> >>>>
> >>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
> >>>>> Hi mapserver devs!
> >>>>>
> >>>>> I got interested in the subject because of the tests made
> recently
> >>>>> by Jeff that shows the potential of the format. I believe it
> >>>>> should be possible to get significant additional performance
> out
> >>>>> of FlatGeobuf in MapServer if it was built in just like
> Shapefile
> >>>>> support is.
> >>>>>
> >>>>> FlatGeobuf C++ implementation is rather small, doesn't
> require any
> >>>>> dependencies, and I don't expect it to require any invasive
> >>>>> changes to existing code.
> >>>>>
> >>>>> Would love to get started on this ASAP, if there are no
> >>>>> objections. :)
> >>>>>
> >>>>> Regards,
> >>>>> Björn Harrtell
> >>>>>
> >> ______
> > _______________________________________________
> > MapServer-dev mailing list
> > MapServer-dev at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
>
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20220518/8e26a778/attachment-0001.htm>
More information about the MapServer-dev
mailing list