[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

Wed May 18 08:09:52 PDT 2022

Nice work all - esp. Björn. Evan has a good point that sounds like it could
be addressed. Couple of comments:

   1. I'd like to see a security impact section added just so it's clear
   that it has been thought about. For example, what is behavior if a corrupt,
   invalid or truncated file is accessed.
   2. Does the flatgeobuf directory contain only code specific to MapServer
   or is there a mix of custom code and 3rd party code? If there is 3rd party
   code could there be a discussion about how that should be maintained over
   time?
   3. Are things pinned to a particular version of flatbuf - I assume yes
   and that should be noted.
   4. I'm assuming there are no MapScript issues - might be nice to
   explicitly state that.

--Steve

On Wed, May 18, 2022 at 6:38 AM Even Rouault <even.rouault at spatialys.com>
wrote:

> I'm 0 on this. While this is always nice to see a perf improvement, I'm
> somewhat concerned by the duplication of code between MapServer and GDAL.
>
> And from a purely technical point of view, one of my worry is that
> symbol clashes might occur between GDAL and MapServer on the common code
> they share in the FlatGeobuf and flatbuffers C++ namespaces. If the
> implementations are not kept in sync (and that will necessarily happen
> at some point because we cannot control which GDAL version is used by
> MapServer), weird things (crashes) could possibly happen at runtime (or
> at build time if doing static builds). That worry could be solved by
> possibly having a mapserver:: prefix in front of those namespace names
> (can be done with some #define trickery. See
>
> https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104
> for potential inspiration)
>
> The interesting take-away from my perspective is that the OGR overhead
> for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one
> interpretation would be that there's some inefficiency in the OGR
> Shapefile driver vs the native MapServer provider.
>
> Even
>
> Le 17/05/2022 à 15:19, Jeff McKenna a écrit :
> > Update: Björn has completed the effort (which now includes several
> > msautotests) and we've created an RFC (137) for the native FlatGeobuf
> > support: https://mapserver.org/development/rfc/ms-rfc-137.html
> >
> > I therefore motion to include FlatGeobuf as a native MapServer driver,
> > per RFC 137, and maintained by Björn Harrtell.
> >
> > starting with my +1
> >
> > Some other interesting updates:
> >
> > - great to see changes here made through testing benefiting so many
> > other projects
> >
> > - here is an updated benchmark result (that is mentioned in the RFC) :
> >
> >
> >   FlatGeobuf (native) 0.008s
> >   Shapefile (native)  0.010s
> >   FlatGeobuf (OGR)    0.013s
> >   Shapefile (OGR)     0.023s
> >   GeoPackage (OGR)    0.042s
> >   SpatiaLite (OGR)    0.045s
> >   PostGIS (native)    0.053s
> >   GeoJSON (OGR)       0.089s
> >
> > (that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)
> >
> > - with all this testing done, and the existing FlatGeobuf
> > documentation effort, it will be very easy to add this to the main
> > documentation now...
> >
> > Thanks,
> >
> >
> > -Björn & Jeff
> >
> >
> >
> >
> >
> > On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
> >> Great point Even, here is an updated result:
> >>
> >>    Shapefile       0.011s
> >>    FlatGeobuf      0.014s
> >>    Shapefile (OGR) 0.024s
> >>    GeoPackage      0.042s
> >>    SpatiaLite      0.045s
> >>    PostGIS         0.053s
> >>    GeoJSON         0.089s
> >>
> >>
> >> -jeff
> >>
> >>
> >>
> >>
> >> On 2022-04-27 2:09 p.m., Even Rouault wrote:
> >>> Jeff,
> >>>
> >>> as a data point, perhaps you could enhance your bench with
> >>> shapefiles through OGR ? It would be interesting to have a sense of
> >>> the OGR overhead (that will be the same for any OGR supported
> >>> datasource) to have an idea if it is really worth the effort to have
> >>> a native FlatGeobuf provider for MapServer w.r.t going through OGR
> >>> (it is hard to beat shapefile, and the current difference between
> >>> shapefile and flatgeobuf is small)
> >>>
> >>> Even
> >>>
> >>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
> >>>> (to followup from our chat on IRC yesterday)
> >>>>
> >>>> To clarify: this would be a new native format in MapServer, such as
> >>>> Shapefile or PostGIS, and the goal would be to not connect to
> >>>> FlatGeobuf through OGR, but instead directly, such as:
> >>>>
> >>>>   DATA countries.fgb
> >>>>
> >>>> instead of:
> >>>>
> >>>>   CONNECTIONTYPE OGR
> >>>>   CONNECTION "countries.fgb"
> >>>>   DATA "countries"
> >>>>
> >>>> This should improve the performance even more.  And it will be easy
> >>>> to update the documentation, as the existing files just need to be
> >>>> updated (see the new page at
> >>>> https://mapserver.org/input/vector/flatgeobuf.html )
> >>>>
> >>>> Great plan Björn!  I look forward to your Pull Request, ha, and
> >>>> will tackle the documentation changes for you.
> >>>>
> >>>> This is also great news for MapServer, and will give us some nice
> >>>> publicity.
> >>>>
> >>>> Thanks so much!
> >>>>
> >>>> -jeff
> >>>>
> >>>>
> >>>>
> >>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
> >>>>> Hi mapserver devs!
> >>>>>
> >>>>> I got interested in the subject because of the tests made recently
> >>>>> by Jeff that shows the potential of the format. I believe it
> >>>>> should be possible to get significant additional performance out
> >>>>> of FlatGeobuf in MapServer if it was built in just like Shapefile
> >>>>> support is.
> >>>>>
> >>>>> FlatGeobuf C++ implementation is rather small, doesn't require any
> >>>>> dependencies, and I don't expect it to require any invasive
> >>>>> changes to existing code.
> >>>>>
> >>>>> Would love to get started on this ASAP, if there are no
> >>>>> objections. :)
> >>>>>
> >>>>> Regards,
> >>>>> Björn Harrtell
> >>>>>
> >> ______
> > _______________________________________________
> > MapServer-dev mailing list
> > MapServer-dev at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20220518/0c40f052/attachment.htm>