[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format
Björn Harrtell
bjorn.harrtell at gmail.com
Thu May 19 00:12:14 PDT 2022
1. Undefined behavior more or less. There is a verifier that could mitigate
security risks in some capacity but it would cost a small bit of
performance. I've not implemented support to use the verifier at this point
(to keep the initial implementation as simple as possible).
2. It contains mapserver specific C wrapper and third party code (upstreams
are FlatGeobuf and transitive dependency flatbuffers)
3. It's currently bleeding edge code wise. If you mean the spec/format
itself it's version 3 (stable since early 2020) and I have no plans for any
breaking change in that regard.
4. I've been careful to make absolute minimal additions to the mapfile
parser. I'm confident it does not introduce anything breaking.
Den ons 18 maj 2022 kl 17:10 skrev Steve Lime <sdlime at gmail.com>:
> Nice work all - esp. Björn. Evan has a good point that sounds like it
> could be addressed. Couple of comments:
>
> 1. I'd like to see a security impact section added just so it's clear
> that it has been thought about. For example, what is behavior if a corrupt,
> invalid or truncated file is accessed.
> 2. Does the flatgeobuf directory contain only code specific to
> MapServer or is there a mix of custom code and 3rd party code? If there is
> 3rd party code could there be a discussion about how that should be
> maintained over time?
> 3. Are things pinned to a particular version of flatbuf - I assume yes
> and that should be noted.
> 4. I'm assuming there are no MapScript issues - might be nice to
> explicitly state that.
>
> --Steve
>
> On Wed, May 18, 2022 at 6:38 AM Even Rouault <even.rouault at spatialys.com>
> wrote:
>
>> I'm 0 on this. While this is always nice to see a perf improvement, I'm
>> somewhat concerned by the duplication of code between MapServer and GDAL.
>>
>> And from a purely technical point of view, one of my worry is that
>> symbol clashes might occur between GDAL and MapServer on the common code
>> they share in the FlatGeobuf and flatbuffers C++ namespaces. If the
>> implementations are not kept in sync (and that will necessarily happen
>> at some point because we cannot control which GDAL version is used by
>> MapServer), weird things (crashes) could possibly happen at runtime (or
>> at build time if doing static builds). That worry could be solved by
>> possibly having a mapserver:: prefix in front of those namespace names
>> (can be done with some #define trickery. See
>>
>> https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104
>> for potential inspiration)
>>
>> The interesting take-away from my perspective is that the OGR overhead
>> for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one
>> interpretation would be that there's some inefficiency in the OGR
>> Shapefile driver vs the native MapServer provider.
>>
>> Even
>>
>> Le 17/05/2022 à 15:19, Jeff McKenna a écrit :
>> > Update: Björn has completed the effort (which now includes several
>> > msautotests) and we've created an RFC (137) for the native FlatGeobuf
>> > support: https://mapserver.org/development/rfc/ms-rfc-137.html
>> >
>> > I therefore motion to include FlatGeobuf as a native MapServer driver,
>> > per RFC 137, and maintained by Björn Harrtell.
>> >
>> > starting with my +1
>> >
>> > Some other interesting updates:
>> >
>> > - great to see changes here made through testing benefiting so many
>> > other projects
>> >
>> > - here is an updated benchmark result (that is mentioned in the RFC) :
>> >
>> >
>> > FlatGeobuf (native) 0.008s
>> > Shapefile (native) 0.010s
>> > FlatGeobuf (OGR) 0.013s
>> > Shapefile (OGR) 0.023s
>> > GeoPackage (OGR) 0.042s
>> > SpatiaLite (OGR) 0.045s
>> > PostGIS (native) 0.053s
>> > GeoJSON (OGR) 0.089s
>> >
>> > (that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)
>> >
>> > - with all this testing done, and the existing FlatGeobuf
>> > documentation effort, it will be very easy to add this to the main
>> > documentation now...
>> >
>> > Thanks,
>> >
>> >
>> > -Björn & Jeff
>> >
>> >
>> >
>> >
>> >
>> > On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
>> >> Great point Even, here is an updated result:
>> >>
>> >> Shapefile 0.011s
>> >> FlatGeobuf 0.014s
>> >> Shapefile (OGR) 0.024s
>> >> GeoPackage 0.042s
>> >> SpatiaLite 0.045s
>> >> PostGIS 0.053s
>> >> GeoJSON 0.089s
>> >>
>> >>
>> >> -jeff
>> >>
>> >>
>> >>
>> >>
>> >> On 2022-04-27 2:09 p.m., Even Rouault wrote:
>> >>> Jeff,
>> >>>
>> >>> as a data point, perhaps you could enhance your bench with
>> >>> shapefiles through OGR ? It would be interesting to have a sense of
>> >>> the OGR overhead (that will be the same for any OGR supported
>> >>> datasource) to have an idea if it is really worth the effort to have
>> >>> a native FlatGeobuf provider for MapServer w.r.t going through OGR
>> >>> (it is hard to beat shapefile, and the current difference between
>> >>> shapefile and flatgeobuf is small)
>> >>>
>> >>> Even
>> >>>
>> >>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
>> >>>> (to followup from our chat on IRC yesterday)
>> >>>>
>> >>>> To clarify: this would be a new native format in MapServer, such as
>> >>>> Shapefile or PostGIS, and the goal would be to not connect to
>> >>>> FlatGeobuf through OGR, but instead directly, such as:
>> >>>>
>> >>>> DATA countries.fgb
>> >>>>
>> >>>> instead of:
>> >>>>
>> >>>> CONNECTIONTYPE OGR
>> >>>> CONNECTION "countries.fgb"
>> >>>> DATA "countries"
>> >>>>
>> >>>> This should improve the performance even more. And it will be easy
>> >>>> to update the documentation, as the existing files just need to be
>> >>>> updated (see the new page at
>> >>>> https://mapserver.org/input/vector/flatgeobuf.html )
>> >>>>
>> >>>> Great plan Björn! I look forward to your Pull Request, ha, and
>> >>>> will tackle the documentation changes for you.
>> >>>>
>> >>>> This is also great news for MapServer, and will give us some nice
>> >>>> publicity.
>> >>>>
>> >>>> Thanks so much!
>> >>>>
>> >>>> -jeff
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
>> >>>>> Hi mapserver devs!
>> >>>>>
>> >>>>> I got interested in the subject because of the tests made recently
>> >>>>> by Jeff that shows the potential of the format. I believe it
>> >>>>> should be possible to get significant additional performance out
>> >>>>> of FlatGeobuf in MapServer if it was built in just like Shapefile
>> >>>>> support is.
>> >>>>>
>> >>>>> FlatGeobuf C++ implementation is rather small, doesn't require any
>> >>>>> dependencies, and I don't expect it to require any invasive
>> >>>>> changes to existing code.
>> >>>>>
>> >>>>> Would love to get started on this ASAP, if there are no
>> >>>>> objections. :)
>> >>>>>
>> >>>>> Regards,
>> >>>>> Björn Harrtell
>> >>>>>
>> >> ______
>> > _______________________________________________
>> > MapServer-dev mailing list
>> > MapServer-dev at lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>> _______________________________________________
>> MapServer-dev mailing list
>> MapServer-dev at lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20220519/e03ed040/attachment-0001.htm>
More information about the MapServer-dev
mailing list