[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format
Jeff McKenna
jmckenna at gatewaygeomatics.com
Wed May 18 13:12:00 PDT 2022
Ah I see your point, I will expand that point more in the RFC now...
Your assumption is correct. Thanks Lars!
-jeff
On 2022-05-18 5:09 p.m., Jeff McKenna wrote:
> Thanks for the positive support Lars, your testing feedback is
> important, and merry Christmas ha.
>
> Regarding your question, please see the section "3.3 Limitations" of the
> RFC which is supposed to handle your question.
>
> -jeff
>
>
>
>
> On 2022-05-18 4:07 p.m., Lars Schylberg wrote:
>> Thanks, Björn and Jeff for the effort to do this.I am eager to start
>> testing soon.As a user I feel that Santa is coming early this year.
>>
>> I also have one question.Is the native FlatGeobuf driver doing the
>> reading without checks, like "VERIFY_BUFFERS" "NO" that you can set in
>> the OGR driver?
>>
>> My inofficial +1
>>
>> Lars Schylberg
>>
>> Den 2022-05-18 kl. 17:09, skrev Steve Lime:
>>> Nice work all - esp. Björn. Evan has a good point that sounds like it
>>> could be addressed. Couple of comments:
>>>
>>> 1. I'd like to see a security impact section added just so it's clear
>>> that it has been thought about. For example, what is behavior if a
>>> corrupt, invalid or truncated file is accessed.
>>> 2. Does the flatgeobuf directory contain only code specific to
>>> MapServer or is there a mix of custom code and 3rd party code? If
>>> there is 3rd party code could there be a discussion about how that
>>> should be maintained over time?
>>> 3. Are things pinned to a particular version of flatbuf - I assume
>>> yes and that should be noted.
>>> 4. I'm assuming there are no MapScript issues - might be nice to
>>> explicitly state that.
>>>
>>> --Steve
>>>
>>> On Wed, May 18, 2022 at 6:38 AM Even Rouault
>>> <even.rouault at spatialys.com> wrote:
>>>
>>> I'm 0 on this. While this is always nice to see a perf
>>> improvement, I'm
>>> somewhat concerned by the duplication of code between MapServer
>>> and GDAL.
>>>
>>> And from a purely technical point of view, one of my worry is that
>>> symbol clashes might occur between GDAL and MapServer on the
>>> common code
>>> they share in the FlatGeobuf and flatbuffers C++ namespaces. If the
>>> implementations are not kept in sync (and that will necessarily
>>> happen
>>> at some point because we cannot control which GDAL version is
>>> used by
>>> MapServer), weird things (crashes) could possibly happen at
>>> runtime (or
>>> at build time if doing static builds). That worry could be solved by
>>> possibly having a mapserver:: prefix in front of those namespace
>>> names
>>> (can be done with some #define trickery. See
>>>
>>> https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104
>>>
>>>
>>> for potential inspiration)
>>>
>>> The interesting take-away from my perspective is that the OGR
>>> overhead
>>> for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one
>>> interpretation would be that there's some inefficiency in the OGR
>>> Shapefile driver vs the native MapServer provider.
>>>
>>> Even
>>>
>>> Le 17/05/2022 à 15:19, Jeff McKenna a écrit :
>>> > Update: Björn has completed the effort (which now includes several
>>> > msautotests) and we've created an RFC (137) for the native
>>> FlatGeobuf
>>> > support: https://mapserver.org/development/rfc/ms-rfc-137.html
>>> >
>>> > I therefore motion to include FlatGeobuf as a native MapServer
>>> driver,
>>> > per RFC 137, and maintained by Björn Harrtell.
>>> >
>>> > starting with my +1
>>> >
>>> > Some other interesting updates:
>>> >
>>> > - great to see changes here made through testing benefiting so
>>> many
>>> > other projects
>>> >
>>> > - here is an updated benchmark result (that is mentioned in the
>>> RFC) :
>>> >
>>> >
>>> > FlatGeobuf (native) 0.008s
>>> > Shapefile (native) 0.010s
>>> > FlatGeobuf (OGR) 0.013s
>>> > Shapefile (OGR) 0.023s
>>> > GeoPackage (OGR) 0.042s
>>> > SpatiaLite (OGR) 0.045s
>>> > PostGIS (native) 0.053s
>>> > GeoJSON (OGR) 0.089s
>>> >
>>> > (that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)
>>> >
>>> > - with all this testing done, and the existing FlatGeobuf
>>> > documentation effort, it will be very easy to add this to the main
>>> > documentation now...
>>> >
>>> > Thanks,
>>> >
>>> >
>>> > -Björn & Jeff
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
>>> >> Great point Even, here is an updated result:
>>> >>
>>> >> Shapefile 0.011s
>>> >> FlatGeobuf 0.014s
>>> >> Shapefile (OGR) 0.024s
>>> >> GeoPackage 0.042s
>>> >> SpatiaLite 0.045s
>>> >> PostGIS 0.053s
>>> >> GeoJSON 0.089s
>>> >>
>>> >>
>>> >> -jeff
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On 2022-04-27 2:09 p.m., Even Rouault wrote:
>>> >>> Jeff,
>>> >>>
>>> >>> as a data point, perhaps you could enhance your bench with
>>> >>> shapefiles through OGR ? It would be interesting to have a
>>> sense of
>>> >>> the OGR overhead (that will be the same for any OGR supported
>>> >>> datasource) to have an idea if it is really worth the effort
>>> to have
>>> >>> a native FlatGeobuf provider for MapServer w.r.t going through
>>> OGR
>>> >>> (it is hard to beat shapefile, and the current difference
>>> between
>>> >>> shapefile and flatgeobuf is small)
>>> >>>
>>> >>> Even
>>> >>>
>>> >>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
>>> >>>> (to followup from our chat on IRC yesterday)
>>> >>>>
>>> >>>> To clarify: this would be a new native format in MapServer,
>>> such as
>>> >>>> Shapefile or PostGIS, and the goal would be to not connect to
>>> >>>> FlatGeobuf through OGR, but instead directly, such as:
>>> >>>>
>>> >>>> DATA countries.fgb
>>> >>>>
>>> >>>> instead of:
>>> >>>>
>>> >>>> CONNECTIONTYPE OGR
>>> >>>> CONNECTION "countries.fgb"
>>> >>>> DATA "countries"
>>> >>>>
>>> >>>> This should improve the performance even more. And it will
>>> be easy
>>> >>>> to update the documentation, as the existing files just need
>>> to be
>>> >>>> updated (see the new page at
>>> >>>> https://mapserver.org/input/vector/flatgeobuf.html )
>>> >>>>
>>> >>>> Great plan Björn! I look forward to your Pull Request, ha, and
>>> >>>> will tackle the documentation changes for you.
>>> >>>>
>>> >>>> This is also great news for MapServer, and will give us some
>>> nice
>>> >>>> publicity.
>>> >>>>
>>> >>>> Thanks so much!
>>> >>>>
>>> >>>> -jeff
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
>>> >>>>> Hi mapserver devs!
>>> >>>>>
>>> >>>>> I got interested in the subject because of the tests made
>>> recently
>>> >>>>> by Jeff that shows the potential of the format. I believe it
>>> >>>>> should be possible to get significant additional performance
>>> out
>>> >>>>> of FlatGeobuf in MapServer if it was built in just like
>>> Shapefile
>>> >>>>> support is.
>>> >>>>>
>>> >>>>> FlatGeobuf C++ implementation is rather small, doesn't
>>> require any
>>> >>>>> dependencies, and I don't expect it to require any invasive
>>> >>>>> changes to existing code.
>>> >>>>>
>>> >>>>> Would love to get started on this ASAP, if there are no
>>> >>>>> objections. :)
>>> >>>>>
>>> >>>>> Regards,
>>> >>>>> Björn Harrtell
>>> >>>>>
>>> >> ______
>>> > _______________________________________________
>>> > MapServer-dev mailing list
>>> > MapServer-dev at lists.osgeo.org
>>> > https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>
>>> -- http://www.spatialys.com
>>> My software is free, but my time generally not.
>>>
>>> _______________________________________________
>>> MapServer-dev mailing list
>>> MapServer-dev at lists.osgeo.org
>>> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>>
> _______________________________________________
> MapServer-dev mailing list
> MapServer-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/mapserver-dev
--
Jeff McKenna
GatewayGeo: Developers of MS4W, MapServer Consulting and Training
co-founder of FOSS4G
http://gatewaygeo.com/
More information about the MapServer-dev
mailing list