[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

Fri May 20 10:00:50 PDT 2022

For the record I can't see the #define trick as reasonable/acceptable if i
understand it correctly. I also have no other ideas, so I'm (very) sorry to
say this is now dead in the water if nothing new comes to light.

Den tors 19 maj 2022 09:13Björn Harrtell <bjorn.harrtell at gmail.com> skrev:

> Thanks Even, I was not aware of the risk of symbol clash. I will try and
> apply the #define trickery.
>
> Den ons 18 maj 2022 kl 13:31 skrev Even Rouault <
> even.rouault at spatialys.com>:
>
>> I'm 0 on this. While this is always nice to see a perf improvement, I'm
>> somewhat concerned by the duplication of code between MapServer and GDAL.
>>
>> And from a purely technical point of view, one of my worry is that
>> symbol clashes might occur between GDAL and MapServer on the common code
>> they share in the FlatGeobuf and flatbuffers C++ namespaces. If the
>> implementations are not kept in sync (and that will necessarily happen
>> at some point because we cannot control which GDAL version is used by
>> MapServer), weird things (crashes) could possibly happen at runtime (or
>> at build time if doing static builds). That worry could be solved by
>> possibly having a mapserver:: prefix in front of those namespace names
>> (can be done with some #define trickery. See
>>
>> https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104
>> for potential inspiration)
>>
>> The interesting take-away from my perspective is that the OGR overhead
>> for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one
>> interpretation would be that there's some inefficiency in the OGR
>> Shapefile driver vs the native MapServer provider.
>>
>> Even
>>
>> Le 17/05/2022 à 15:19, Jeff McKenna a écrit :
>> > Update: Björn has completed the effort (which now includes several
>> > msautotests) and we've created an RFC (137) for the native FlatGeobuf
>> > support: https://mapserver.org/development/rfc/ms-rfc-137.html
>> >
>> > I therefore motion to include FlatGeobuf as a native MapServer driver,
>> > per RFC 137, and maintained by Björn Harrtell.
>> >
>> > starting with my +1
>> >
>> > Some other interesting updates:
>> >
>> > - great to see changes here made through testing benefiting so many
>> > other projects
>> >
>> > - here is an updated benchmark result (that is mentioned in the RFC) :
>> >
>> >
>> >   FlatGeobuf (native) 0.008s
>> >   Shapefile (native)  0.010s
>> >   FlatGeobuf (OGR)    0.013s
>> >   Shapefile (OGR)     0.023s
>> >   GeoPackage (OGR)    0.042s
>> >   SpatiaLite (OGR)    0.045s
>> >   PostGIS (native)    0.053s
>> >   GeoJSON (OGR)       0.089s
>> >
>> > (that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)
>> >
>> > - with all this testing done, and the existing FlatGeobuf
>> > documentation effort, it will be very easy to add this to the main
>> > documentation now...
>> >
>> > Thanks,
>> >
>> >
>> > -Björn & Jeff
>> >
>> >
>> >
>> >
>> >
>> > On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
>> >> Great point Even, here is an updated result:
>> >>
>> >>    Shapefile       0.011s
>> >>    FlatGeobuf      0.014s
>> >>    Shapefile (OGR) 0.024s
>> >>    GeoPackage      0.042s
>> >>    SpatiaLite      0.045s
>> >>    PostGIS         0.053s
>> >>    GeoJSON         0.089s
>> >>
>> >>
>> >> -jeff
>> >>
>> >>
>> >>
>> >>
>> >> On 2022-04-27 2:09 p.m., Even Rouault wrote:
>> >>> Jeff,
>> >>>
>> >>> as a data point, perhaps you could enhance your bench with
>> >>> shapefiles through OGR ? It would be interesting to have a sense of
>> >>> the OGR overhead (that will be the same for any OGR supported
>> >>> datasource) to have an idea if it is really worth the effort to have
>> >>> a native FlatGeobuf provider for MapServer w.r.t going through OGR
>> >>> (it is hard to beat shapefile, and the current difference between
>> >>> shapefile and flatgeobuf is small)
>> >>>
>> >>> Even
>> >>>
>> >>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
>> >>>> (to followup from our chat on IRC yesterday)
>> >>>>
>> >>>> To clarify: this would be a new native format in MapServer, such as
>> >>>> Shapefile or PostGIS, and the goal would be to not connect to
>> >>>> FlatGeobuf through OGR, but instead directly, such as:
>> >>>>
>> >>>>   DATA countries.fgb
>> >>>>
>> >>>> instead of:
>> >>>>
>> >>>>   CONNECTIONTYPE OGR
>> >>>>   CONNECTION "countries.fgb"
>> >>>>   DATA "countries"
>> >>>>
>> >>>> This should improve the performance even more.  And it will be easy
>> >>>> to update the documentation, as the existing files just need to be
>> >>>> updated (see the new page at
>> >>>> https://mapserver.org/input/vector/flatgeobuf.html )
>> >>>>
>> >>>> Great plan Björn!  I look forward to your Pull Request, ha, and
>> >>>> will tackle the documentation changes for you.
>> >>>>
>> >>>> This is also great news for MapServer, and will give us some nice
>> >>>> publicity.
>> >>>>
>> >>>> Thanks so much!
>> >>>>
>> >>>> -jeff
>> >>>>
>> >>>>
>> >>>>
>> >>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
>> >>>>> Hi mapserver devs!
>> >>>>>
>> >>>>> I got interested in the subject because of the tests made recently
>> >>>>> by Jeff that shows the potential of the format. I believe it
>> >>>>> should be possible to get significant additional performance out
>> >>>>> of FlatGeobuf in MapServer if it was built in just like Shapefile
>> >>>>> support is.
>> >>>>>
>> >>>>> FlatGeobuf C++ implementation is rather small, doesn't require any
>> >>>>> dependencies, and I don't expect it to require any invasive
>> >>>>> changes to existing code.
>> >>>>>
>> >>>>> Would love to get started on this ASAP, if there are no
>> >>>>> objections. :)
>> >>>>>
>> >>>>> Regards,
>> >>>>> Björn Harrtell
>> >>>>>
>> >> ______
>> > _______________________________________________
>> > MapServer-dev mailing list
>> > MapServer-dev at lists.osgeo.org
>> > https://lists.osgeo.org/mailman/listinfo/mapserver-dev
>>
>> --
>> http://www.spatialys.com
>> My software is free, but my time generally not.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/mapserver-dev/attachments/20220520/3c0151ac/attachment.htm>