[mapserver-dev] Proposal/request to implement FlatGeobuf as built-in format

[Jeff McKenna]

Update: Björn has completed the effort (which now includes several 
msautotests) and we've created an RFC (137) for the native FlatGeobuf 
support: https://mapserver.org/development/rfc/ms-rfc-137.html

I therefore motion to include FlatGeobuf as a native MapServer driver, 
per RFC 137, and maintained by Björn Harrtell.

starting with my +1

Some other interesting updates:

- great to see changes here made through testing benefiting so many 
other projects

- here is an updated benchmark result (that is mentioned in the RFC) :


   FlatGeobuf (native) 0.008s
   Shapefile (native)  0.010s
   FlatGeobuf (OGR)    0.013s
   Shapefile (OGR)     0.023s
   GeoPackage (OGR)    0.042s
   SpatiaLite (OGR)    0.045s
   PostGIS (native)    0.053s
   GeoJSON (OGR)       0.089s

(that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)

- with all this testing done, and the existing FlatGeobuf documentation 
effort, it will be very easy to add this to the main documentation now...

Thanks,


-Björn & Jeff





On 2022-04-27 3:07 p.m., Jeff McKenna wrote:
> Great point Even, here is an updated result:
> 
>    Shapefile       0.011s
>    FlatGeobuf      0.014s
>    Shapefile (OGR) 0.024s
>    GeoPackage      0.042s
>    SpatiaLite      0.045s
>    PostGIS         0.053s
>    GeoJSON         0.089s
> 
> 
> -jeff
> 
> 
> 
> 
> On 2022-04-27 2:09 p.m., Even Rouault wrote:
>> Jeff,
>>
>> as a data point, perhaps you could enhance your bench with shapefiles 
>> through OGR ? It would be interesting to have a sense of the OGR 
>> overhead (that will be the same for any OGR supported datasource) to 
>> have an idea if it is really worth the effort to have a native 
>> FlatGeobuf provider for MapServer w.r.t going through OGR (it is hard 
>> to beat shapefile, and the current difference between shapefile and 
>> flatgeobuf is small)
>>
>> Even
>>
>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :
>>> (to followup from our chat on IRC yesterday)
>>>
>>> To clarify: this would be a new native format in MapServer, such as 
>>> Shapefile or PostGIS, and the goal would be to not connect to 
>>> FlatGeobuf through OGR, but instead directly, such as:
>>>
>>>   DATA countries.fgb
>>>
>>> instead of:
>>>
>>>   CONNECTIONTYPE OGR
>>>   CONNECTION "countries.fgb"
>>>   DATA "countries"
>>>
>>> This should improve the performance even more.  And it will be easy 
>>> to update the documentation, as the existing files just need to be 
>>> updated (see the new page at 
>>> https://mapserver.org/input/vector/flatgeobuf.html )
>>>
>>> Great plan Björn!  I look forward to your Pull Request, ha, and will 
>>> tackle the documentation changes for you.
>>>
>>> This is also great news for MapServer, and will give us some nice 
>>> publicity.
>>>
>>> Thanks so much!
>>>
>>> -jeff
>>>
>>>
>>>
>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:
>>>> Hi mapserver devs!
>>>>
>>>> I got interested in the subject because of the tests made recently 
>>>> by Jeff that shows the potential of the format. I believe it should 
>>>> be possible to get significant additional performance out of 
>>>> FlatGeobuf in MapServer if it was built in just like Shapefile 
>>>> support is.
>>>>
>>>> FlatGeobuf C++ implementation is rather small, doesn't require any 
>>>> dependencies, and I don't expect it to require any invasive changes 
>>>> to existing code.
>>>>
>>>> Would love to get started on this ASAP, if there are no objections. :)
>>>>
>>>> Regards,
>>>> Björn Harrtell
>>>>
> ______