<div dir="ltr"><div>1. Undefined behavior more or less. There is a verifier that could mitigate security risks in some capacity but it would cost a small bit of performance. I've not implemented support to use the verifier at this point (to keep the initial implementation as simple as possible).</div><div>2. It contains mapserver specific C wrapper and third party code (upstreams are FlatGeobuf and transitive dependency flatbuffers)</div><div>3. It's currently bleeding edge code wise. If you mean the spec/format itself it's version 3 (stable since early 2020) and I have no plans for any breaking change in that regard.</div><div>4. I've been careful to make absolute minimal additions to the mapfile parser. I'm confident it does not introduce anything breaking.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Den ons 18 maj 2022 kl 17:10 skrev Steve Lime <<a href="mailto:sdlime@gmail.com">sdlime@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Nice work all - esp. Björn. Evan has a good point that sounds like it could be addressed. Couple of comments:<div><ol><li>I'd like to see a security impact section added just so it's clear that it has been thought about. For example, what is behavior if a corrupt, invalid or truncated file is accessed.</li><li>Does the flatgeobuf directory contain only code specific to MapServer or is there a mix of custom code and 3rd party code? If there is 3rd party code could there be a discussion about how that should be maintained over time? </li><li>Are things pinned to a particular version of flatbuf - I assume yes and that should be noted.</li><li>I'm assuming there are no MapScript issues - might be nice to explicitly state that.</li></ol>--Steve <br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 18, 2022 at 6:38 AM Even Rouault <<a href="mailto:even.rouault@spatialys.com" target="_blank">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I'm 0 on this. While this is always nice to see a perf improvement, I'm <br>
somewhat concerned by the duplication of code between MapServer and GDAL.<br>
<br>
And from a purely technical point of view, one of my worry is that <br>
symbol clashes might occur between GDAL and MapServer on the common code <br>
they share in the FlatGeobuf and flatbuffers C++ namespaces. If the <br>
implementations are not kept in sync (and that will necessarily happen <br>
at some point because we cannot control which GDAL version is used by <br>
MapServer), weird things (crashes) could possibly happen at runtime (or <br>
at build time if doing static builds). That worry could be solved by <br>
possibly having a mapserver:: prefix in front of those namespace names <br>
(can be done with some #define trickery. See <br>
<a href="https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104" rel="noreferrer" target="_blank">https://github.com/OSGeo/PROJ/blob/9f89c7288e44c64ba234a299b1d5528a54d9d61e/include/proj/util.hpp#L104</a> <br>
for potential inspiration)<br>
<br>
The interesting take-away from my perspective is that the OGR overhead <br>
for FlatGeobuf is only 5 ms, but 13 ms for Shapefile. So one <br>
interpretation would be that there's some inefficiency in the OGR <br>
Shapefile driver vs the native MapServer provider.<br>
<br>
Even<br>
<br>
Le 17/05/2022 à 15:19, Jeff McKenna a écrit :<br>
> Update: Björn has completed the effort (which now includes several <br>
> msautotests) and we've created an RFC (137) for the native FlatGeobuf <br>
> support: <a href="https://mapserver.org/development/rfc/ms-rfc-137.html" rel="noreferrer" target="_blank">https://mapserver.org/development/rfc/ms-rfc-137.html</a><br>
><br>
> I therefore motion to include FlatGeobuf as a native MapServer driver, <br>
> per RFC 137, and maintained by Björn Harrtell.<br>
><br>
> starting with my +1<br>
><br>
> Some other interesting updates:<br>
><br>
> - great to see changes here made through testing benefiting so many <br>
> other projects<br>
><br>
> - here is an updated benchmark result (that is mentioned in the RFC) :<br>
><br>
><br>
> FlatGeobuf (native) 0.008s<br>
> Shapefile (native) 0.010s<br>
> FlatGeobuf (OGR) 0.013s<br>
> Shapefile (OGR) 0.023s<br>
> GeoPackage (OGR) 0.042s<br>
> SpatiaLite (OGR) 0.045s<br>
> PostGIS (native) 0.053s<br>
> GeoJSON (OGR) 0.089s<br>
><br>
> (that was with yesterday's main, GDAL 3.4.3, with MS4W on Windows)<br>
><br>
> - with all this testing done, and the existing FlatGeobuf <br>
> documentation effort, it will be very easy to add this to the main <br>
> documentation now...<br>
><br>
> Thanks,<br>
><br>
><br>
> -Björn & Jeff<br>
><br>
><br>
><br>
><br>
><br>
> On 2022-04-27 3:07 p.m., Jeff McKenna wrote:<br>
>> Great point Even, here is an updated result:<br>
>><br>
>> Shapefile 0.011s<br>
>> FlatGeobuf 0.014s<br>
>> Shapefile (OGR) 0.024s<br>
>> GeoPackage 0.042s<br>
>> SpatiaLite 0.045s<br>
>> PostGIS 0.053s<br>
>> GeoJSON 0.089s<br>
>><br>
>><br>
>> -jeff<br>
>><br>
>><br>
>><br>
>><br>
>> On 2022-04-27 2:09 p.m., Even Rouault wrote:<br>
>>> Jeff,<br>
>>><br>
>>> as a data point, perhaps you could enhance your bench with <br>
>>> shapefiles through OGR ? It would be interesting to have a sense of <br>
>>> the OGR overhead (that will be the same for any OGR supported <br>
>>> datasource) to have an idea if it is really worth the effort to have <br>
>>> a native FlatGeobuf provider for MapServer w.r.t going through OGR <br>
>>> (it is hard to beat shapefile, and the current difference between <br>
>>> shapefile and flatgeobuf is small)<br>
>>><br>
>>> Even<br>
>>><br>
>>> Le 26/04/2022 à 12:57, Jeff McKenna a écrit :<br>
>>>> (to followup from our chat on IRC yesterday)<br>
>>>><br>
>>>> To clarify: this would be a new native format in MapServer, such as <br>
>>>> Shapefile or PostGIS, and the goal would be to not connect to <br>
>>>> FlatGeobuf through OGR, but instead directly, such as:<br>
>>>><br>
>>>> DATA countries.fgb<br>
>>>><br>
>>>> instead of:<br>
>>>><br>
>>>> CONNECTIONTYPE OGR<br>
>>>> CONNECTION "countries.fgb"<br>
>>>> DATA "countries"<br>
>>>><br>
>>>> This should improve the performance even more. And it will be easy <br>
>>>> to update the documentation, as the existing files just need to be <br>
>>>> updated (see the new page at <br>
>>>> <a href="https://mapserver.org/input/vector/flatgeobuf.html" rel="noreferrer" target="_blank">https://mapserver.org/input/vector/flatgeobuf.html</a> )<br>
>>>><br>
>>>> Great plan Björn! I look forward to your Pull Request, ha, and <br>
>>>> will tackle the documentation changes for you.<br>
>>>><br>
>>>> This is also great news for MapServer, and will give us some nice <br>
>>>> publicity.<br>
>>>><br>
>>>> Thanks so much!<br>
>>>><br>
>>>> -jeff<br>
>>>><br>
>>>><br>
>>>><br>
>>>> On 2022-04-25 5:57 p.m., Björn Harrtell wrote:<br>
>>>>> Hi mapserver devs!<br>
>>>>><br>
>>>>> I got interested in the subject because of the tests made recently <br>
>>>>> by Jeff that shows the potential of the format. I believe it <br>
>>>>> should be possible to get significant additional performance out <br>
>>>>> of FlatGeobuf in MapServer if it was built in just like Shapefile <br>
>>>>> support is.<br>
>>>>><br>
>>>>> FlatGeobuf C++ implementation is rather small, doesn't require any <br>
>>>>> dependencies, and I don't expect it to require any invasive <br>
>>>>> changes to existing code.<br>
>>>>><br>
>>>>> Would love to get started on this ASAP, if there are no <br>
>>>>> objections. :)<br>
>>>>><br>
>>>>> Regards,<br>
>>>>> Björn Harrtell<br>
>>>>><br>
>> ______<br>
> _______________________________________________<br>
> MapServer-dev mailing list<br>
> <a href="mailto:MapServer-dev@lists.osgeo.org" target="_blank">MapServer-dev@lists.osgeo.org</a><br>
> <a href="https://lists.osgeo.org/mailman/listinfo/mapserver-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
<br>
-- <br>
<a href="http://www.spatialys.com" rel="noreferrer" target="_blank">http://www.spatialys.com</a><br>
My software is free, but my time generally not.<br>
<br>
_______________________________________________<br>
MapServer-dev mailing list<br>
<a href="mailto:MapServer-dev@lists.osgeo.org" target="_blank">MapServer-dev@lists.osgeo.org</a><br>
<a href="https://lists.osgeo.org/mailman/listinfo/mapserver-dev" rel="noreferrer" target="_blank">https://lists.osgeo.org/mailman/listinfo/mapserver-dev</a><br>
</blockquote></div>
</blockquote></div></div>