[gdal-dev] Esri REST PBF support

[Craig de Stigter]

Thanks Even

I've spent this morning looking at the source code - you're definitely
right that even though the PBF format is a similar *structure* to the JSON
format, the code won't be easily adapted to support both without becoming
extremely messy. So a separate driver is likely better.

The definitions in gpb.h look useful for this.

I will have to think harder at some point about how the two drivers will
interact - it would be best if the new driver could fallback to ESRIJSON if
the URL it has been given doesn't actually have PBF support. And of course
the service and layer definition endpoints don't support PBF at all, only
the query endpoints do - so the new driver will definitely need to re-use
some of the code from ESRIJSON to do all that.

I won't be building this this week, but I'll keep you informed when I get
to it.

Thanks
Craig


On Wed, 23 Jul 2025 at 23:04, Even Rouault <even.rouault at spatialys.com>
wrote:

> Craig,
>
> What would you be your idea: decode the PBF into JSON ? otherwise re-using
> the existing code of the ESRI JSON driver is going to be difficult as it
> assumes JSON everywhere.
>
> For PBF reading, we have ogr/ogrsf_frmts/gpb.h used both by the OSM PBF
> driver and the MVT driver. This is probably a bit tedious to use than code
> generated by the protobuf compiler, but at least this saves an extra
> dependency.
>
> I'm a bit skeptical about the ability of a LLM to generate code that will
> insert nicely into GDAL, but let's see...
>
> Even
> Le 23/07/2025 à 00:04, Craig de Stigter via gdal-dev a écrit :
>
> Hi folks
>
> OGR has support for ESRI FeatureService endpoints via the ESRI JSON
> <https://gdal.org/en/stable/drivers/vector/esrijson.html> driver.
> I am wondering if there would be interest in adding PBF support to that
> driver, so that we could ingest the more efficient PBF versions of those
> endpoints.
>
> As far as I can tell, this is *different* and unrelated to the OSM PBF
> format that GDAL already supports.
>
> Feature servers we have come across that support PBF seem to return query
> results much faster and more reliably in PBF rather than JSON.
>
> We have some Python code internally (not using GDAL) that handles the PBF,
> but it performs poorly due to being computation code in Python. I am
> investigating ways to speed it up using other languages. As part of this I
> may be in a position to convert it to C++ and contribute it to GDAL, if
> there is interest. I am no C++ developer but am becoming proficient with
> Claude Code :)
>
> I would lean towards implementing the PBF support as a by-default
> optimisation on the existing ESRIJSON driver, if the server advertises
> support for PBF - with a config option to force use of JSON if necessary. A
> separate driver would be an alternative but seems unnecessary since the PBF
> structure is quite similar to the JSON structure (the main difference is
> that geometries are encoded more efficiently)
>
> I haven't given much time towards investigating how to do this in GDAL,
> but would appreciate any tips or discussion.
>
> --
> Regards,
> Craig
>
> Platform Engineer
> Koordinates
> koordinates.com / @koordinates <https://twitter.com/koordinates>
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
Regards,
Craig

Platform Engineer
Koordinates
koordinates.com / @koordinates <https://twitter.com/koordinates>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20250724/7d7c02b0/attachment.htm>