[PROJ] [gdal-dev] RFC 102: Embedding resource files into libgdal (and PROJ)

[Kristian Evers]

Thanks for the reminder, Even. I haven't had time to properly look at the
RFC before now and I might have missed some of the discussions so please
excuse me if I'm repeating something that's already been discussed. I'm
only concerned about the PROJ specifics of this RFC, so I've cut off the
GDAL list from this thread to not generate too much noise.

First of all, I think a PROJ-specific RFC is warranted. The contents should
be largely the same, obviously without the very GDAL-oriented details such
as affected files etc. I’d like the RFC to touch on the relation to [RFC-3](
https://proj.org/en/9.5/community/rfc/rfc-3.html) as we are now talking
about using cutting edge features of C23 and the purpose of RFC3 is to
avoid that. I don’t think there’s a conflict as it seems you’ve managed to
write the code in a backwards compatible manner. It does mean there’s a
bunch of if-compiler-can-embed-files guards which would be great to get rid
of once this functionality is mainstream. A rough plan for when that can be
done would be great (as well as an accompanying GitHub issue so we remember
to do it).

As for the actual embedded files. Is it only contained to proj.db and
proj.ini in the case of PROJ? Or are there other files that potentially
would be embedded? I suppose some of the smaller files like the ITRFxxxx
files potentially could be useful. The RFC doesn’t mention contents from
PROJ-data - I’m sure someone eventually will want to embed those as well,
so maybe put in a sentence or two about that (and why it’s probably a bad
idea).

Lastly, I think it would be good to touch on *when* you might want to embed
resource files. I know this has been requested in the past but where it is
actually useful I’m not sure. I’m probably not alone in that, so if the
docs could include such an explanation it would be great. Could also be
added to the RFC to make it a little bit easier to digest the thing before
there’s a vote to adopt it.

/Kristian

On Wed, 9 Oct 2024 at 18:12, Even Rouault via PROJ <proj at lists.osgeo.org>
wrote:

> Hi,
>
> Proposed implementations are, as far as I can tell, feature complete now
> (including docs & CI):
>
> - GDAL: https://github.com/OSGeo/gdal/pull/10972 . I propose that we
> target 3.11. We are probably too close to the feature freeze of 3.10 and
> would otherwise need to rush things, which feels unnecessary.
>
> - PROJ: https://github.com/OSGeo/PROJ/pull/4265 . Embedding has also
> been extended to proj.ini , in addition to proj.db (both available in
> non-C23 or C23 modes)
>
> I've also updated the RFC text to reflect the current state of things.
>
> I let more time for people to review before submitting to vote, probably
> end of next week (as usual, tell if you need more time)
>
> Even
>
> Le 03/10/2024 à 19:30, Even Rouault via PROJ a écrit :
> > Hi,
> >
> > I've now a PROJ branch with proj.db embedding working at
> > https://github.com/OSGeo/PROJ/pull/4265. And contrary to the current
> > wording of the RFC text, I've actually invested in making it work with
> > non-C23 compilers too (C23 mechanism used when available), by adapting
> >
> https://gitlab.com/jhamberg/cmake-examples/-/blob/master/cmake/FileEmbed.cmake
> > and optimizing it because its performance at generating a .c file from
> > a binary was not acceptable on multi-megabyte large files like
> > proj.db.  Non-C23 embedding works fine with older gcc, clang, MSVC.
> > The proj_db.c generation from proj.db runs in 8 seconds, and building
> > it with gcc in 18 seconds (looking at build times in CI, that seems to
> > be fast with MSVC too)
> >
> > But I won't (probably) support non-C23 embedding for GDAL: that would
> > be too much complication w.r.t benefit. Only a "few" GDAL drivers need
> > resource files, whereas access to proj.db is a strong requirement.
> >
> > Kurt, I had to modify the memOpen() and the memvfs_init() functions of
> > the https://www.sqlite.org/src/doc/tip/ext/misc/memvfs.c (I did other
> > changes in memvfs_init() to avoid making it the default VFS, and
> > registering it per db-connection to avoid a global registration),
> > otherwise some complex SQL requests that require creating a temporary
> > file wouldn't work, such as the ones of 'bin/projinfo --list-crs' or
> > 'bin/cct "this is a bogus CRS meant to trigger a syntax error in
> > proj_create"'  (with sqlite 3.46 of fedora:rawhide). Is it something
> > you ran into? Cf the changes in memvfs.c in
> >
> https://github.com/rouault/PROJ/commit/e7fcf162f6968c48587836f537b506708112e7d4
> >
> > With those changes, ctest pass
> >
> > Even
> >
> >
> > http://www.spatialys.com
> > My software is free, but my time generally not.
> >
> > _______________________________________________
> > PROJ mailing list
> > PROJ at lists.osgeo.org
> > https://lists.osgeo.org/mailman/listinfo/proj
>
> --
> http://www.spatialys.com
> My software is free, but my time generally not.
>
> _______________________________________________
> PROJ mailing list
> PROJ at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/proj
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/proj/attachments/20241010/9b0628fe/attachment.htm>