[PROJ] Management of datum grid files.

Tue Nov 26 09:09:31 PST 2019

Comments inline below

Cheers
Chris

> -----Original Message-----
> From: Even Rouault [mailto:even.rouault at spatialys.com]
> Sent: Tuesday, 26 November 2019 12:21 p.m.
> To: proj at lists.osgeo.org
> Cc: Chris Crook
> Subject: Re: [PROJ] Management of datum grid files.
>
> Chris,
>
> Interesting points !
>
> >  doubt that many Australian users will be interested in the NZ grids,
> > and
> vice versa.  (Particularly of course as we do not share a land boundary).
>
> We had to try to find some classification, to have a reasonable number of
> packages of a reasonable size. This is not perfect. For example for French
> grids, they are stored in -europe even for the oversea territories in North
> America, Carribeans, Indian ocean...
>
> > So for the
> > most part users will need to download the zip file from the URL on the
> > PROJ pages, and install manually.
>
> Yes, the "advanced" PROJ API or the projinfo utility also gives a hint currently,
> like "Grid GDA94_GDA2020_conformal.gsb needed but not found on the
> system. Can be obtained from the proj-datumgrid-oceania package at
> https:// download.osgeo.org/proj/proj-datumgrid-oceania-latest.zip"
>
> > Initially I thought I wondered about create a directory in
> > proj-datumgrid for NZ grids and a  new product proj-datumgrid-nz.
>
> I think our current release manager would probably object that having to
> handle more packages will be overhead for him (as we need to keep track of
> their versionning). We would certainly add a "africa", "south-america" and
> "asia" subpackages if we have datasets for them, but having to create
> separate packages for each agency would be a larger maintainance and
> distribution burden.

Yes .. that was why I was thinking it made more sense for LINZ to manage it!

Versioning is another question.  I was pondering this when I was writing the
original email, but thought it was already long enough.  Maybe the installer downloading
an agency zip file could require a VERSION file or something similar so that the version
can be reported to users.   Certainly as a datum maintainer we would want to
provide version information to be confident of what users were testing.

>
> > But since LINZ (NZ
> > geodetic agency) will be publishing these in any case, would it make
> > more sense for LINZ to maintain and host the proj-datumgrid-nz
> > product, and for that simply to be referenced in proj documentation?
>
> Several points:
> - I suppose that depends on the PROJ community trusts the data producer
> (I'm talking in general here. Nothing specific about LINZ !) to maintain its
> content, and at a stable location. One concern I have is with superseded
> grids. There are some agencies that tend to hide old grids when they release
> a new one. Which can be problematic for reproducibility, or even if you just
> use an old version of PROJ whose database hasn't been updated to point at
> the new grid. By having the grids "physically" in the git repo of proj-
> datumgrid we as as project can decide about their fate.

I think ultimately the users have to trust the authority managing the local datum definition,
LINZ in New Zealand.   However maybe the information installed into proj should not just
be the location of the grid files, but also the contact details for the responsible authority.
That way the install_proj_datumgrid utility could provide the users with relevant information
if the source files are unavailable or do not provide the expected transformation grids.

Users, or agencies, or the proj project itself could certainly monitor the that the requisite
grids are available and readable (eg in the correct format), and notify the appropriate agencies
if not.

> - Up to now data producers tended to deliver their datasets in "random"
> formats and we had to do a conversion to something PROJ could ingest.
> - We need to be really sure about the filename of the grid. What EPSG
> mentions in its database and what data producers actually deliver tend to be
> different, so we have a database table to do that mapping. Even slight
> changes like foo.gsb becoming FOO.GSB could break PROJ good working.

Totally agree - and certainly we haven't sorted out the LINZ grids and names yet.
Our preference is to fix this at source (ie EPSG) rather than adding a mapping
in proj.  Otherwise we end up with two places to maintain.  This is something we
will be addressing very soon!

Also I don't think that the two approaches are mutually exclusive.  An agency such
as LINZ could either choose to install files directly into proj-datumgrid or just to install
the url/contact details for use by an installer.  Or if the agency is not able or willing to
maintain the grids then maybe local users would be able issue pull requests to proj-datumgrid.

Also global data sets (eg EGM geoid models) may be best managed within proj-datumgrid.

Where possible though it makes sense to me that the proj is responsible for the infrastructure
and frameworks, and agencies can provide the data that it uses.

Of course LINZ can in any case publish a proj-datumgrid-nz.zip file for use by the NZ
community and others in any case.  It just feels like it would make sense to have a framework
that directly supports that!

> - By having the grids stored in proj-datumgrid, we can also check their
> licensing situation before inclusion.
>
> > The current initiative to deliver grids via a CDN
> >
> (https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_grid/doc
> > s/sou
> > rce/community/rfc/rfc-4.rst) will simplify this, but  the current
> > proposal doesn't replace proj-datumgrid so I think this will be an
> > issue for a bit longer.
>
> Yes, in my mind, proj-datumgrid will remain the "master", and we will have a
> script that will resync its content onto the cloud storage we will have access
> to.

So possibly the approach for agency hosted data would be for the resync script
to download from the authoritative site, verify the contents, and post to cloud
storage if it passes file name and format tests, maybe version and licence information
 etc.

> > To make this work well it would be useful to have a command line tool
> > for installing grids that is installed with PROJ.  Say (sudo)
> > install_proj_datumgrid <region>.   This could have a configuration file
> > which defines the URL of the authoritative source for each region so
> > that "install_proj_datumgrid nz" would download and install a file
> > from (say)
> > www.geodesy.linz.govt.nz/download/proj-datumgrid/proj-datumgrid-
> nz.zip
> > <http
> > ://www.geodesy.linz.govt.nz/download/proj-datumgrid/proj-datumgrid-
> nz.zip>.
> > This could work well with the existing products too, eg
> > install_proj_datumgrid oceania.   (This could be a trivial shell script for
> > linux - sure for other OS targets)
>
> A python script then :-) It would probably also need the path to the data
> (PROJ_LIB) directory, as this can vary between binary distributions. Or we
> could also have a --user flag (like pip install --user) that would install them in
> the ${XDG_DATA_HOME}/proj directory mentionned in RFC4 to hold the
> network cache, and the default search path PROJ would also look into
> that.

I like python :-)  I'm not sure if it gives sufficient platform independence?
Do any of the proj binaries have a function to report the default value for PROJ_LIB
that a python utility could use if it is not explicitly set?

> All this discussion shows that for use cases where people will be able and
> willing to rely on network access, just telling PROJ "go and fetch the grids
> wherever they are stored" should be a huge usability improvement.
>
> Even
>
> --
> Spatialys - Geospatial professional services http://www.spatialys.com

________________________________

This message contains information, which may be in confidence and may be subject to legal privilege. If you are not the intended recipient, you must not peruse, use, disseminate, distribute or copy this message. If you have received this message in error, please notify us immediately (Phone 0800 665 463 or info at linz.govt.nz) and destroy the original message. LINZ accepts no responsibility for changes to this email, or for any attachments, after its transmission from LINZ. Thank You.