[PROJ] Management of datum grid files.

[Even Rouault]

Chris,

Interesting points !

>  doubt that many Australian users will be interested in the NZ grids, and 
vice versa.  (Particularly of course as we do not share a land boundary).

We had to try to find some classification, to have a reasonable number of 
packages of a reasonable size. This is not perfect. For example for French 
grids, they are stored in -europe even for the oversea territories in North 
America, Carribeans, Indian ocean...

> So for the
> most part users will need to download the zip file from the URL on the PROJ
> pages, and install manually.

Yes, the "advanced" PROJ API or the projinfo utility also gives a hint 
currently, like
"Grid GDA94_GDA2020_conformal.gsb needed but not found on the system. Can be 
obtained from the proj-datumgrid-oceania package at https://
download.osgeo.org/proj/proj-datumgrid-oceania-latest.zip"
 
> Initially I thought I wondered about create a directory in proj-datumgrid
> for NZ grids and a  new product proj-datumgrid-nz. 

I think our current release manager would probably object that having to 
handle more packages will be overhead for him (as we need to keep track of 
their versionning). We would certainly add a "africa", "south-america" and 
"asia" subpackages if we have datasets for them, but having to create separate 
packages for each agency would be a larger maintainance and distribution 
burden.

> But since LINZ (NZ
> geodetic agency) will be publishing these in any case, would it make more
> sense for LINZ to maintain and host the proj-datumgrid-nz product, and for
> that simply to be referenced in proj documentation?

Several points:
- I suppose that depends on the PROJ community trusts the data producer (I'm 
talking in general here. Nothing specific about LINZ !) to maintain its 
content, and at a stable location. One concern I have is with superseded 
grids. There are some agencies that tend to hide old grids when they release a 
new one. Which can be problematic for reproducibility, or even if you just use 
an old version of PROJ whose database hasn't been updated to point at the new 
grid. By having the grids "physically" in the git repo of proj-datumgrid we as 
as project can decide about their fate.
- Up to now data producers tended to deliver their datasets in "random" 
formats and we had to do a conversion to something PROJ could ingest.
- We need to be really sure about the filename of the grid. What EPSG mentions 
in its database and what data producers actually deliver tend to be different, 
so we have a database table to do that mapping. Even slight changes like 
foo.gsb becoming FOO.GSB could break PROJ good working.
- By having the grids stored in proj-datumgrid, we can also check their 
licensing situation before inclusion.

> The current initiative to deliver grids via a CDN
> (https://github.com/rouault/PROJ/blob/rfc4_remote_and_geotiff_grid/docs/sou
> rce/community/rfc/rfc-4.rst) will simplify this, but  the current proposal
> doesn't replace proj-datumgrid so I think this will be an issue for a bit
> longer.

Yes, in my mind, proj-datumgrid will remain the "master", and we will have a  
script that will resync its content onto the cloud storage we will have access 
to.

> To make this work well it would be useful to have a command line tool for
> installing grids that is installed with PROJ.  Say (sudo)
> install_proj_datumgrid <region>.   This could have a configuration file
> which defines the URL of the authoritative source for each region so that
> "install_proj_datumgrid nz" would download and install a file from (say)
> www.geodesy.linz.govt.nz/download/proj-datumgrid/proj-datumgrid-nz.zip<http
> ://www.geodesy.linz.govt.nz/download/proj-datumgrid/proj-datumgrid-nz.zip>. 
> This could work well with the existing products too, eg
> install_proj_datumgrid oceania.   (This could be a trivial shell script for
> linux - sure for other OS targets)

A python script then :-) It would probably also need the path to the data 
(PROJ_LIB) directory, as this can vary between binary distributions. Or we 
could also have a --user flag (like pip install --user) that would install 
them in the ${XDG_DATA_HOME}/proj directory mentionned in RFC4 to hold the 
network cache, and the default search path of PROJ would also look into that.

All this discussion shows that for use cases where people will be able and 
willing to rely on network access, just telling PROJ "go and fetch the grids 
wherever they are stored" should be a huge usability improvement.

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com