[PROJ] PROJ grid files CDN

[Howard Butler]

> On Sep 13, 2019, at 12:45 PM, Greg Troxel <gdt at lexort.com> wrote:
> 
>  I really don't understand the notion of incorrect results from missing  shift files as other than a bug. 

That's why I scare quoted "incorrect", and Even's reply covers it thoroughly. If we were to error in the face of missing grids, the only thing a user without system access can do is to fully replicate PROJ_LIB somewhere locally and then copy in their own grids. It is massively inconvenient and easy to screw up.

>  One issue with grids seems to be licensing.  I have not investigated  deeply, but it seems some have terms that make it difficult to
>  distribute, and some are non-Free (being an issue for Debian, perhaps,  an in pkgsrc requiring a non-Free license tag, sort of the same thing).  Are all of the grids you are talking about able to be  distributed (verbatim) via no-cost internet downloads?  

I propose we wouldn't distribute anything via CDN that wouldn't meet Debian's notion of "free", and I would think that a distribution approach like this, if especially convenient, would encourage some of the licensing laggards of grids to follow along. 

> As part of a paid-for CDROM that aggregates many things?

If you were to carefully inspect some of the paid-for CDROMs being distributed today, you're likely going to find these grid files regardless of the specific licensing language on some specific grids.

>  My impression is that grids are not included in proj packages because  they are large, in addition to the above.

Yes that's correct. They can be huge for the full set. That's why I'm proposing allowing a lazily-fetched CDN approach.

>  I don't follow the release cadence point at all.  Yes, proj has  releases, grid shifts have releases, and these all make their way into
>  packaging systems, which then have releases.  Generally people want to  run more recent versions, except for some people that choose to run  old software (which they call LTS).

You practically never want old grid data, unless you're trying to replicate something that happened in the past. This also brings up the problem of versioning of the grids, which is not handled currently.

>  Your point about packaging systems not having grid shift packages due  to size is fair; it's not clear how to deal with that.

Let the packagers continue to bulk copy the redistributable grids into packages and place them in PROJ_LIB. Fully unzipped, its ~650mb and growing. They can snap their packages and versions from the CDN.

> So a user  running proj would not be able to write to the system proj directory.  Do you envision the files going into some per-user directory in their  homedir?

Implementation detail, but yes, something like that.

>  Generally, packaging systems consider it a bug when programs in  packages do automatic downloading.  Partly this is because one should  be able to install a package to an offline computer.

That's why I propose it to be compile-time and runtime off-by-default. Specific packaging systems will have to determine if it is worth it to its users to do this. For front-end user-oriented applications where the users are not typically versed in the minutiae of geodesic transforms, I suspect the answer will be yes.

Howard