[PROJ] PROJ grid files CDN

[Greg Troxel]

Howard Butler <howard at hobu.co> writes:

> A few days ago, I tweeted [1] looking for a CDN partner to help us
> incrementally distribute the shift files that PROJ uses. After that
> generated some connections from organizations willing to help, I'm now
> asking the community if there is support for such an idea.
>
> The current software approach PROJ uses to apply grid shifts works
> just fine, but it has one significant deficiency that can cause PROJ
> to generate "incorrect" results – the need for proper grid shift
> files. Incorrect results can happen due to missing shift data that
> users didn't know they needed to download and put in a magical
> directory. Some distributions do not ship full copies of the shift
> files due to their size, and new versions of the grid shift files are
> released at a different release cadence than the releases of
> distributions, the EPSG database, or PROJ itself.

That raises several issues:

  I really don't understand the notion of incorrect results from missing
  shift files as other than a bug.  I know we've discussed this before,
  but it seems that a transform that needs grid files should use grid
  files, and fail if they are not there.  I realize there is some notion
  of lower and higher accuracy transforms, but it seems dangerous to
  have different outcomes based on grids being installed (vs flags
  asking to not use them).  I acknowledge that this is a preference to
  repeatable results over convenience.

  One issue with grids seems to be licensing.  I have not investigated
  deeply, but it seems some have terms that make it difficult to
  distribute, and some are non-Free (being an issue for Debian, perhaps,
  an in pkgsrc requiring a non-Free license tag, sort of the same
  thing).  Are all of the grids you are talking about able to be
  distributed (verbatim) via no-cost internet downloads?  As part of a
  paid-for CDROM that aggregates many things?

  My impression is that grids are not included in proj packages because
  they are large, in addition to the above.

  In general, within the context of a packaging system, the right thing
  for additional files is to have packages for them, rather than
  inventing a sort of packaging system managed by some particular
  program.

  I don't follow the release cadence point at all.  Yes, proj has
  releases, grid shifts have releases, and these all make their way into
  packaging systems, which then have releases.  Generally people want to
  run more recent versions, except for some people that choose to run
  old software (which they call LTS).

  Your point about packaging systems not having grid shift packages due
  to size is fair; it's not clear how to deal with that.

> The current management of the grid shift files would be improved for
> many users by providing an optional online web service alternative for
> obtaining shift information. Some benefits of a web service approach
> include:
>
> * users no longer have to manually fetch grid files and place them in PROJ_LIB
> * full and accurate capability of the software would no longer require GBs of grid shift files
> * the web service can manage and provide proper versioning for the shift files
> * cache build up of the grid files could happen lazily so users end up locally mirroring what they actually use

Further issues:

  Generally, packaging systems install proj in a system directory, such
  that regular users cannot write the directories and files.  So a user
  running proj would not be able to write to the system proj directory.
  Do you envision the files going into some per-user directory in their
  homedir?

  Generally, packaging systems consider it a bug when programs in
  packages do automatic downloading.  Partly this is because one should
  be able to install a package to an offline computer.

  If what you're really talking about first is a CDN URL to download
  grids, mirroring the current download area, that sounds fine, but it
  seems somewhat separable from the autodownload notion.

> I recognize that many do not want PROJ reaching out to a web service,
> and I would propose that the machinery to do this would be optionally
> compiled and optionally activated via environment variable or some
> similar mechanism. However, a significant portion of
> don't-even-know-they're-using-PROJ users could benefit from PROJ
> having the optional ability to do its best in application of grid
> shifts.

Agreed that if it happens it should be opt in.

> Does the PROJ community support such an idea? How does management of
> grid shift data impact your ability to use PROJ, and what ideas do you
> have to help us improve it?  If the feedback on this proposal is
> generally positive, I will work with Even on writing an RFC of a
> proposed implementation.