[MetaCRS] CS-Map RFC 1 - Creation of a GDC file to handle French grid files

Tue Oct 28 18:43:00 EDT 2008

Hello All . . .

>1) My understanding is that NTv2 files already provide a heirarchical
>set of grids.  Would it be sensible to convert the gr3df97a.txt
>and the metropolitan NTv2 grid shift files into a single NTv2 for all of
>France?

With an exception, I have always avoided altering the data files of governmental agencies which supply such.  I've always have taken the position that we use these files unaltered in any way.  The exception is that the three (or is it four) data file formats which are in text form and are quite laborious to parse, CS-MAP will build a binary rendition of the file and use it provided that the binary version is newer than the text version from which it is derived.  These exceptions are the Japanese, French, and OSGB data files.

>2) As a followon, does CS-Map have mechanism for creating NTv2 files? PROJ.4
>can read them but I haven't tried producing them.

NO.  Being consistent with the above, CS-MAP does not produce data files (or modify them) in any way.

>3) Could you provide an example of the proposed GDC file, possible with
>references to a .GDC specification or notes?

Datum conversions that rely on the existence of one or more grid data files use the concept of a Geodetic Data Catalog to provide the name and location of the actual grid data files which are to be used.  An ordinary text file with the .gdc extension is used for this purpose.  These files are also referred to as .gdc files.

Since these files are simple text files, they can be created and/or modified using any text editor, such as Notepad.  You can use a word processor if you are careful to save the file as an ordinary text file.  Each line of text in the file provides the Coordinate Conversion System with the name of the file, the location of the file, and some information as to how the file is to be used.  Up to four data elements can be specified.  Only the first element, the file name and location is a required element.

In most cases, the Geodetic Data Catalogs provided by your software's distribution are fine for general use.  However, there are definite cases where you will want to tweak these files for your own unique situation.  These situations are the several cases where there are several data files involved, and the coverage of these files overlap.

For example, in the case of NAD27 to NAD83 (the letters NAD refer to North American Datum) conversion, there are several files.  The U. S. files by themselves do not present a problem as they do not overlap.  However, when we add the Canadian National Transformation file to the catalog, we have a lot of geography where the data files overlap.  If the use of either data file in the case of overlap produced the same results, we would not have a problem.  However, this is rarely the case.  Therefore, users often wish to specify which file gets used in the case of overlap.  This is one of the functions of the Geodetic Data Catalog.
For each point which is to be converted, the Coordinate Conversion System will look at all the files listed in the appropriate Geodetic Data Catalog.  In selecting a file to use, the Coordinate Conversion System will first determine which of the files listed actually cover the geographic location of the point being converted.  Of the files providing coverage, the Coordinate Conversion System will then select the files with the densest grid; that is the grid shift data file in which the grid cells are the smallest.  (It is presumed that the smaller the grid cell, the more accurate the data in the file.)  In the case where two or more files meet these initial criteria, the Coordinate Conversion System will choose the first file listed in the Geodetic Data Catalog.
While the Coordinate Conversion System can determine the grid density of most all datum shift grid files, users can use the Geodetic Data Catalog to override the actual grid density value to control the datum shift grid file selection process.  Further, users can alter the order in which the datum shift grid files are listed to further control the datum shift grid file selection process.  The following sections describe how one can use the interactive Geodetic Data Catalog Editor, or simply use a text editor on the .gdc file, to accomplish these tasks.

Geodetic Data Catalogs can also be used to improve the performance of the Coordinate Conversion System.  For example, the HARN data catalog file must contain as many as 22 file references.  If your data conversions are limited to a single state, the unused files may be removed and performance of HARN conversions improved somewhat.

>4) Have you considered having a syntax in the .GDC file that indicates
>'use this grid shift file if available, otherwise pass on to the next
>option'?

No.  In the current implementation, if a file is referenced in the .gdc file, it must exist or you get a hard error.  What is implemented is the concept of a fallback.  If the point being converted is outside the coverage of any and all files listed in the .gdc file, the fallback is used to perform the conversion.  Fal;backs are generally restricted to datum transformations of the type which are limited by a specific coverage.  When a fallback is used, a +1 (rather than 0 for normal and -1 for fatal) status is returned.

Norm

-----Original Message-----
From: metacrs-bounces at lists.osgeo.org [mailto:metacrs-bounces at lists.osgeo.org] On Behalf Of Hugues Wisniewski
Sent: Tuesday, October 28, 2008 3:44 PM
To: Frank Warmerdam (External)
Cc: metacrs at lists.osgeo.org
Subject: RE: [MetaCRS] CS-Map RFC 1 - Creation of a GDC file to handle French grid files

Hi Frank,

>As I understand it the .gdc "catalog" mechanism is already used in CS-Map.
>It seems the only thing being done is to create a catalog file for the NTF
>datum.  Is that right?
Yes, the mechanism is already in place for other places in the world.
Since there was only one grid file for France in the past, there was no need for a gdc file at that time and that single grid file was harcoded inside CS-Map.

>1) My understanding is that NTv2 files already provide a heirarchical
>set of grids.  Would it be sensible to convert the gr3df97a.txt
>and the metropolitan NTv2 grid shift files into a single NTv2 for all of
>France?
I suppose Norm could answer this in more detail but I see a few problems doing this:
     - we do not have the right to redistribute the NTV2 grid files whereas we have the right to redistribute gr3df97a.txt
     - grid files come from official organisms like the IGN in the current scenario, and we do not change their content. When a new one comes, it can be
       added directly to the grid folder, or it can replace an old version of the same grid file
     - The NTV2 grid files cover specific area, the metropolitan areas, and each one of them overlaps the area covered by gr3df97a.txt by 100%
       With the use of GDC file, the end user can change the order in which the grid files will be used in case of an overlap, comment out a grid file if
       it doesn't have to be used, or add new grid files in the future if they become available
The main advantage of the gdc files is that they are text files editable by the end user to change the behavior of the conversion technique.

>2) As a followon, does CS-Map have mechanism for creating NTv2 files? PROJ.4
>can read them but I haven't tried producing them.
I don't think so, but Norm can confirm that

>3) Could you provide an example of the proposed GDC file, possible with
>references to a .GDC specification or notes?
http://svn.osgeo.org/metacrs/csmap/trunk/CsMapDev/Dictionaries/Nad27ToNad83.gdc
You will see that some grid files are commented out and the order of the Nadcon files can be changed by the end user when the data is in an area that is overlapped by multiple grid files. The first grid file in the list is the one that will be chosen if there is an overlap.

>4) Have you considered having a syntax in the .GDC file that indicates
>'use this grid shift file if available, otherwise pass on to the next
>option'?
That's quite an interesting feature but I don't think that's possible right now since we have to comment out the grid files that we cannot redistribute.
This triggers some testing problems when we forget to uncomment those grid files used in lab to test the library on a test machine that has the grid files available. That would likely be a nice add on to the gdc file management inside the CS-Map code as we would always use the same files for testing and distribution without constantly editing them.

Hugues

-----Original Message-----
From: Frank Warmerdam (External)
Sent: Tuesday, October 28, 2008 12:59 PM
To: Hugues Wisniewski
Cc: metacrs at lists.osgeo.org
Subject: Re: [MetaCRS] CS-Map RFC 1 - Creation of a GDC file to handle French grid files

Hugues Wisniewski wrote:
> Hi,
>
>
>
> I would like to submit "CS-Map RFC 1 - Creation of a GDC file to handle
> French grid files" for review and comment:
>
> http://trac.osgeo.org/csmap/wiki/CsMapRfc1

Hugues,

I have reviewed this RFC and I see no obvious problem with it.

As I understand it the .gdc "catalog" mechanism is already used in CS-Map.
It seems the only thing being done is to create a catalog file for the NTF
datum.  Is that right?

In this regard I might suggest an official RFC may not be required.

However, since the topic is raised I do have a few questions, mostly oriented
towards increasing integration of grid shift datasets between different
packages.

1) My understanding is that NTv2 files already provide a heirarchical
set of grids.  Would it be sensible to convert the gr3df97a.txt
and the metropolitan NTv2 grid shift files into a single NTv2 for all of
France?

2) As a followon, does CS-Map have mechanism for creating NTv2 files? PROJ.4
can read them but I haven't tried producing them.

3) Could you provide an example of the proposed GDC file, possible with
references to a .GDC specification or notes?

4) Have you considered having a syntax in the .GDC file that indicates
'use this grid shift file if available, otherwise pass on to the next
option'?  In PROJ.4 we use a syntax like:

   +nadgrids=@ntv2_0.gsb,ntv1_can.dat

to indicate use ntv2_0.gsb if it is available, but if not try the next
option.  But since the ntv_1_can.dat is not prefixed with '@' it is
not considered to be optional and an error will occur if it is not
found.  I'm not suggesting the same syntax be used, but perhaps that
this approach would be valuable in CSMap .GDC files so that users don't
need to edit a text file when they add the local grid shift files.

---

So what I'm really hoping to get to is a situation where we have more
sophisticated grid shift file management tools (creation, etc), and
where the same grid shift files can be used with CS-Map, PROJ.4,
and perhaps other packages like GeoTools.  If it is expressive, and
not too complicated perhaps I could add .GDC format support to
PROJ.4 and be able to more easily take advantage of CS-Map datum
shift stuff.

Best regards,
--
---------------------------------------+--------------------------------------
I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush    | Geospatial Programmer for Rent

_______________________________________________
MetaCRS mailing list
MetaCRS at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/metacrs