[gdal-dev] proposed changes to netcdf export behaviour (perhaps RFC)

[Etienne Tourigny]

Hi Frank,

On Tue, Oct 18, 2011 at 7:22 PM, Frank Warmerdam <warmerdam at pobox.com> wrote:
> On Tue, Oct 18, 2011 at 1:10 PM, Etienne Tourigny
> <etourigny.dev at gmail.com> wrote:
>> 2) Change to a default south-up orientation for NetCDF data
>
> Etienne,
>
> Could you expand on this a bit?  In particular I'm wondering if:
>
> 1) Is a south-up file represented as south up to GDAL applications?
> That is, will geotransform[5] be negative or positive?


Files created by the driver have a negative  pixel height
(geotransform[5]) when opened with GDAL , so I guess that means it
looks like it is north up.

When the driver opens a south-up file, it has to invert the y axis
upon import to make it compatible with GDAL's data model.

For example, a gtiff landsat that has the following geo info:
Origin = (625970.000000000000000,8981830.000000000000000)
Pixel Size = (30.000000000000000,-30.000000000000000)

Which is the same when translated to netcdf and the netcdf is read by GDAL.

>
> 2) Depending on the answer to the above, are there any performance
> implications when gdal_translate'ing from and to north north up datasets
> in other formats?
>

The difference is that it has to loop over the Y Axis backwards when
reading or writing, so if a file is not organised by scanlines there
will be thrashing (as far as I understand, I'm not an I/O expert).

Our new implementation has a BOTTOM_UP option (default: yes), so the
user has an option to write it in GDAL's native order, if performance
is an issue.

For what it gives, here are some numbers for gdal_translate on a test
dataset (22239 x 33359 INT) . The netcdf creation takes longer than
gtiff, but the difference between south-up and north-up is small.

gtiff-> gtiff
real    0m34.251s
user    0m1.550s
sys     0m3.710s

gtiff->netcdf (BOTTOM_UP=YES)
real    1m3.869s
user    0m2.520s
sys     0m5.300s

gtiff->netcdf (BOTTOM_UP=NO)
real    1m35.077s
user    0m3.050s
sys     0m6.150s

gtiff->netcdf (driver in svn trunk, north-up)
real    2m4.186s
user    0m5.110s
sys     0m7.130s


>From our understanding of NetCDF 3's data organisation, there's not a
big penalty if you don't start reading at the start of the file, it's
supposed to be a fairly random-access format. I don't know about other
formats supported by GDAL, though.

As part of further changes (NETCDF-4 format and DEFLATE compression),
we will look more into performance issues, hopefully processing time
will get smaller.


> The changes to the driver seem major enough and with backward
> compatibility issues such that I think an RFC is reasonable.  Calling
> for a vote on the existing document will likely be sufficient after
> any discussion here.
>

I'd like to add that the backward-compat issues are rather rare, in
most cases the old driver (since gdal 1.6) will read these files fine
(as CF-1 projection and geotransform info is supported by the old
driver). The -co options BOTTOM_UP and WRITE_GDAL_TAGS (default yes)
can make files that will be similar to what the current driver does.
Exceptions occur when there is no projection or geotransform, and
perhaps because of new (correct) export and import of certain
projections.

I agree that the changes are significant (new -co options, default
south-up export and change in projection parameters), so it's up to you guys.

Many thanks, Etienne

> Best regards,
> --
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam, warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | Geospatial Software Developer
>