[gdal-dev] proposed changes to netcdf export behaviour (perhaps RFC)

[Etienne Tourigny]

On Wed, Oct 19, 2011 at 3:34 AM, Frank Warmerdam <warmerdam at pobox.com> wrote:
> On 11-10-18 10:19 PM, Etienne Tourigny wrote:
>>
>> Hi Frank,
>>
>> On Tue, Oct 18, 2011 at 7:22 PM, Frank Warmerdam<warmerdam at pobox.com>
>>  wrote:
>>>
>>> On Tue, Oct 18, 2011 at 1:10 PM, Etienne Tourigny
>>> <etourigny.dev at gmail.com>  wrote:
>>>>
>>>> 2) Change to a default south-up orientation for NetCDF data
>>>
>>> Etienne,
>>>
>>> Could you expand on this a bit?  In particular I'm wondering if:
>>>
>>> 1) Is a south-up file represented as south up to GDAL applications?
>>> That is, will geotransform[5] be negative or positive?
>>
>>
>> Files created by the driver have a negative  pixel height
>> (geotransform[5]) when opened with GDAL , so I guess that means it
>> looks like it is north up.
>>
>> When the driver opens a south-up file, it has to invert the y axis
>> upon import to make it compatible with GDAL's data model.
>>
>> For example, a gtiff landsat that has the following geo info:
>> Origin = (625970.000000000000000,8981830.000000000000000)
>> Pixel Size = (30.000000000000000,-30.000000000000000)
>>
>> Which is the same when translated to netcdf and the netcdf is read by
>> GDAL.\
>
> Etienne,
>
> OK, it seems that to applications the change to producing
> bottom-up files will not be particularly disruptive.

yes, and as netcdf is more or less random access the performance hit
should not be significant.

>
>> For what it gives, here are some numbers for gdal_translate on a test
>> dataset (22239 x 33359 INT) . The netcdf creation takes longer than
>> gtiff, but the difference between south-up and north-up is small.
>>
>> gtiff->  gtiff
>> real    0m34.251s
>> user    0m1.550s
>> sys     0m3.710s
>>
>> gtiff->netcdf (BOTTOM_UP=YES)
>> real    1m3.869s
>> user    0m2.520s
>> sys     0m5.300s
>>
>> gtiff->netcdf (BOTTOM_UP=NO)
>> real    1m35.077s
>> user    0m3.050s
>> sys     0m6.150s
>>
>> gtiff->netcdf (driver in svn trunk, north-up)
>> real    2m4.186s
>> user    0m5.110s
>> sys     0m7.130s
>>
>
> I'm a bit confused about the above.  How are the "driver in svn trunk,
> north-up" case and the "BOTTOM_UP=YES" case different?  Why is
> "BOTTOM_UP=NO" case slower than "BOTTOM_UP_YES" case?
>

Frank, Sorry about the confusion

The driver in svn produces north-up, as does BOTTOM_UP=NO, so the
result is identical numerically. BOTTUM_UP=YES produces south-up

For the numbers it seems to be rather random, I can never get really
close values (maybe it has to do with caching).

The tests I made were with a slight optimisation which isn't in the
old driver.  Shows that we have been making progress on a few fronts!

I reverted that and here is what I got, for 2 separate runs:

Basically north-up and south-up are pretty much at the same order (5s
user, 6s sys) , and also the time is consistent with older driver (5s
user, 7s sys)

tourigny at supernova: /data/research/work/gdal-netcdf $ time
gdal_translate -of netcdf -co WRITE_BOTTOMUP=1 -co WRITE_LONLAT=0
mcd45-100.tif mcd45-100-1.nc
real	2m17.040s
user	0m4.160s
sys	0m5.860s

tourigny at supernova: /data/research/work/gdal-netcdf $ time
gdal_translate -of netcdf -co WRITE_BOTTOMUP=0 -co WRITE_LONLAT=0
mcd45-100.tif mcd45-100-2.nc
real	1m55.951s
user	0m4.890s
sys	0m6.900s

tourigny at supernova: /data/research/work/gdal-netcdf $ time
gdal_translate -of netcdf -co WRITE_BOTTOMUP=1 -co WRITE_LONLAT=0
mcd45-100.tif mcd45-100-1.nc
real	3m0.683s
user	0m6.320s
sys	0m9.700s

tourigny at supernova: /data/research/work/gdal-netcdf $ time
gdal_translate -of netcdf -co WRITE_BOTTOMUP=0 -co WRITE_LONLAT=0
mcd45-100.tif mcd45-100-2.nc
real	1m50.754s
user	0m4.870s
sys	0m7.480s


> In any event, at least for this case the differences aren't devastating
> and there are options to control what is done.  I am supportive of changes
> that produce a more interoperable  result.

I have been considering using a single Config Option to select which
workflow the user wants (strictly GDAL, more back-compat and more
efficient in GDAL vs. CF with less back-compact)

>
> As PSC chair I will not be offended if you proceed with netcdf changes
> *without* a formal RFC type vote on the changes.  The changes are fixing
> bugs and don't cause real backward compatability issues and I'm willing to
> treat you as the authority/owner for the driver.

OK Frank, many thanks for you support and trust

>
> Best regards,
> --
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/warmerda
> and watch the world go round - Rush    | Geospatial Software Developer
>
>