[gdal-dev] Making a small(er) test netcdf from a netcdf with subdatasets?

[Kurt Schwehr]

Thanks!  That gets it down to 268K!

ncks -4 --deflate 9 -v water_u,water_v -d depth,1,2,1 -d lon,0,360,10
hycom_glb_911_2015110200_t000_uv3z.nc out4.nc

On Thu, Nov 19, 2015 at 12:22 PM, Julien Demaria <Julien.Demaria at acri-st.fr>
wrote:

> If you want to reduce again your file you should be able to keep 0..360
> but change the step by decimating: -d lon,0,360,10 for example to keep only
> 37 longitudes.
> http://research.jisao.washington.edu/data_sets/nco/#example10
>
> Julien
>
> ------------------------------
> *De :* Kurt Schwehr [schwehr at gmail.com]
> *Date d'envoi :* jeudi 19 novembre 2015 21:10
> *À :* Julien Demaria
> *Cc :* gdal dev
> *Objet :* Re: RE : [gdal-dev] Making a small(er) test netcdf from a
> netcdf with subdatasets?
>
> Julien,
>
> Bummer that I broke you patches.  I don't expect to do much more in frmts
> for a while.  My big sprint is done and I will likely only do a couple
> hours each Friday.  My focus will likely be finishing ports/ and working on
> gcore and then ogr.
>
> Here is what I currently have based on comments from gis stack exchange.
> In my case, I sadly can't use your lon/lat bounds shrinking as need to test
> my code that shifts the data from 0..360 to -180..180.
>
> ncks -4 --deflate 9 -v water_u,water_v -d depth,1,2,1
> hycom_glb_911_2015110200_t000_uv3z.nc out.nc
>
> That takes 1.4GB down to 33MB.
>
> -kurt
>
> On Thu, Nov 19, 2015 at 11:56 AM, Julien Demaria <
> Julien.Demaria at acri-st.fr> wrote:
>
>> Dear Kurt,
>>
>> Yes you can do that using ncks from nco by specifying the subset on each
>> dimension:
>> ncks -d lon,340.0,10.0 -d lat,10.0,20.0 in.nc out.nc
>> another solution is to do ncdump -h into a file, editing the ascii to
>> reduce dimensions and do ncgen to generate the NetCDF. By default it will
>> fill with fillvalue, but you can also add data at the end in the ascii file.
>>
>> Using gdal I think it is impossible to keep the exact input structure but
>> I'm not sure 100%.
>>
>> By the way, do you plan to do other improvements/big
>> refactorings/cleanings in short term on the NetCDF driver? I still plan to
>> finalize this
>> http://lists.osgeo.org/pipermail/gdal-dev/2015-February/040973.html
>> I have to add unit tests to finalize this, but I have seen that your
>> recent commits completely break my patches, but this is my fault because I
>> should have done before, but time is missing... :-(
>>
>> Cheers,
>> Julien
>>
>> ------------------------------
>> *De :* gdal-dev [gdal-dev-bounces at lists.osgeo.org] de la part de Kurt
>> Schwehr [schwehr at gmail.com]
>> *Date d'envoi :* jeudi 19 novembre 2015 20:22
>> *À :* gdal dev
>> *Objet :* [gdal-dev] Making a small(er) test netcdf from a netcdf with
>> subdatasets?
>>
>> Also asked here:
>> http://gis.stackexchange.com/questions/170882/how-can-a-make-a-netcdf-with-subgroups-smaller
>>
>> Is there an easy way to make a netcdf with subdatasets but shrink it
>> without resorting to C or Fortan?
>>
>> I'm trying to write a test for code that works with hycom netcdf global
>> ocean circulation data. The source file 2 subdatasets:
>>
>> gdalinfo hycom_glb_911_2015110200_t000_uv3z.nc | grep _NAME
>>   SUBDATASET_1_NAME=NETCDF:"hycom_glb_911_2015110200_t000_uv3z.nc
>> ":water_u
>>   SUBDATASET_2_NAME=NETCDF:"hycom_glb_911_2015110200_t000_uv3z.nc
>> ":water_v
>> With 40 depth bands:
>>
>> gdalinfo NETCDF:hycom_glb_911_2015110200_t000_uv3z.nc:water_u | grep
>> '^Band' | head -3
>> Band 1 Block=4500x1 Type=Int16, ColorInterp=Undefined
>> Band 2 Block=4500x1 Type=Int16, ColorInterp=Undefined
>> Band 3 Block=4500x1 Type=Int16, ColorInterp=Undefined
>> I've got some code written the does things like calculate the speed in
>> m/s from the UV and creates a geotiff, etc. The original files are much too
>> large for using in unittests. Is there an easy way with gdal, nco or ? to
>> keep the structure and metadata, but create a much smaller file? I'm open
>> to dropping all but the first two bands and writing a constant value into
>> each band + turning on deflate compression.
>>
>> Trying with GDAL:
>>
>> gdal_translate --version  # At head -> r31584
>> GDAL 2.1.0dev, released 2015/99/99
>>
>> gdal_translate hycom_glb_911_2015110200_t000_uv3z.nc
>> hycom_glb_911_2015110200_t000_uv3z-try2.nc -of netcdf -co
>> compress=deflate
>> Input file contains subdatasets. Please, select one of them for reading.
>> The originals are quite a bit large for tests:
>>
>> ls -lh hycom_glb_911_2015110200_t000_uv3z.nc
>> -rw-r----- 1 schwehr group 1.4G Nov 19 09:13
>> hycom_glb_911_2015110200_t000_uv3z.nc
>>
>> Thanks,
>> -kurt
>>
>
>
>
> --
> --
> http://schwehr.org
>



-- 
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20151119/5d81d728/attachment.html>