[gdal-dev] Making a small(er) test netcdf from a netcdf with subdatasets?

[Julien Demaria]

If you want to reduce again your file you should be able to keep 0..360 but change the step by decimating: -d lon,0,360,10 for example to keep only 37 longitudes.
http://research.jisao.washington.edu/data_sets/nco/#example10

Julien

________________________________
De : Kurt Schwehr [schwehr at gmail.com]
Date d'envoi : jeudi 19 novembre 2015 21:10
À : Julien Demaria
Cc : gdal dev
Objet : Re: RE : [gdal-dev] Making a small(er) test netcdf from a netcdf with subdatasets?

Julien,

Bummer that I broke you patches.  I don't expect to do much more in frmts for a while.  My big sprint is done and I will likely only do a couple hours each Friday.  My focus will likely be finishing ports/ and working on gcore and then ogr.

Here is what I currently have based on comments from gis stack exchange. In my case, I sadly can't use your lon/lat bounds shrinking as need to test my code that shifts the data from 0..360 to -180..180.

ncks -4 --deflate 9 -v water_u,water_v -d depth,1,2,1 hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc> out.nc<http://out.nc>

That takes 1.4GB down to 33MB.

-kurt

On Thu, Nov 19, 2015 at 11:56 AM, Julien Demaria <Julien.Demaria at acri-st.fr<mailto:Julien.Demaria at acri-st.fr>> wrote:
Dear Kurt,

Yes you can do that using ncks from nco by specifying the subset on each dimension:
ncks -d lon,340.0,10.0 -d lat,10.0,20.0 in.nc<http://in.nc> out.nc<http://out.nc>
another solution is to do ncdump -h into a file, editing the ascii to reduce dimensions and do ncgen to generate the NetCDF. By default it will fill with fillvalue, but you can also add data at the end in the ascii file.

Using gdal I think it is impossible to keep the exact input structure but I'm not sure 100%.

By the way, do you plan to do other improvements/big refactorings/cleanings in short term on the NetCDF driver? I still plan to finalize this
http://lists.osgeo.org/pipermail/gdal-dev/2015-February/040973.html
I have to add unit tests to finalize this, but I have seen that your recent commits completely break my patches, but this is my fault because I should have done before, but time is missing... :-(

Cheers,
Julien

________________________________
De : gdal-dev [gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>] de la part de Kurt Schwehr [schwehr at gmail.com<mailto:schwehr at gmail.com>]
Date d'envoi : jeudi 19 novembre 2015 20:22
À : gdal dev
Objet : [gdal-dev] Making a small(er) test netcdf from a netcdf with subdatasets?

Also asked here: http://gis.stackexchange.com/questions/170882/how-can-a-make-a-netcdf-with-subgroups-smaller

Is there an easy way to make a netcdf with subdatasets but shrink it without resorting to C or Fortan?

I'm trying to write a test for code that works with hycom netcdf global ocean circulation data. The source file 2 subdatasets:

gdalinfo hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc> | grep _NAME
  SUBDATASET_1_NAME=NETCDF:"hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc>":water_u
  SUBDATASET_2_NAME=NETCDF:"hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc>":water_v
With 40 depth bands:

gdalinfo NETCDF:hycom_glb_911_2015110200<tel:2015110200>_t000_uv3z.nc:water_u | grep '^Band' | head -3
Band 1 Block=4500x1 Type=Int16, ColorInterp=Undefined
Band 2 Block=4500x1 Type=Int16, ColorInterp=Undefined
Band 3 Block=4500x1 Type=Int16, ColorInterp=Undefined
I've got some code written the does things like calculate the speed in m/s from the UV and creates a geotiff, etc. The original files are much too large for using in unittests. Is there an easy way with gdal, nco or ? to keep the structure and metadata, but create a much smaller file? I'm open to dropping all but the first two bands and writing a constant value into each band + turning on deflate compression.

Trying with GDAL:

gdal_translate --version  # At head -> r31584
GDAL 2.1.0dev, released 2015/99/99

gdal_translate hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc> hycom_glb_911_2015110200_t000_uv3z-try2.nc<http://hycom_glb_911_2015110200_t000_uv3z-try2.nc> -of netcdf -co compress=deflate
Input file contains subdatasets. Please, select one of them for reading.
The originals are quite a bit large for tests:

ls -lh hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc>
-rw-r----- 1 schwehr group 1.4G Nov 19 09:13 hycom_glb_911_2015110200_t000_uv3z.nc<http://hycom_glb_911_2015110200_t000_uv3z.nc>

Thanks,
-kurt



--
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20151119/a5557030/attachment-0001.html>