[gdal-dev] Making a small(er) test netcdf from a netcdf with subdatasets?

[Kurt Schwehr]

Julien,

Bummer that I broke you patches.  I don't expect to do much more in frmts
for a while.  My big sprint is done and I will likely only do a couple
hours each Friday.  My focus will likely be finishing ports/ and working on
gcore and then ogr.

Here is what I currently have based on comments from gis stack exchange. In
my case, I sadly can't use your lon/lat bounds shrinking as need to test my
code that shifts the data from 0..360 to -180..180.

ncks -4 --deflate 9 -v water_u,water_v -d depth,1,2,1
hycom_glb_911_2015110200_t000_uv3z.nc out.nc

That takes 1.4GB down to 33MB.

-kurt

On Thu, Nov 19, 2015 at 11:56 AM, Julien Demaria <Julien.Demaria at acri-st.fr>
wrote:

> Dear Kurt,
>
> Yes you can do that using ncks from nco by specifying the subset on each
> dimension:
> ncks -d lon,340.0,10.0 -d lat,10.0,20.0 in.nc out.nc
> another solution is to do ncdump -h into a file, editing the ascii to
> reduce dimensions and do ncgen to generate the NetCDF. By default it will
> fill with fillvalue, but you can also add data at the end in the ascii file.
>
> Using gdal I think it is impossible to keep the exact input structure but
> I'm not sure 100%.
>
> By the way, do you plan to do other improvements/big
> refactorings/cleanings in short term on the NetCDF driver? I still plan to
> finalize this
> http://lists.osgeo.org/pipermail/gdal-dev/2015-February/040973.html
> I have to add unit tests to finalize this, but I have seen that your
> recent commits completely break my patches, but this is my fault because I
> should have done before, but time is missing... :-(
>
> Cheers,
> Julien
>
> ------------------------------
> *De :* gdal-dev [gdal-dev-bounces at lists.osgeo.org] de la part de Kurt
> Schwehr [schwehr at gmail.com]
> *Date d'envoi :* jeudi 19 novembre 2015 20:22
> *À :* gdal dev
> *Objet :* [gdal-dev] Making a small(er) test netcdf from a netcdf with
> subdatasets?
>
> Also asked here:
> http://gis.stackexchange.com/questions/170882/how-can-a-make-a-netcdf-with-subgroups-smaller
>
> Is there an easy way to make a netcdf with subdatasets but shrink it
> without resorting to C or Fortan?
>
> I'm trying to write a test for code that works with hycom netcdf global
> ocean circulation data. The source file 2 subdatasets:
>
> gdalinfo hycom_glb_911_2015110200_t000_uv3z.nc | grep _NAME
>   SUBDATASET_1_NAME=NETCDF:"hycom_glb_911_2015110200_t000_uv3z.nc":water_u
>   SUBDATASET_2_NAME=NETCDF:"hycom_glb_911_2015110200_t000_uv3z.nc":water_v
> With 40 depth bands:
>
> gdalinfo NETCDF:hycom_glb_911_2015110200_t000_uv3z.nc:water_u | grep
> '^Band' | head -3
> Band 1 Block=4500x1 Type=Int16, ColorInterp=Undefined
> Band 2 Block=4500x1 Type=Int16, ColorInterp=Undefined
> Band 3 Block=4500x1 Type=Int16, ColorInterp=Undefined
> I've got some code written the does things like calculate the speed in m/s
> from the UV and creates a geotiff, etc. The original files are much too
> large for using in unittests. Is there an easy way with gdal, nco or ? to
> keep the structure and metadata, but create a much smaller file? I'm open
> to dropping all but the first two bands and writing a constant value into
> each band + turning on deflate compression.
>
> Trying with GDAL:
>
> gdal_translate --version  # At head -> r31584
> GDAL 2.1.0dev, released 2015/99/99
>
> gdal_translate hycom_glb_911_2015110200_t000_uv3z.nc
> hycom_glb_911_2015110200_t000_uv3z-try2.nc -of netcdf -co compress=deflate
> Input file contains subdatasets. Please, select one of them for reading.
> The originals are quite a bit large for tests:
>
> ls -lh hycom_glb_911_2015110200_t000_uv3z.nc
> -rw-r----- 1 schwehr group 1.4G Nov 19 09:13
> hycom_glb_911_2015110200_t000_uv3z.nc
>
> Thanks,
> -kurt
>



-- 
--
http://schwehr.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20151119/77ede6d8/attachment.html>