[gdal-dev] How to represent multi-dimensional array

[Christopher Barker]

Thanks to Michael, Joaquim, Ivan, and Jason.

I'll explore some of the tools and suggestions you made.

Michael Sumner wrote:
> NetCDF will tend to store dimensions in
> reverse order to the natural one, and I think GDAL reverses that - but
> you can tell by the dimension and number of your bands, and the named
> metadata on the GDAL bands.

yup -- I figured that out - it got better when we re-ordered to (t, z, 
y, x).

> NetCDF cannot store multi-attribute arrays (it will store several
> same-size, same-metadata arrays for that purpose),

Actually, I think it can -- though maybe that's only for netcdf4 (based 
on hdf5), but the conventions don't suggest you do that -- particularly 
if different attributes are different data types.

> Manifold reads in multiple rasters

> Eonfusion will do its best to read the array in its natural state

> R can read NetCDF natively or with GDAL (RNetCDF, ncdf, rgdal

We're not really looking for other tools at this point -- we are doing 
visualization with IDV, which handles 4-d data in netcdf just fine. This 
part is all about getting this data into Arc.

I may deal with it by figuring out what we really need to do in Arc, and 
just exporting that part of the data -- we're writing these files with 
Python anyway, so doing some pre-processing there won't be a big deal.

Joaquim Luis wrote:
> Don't know if this is what you are looking for but if those netCDF files 
> are of a similar type that one can get from the poet site 
> (http://poet.jpl.nasa.gov/), Mirone has a tool called "Aquamoto" (a tool 
> original developed to show time stamps of a tsunami propagation models) 
> that loads those files and show their content interactively with the 
> help of a slider.

Not much help for this, but it's a cool tool, thanks for the link.

Jason Roberts wrote:
> I have some experience trying to get ArcGIS to work well with time series
> satellite imagery and 4D ocean models (e.g. HYCOM, ROMS).

Exactly our situation, here.

> I am part of a working group initiated
> by ESRI and led by an ESRI program manager (Nawajish Noman) that is trying,
> in essence, to get the community of users who use both ArcGIS and netCDF to
> develop some Python geoprocessing tools for ArcGIS that provide more
> functionality than out-of-box tools already in ArcGIS.

cool -- do you have contact information for that project? Is there code 
anywhere we can get at it?

> 1. The Make NetCDF Raster Layer tool can represent 3D netCDF variables as
> multiband raster layers.

We'll give this a try. It may be what our GIS person is using already, 
but I got a bit confused by the "Dimension Values parameter". We'll poke 
at it some more.

> The trick is your netCDF has to meet a bunch
> of constraints for ArcGIS to recognize it. It has to have square cells.

bingo -- that was one of our key problems -- wait -- they have to be 
"square" -- rectangular won't do? arrgg!

Anyway, the way we have it now the cells ar rectangular in meters, but 
we un-projectee them to lat-long, so they are no longer simle 
rectangular -- I think I may change this an output in meters, with the 
projection info. But if it has to be square, we're kind of dea in the 
water...

> It
> has adhere to the CF or COARDS conventions (I forget which versions)

that we do have.

> 2. Under contract to NOAA, Applied Science Associates built a couple of
> tools that might be useful: the Environmental Data Connector (EDC) and
> TimeSlider Extension. Download from
> http://www.asascience.com/software/downloads/index.shtml, see other parts of
> the website for more info. EDC was built to download multidimensional
> OPeNDAP datasets into multiband rasters.

Hmm -- did know about those tools, but didn't realize that EDC was 
OPenDAP based -- nice to know.

> TimeSlider is a UI extension to
> help with playback of time-series data.

That too -- by the way, I'm pretty sure the Coast Guard funded a bunhc 
of that, for their Search and Rescue tools.

> I think they can both work with
> netCDFs directly, not just OPeNDAP.

That I didn't know -- we'll try that out.

> 3. If you don't want to use netCDFs, you can fake multidimensionality for
> some scenarios by building a raster catalog with columns for the time and
> depth.

I have no idea how to do that -- can GDAL build a raster catalog?

> 4. My group, the Duke University Marine Geospatial Ecology Lab, is currently
> building 3D and 4D awareness into a collection of tools we publish, Marine
> Geospatial Ecology Tools (MGET, see
> http://code.nicholas.duke.edu/projects/mget), built in Python on GDAL and
> other FOSS packages.

Very cool. I'll keep an eye on that.

Thanks for everyone's help,

-Chris



-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov