[gdal-dev] How to represent multi-dimensional array

[Christopher Barker]

Jason Roberts wrote:
>>> The trick is your netCDF has to meet a bunch
>>> of constraints for ArcGIS to recognize it. It has to have square cells.
>> bingo -- that was one of our key problems -- wait -- they have to be 
>> "square" -- rectangular won't do? arrgg!
> 
> Oops, I'm pretty sure you're right, they can be rectangular so long
> as the increment is constant. I recall working with some climate model
> output that was rectangular with a constant increment (2 deg latitude, 3
> deg longitude). I checked the documentation and it confirms this.

That's better, and makes sense -- other raster data works that way.

> The limitation is that it cannot work with rectangular cells that
> have  an irregular increment

which indeed is what we were trying to do. Actually, the cells were 
rectangular in projected coordinates (where we did the computation), but 
weren't in lat-long, which is what we were exporting. So I should be 
able to fix that.

-thanks,
  -Chris

> Jason
> 
> 
> -----Original Message-----
> From: gdal-dev-bounces at lists.osgeo.org [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Christopher Barker
> Sent: Saturday, June 19, 2010 5:45 PM
> To: gdal-dev at lists.osgeo.org
> Subject: Re: [gdal-dev] How to represent multi-dimensional array
> 
> Thanks to Michael, Joaquim, Ivan, and Jason.
> 
> I'll explore some of the tools and suggestions you made.
> 
> Michael Sumner wrote:
>> NetCDF will tend to store dimensions in
>> reverse order to the natural one, and I think GDAL reverses that - but
>> you can tell by the dimension and number of your bands, and the named
>> metadata on the GDAL bands.
> 
> yup -- I figured that out - it got better when we re-ordered to (t, z, 
> y, x).
> 
>> NetCDF cannot store multi-attribute arrays (it will store several
>> same-size, same-metadata arrays for that purpose),
> 
> Actually, I think it can -- though maybe that's only for netcdf4 (based 
> on hdf5), but the conventions don't suggest you do that -- particularly 
> if different attributes are different data types.
> 
>> Manifold reads in multiple rasters
> 
>> Eonfusion will do its best to read the array in its natural state
> 
>> R can read NetCDF natively or with GDAL (RNetCDF, ncdf, rgdal
> 
> We're not really looking for other tools at this point -- we are doing 
> visualization with IDV, which handles 4-d data in netcdf just fine. This 
> part is all about getting this data into Arc.
> 
> I may deal with it by figuring out what we really need to do in Arc, and 
> just exporting that part of the data -- we're writing these files with 
> Python anyway, so doing some pre-processing there won't be a big deal.
> 
> Joaquim Luis wrote:
>> Don't know if this is what you are looking for but if those netCDF files 
>> are of a similar type that one can get from the poet site 
>> (http://poet.jpl.nasa.gov/), Mirone has a tool called "Aquamoto" (a tool 
>> original developed to show time stamps of a tsunami propagation models) 
>> that loads those files and show their content interactively with the 
>> help of a slider.
> 
> Not much help for this, but it's a cool tool, thanks for the link.
> 
> Jason Roberts wrote:
>> I have some experience trying to get ArcGIS to work well with time series
>> satellite imagery and 4D ocean models (e.g. HYCOM, ROMS).
> 
> Exactly our situation, here.
> 
>> I am part of a working group initiated
>> by ESRI and led by an ESRI program manager (Nawajish Noman) that is trying,
>> in essence, to get the community of users who use both ArcGIS and netCDF to
>> develop some Python geoprocessing tools for ArcGIS that provide more
>> functionality than out-of-box tools already in ArcGIS.
> 
> cool -- do you have contact information for that project? Is there code 
> anywhere we can get at it?
> 
>> 1. The Make NetCDF Raster Layer tool can represent 3D netCDF variables as
>> multiband raster layers.
> 
> We'll give this a try. It may be what our GIS person is using already, 
> but I got a bit confused by the "Dimension Values parameter". We'll poke 
> at it some more.
> 
>> The trick is your netCDF has to meet a bunch
>> of constraints for ArcGIS to recognize it. It has to have square cells.
> 
> bingo -- that was one of our key problems -- wait -- they have to be 
> "square" -- rectangular won't do? arrgg!
> 
> Anyway, the way we have it now the cells ar rectangular in meters, but 
> we un-projectee them to lat-long, so they are no longer simle 
> rectangular -- I think I may change this an output in meters, with the 
> projection info. But if it has to be square, we're kind of dea in the 
> water...
> 
>> It
>> has adhere to the CF or COARDS conventions (I forget which versions)
> 
> that we do have.
> 
>> 2. Under contract to NOAA, Applied Science Associates built a couple of
>> tools that might be useful: the Environmental Data Connector (EDC) and
>> TimeSlider Extension. Download from
>> http://www.asascience.com/software/downloads/index.shtml, see other parts of
>> the website for more info. EDC was built to download multidimensional
>> OPeNDAP datasets into multiband rasters.
> 
> Hmm -- did know about those tools, but didn't realize that EDC was 
> OPenDAP based -- nice to know.
> 
>> TimeSlider is a UI extension to
>> help with playback of time-series data.
> 
> That too -- by the way, I'm pretty sure the Coast Guard funded a bunhc 
> of that, for their Search and Rescue tools.
> 
>> I think they can both work with
>> netCDFs directly, not just OPeNDAP.
> 
> That I didn't know -- we'll try that out.
> 
>> 3. If you don't want to use netCDFs, you can fake multidimensionality for
>> some scenarios by building a raster catalog with columns for the time and
>> depth.
> 
> I have no idea how to do that -- can GDAL build a raster catalog?
> 
>> 4. My group, the Duke University Marine Geospatial Ecology Lab, is currently
>> building 3D and 4D awareness into a collection of tools we publish, Marine
>> Geospatial Ecology Tools (MGET, see
>> http://code.nicholas.duke.edu/projects/mget), built in Python on GDAL and
>> other FOSS packages.
> 
> Very cool. I'll keep an eye on that.
> 
> Thanks for everyone's help,
> 
> -Chris
> 
> 
> 


-- 
Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov