[gdal-dev] gridding binary data

[David Hoese]

Hey Jan,

Thanks for the response. My process will need to be automated so excel 
won't help.  I also will be working with potentially large amounts of 
data (200MB - ~4GB).  I would like the output to be in a flat binary 
format (just the data), but if I can pull the data out of another format 
that will do for now.  The output data will be used in other software 
that I have written to place the data in any format I request (geotiff, 
AWIPS compatible netCDF files, etc).  I was hoping to use gdal_grid for 
its algorithms to do the interpolation from input swath pixels to output 
grid pixels.  I am working with "raw" satellite observation data that is 
not equidistant (so it can't be put through gdalwarp which expects an 
equidistant grid as input).

The x and y arrays are created by software I have written.  The values 
in these arrays state the x and y offset from the grid origin of where 
the image pixel is.  For example, a single pixel might be at column 1.14 
and row 1.6 of the grid I am mapping to. I would like gdal_grid (or some 
other code) to use an algorithm (nearest neighbor, averaging, etc) to 
interpolate this image pixel value onto the surrounding grid points.  
Does that make sense? It's difficult for me to describe since I've been 
working with it so much lately.

The start-to-finish goal is essentially what gdalwarp does when 
remapping one grid to another, but this doesn't make the assumption that 
the data points are equidistant.  The gridding utility I'm currently 
using is old and only offers a single algorithm for interpolation.  That 
algorithm does not produce "pretty" images for a specific instrument's 
data so I'm looking for alternatives without having to hack up the old 
utility.  Maybe looking at it this way will make more sense:

(lat,lon,projection) -> my software -> (y,x)
(x,y,image_in) -> gdal_grid -> (gridded_image_out)

Maybe I'll just write the python wrapper for GDALGridCreate since the 
rest of my software is python.  As far as I know GDAL doesn't have 
anything else that does gridding and doesn't have anything that can do 
non-equidistant data points. Thanks for the help so far. Man this email 
got long fast.

-Dave

On 1/22/13 2:58 PM, Jan Heckman wrote:
> Hi Dave,
> So you have effectively three arrays, one for pixel values, one for x, 
> one for y (probably offsets), which are coordinated and you want to 
> generate a raster or grid which you can display / use in a GIS?
> Sort of sparse array to be converted to a full array, effectively?
> It sounds crazy, maybe, but consider converting these to separated 
> text (numbers), bring them into excel (columns), tell (arc)gis to 
> convert to a (point) shapefile and finally, if necessary, convert that 
> to a raster.
> If this is the job, it can alternatively be done quite simply by 
> writing a small procedure (C++ or python (or something else, I'm 
> sure)), without using excel etc, as long as you know the output format 
> (and maybe have a geoprocessor to call upon). A simple output format 
> is .bil. This is roughly simlar to shapefiles, with some supporting 
> files to use it in a GIS. Possibly flat 32 bit float (.flt) is 
> understood by gdal as well.
> Sorry that I can't help you with the gdal/ogr commands.
> If you don't get a helpful gdal command, give me a yell with an example.
> Jan
>
> On Sun, Jan 20, 2013 at 9:51 PM, David Hoese <dhoese at gmail.com 
> <mailto:dhoese at gmail.com>> wrote:
>
>     I have 3 flat binary files that I would like to use to grid data.
>     The 3 binary files are the "scattered" image data pixels and the
>     other two specify the fractional column and row for each input
>     pixel in the output grid.  So the image pixel in the first file at
>     offset 0 is mapped to the grid column at offset 0 (from the
>     columns file) and the grid row at offset 0 (from the rows file).
>
>     One way I think I can get the data gridded is to use the
>     "gdal_grid" utility, but I'm not really sure how to get the flat
>     binary files to be accepted by "gdal_grid".  I thought I could use
>     VRT files, but I'm not exactly sure how I would do that.  I wasn't
>     sure since the documentation here
>     http://www.gdal.org/ogr/drv_vrt.html says that I need to specify a
>     "SrcLayer", but binary files don't have layers. So then would I
>     need an intermediate VRT file to define layers for the binaries?
>      If I need an intermediate VRT maybe I should just push the data
>     into a geotiff (with no projection info) and try using that?
>      Could someone clear this up for me?
>
>     My other question is has there been any work done to make a python
>     wrapper for GDALGridCreate().  I found email threads and a ticket
>     (http://trac.osgeo.org/gdal/ticket/2023) talking about creating
>     one, but then conversation just stops and those are from 5 years
>     ago.  I didn't notice anything that screamed "this is impossible"
>     when skimming through those threads and the ticket.  I'm also sure
>     a lot has changed for GDAL and numpy in the last 5 years.  If it's
>     a time thing I'm willing to add it if someone can point me in the
>     right direction to start (how-to on adding to GDAL).
>
>     Thanks.
>
>     -Dave
>     _______________________________________________
>     gdal-dev mailing list
>     gdal-dev at lists.osgeo.org <mailto:gdal-dev at lists.osgeo.org>
>     http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20130122/19436426/attachment.html>