[gdal-dev] Fast Pixel Access

[Daniel Clewley]

Hi David,

Following on from the VRT / Bigtiff comparison Jukka posted. Have you considered storing the data as a single KEA format file, which is based on HDF5?

I have the National Elevation Dataset for the US, which comprises 3,605 1 x 1 degree tiles at 1 arc sec resolution. I first created a VRT then used gdal_translate to convert to a KEA file. Total size is 421,212 x 252,012 pixels, 77 GB. I also built overviews for a fast display (this took a long time and I don't think is needed for your case).

I've just tried using gdallocationinfo on the file to get pixel information and it takes < 0.5s to get the pixel value back. 

The KEA library and GDAL driver source are available from:

https://bitbucket.org/chchrsc/kealib/

and the format is described in:

Peter Bunting, Sam Gillingham, The KEA image file format, Computers & Geosciences, Volume 57, August 2013, Pages 54-58, ISSN 0098-3004, http://dx.doi.org/10.1016/j.cageo.2013.03.025.

If you don't mind having one massive file (in addition to the individual tiles, which could be archived), this might work for your use case. 

Thanks,

Dan

> 
> Message: 10
> Date: Fri, 31 Jan 2014 16:15:53 +0000
> From: "David Baker (Geoscience)" <david.m.baker at chk.com>
> To: "'gdal-dev at lists.osgeo.org'" <gdal-dev at lists.osgeo.org>
> Subject: [gdal-dev] Fast Pixel Access
> Message-ID:
> 	<2A18A4344312134B937DF938D992264A0508F250 at OKCEXHPRD122.chkenergy.net>
> Content-Type: text/plain; charset="us-ascii"
> 
> Dev's,
> 
> I have a set of 55,501 bil files in a single directory.  They are DEMS data that cover the US in 7.5 minute tiles.  I would like to randomly access elevations at a given lat/lon's from the whole dataset.  I created a vrt file from the directory of bil files, and have been able to access the elevation at a given lat/lon using gdallocationinfo, but because of the size of the dataset, this operation is somewhat slow.  Can the vrt be indexed? Or, is there a faster, better way to access the pixels?  I would first like to do this with the utilities before diving into code (C#).  The files are regularly named base on their location within a 1 arc-second grid.
> 
> Thanks,
> David
> 
> David M. Baker
> Senior Advisor - Geoscience Technology
> Chesapeake Energy Corporation
> david.m.baker at chk.com<mailto:david.m.baker at chk.com>
> 
> 
>