[gdal-dev] Fast Pixel Access

[David Baker (Geoscience)]

Dan,

I had not heard of the KEA format and it looks promising, accept the need to compile.  I am hoping to this with out-of-the-box GDAL.  I also did not see a license statement.  GDAL does support HDF5 (another format I am not familiar with), but it looks like the limit is 2GB for the built in driver.  My dataset is also covers the US with 55,501 x 7.5 minutes tiles at 0.15012 arc second resolution (~5m), a total size 2,289,001 x 756,001 pixels, 1.8T.  Creating a single dataset from the tiles can be done, but in our environment, is not cheap.

David

-----Original Message-----
From: gdal-dev-bounces at lists.osgeo.org [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Daniel Clewley
Sent: Saturday, February 01, 2014 1:45 PM
To: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Fast Pixel Access

Hi David,

Following on from the VRT / Bigtiff comparison Jukka posted. Have you considered storing the data as a single KEA format file, which is based on HDF5?

I have the National Elevation Dataset for the US, which comprises 3,605 1 x 1 degree tiles at 1 arc sec resolution. I first created a VRT then used gdal_translate to convert to a KEA file. Total size is 421,212 x 252,012 pixels, 77 GB. I also built overviews for a fast display (this took a long time and I don't think is needed for your case).

I've just tried using gdallocationinfo on the file to get pixel information and it takes < 0.5s to get the pixel value back.

The KEA library and GDAL driver source are available from:

https://bitbucket.org/chchrsc/kealib/

and the format is described in:

Peter Bunting, Sam Gillingham, The KEA image file format, Computers & Geosciences, Volume 57, August 2013, Pages 54-58, ISSN 0098-3004, http://dx.doi.org/10.1016/j.cageo.2013.03.025.

If you don't mind having one massive file (in addition to the individual tiles, which could be archived), this might work for your use case.

Thanks,

Dan

>
> Message: 10
> Date: Fri, 31 Jan 2014 16:15:53 +0000
> From: "David Baker (Geoscience)" <david.m.baker at chk.com>
> To: "'gdal-dev at lists.osgeo.org'" <gdal-dev at lists.osgeo.org>
> Subject: [gdal-dev] Fast Pixel Access
> Message-ID:
>       <2A18A4344312134B937DF938D992264A0508F250 at OKCEXHPRD122.chkenergy.net>
> Content-Type: text/plain; charset="us-ascii"
>
> Dev's,
>
> I have a set of 55,501 bil files in a single directory.  They are DEMS data that cover the US in 7.5 minute tiles.  I would like to randomly access elevations at a given lat/lon's from the whole dataset.  I created a vrt file from the directory of bil files, and have been able to access the elevation at a given lat/lon using gdallocationinfo, but because of the size of the dataset, this operation is somewhat slow.  Can the vrt be indexed? Or, is there a faster, better way to access the pixels?  I would first like to do this with the utilities before diving into code (C#).  The files are regularly named base on their location within a 1 arc-second grid.
>
> Thanks,
> David
>
> David M. Baker
> Senior Advisor - Geoscience Technology
> Chesapeake Energy Corporation
> david.m.baker at chk.com<mailto:david.m.baker at chk.com>
>
>
>

_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

________________________________

This email (and attachments if any) is intended only for the use of the individual or entity to which it is addressed, and may contain information that is confidential or privileged and exempt from disclosure under applicable law. If the reader of this email is not the intended recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return email and destroy all copies of the email (and attachments if any).