[gdal-dev] Fast Pixel Access

[David Baker (Geoscience)]

Evan,

I am not sure how to profile as I do not have access to the code to profile.  I did do a timing test...

vrt file = 22,970 KB
bil file = 35,180 KB * 55,501

I piped five locations from the loc.txt file:
-96.0 36.0
-98.0 37.0
-100.0 38.0
-99.0 39.0
-101.0 35.0

gdallocationinfo -valonly -geoloc intermap.vrt < loc.txt
189.841857910156        25.5 sec
384.857452392578        22.6 sec
762.015930175781        22.9 sec
550.719116210938        23.6 sec
883.637023925781        22.9 sec

Note: I used a lap timer on my iPhone to capture the split times as the results appeared in the console window.  Does this give any insight?

David

-----Original Message-----
From: gdal-dev-bounces at lists.osgeo.org [mailto:gdal-dev-bounces at lists.osgeo.org] On Behalf Of Even Rouault
Sent: Saturday, February 01, 2014 1:28 AM
To: Brian Case
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] Fast Pixel Access

Le samedi 01 février 2014 00:23:13, Brian Case a écrit :
> evenr
>
>
> what about the use of a tileindex?

You really mean a tileindex as produced by gdaltindex ? Well, that's not
exactly the same beast as a VRT, but yes if it was recognized as a GDAL
dataset then you could potentially save the cost of XML parsing. One could
imagine that the VRT driver would accept a tileindex as an altenate connection
string.

Anyway it would be interesting to first profile where the time is spent in David
use case. If it's in the XML parsing, then I can't see what could be easily
improved in that area. If it's the intersection, then there's potential for
improvement.

> seems an intersection with a set of
> polys first would be quick


>
> brian
>
> On Fri, 2014-01-31 at 19:30 +0100, Even Rouault wrote:
> > Le vendredi 31 janvier 2014 17:15:53, David Baker (Geoscience) a écrit :
> > > Dev's,
> > >
> > > I have a set of 55,501 bil files in a single directory.  They are DEMS
> > > data that cover the US in 7.5 minute tiles.  I would like to randomly
> > > access elevations at a given lat/lon's from the whole dataset.  I
> > > created a vrt file from the directory of bil files, and have been able
> > > to access the elevation at a given lat/lon using gdallocationinfo, but
> > > because of the size of the dataset, this operation is somewhat slow.
> > > Can the vrt be indexed?
> >
> > No, it isn't currently, although I think it could be improved to have a
> > in- memory index with moderate effort.
> >
> > But are you sure the slowness is due to the lack of index ? 55,000 is a
> > big number, but not that big. Maybe the slowness just comes from the
> > opening time (XML parsing) of such a big VRT. That would need to be
> > profiled to be sure where the bottleneck is.
> >
> > > Or, is there a faster, better way to access the pixels?  I would
> > > first like to do this with the utilities before diving into code (C#).
> > > The files are regularly named base on their location within a 1
> > > arc-second grid.
> > >
> > > Thanks,
> > > David
> > >
> > > David M. Baker
> > > Senior Advisor - Geoscience Technology
> > > Chesapeake Energy Corporation
> > > david.m.baker at chk.com<mailto:david.m.baker at chk.com>
> > >
> > >
> > > ________________________________
> > >
> > > This email (and attachments if any) is intended only for the use of the
> > > individual or entity to which it is addressed, and may contain
> > > information that is confidential or privileged and exempt from
> > > disclosure under applicable law. If the reader of this email is not
> > > the intended recipient, or the employee or agent responsible for
> > > delivering this message to the intended recipient, you are hereby
> > > notified that any dissemination, distribution or copying of this
> > > communication is strictly prohibited. If you have received this
> > > communication in error, please notify the sender immediately by return
> > > email and destroy all copies of the email (and attachments if any).

--
Geospatial professional services
http://even.rouault.free.fr/services.html
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

________________________________

This email (and attachments if any) is intended only for the use of the individual or entity to which it is addressed, and may contain information that is confidential or privileged and exempt from disclosure under applicable law. If the reader of this email is not the intended recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify the sender immediately by return email and destroy all copies of the email (and attachments if any).