[gdal-dev] In-memory VRT performance

[Sean Gillies]

Hi all,

I have a GeoTIFF in S3 and am trying to access it via a VRT. I'm
constructing the VRT in Python and then passing the string directly to GDAL
like

    GDALOpenEx("<VRTDataset>...</VRTDataset>", ...)

I'm getting the results I expect, but very slowly. In my logs, it seems
that GDAL is fetching the entire GeoTIFF and copying it to /vsimem/ before
reading any data.

DEBUG:rasterio._base:Entering Dataset <open DatasetBase name='<VRTDataset
rasterXSize="48949" rasterYSize="15633"><SRS>PROJCS["WGS 84 / UTM zone
46N",GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS
84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AUTHORITY["EPSG","4326"]],PROJECTION["Transverse_Mercator"],PARAMETER["latitude_of_origin",0],PARAMETER["central_meridian",93],PARAMETER["scale_factor",0.9996],PARAMETER["false_easting",500000],PARAMETER["false_northing",0],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AXIS["Easting",EAST],AXIS["Northing",NORTH],AUTHORITY["EPSG","32646"]]</SRS><GeoTransform>242785.74302396187,0.018557199999999992,0.0,2634995.766347841,0.0,-0.018557199999991995</GeoTransform><VRTRasterBand
band="1"
dataType="Byte"><ColorInterp>Red</ColorInterp><ComplexSource><SourceFilename
relativeToVRT="0">
http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif</SourceFilename><SourceBand>1</SourceBand><SourceProperties
BlockXSize="512" BlockYSize="512" RasterXSize="48949" RasterYSize="15633"
dataType="Byte" /><SrcRect xOff="0" xSize="48948" yOff="0" ySize="15632"
/><DstRect xOff="0.0" xSize="48948" yOff="-0.0" ySize="15632"
/></ComplexSource></VRTRasterBand><VRTRasterBand band="2"
dataType="Byte"><ColorInterp>Green</ColorInterp><ComplexSource><SourceFilename
relativeToVRT="0">
http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif</SourceFilename><SourceBand>2</SourceBand><SourceProperties
BlockXSize="512" BlockYSize="512" RasterXSize="48949" RasterYSize="15633"
dataType="Byte" /><SrcRect xOff="0" xSize="48948" yOff="0" ySize="15632"
/><DstRect xOff="0.0" xSize="48948" yOff="-0.0" ySize="15632"
/></ComplexSource></VRTRasterBand><VRTRasterBand band="3"
dataType="Byte"><ColorInterp>Blue</ColorInterp><ComplexSource><SourceFilename
relativeToVRT="0">
http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif</SourceFilename><SourceBand>3</SourceBand><SourceProperties
BlockXSize="512" BlockYSize="512" RasterXSize="48949" RasterYSize="15633"
dataType="Byte" /><SrcRect xOff="0" xSize="48948" yOff="0" ySize="15632"
/><DstRect xOff="0.0" xSize="48948" yOff="-0.0" ySize="15632"
/></ComplexSource></VRTRasterBand><MaskBand><VRTRasterBand
dataType="Byte"><SimpleSource><SourceFilename relativeToVRT="0">
http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif</SourceFilename><OpenOptions><OOI
key="OVERVIEW_LEVEL">AUTO</OOI></OpenOptions><SourceBand>mask,1</SourceBand><SourceProperties
BlockXSize="512" BlockYSize="512" RasterXSize="48949" RasterYSize="15633"
dataType="Byte" /><SrcRect xOff="0" xSize="48948" yOff="0" ySize="15632"
/><DstRect xOff="0.0" xSize="48948" yOff="-0.0" ySize="15632"
/></SimpleSource></VRTRasterBand></MaskBand></VRTDataset>' mode='r'>
context.
DEBUG:rasterio._io:IO window xoff=0.0 yoff=0.0 width=48948.0 height=15632.0
DEBUG:rasterio._gdal:CPLE_None in HTTP: Fetch(
http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif
)
GDAL: GDALOpen(/vsimem/http_1/3fa181a1-425c-4e53-b49f-ea72884da097.tif,
this=0x101e9f4c0) succeeds as GTiff.
DEBUG:rasterio._gdal:CPLE_None in GDAL: GDALOpen(
http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif,
this=0x101e9f4c0) succeeds as HTTP.
DEBUG:rasterio._gdal:CPLE_None in GTiff: ScanDirectories()
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 24474x7816 overview.
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 12237x3908 overview.
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 6119x1954 overview.
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 3060x977 overview.
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 1530x489 overview.
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 765x245 overview.
DEBUG:rasterio._gdal:CPLE_None in GTiff: Opened 383x123 overview.

Once the entire GeoTIFF has been download, the overviews are accessed as
I'd expect, but too late to be useful.

Is there a way to bypass creation of the /vsimem/ copy of the source
dataset when passing a VRT XML doc like this to GDALOpenEx?

Thanks in advance,

-- 
Sean Gillies
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20180615/683d68a8/attachment-0001.html>