[gdal-dev] Using /vsicurl and /vsizip with zip files and signed S3 links

[Scott Arko]

Hello GDAL Developers,


I am having trouble using the /vsis3 or /vsicurl driver with S3 signed
links and can't find any details in the archive of this list. I have a
signed S3 link that is a zip file containing a variety of other files,
including a geotiff, which is what I'm really after.   I would like to be
able to access the geotiff file only using a combination of /vsizip and
/vsicurl.   I am using GDAL version 2.2.1 on MacOS.  An example link is
below:

https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D

If I do a gdalinfo on this file using vsizip and vsicurl it seems to
recognize the file and gives me links to use to access the individual
elements.
gdalinfo /vsizip//vsicurl/"
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D
"

ERROR 6: Support only 1 file in archive file /vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D
when no explicit in-archive filename is specified

You could try one of the following :

  /vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.tif

  /vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.jpg

  /vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.jpg.aux.xml

  /vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/ESA_citation.txt

  /vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.wld


Apologies for the funky formatting on this result.  However, if I try to
run gdalinfo on one of the recommended links (the geotiff), I get the
following (with --debug ON)

gdalinfo --debug ON /vsizip//vsicurl/"
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.tif
"

GDAL: Auto register /opt/local/lib/gdalplugins/gdal_KEA.dylib using
GDALRegister_KEA.

GDAL: Assuming DCAP_RASTER for driver KEA. Please fix it.

GNM: GNMRegisterAllInternal

GNM: RegisterGNMFile

GNM: RegisterGNMdatabase

VSICURL: GetFileSize(
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.tif)=0
response_code=403

VSICURL: GetFileSize(
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.tif.xml)=0
response_code=403

ERROR 4: `/vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.tif'
does not exist in the file system, and is not recognized as a supported
dataset name.

gdalinfo failed - unable to open '/vsizip//vsicurl/
https://gsfc-ngap-hyp3-product-prod.s3.amazonaws.com/547/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.zip?AWSAccessKeyId=AKIAIYGQSCGFDA6WYRIQ&Expires=1824498425&Signature=WkQJbs%2FE6vLttvAAYt2fXlKOeKk%3D/S1B_IW_GRDH_1SDV_20170903T223511_20170903T223539_007233_00CC06_E9A8-POEORB-24-RGB.tif
'.

I'm wondering if there is something I am doing wrong, something in the way
gdal handles S3 signed links, or perhaps the functionality I'm looking for
just doesn't exist yet.  Thank you very much for any thoughts you might
have.

Kindest Regards,
Scott Arko



-- 
Scott Arko
Alaska Satellite Facility Deputy Director
University of Alaska Fairbanks

903 Koyukuk Drive
Fairbanks, AK 99775
Phone: (907) 474 - 5570
email:  saarko at alaska.edu
http://www.asf.alaska.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20171031/f0b8e09f/attachment-0001.html>