[gdal-dev] GDAL vsicurl with query string

[Even Rouault]

Le lundi 25 août 2014 20:23:14, JDzialo John a écrit :
> Thanks.
> 
> I think an hour may be the limit but I set it for 10 hours so if it's
> allowed this link should be good until midnight tonight...
> 
> https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?Signa
> ture=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=A
> KIAJF3TTVKIEFQXGZ3Q
> 
> Thanks for the debug command I appreciate your help.  I tried the command
> and received the following error...
> 
> PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE YES
> "/vsicurl/https://parcel-test.s3.amazonaws.com/
> test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs
> %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T TVKIEFQXGZ3Q"
> VSICURL:
> GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/s
> tring) * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 (#0)
> * libcurl is now using a weak random seed!
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
> failed * Closing connection #0
> * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 (#0)
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify
> failed * Closing connection #0
> VSICURL:
> GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/string/obj
> ect.jpg?Signature=7SPVoYI84N2YF5O0vhj
> vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q)=0 
> response_code=0 ERROR 4:
> `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/objec
> t.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
> v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q' does
> not exist in the file system, and is not recognised as a supported dataset
> name.
> 
> gdalinfo failed - unable to open
> '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/objec
> t.jpg?Signatu
> re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AK
> IAJF3TTVKIEFQXGZ3Q'.
> 

Interesting. I don't have that error (but others ;-)). There might be a 
problem with the certificate, and curl being not able to check it.

I believe that "--config GDAL_HTTP_UNSAFESSL YES" should bypass certificate 
checks.

Assuming this solves this issue, and you'll get the same errors as on my 
machine, you'll see that the server doesn't apparently accept HEAD requests 
(this seems to be a constant issue with S3 storage)

> HEAD 
/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q 
HTTP/1.1
Host: parcel-test.s3.amazonaws.com
Accept: */*

< HTTP/1.1 403 Forbidden

You can solve this by adding --config CPL_VSIL_CURL_USE_HEAD NO

And, optionnaly, to make it faster, add --config GDAL_DISABLE_READDIR_ON_OPEN 
EMPTY_DIR

With the last 2 applied :

$ gdalinfo  "/vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q" 
--config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_VSIL_CURL_USE_HEAD 
NO
Driver: JPEG/JPEG JFIF
Files: /vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
Size is 974, 647
Coordinate System is `'
Metadata:
  EXIF_ExifVersion=0220
  EXIF_PixelXDimension=974
  EXIF_PixelYDimension=647
  EXIF_Software=Google
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  647.0)
Upper Right (  974.0,    0.0)
Lower Right (  974.0,  647.0)
Center      (  487.0,  323.5)
Band 1 Block=974x1 Type=Byte, ColorInterp=Red
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=974x1 Type=Byte, ColorInterp=Green
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=974x1 Type=Byte, ColorInterp=Blue
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG

Even

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com