[gdal-dev] GDAL vsicurl with query string

[JDzialo John]

Weird...

I'm still having an issue with the certificate using GDAL_HTTP_UNSAFESSL YES.  It seems to try to verify the cert with this set or not.

I wonder why you are not seeing that error at all?  

Is there any other why of setting up gdalinfo to not verify the cert?  Is there a list of GDAL config options somewhere I could go through?



-----Original Message-----
From: Even Rouault [mailto:even.rouault at spatialys.com] 
Sent: Monday, August 25, 2014 2:40 PM
To: JDzialo John
Cc: gdal-dev at lists.osgeo.org
Subject: Re: [gdal-dev] GDAL vsicurl with query string

Le lundi 25 août 2014 20:23:14, JDzialo John a écrit :
> Thanks.
> 
> I think an hour may be the limit but I set it for 10 hours so if it's 
> allowed this link should be good until midnight tonight...
> 
> https://parcel-test.s3.amazonaws.com/test/key/value/string/object.jpg?
> Signa 
> ture=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKe
> yId=A
> KIAJF3TTVKIEFQXGZ3Q
> 
> Thanks for the debug command I appreciate your help.  I tried the 
> command and received the following error...
> 
> PS C:\Users\jdzialoex> gdalinfo --debug on --config CPL_CURL_VERBOSE 
> YES "/vsicurl/https://parcel-test.s3.amazonaws.com/
> test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%
> 2FOs %3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3T TVKIEFQXGZ3Q"
> VSICURL:
> GetFileList(/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/val
> ue/s
> tring) * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 
> (#0)
> * libcurl is now using a weak random seed!
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate 
> verify failed * Closing connection #0
> * timeout on name lookup is not supported
> * About to connect() to parcel-test.s3.amazonaws.com port 443 (#0)
> *   Trying 54.231.2.217... * connected
> * Connected to parcel-test.s3.amazonaws.com (54.231.2.217) port 443 
> (#0)
> * SSL certificate problem, verify that the CA cert is OK. Details:
> error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate 
> verify failed * Closing connection #0
> VSICURL:
> GetFileSize(https://parcel-test.s3.amazonaws.com/test/key/value/string
> /obj
> ect.jpg?Signature=7SPVoYI84N2YF5O0vhj
> vDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
> )=0
> response_code=0 ERROR 4:
> `/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/o
> bjec
> t.jpg?Signature=7SPVoYI84N2YF5O0vhjvD
> v%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q' 
> does not exist in the file system, and is not recognised as a 
> supported dataset name.
> 
> gdalinfo failed - unable to open
> '/vsicurl/https://parcel-test.s3.amazonaws.com/test/key/value/string/o
> bjec
> t.jpg?Signatu
> re=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyI
> d=AK
> IAJF3TTVKIEFQXGZ3Q'.
> 

Interesting. I don't have that error (but others ;-)). There might be a problem with the certificate, and curl being not able to check it.

I believe that "--config GDAL_HTTP_UNSAFESSL YES" should bypass certificate checks.

Assuming this solves this issue, and you'll get the same errors as on my machine, you'll see that the server doesn't apparently accept HEAD requests (this seems to be a constant issue with S3 storage)

> HEAD
/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
HTTP/1.1
Host: parcel-test.s3.amazonaws.com
Accept: */*

< HTTP/1.1 403 Forbidden

You can solve this by adding --config CPL_VSIL_CURL_USE_HEAD NO

And, optionnaly, to make it faster, add --config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR

With the last 2 applied :

$ gdalinfo  "/vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q" 
--config GDAL_DISABLE_READDIR_ON_OPEN EMPTY_DIR --config CPL_VSIL_CURL_USE_HEAD NO
Driver: JPEG/JPEG JFIF
Files: /vsicurl/https://parcel-
test.s3.amazonaws.com/test/key/value/string/object.jpg?Signature=7SPVoYI84N2YF5O0vhjvDv%2FU%2FOs%3D&Expires=1409026574&AWSAccessKeyId=AKIAJF3TTVKIEFQXGZ3Q
Size is 974, 647
Coordinate System is `'
Metadata:
  EXIF_ExifVersion=0220
  EXIF_PixelXDimension=974
  EXIF_PixelYDimension=647
  EXIF_Software=Google
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0,  647.0)
Upper Right (  974.0,    0.0)
Lower Right (  974.0,  647.0)
Center      (  487.0,  323.5)
Band 1 Block=974x1 Type=Byte, ColorInterp=Red
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=974x1 Type=Byte, ColorInterp=Green
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=974x1 Type=Byte, ColorInterp=Blue
  Overviews: 487x324, 244x162
  Image Structure Metadata:
    COMPRESSION=JPEG

Even

--
Spatialys - Geospatial professional services http://www.spatialys.com