[gdal-dev] cURL response content output

[Daniel Evans]

Hi again,

To loop back in the mailing list on the conversation Even and I had on the
PR, the answer to my original question (at least in the /vsis3/ use case)
was to enable CPL_DEBUG, which outputs the raw response in these sorts of
error cases, e.g.:

"CPLE_None in S3: <?xml version=\"1.0\"
encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The
specified bucket does not
exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName><RequestId>...</RequestId><HostId>...</HostId></Error>"

I'd been thrown off by the fact that rasterio captures GDAL's CPL_DEBUG
logs and re-emits them in Python, but CPL_CURL_VERBOSE messages aren't
captured in the same way. Some quirks in the configuration of the logging
on my end meant the DEBUG log messages were being discarded when I thought
they weren't.

Regards,
Daniel

On Wed, 6 Jul 2022 at 18:55, Even Rouault <even.rouault at spatialys.com>
wrote:

> Daniel,
>
> https://github.com/OSGeo/gdal/pull/6028 should help, but with the
> restriction indicated in the commit message.
>
> Even
> Le 06/07/2022 à 13:13, Daniel Evans a écrit :
>
> Hi,
>
> I'm trying to diagnose some issues with accessing remote datasets in GDAL
> - I believe the problem is unrelated to GDAL itself, but it's proving hard
> to debug.
>
> By enabling CPL_CURL_VERBOSE, I can see that the server is kicking back a
> 400 Bad Request status, but the debug output doesn't include the actual
> response content, which would allow me to understand the problem in more
> detail. Is there a way to get GDAL/cURL to output the response content for
> failed requests?
>
>
> By way of example, if I attempt to access a nonexistent file on S3 with
> cURL directly, I get the 404 error code and the descriptive XML error
> message at the end:
>
> $ curl -v https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif
> [...]
> < HTTP/1.1 404 Not Found
> < Content-Type: application/xml
> < Transfer-Encoding: chunked
> < Date: Wed, 06 Jul 2022 11:09:09 GMT
> < Server: AmazonS3
> --- these lines ---
> <
> <?xml version="1.0" encoding="UTF-8"?>
> <Error><Code>NoSuchBucket</Code><Message>The specified bucket does not
> exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName>[...]</Error>
> --- these lines ---
>
>
> In contrast, the output from GDAL with CPL_CURL_VERBOSE excludes the XML
> response:
>
> < HTTP/1.1 404 Not Found
> < Content-Type: application/xml
> < Transfer-Encoding: chunked
> < Date: Wed, 06 Jul 2022 11:04:16 GMT
> < Server: AmazonS3
>
> --- no XML output here ---
>
> * Connection #1 to host a-very-very-fake-s3-bucket.s3.amazonaws.com left
> intact
> * Couldn't find host a-very-very-fake-s3-bucket.s3.amazonaws.com in the
> .netrc file; using defaults
> * Found bundle for host a-very-very-fake-s3-bucket.s3.amazonaws.com:
> 0x25b5080 [can pipeline]
> * Re-using existing connection! (#1) with host
> a-very-very-fake-s3-bucket.s3.amazonaws.com
> * Connected to a-very-very-fake-s3-bucket.s3.amazonaws.com
> (52.216.207.115) port 443 (#1)
> > GET /?delimiter=%2F&max-keys=100&prefix=example.tif%2F HTTP/1.1
> [...]
>
> Regards,
> Daniel
>
> _______________________________________________
> gdal-dev mailing listgdal-dev at lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220712/23c43f85/attachment-0001.htm>