[gdal-dev] cURL response content output

[Daniel Evans]

Hi,

I'm trying to diagnose some issues with accessing remote datasets in GDAL -
I believe the problem is unrelated to GDAL itself, but it's proving hard to
debug.

By enabling CPL_CURL_VERBOSE, I can see that the server is kicking back a
400 Bad Request status, but the debug output doesn't include the actual
response content, which would allow me to understand the problem in more
detail. Is there a way to get GDAL/cURL to output the response content for
failed requests?


By way of example, if I attempt to access a nonexistent file on S3 with
cURL directly, I get the 404 error code and the descriptive XML error
message at the end:

$ curl -v https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif
[...]
< HTTP/1.1 404 Not Found
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 06 Jul 2022 11:09:09 GMT
< Server: AmazonS3
--- these lines ---
<
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not
exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName>[...]</Error>
--- these lines ---


In contrast, the output from GDAL with CPL_CURL_VERBOSE excludes the XML
response:

< HTTP/1.1 404 Not Found
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Wed, 06 Jul 2022 11:04:16 GMT
< Server: AmazonS3

--- no XML output here ---

* Connection #1 to host a-very-very-fake-s3-bucket.s3.amazonaws.com left
intact
* Couldn't find host a-very-very-fake-s3-bucket.s3.amazonaws.com in the
.netrc file; using defaults
* Found bundle for host a-very-very-fake-s3-bucket.s3.amazonaws.com:
0x25b5080 [can pipeline]
* Re-using existing connection! (#1) with host
a-very-very-fake-s3-bucket.s3.amazonaws.com
* Connected to a-very-very-fake-s3-bucket.s3.amazonaws.com (52.216.207.115)
port 443 (#1)
> GET /?delimiter=%2F&max-keys=100&prefix=example.tif%2F HTTP/1.1
[...]

Regards,
Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20220706/9e060c74/attachment.htm>