<div dir="ltr">Hi again,<div><br></div><div>To loop back in the mailing list on the conversation Even and I had on the PR, the answer to my original question (at least in the /vsis3/ use case) was to enable CPL_DEBUG, which outputs the raw response in these sorts of error cases, e.g.:</div><div><br></div><div>"<span style="color:rgb(29,28,29);font-family:Monaco,Menlo,Consolas,"Courier New",monospace;font-size:12px;font-variant-ligatures:none;white-space:pre-wrap">CPLE_None in S3: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName><RequestId>...</RequestId><HostId>...</HostId></Error>"</span></div><div><br></div><div>I'd been thrown off by the fact that rasterio captures GDAL's CPL_DEBUG logs and re-emits them in Python, but CPL_CURL_VERBOSE messages aren't captured in the same way. Some quirks in the configuration of the logging on my end meant the DEBUG log messages were being discarded when I thought they weren't.</div><div><br></div><div>Regards,</div><div>Daniel</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 6 Jul 2022 at 18:55, Even Rouault <<a href="mailto:even.rouault@spatialys.com">even.rouault@spatialys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Daniel,</p>
<p><a href="https://github.com/OSGeo/gdal/pull/6028" target="_blank">https://github.com/OSGeo/gdal/pull/6028</a> should help, but with the
restriction indicated in the commit message.</p>
<p>Even<br>
</p>
<div>Le 06/07/2022 à 13:13, Daniel Evans a
écrit :<br>
</div>
<blockquote type="cite">
<div dir="ltr">Hi,
<div><br>
</div>
<div>I'm trying to diagnose some issues with accessing remote
datasets in GDAL - I believe the problem is unrelated to GDAL
itself, but it's proving hard to debug.</div>
<div><br>
</div>
<div>By enabling CPL_CURL_VERBOSE, I can see that the server is
kicking back a 400 Bad Request status, but the debug output
doesn't include the actual response content, which would allow
me to understand the problem in more detail. Is there a way to
get GDAL/cURL to output the response content for failed
requests?</div>
<div><br>
</div>
<div><br>
</div>
<div>By way of example, if I attempt to access a nonexistent
file on S3 with cURL directly, I get the 404 error code and
the descriptive XML error message at the end:<br>
<br>
</div>
<div>$ curl -v <a href="https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif" target="_blank">https://a-very-very-fake-s3-bucket.s3.amazonaws.com/example.tif</a><br>
</div>
<div>[...]</div>
<div>< HTTP/1.1 404 Not Found<br>
< Content-Type: application/xml<br>
< Transfer-Encoding: chunked<br>
< Date: Wed, 06 Jul 2022 11:09:09 GMT<br>
< Server: AmazonS3<br>
--- these lines ---<br>
</div>
<div><</div>
<div><?xml version="1.0" encoding="UTF-8"?><br>
</div>
<div><Error><Code>NoSuchBucket</Code><Message>The
specified bucket does not
exist</Message><BucketName>a-very-very-fake-s3-bucket</BucketName>[...]</Error><br>
</div>
<div>--- these lines ---</div>
<div><br>
</div>
<div><br>
</div>
<div>In contrast, the output from GDAL with CPL_CURL_VERBOSE
excludes the XML response:</div>
<div><br>
</div>
<div>< HTTP/1.1 404 Not Found<br>
< Content-Type: application/xml<br>
< Transfer-Encoding: chunked<br>
< Date: Wed, 06 Jul 2022 11:04:16 GMT<br>
< Server: AmazonS3<br>
</div>
<div><br>
</div>
<div>--- no XML output here ---</div>
<div><br>
</div>
<div>* Connection #1 to host <a href="http://a-very-very-fake-s3-bucket.s3.amazonaws.com" target="_blank">a-very-very-fake-s3-bucket.s3.amazonaws.com</a>
left intact<br>
* Couldn't find host <a href="http://a-very-very-fake-s3-bucket.s3.amazonaws.com" target="_blank">a-very-very-fake-s3-bucket.s3.amazonaws.com</a>
in the .netrc file; using defaults<br>
* Found bundle for host <a href="http://a-very-very-fake-s3-bucket.s3.amazonaws.com" target="_blank">a-very-very-fake-s3-bucket.s3.amazonaws.com</a>:
0x25b5080 [can pipeline]<br>
* Re-using existing connection! (#1) with host <a href="http://a-very-very-fake-s3-bucket.s3.amazonaws.com" target="_blank">a-very-very-fake-s3-bucket.s3.amazonaws.com</a><br>
* Connected to <a href="http://a-very-very-fake-s3-bucket.s3.amazonaws.com" target="_blank">a-very-very-fake-s3-bucket.s3.amazonaws.com</a>
(52.216.207.115) port 443 (#1)<br>
> GET
/?delimiter=%2F&max-keys=100&prefix=example.tif%2F
HTTP/1.1<br>
</div>
<div>[...]</div>
<div><br>
</div>
<div>Regards,</div>
<div>Daniel</div>
</div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
gdal-dev mailing list
<a href="mailto:gdal-dev@lists.osgeo.org" target="_blank">gdal-dev@lists.osgeo.org</a>
<a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev" target="_blank">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a>
</pre>
</blockquote>
<pre cols="72">--
<a href="http://www.spatialys.com" target="_blank">http://www.spatialys.com</a>
My software is free, but my time generally not.</pre>
</div>
</blockquote></div>