[gdal-dev] request limit on netcdf server ?

[Michael Sumner]

I'm having trouble with remote dsn from here:

https://esgf.nci.org.au/thredds/catalog/esgcet/cmip5/output1/CSIRO-BOM/ACCESS1-3/rcp85/day/ocean/day/r1i1p1/cmip5.output1.CSIRO-BOM.ACCESS1-3.rcp85.day.ocean.day.r1i1p1.v20120413.html#cmip5.output1.CSIRO-BOM.ACCESS1-3.rcp85.day.ocean.day.r1i1p1.v20120413

The support tells me I have been blocked for requests from my IP that
exceed 300 in 5min. Apparently, daily you are cleared from the blocklist
but with experiments today I've had two machines blocked (and now manually
cleared). Still, it's hard to test :0

I've stripped down to this dsn, I thought limiting  the driver might limit
the requests because it won't cascade down the driver identification (but
I'm not sure if -if adds anymore than the "NETCDF:" is already doing). At
any rate I need to target a subdataset so I need that syntax.

"vrt://NETCDF:\"/vsicurl/
https://esgf.nci.org.au/thredds/fileServer/master/CMIP5/output1/CSIRO-BOM/ACCESS1-3/rcp85/day/ocean/day/r1i1p1/v20120413/tos/tos_day_ACCESS1-3_rcp85_r1i1p1_20560101-20651231.nc\
":tos?bands=1&if=NETCDF"

It takes quite a while to read one band, but the warper and RasterIO result
is good.  I think the machine is now blocked again, I wasn't able to also
run gdalinfo after warper and straight read.

I'm just doing standalone from-scratch opens, not with a persistent dataset
- I assume that would reduce the request load, which I'll try in python
when I have access again.

Questions:

1. Are there other things I can pursue to reduce my request load on the
server?

2. Can this actually be faster?  RasterIO for one band 360x300 was
minutes.  (I appreciate this might be better suited to mdim, but that seems
similarly problematic in terms of requests and performance).

I've tried the config options   GDAL_NETCDF_VERIFY_DIMS / STRICT and
GDAL_NETCDF_IGNORE_XY_AXIS_NAME_CHECK=/YES and that does seem to have
helped the warper request but RasterIO was still slow.
.
This was on GDAL 3.8.0dev-414270a94b-dirty, released 2023/05/09 (debug
build)

I see this repeated warning

Warning 1: dimension #2 (i) is not a Longitude/X dimension.
Warning 1: dimension #1 (j) is not a Latitude/Y dimension.
Warning 1: dimension #1 (i) is not a Longitude/X dimension.
Warning 1: dimension #0 (j) is not a Latitude/Y dimension.
Warning 1: dimension #1 (i) is not a Longitude/X dimension.
Warning 1: dimension #0 (j) is not a Latitude/Y dimension.

Thank you

-- 
Michael Sumner
Software and Database Engineer
Australian Antarctic Division
Hobart, Australia
e-mail: mdsumner at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230510/24a285d5/attachment-0001.htm>