[gdal-dev] request limit on netcdf server ?

[Even Rouault]

Michael,

analysis and improvements in https://github.com/OSGeo/gdal/pull/7737

Even

Le 10/05/2023 à 09:03, Michael Sumner a écrit :
> I'm having trouble with remote dsn from here:
>
> https://esgf.nci.org.au/thredds/catalog/esgcet/cmip5/output1/CSIRO-BOM/ACCESS1-3/rcp85/day/ocean/day/r1i1p1/cmip5.output1.CSIRO-BOM.ACCESS1-3.rcp85.day.ocean.day.r1i1p1.v20120413.html#cmip5.output1.CSIRO-BOM.ACCESS1-3.rcp85.day.ocean.day.r1i1p1.v20120413
>
> The support tells me I have been blocked for requests from my IP that 
> exceed 300 in 5min. Apparently, daily you are cleared from the 
> blocklist but with experiments today I've had two machines blocked 
> (and now manually cleared). Still, it's hard to test :0
>
> I've stripped down to this dsn, I thought limiting  the driver might 
> limit the requests because it won't cascade down the driver 
> identification (but I'm not sure if -if adds anymore than the 
> "NETCDF:" is already doing). At any rate I need to target a subdataset 
> so I need that syntax.
>
> "vrt://NETCDF:\"/vsicurl/https://esgf.nci.org.au/thredds/fileServer/master/CMIP5/output1/CSIRO-BOM/ACCESS1-3/rcp85/day/ocean/day/r1i1p1/v20120413/tos/tos_day_ACCESS1-3_rcp85_r1i1p1_20560101-20651231.nc\ 
> <https://esgf.nci.org.au/thredds/fileServer/master/CMIP5/output1/CSIRO-BOM/ACCESS1-3/rcp85/day/ocean/day/r1i1p1/v20120413/tos/tos_day_ACCESS1-3_rcp85_r1i1p1_20560101-20651231.nc\>":tos?bands=1&if=NETCDF"
>
> It takes quite a while to read one band, but the warper and RasterIO 
> result is good.  I think the machine is now blocked again, I wasn't 
> able to also run gdalinfo after warper and straight read.
>
> I'm just doing standalone from-scratch opens, not with a persistent 
> dataset - I assume that would reduce the request load, which I'll try 
> in python when I have access again.
>
> Questions:
>
> 1. Are there other things I can pursue to reduce my request load on 
> the server?
>
> 2. Can this actually be faster?  RasterIO for one band 360x300 was 
> minutes.  (I appreciate this might be better suited to mdim, but that 
> seems similarly problematic in terms of requests and performance).
>
> I've tried the config options   GDAL_NETCDF_VERIFY_DIMS / STRICT and 
> GDAL_NETCDF_IGNORE_XY_AXIS_NAME_CHECK=/YES and that does seem to have 
> helped the warper request but RasterIO was still slow.
> .
> This was on GDAL 3.8.0dev-414270a94b-dirty, released 2023/05/09 (debug 
> build)
>
> I see this repeated warning
>
> Warning 1: dimension #2 (i) is not a Longitude/X dimension.
> Warning 1: dimension #1 (j) is not a Latitude/Y dimension.
> Warning 1: dimension #1 (i) is not a Longitude/X dimension.
> Warning 1: dimension #0 (j) is not a Latitude/Y dimension.
> Warning 1: dimension #1 (i) is not a Longitude/X dimension.
> Warning 1: dimension #0 (j) is not a Latitude/Y dimension.
>
> Thank you
>
> -- 
> Michael Sumner
> Software and Database Engineer
> Australian Antarctic Division
> Hobart, Australia
> e-mail: mdsumner at gmail.com
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20230510/93b15778/attachment-0001.htm>