[gdal-dev] RANGE Requests on large Rasters with CDN (AWS Cloudfront)

[Even Rouault]

Sean

I'm not familiar with Cloudfront specifics, but reading 
https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html, 
my understanding is that while Cloudfront will not attempt at caching 
files larger than 50 GB, it is able to forward HTTP Get Range requests 
smaller than 50 GB to the origin. And you would have to design a 
particularly unfriendly file for GDAL to have to download chunks of > 50 
GB !

Otherwise yes the strategy of using a VRT mosaicing tiles < 50 GB each 
should work fine if you want to benefit from Cloudfront caching. The 
easiest to tile your raster is to use gdal_retile, but be careful to 
make sure each tile fit into RAM (as gdal_retile reads & writes each 
tiles in a single time). To create large tiles, you'll rather have to 
use gdal_translate -srcwin in a loop iterating over raster coordinates.

Even

Le 09/09/2024 à 16:06, Sean Turner via gdal-dev a écrit :
> Hi all,
>
> Curious if anyone is doing something similar.
>
> We've got our data in an S3 Bucket that is fronted by Cloudfront. From 
> there we can use the GDAL `GDAL_HTTP_HEADER_FILE` environment variable 
> to point to a file containing Cloudfront signed cookies (which 
> control access to files within Cloudfront).
>
> Cloudfront supports a maximum file size of 50gb. Trying to add a 
> raster layer that is greater than 50gb immediately results in an error.
>
> Is anyone doing something similar and have a work around? AWS 
> Documentation mentions RANGE requests where a range of incremental 
> 50gb are requested from Cloudfront and downloaded simultaneously. Is 
> this supported by GDAL (QGIS)?
>
> Otherwise also considering using a VRT? 
> https://gdal.org/en/latest/drivers/raster/vrt.html
>
> Open to any and all suggestions!
>
> Cheers. Also, sorry I haven't done one of these mailing lists before, 
> apologies if this submission is in the wrong place.
>
> All the best,
> Sean
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.