[gdal-dev] Indexing lots of remote COGs for use with gdal raster clip and QGIS

[Robin Wilson]

Hi,

I have a lot of COGs hosted online in HTTP RANGE compatible storage. I want to create a catalog/index of them so that GDAL and tools that use GDAL (like QGIS) can extract the relevant data from the COGs for specific areas. For example, I’d like to use `gdal raster clip` to clip this big ‘virtual raster’ to a specific bbox or polygon. Ideally I’d also like to be able to view the big raster on-the-fly in QGIS, but I’m aware that may be more difficult. My tiles are ’true tiles’ in that they tessellate properly and each location is in a single file.

I’ve been looking at the GDAL Tile Index format, but have been getting a bit confused, and some things haven’t been working quite how I expect. A few questions below:

1. Is there an easy way to generate a GDAL Tile Index file from a file containing a list of remote URLs? I couldn’t find the right way to specify a file containing URLs on the command line, so I ended up writing a short Python script that loads the list of URLs from a file and then passes them to gdal.TileIndex. Is that the best way to do this? One disadvantage of this method is that I can’t seem to get a progress bar to display.

2. My current way of doing that requires GDAL loading the bbox of each file by requesting it over the network, which is fairly slow, and I have thousands of files to do this for. Is there a way to get the GTI creation to use local files on my hard disk for getting the extent of each file, but save the location as a /vsicurl/ URL to the remote file? I’ve found --base-url in `gdal raster index` though I’m slightly confused by the description of it ("Base URL for STAC-GeoParquet the href property of the id property in a STAC-GeoParquet index.” - does that do what I want and put a base URL in front of each file it finds?). That option doesn’t seem to be present in `gdal driver gti create` though, and seems like it may be STAC-GeoParquet specific.

3. Once I’ve created the file, I can do `gdal raster clip` operations on the command-line quite quickly, but if I load the GTI raster into QGIS then QGIS just hangs. I hoped it would just request the relevant COGs for the area I was looking at. I’ve tried loading it in by dragging and dropping the .gti.gpkg file and then selecting the raster in the layers popup, and also adding it via Add Layer->Raster and putting ‘GTI:<path>’. I have a web map working which loads these COG files on the fly depending on where you’re looking (using a minimal STAC catalog in a JSON file) and was hoping to have the same thing working in QGIS, without the effort of setting up something like titiler or titiler-pgstac, as I’m currently managing to get away without running any servers  (as opposed to static data stores) for this data.

If there’s an alternative method I should be looking at for doing either part of this (the clipping or the viewing in QGIS) then let me know - I’m aware I may be pushing the boundaries of what can be sensibly done with the GTI format.

Best regards,

Robin


Dr Robin Wilson

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20260901/57f99fbd/attachment-0001.htm>