[gdal-dev] GDAL WMTS get tiles no documentation

Fri May 31 07:14:11 PDT 2024

Hi,

You do not need to use GDALTranslate even if may be convenient. In the RasterIO() function GDALRasterBand C++ API — GDAL documentation<https://gdal.org/api/gdalrasterband_cpp.html> the region is selected by using XOff, YOff, XSize and YSize which map directly with -srcwin. But I was thinking that very soon you’ll ask about how to convert a georeferenced coordinates into pixel space, and gdal_translate supports that. So maybe you want to read the source code https://github.com/OSGeo/gdal/blob/master/apps/gdal_translate_lib.cpp. And meanwhile you can read and convert data from WMTS with gdal_translate, enjoy in seeing that it works, and be confident that you will be able to do the same and more with your own code. If programmers think that way, I do not know.

-Jukka-

Lähettäjä: Michał Kowalczuk <michkowalczuk at gmail.com>
Lähetetty: perjantai 31. toukokuuta 2024 16.32
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi>
Kopio: Even Rouault <even.rouault at spatialys.com>; gdal-dev at lists.osgeo.org
Aihe: Re: [gdal-dev] GDAL WMTS get tiles no documentation

Thank you Rukka,
Since -srcwin and -projwin are options only for gdal_translate utility, your suggestion leads me to use GDALTranslate() function with these parameters.

This means that I cannot use directly created WMTS dataset to retrieve data (using GDALDatasetRasterIO), but create a new (in-memory or file-based) for each map extent change.

And this is the standard way of doing it?
Do I understand it correctly?

Michal

pt., 31 maj 2024 o 15:13 Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> napisał(a):
Hi,

Before starting to program you can familiar yourself with the topic by using gdal_translate gdal_translate — GDAL documentation<https://gdal.org/programs/gdal_translate.html>.
Play with the -srcwin and -projwin / -projwin_srs options and gather some trust that is is possible to select and download any part of the large WMTS layer. Add “-- debug on” and you will see that GDAL downloads tiles separately just like QGIS. Actually, with WMTS it is the only way to download data. Maybe you have read the standard already https://portal.ogc.org/files/?artifact_id=35326. It is not necessary because GDAL knows what to do, but a review every now and then does not harm. Especially the TileMatrix and TileMatrixSet things are not self-evident, but they provide the answer to the question about how a WMTS client knows which row, col, and zoom to download.

Do not touch the DataWindow. It defines the extent of the whole WMTS layer and it is used for example if gdal_translate is run without spatial selection (-srcwin or -projwin). Touch in your future code things which are equivalent to -srcwin or -projwin of gdal_translate.

-Jukka Rahkonen-

Lähettäjä: Michał Kowalczuk <michkowalczuk at gmail.com<mailto:michkowalczuk at gmail.com>>
Lähetetty: perjantai 31. toukokuuta 2024 15.45
Vastaanottaja: Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>>
Kopio: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: Re: [gdal-dev] GDAL WMTS get tiles no documentation

Hi, thank you for your replies!
As I'm still a newbie especially with WMTS. I need to take a step back to the basics. Let's forget about multithreading for now.

  *   I'm in the stage that I can open a dataset using WMTS service path from data provider
  *   I can list subdatasets and open the selected layer as a new dataset using path from SUBDATASET_id_NAME.
  *   This dataset reports large height and width values, which is obvious.
  *   The first question is how to read any fragment from this dataset using GDALDatasetRasterIO.
Is there a similar way to SpatialFIlter from vector layers?
I see in the docs there is a <DataWindow> tag for a local service descriptions which defines extents of the data. Does it mean that for each map extent change I need to create a new local XML file and open a new dataset to read demanded bbox?
Is this the only way for doing it?
  *   From other site I noticed that QGIS downloads tiles separately? But how it knows which row/col and zoom should be used in GET request? Does it use GDAL for WMTS reading at all?
[cid:image001.png at 01DAB379.1058A2F0]
I will be grateful for any tip and trucks for a newbie ;-)

Regards,
Michal

śr., 29 maj 2024 o 12:41 Even Rouault <even.rouault at spatialys.com<mailto:even.rouault at spatialys.com>> napisał(a):

Hi,

The WMS cache (which is used underneath by the WMTS driver) writes each tiles in a separate file. So if you use multi-threaded, as long as you read in parallel different GDALDataset* object on the WMTS filename, and read disjoint areas, that should be fine. There might be a slight risk of things going bad if you happened to read the same tile from different GDALDataset* object at the same time, as it could potentially read a partially written cached tile, or concurrently write the tile in the cache and corrupt it. The logic is in frmts/wms/gdalwmscache.cpp

Even
Le 29/05/2024 à 10:14, Michał Kowalczuk via gdal-dev a écrit :
Because this is not clear in documentation, especially in WMTS driver docs, my question is strictly about this issue to programmers who have already explored this topic.

Regards
Michal

W dniu śr., 29.05.2024 o 10:04 Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>> napisał(a):
Hi,

No, I am not sure because I do not program myself. Reading this document https://gdal.org/user/multithreading.html makes me feel that it is possible to do some things in parallel, but the programmer must know how to do it right.

-Jukka Rahkonen-

Lähettäjä: Javier Jimenez Shaw <j1 at jimenezshaw.com<mailto:j1 at jimenezshaw.com>>
Lähetetty: keskiviikko 29. toukokuuta 2024 10.41
Vastaanottaja: Rahkonen Jukka <jukka.rahkonen at maanmittauslaitos.fi<mailto:jukka.rahkonen at maanmittauslaitos.fi>>
Kopio: Michał Kowalczuk <michkowalczuk at gmail.com<mailto:michkowalczuk at gmail.com>>; gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: Re: [gdal-dev] GDAL WMTS get tiles no documentation

On Wed, 29 May 2024 at 08:59, Rahkonen Jukka via gdal-dev <gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>> wrote:
Hi,

When you have a RasterBand  from the WMTS data source, it is abstracted and you can read the raster data just like from any other data source and raster band https://gdal.org/tutorials/raster_api_tut.html#reading-raster-data “There are a few ways to read raster data, but the most common is via the GDALRasterBand::RasterIO() method. This method will automatically take care of data type conversion, up/down sampling and windowing.” GDAL knows which tiles to read.

I do not know if the WMTS driver can do parallel tile downloads. If not, it is possible to run many RasterIO() at the same time, each reading data from a different window like in this rasterio document https://rasterio.readthedocs.io/en/latest/topics/concurrency.html.

Jukka, are you sure you can run several RasterIO in parallel (on the same dataset)? in GDAL GeoTIFF you cannot: the cache may be corrupted. In that case I open several datasets over the same file. I do not know about WMTS.

-Jukka Rahkonen-

Lähettäjä: gdal-dev <gdal-dev-bounces at lists.osgeo.org<mailto:gdal-dev-bounces at lists.osgeo.org>> Puolesta Michal Kowalczuk via gdal-dev
Lähetetty: keskiviikko 29. toukokuuta 2024 9.08
Vastaanottaja: gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
Aihe: [gdal-dev] GDAL WMTS get tiles no documentation

Hi GDAL fellows
This is my first post on this mailing list, so I'm asking for understanding.

As all we know the purpose of using WMTS over WMS, I'd like to implement parallel downloading tiles from service using C API.
In my opinion GDAL documentation (https://gdal.org/drivers/raster/wmts.html) says nothing on this topic.
I can get capabilities from WMTS, I can open the selected subdataset but how to get tiles for given extent? I could not find any information how to do it, even in the GDAL tests on github.

I am kindly asking for tips. It also can be in python. How using pure GDAL API fetch tiles to dynamically complete the displayed map. This is my goal.

Thank you!
Michal
_______________________________________________
gdal-dev mailing list
gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>
https://lists.osgeo.org/mailman/listinfo/gdal-dev

_______________________________________________

gdal-dev mailing list

gdal-dev at lists.osgeo.org<mailto:gdal-dev at lists.osgeo.org>

https://lists.osgeo.org/mailman/listinfo/gdal-dev

--

http://www.spatialys.com<http://www.spatialys.com/>

My software is free, but my time generally not.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240531/e26388f4/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 25778 bytes
Desc: image001.png
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20240531/e26388f4/attachment-0001.png>