<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML-esimuotoiltu Char";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";
mso-fareast-language:FI;}
span.HTML-esimuotoiltuChar
{mso-style-name:"HTML-esimuotoiltu Char";
mso-style-priority:99;
mso-style-link:HTML-esimuotoiltu;
font-family:Consolas;
mso-fareast-language:EN-US;}
span.Shkpostityyli22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="#0563C1" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US">At least in this case having </span><span lang="EN-US">INITIAL_REQUEST_PAGE_SIZE=number option would resolve the practical problem. We have collections with millions of features and therefore large page size is essential
when downloading the whole collection. On the other hand we have complicated geometries like lake polygons and reading 10000 large geometries for resolving the schema is pretty heavy. And we have 126 collections in the service that makes 126 x 10000 features
read for resolving the schemas if the aim is to clip a small area from all collections into GeoPackage.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">-Jukka Rahkonen-</span><span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="mso-fareast-language:FI">Lähettäjä:</span></b><span style="mso-fareast-language:FI"> Even Rouault <even.rouault@spatialys.com>
<br>
<b>Lähetetty:</b> maanantai 27. syyskuuta 2021 16.47<br>
<b>Vastaanottaja:</b> Rahkonen Jukka (MML) <jukka.rahkonen@maanmittauslaitos.fi>; 'gdal-dev@lists.osgeo.org' <gdal-dev@lists.osgeo.org><br>
<b>Aihe:</b> Re: [gdal-dev] Does OAPIF paging work as supposed?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Jukka,<span style="mso-fareast-language:FI"><o:p></o:p></span></p>
<p>your analysis is completely correct. Whether this is expected or not probably depends on situations. Should we have a INITIAL_REQUEST_PAGE_SIZE=number open option to overload the number of features to retrieve specifically in the first request... ??<o:p></o:p></p>
<p>Regarding the spatial filter, it is passed through the OGR API generally after having queried the schema, and for most OGR datasources it wouldn't influence the schema, so there isn't much that can be done here, except maybe adding a BBOX=west,south,east,north
open option.<o:p></o:p></p>
<p>One option to avoid both issues would be for the service to publish DescribedBy links at the collection level that would point to a XML schema (using a GML Simple Feature schema profile, such as the one understood by the GML driver) or a JSON schema (not
"too" complicated too). Both are handled by the driver.<o:p></o:p></p>
<p>Even<o:p></o:p></p>
<div>
<p class="MsoNormal">Le 27/09/2021 à 15:21, Rahkonen Jukka (MML) a écrit :<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span lang="EN-US">Hi,</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">I tried to read a relatively small BBOX from an OAPIF server but the process feels rather slow and I do not quite understand what I am seeing in the log.</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">ogr2ogr -f GPKG test.gpkg OAPIF:<a href="https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/?api-key=xxxx">https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/?api-key=xxxx</a>
-spat 25 65 25.1 65.1 -oo PAGE_SIZE=10000 --debug on --config cpl_curl_verbose yes</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Excerpts from the log:</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">HTTP: Fetch(<a href="https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key=xxxx&f=json&limit=10000">https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key=xxxx&f=json&limit=10000</a>)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">HTTP: These HTTP headers were set: Accept: application/geo+json, application/json</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">…</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">> GET /maastotiedot/features/v1/collections/osoitepiste/items?api-key=xxxx&f=json&limit=10000 HTTP/1.1</span><o:p></o:p></p>
<p class="MsoNormal">Host: avoin-paikkatieto.maanmittauslaitos.fi<o:p></o:p></p>
<p class="MsoNormal">Accept-Encoding: gzip<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Accept: application/geo+json, application/json</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">* Mark bundle as not supporting multiuse</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">< HTTP/1.1 200 OK</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">…</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">GeoJSON: First pass: 56.54 %</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">GeoJSON: First pass: 100.00 %</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">HTTP: Fetch(<a href="https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key=xxxx&f=json&limit=10000&bbox=25,65,25.1000000000000014,65.0999999999999943">https://avoin-paikkatieto.maanmittauslaitos.fi/maastotiedot/features/v1/collections/osoitepiste/items?api-key=xxxx&f=json&limit=10000&bbox=25,65,25.1000000000000014,65.0999999999999943</a>)</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">…</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">> GET /maastotiedot/features/v1/collections/osoitepiste/items?api-key=xxxx&f=json&limit=10000&bbox=25,65,25.1000000000000014,65.0999999999999943 HTTP/1.1</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">…</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">< Content-Length: 309</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">GDALVectorTranslate: 0 features written in layer 'osoitepiste'</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">Do I read right that GDAL is first reading one page, in this time 10000 features without BBOX, perhaps for resolving the schema, and then makes a new query with BBOX? In this case the BBOX query finds nothing. Reading
10000 features on the first round and then discarding everything feels too expensive. Could it be enough to read for example 10 features that is the default page size on the first round instead of the full page?</span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US"> </span><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-US">-Jukka Rahkonen-</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:FI"><br>
<br>
<o:p></o:p></span></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>gdal-dev mailing list<o:p></o:p></pre>
<pre><a href="mailto:gdal-dev@lists.osgeo.org">gdal-dev@lists.osgeo.org</a><o:p></o:p></pre>
<pre><a href="https://lists.osgeo.org/mailman/listinfo/gdal-dev">https://lists.osgeo.org/mailman/listinfo/gdal-dev</a><o:p></o:p></pre>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></pre>
<pre>My software is free, but my time generally not.<o:p></o:p></pre>
</div>
</body>
</html>