<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.Shkpostityyli18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.Shkpostityyli19
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 2.0cm 70.85pt 2.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Even,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">When Geopackage or Spatialite db are created with ogr2ogr there is also ogc_fid that could be used for fast range queries.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I decided already not to bother and to forget the whole thing because I was just playing with Geopackage without a direct
use case. But then I was reading more and it seems that LIMIT + OFFSET is slow also in PostgreSQL for the same reason. What I see around line 2358 in
<a href="https://github.com/mapserver/mapserver/blob/branch-7-0/mapogr.cpp">https://github.com/mapserver/mapserver/blob/branch-7-0/mapogr.cpp</a> makes me think that those are used by Mapserver and that paging with cursor
<a href="https://www.postgresql.org/docs/9.1/static/sql-declare.html">https://www.postgresql.org/docs/9.1/static/sql-declare.html</a> could be faster with PostGIS. What do you think?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">-Jukka-<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Even Rouault wrote:<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt"><span style="font-size:9.0pt;font-family:"Courier New"">On mardi 20 décembre 2016 10:36:55 CET Rahkonen Jukka (MML) wrote:<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> Hi,<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">>
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> It seems that because of how SQLite works it is not possible to make fast<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> paging for big tables by using simple SQL with LIMIT and OFFSET. Making a<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> query as "LIMIT 100 OFFSET 1000000" makes SQLite to read 1000100 rows and<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> throw away the first million ones. The bigger the offset the slower the<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> query.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">>
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> I guess that Mapserver must still use LIMIT and OFFSET because they work<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> even with combined attribute and spatial filters. But I wonder if it could<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> be possible to make a WFS 2.0 stored query for a not so uncommon use case<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> of loading pages without other filters from very big tables. This stored<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> query would also take count and startindex as inputs but place them into a<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> query like<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">>
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> SELECT * FROM table WHERE rowid>startindex AND rowid<=(startindex+count)<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">>
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> An optional parameter could be used for defining some other unique, numeric<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> and indexed attribute for paging. This query should work very fine with a<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> SQLite table that is freshly created for example with ogr2ogr because then<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> rowids start from one and they do not have gaps. If table has been edited<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> there may be missing rowids but that is not critical because query would<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> just return sometimes less rows than "count".<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">>
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> Unfortunately by reading<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New"">>
</span><span style="font-size:9.0pt;font-family:"Courier New""><a href="http://mapserver.org/ogc/wfs_server.html#stored-queries-wfs-2-0"><span lang="EN-US">http://mapserver.org/ogc/wfs_server.html#stored-queries-wfs-2-0</span></a></span><span lang="EN-US" style="font-size:9.0pt;font-family:"Courier New"">
it is not<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> obvious for me how to write such "GetPageByRowid" query and I would<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">> appreciate getting some help.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">I'm not completely sure it is legal to reuse the standard STARTINDEX and COUNT parameters as query parameters, but that would
probably work in the current implementation. But you'll be stuck with the addition of the startindex and count. There's no way to do that in a Filter understood by MapServer. You could however have a startindex and lastindex params. You would need to have
the rowid column explictly exposed as a field with something like DATA "SELECT rowid as the_rowid, * FROM table". Hum but then you'd get in the slow path since that wouldn't be recognized as a regular OGR layer.<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">Even<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-paragraph-type:empty;-qt-block-indent:0">
<span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">--
<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New"">Spatialys - Geospatial professional services<o:p></o:p></span></p>
<p style="margin:0cm;margin-bottom:.0001pt;-qt-block-indent:0;-qt-user-state:0"><span style="font-size:9.0pt;font-family:"Courier New""><a href="http://www.spatialys.com">http://www.spatialys.com</a><o:p></o:p></span></p>
</div>
</body>
</html>