<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
tt
{mso-style-priority:99;
font-family:"Courier New";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'>Yes, this works nicely. Thanks! Didn’t know about the ‘WITH’ statement<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> postgis-users [mailto:postgis-users-bounces@lists.osgeo.org] <b>On Behalf Of </b>Rémi Cura<br><b>Sent:</b> vrijdag 16 december 2016 18:33<br><b>To:</b> PostGIS Users Discussion <postgis-users@lists.osgeo.org><br><b>Subject:</b> Re: [postgis-users] Query choces on searching too small area<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal><span style='font-family:"Courier New"'>Hey, there is a dedicated "slow query" protocol on postgres user list, and its quite sane.<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Courier New"'>For instance, it would suggest you to give the version number you use, your hardware, etc etc.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>About you query, I guess your photo are geotagged (i.e. each photo is a point, and maybe you have a precision attribute).<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>Using ST_Intersects thus could be replaced by ST_DWithin with your precision / default tolerancey, which is safer.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>Index not kicking may have many causes, such as outdated statistics (have you vacuum analyse -ed often?), wrong config regarding your hardware (</span><tt><span style='font-size:10.0pt'>seq_page_cost, and so</span></tt><span style='font-family:"Courier New"'>),<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Courier New"'>poorly written query, etc.<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Courier New"'>Anyway your query should not freeze on only 500k geometries, so I'm also guessing that in the table "photo" you not only store photo point / geometry, but also the binary of the photo, which is bound to be dangerous.<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>So steps to fix your problem<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> - update postgres / postgis if you can<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> - check stats / vacuum<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> - check postgres config<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> - rewrite your query for a better form (see example 1 )<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> - post on list, this might be a bug<br> - rewrite query to force to perform first geometry test then the other (example query 2)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> - create a "proxy" photo_proxy table that contains only photo_id and photo.geometry<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Courier New"'> - force use of index via settings (usually a very bad idea)<br> - ...<br><br><o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>Here is how your query could be simplified :<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>SELECT <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>"id", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>"filename", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>ST_AsText("geometry") AS "geometry", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>ST_AsText("center") AS "center", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>"angle"<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>FROM "photo" <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>WHERE (<b>ST_DWithin</b>("geometry", st_GeomFromText( 'POINT(4.5063099203616 51.923602970634)', 4326)<b>,your_precision</b>)) <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>AND "id" IN (<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:18.7pt;text-indent:36.0pt'>SELECT DISTINCT "photoId" <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:18.7pt;text-indent:36.0pt'>FROM "photoMetadata" <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:54.7pt'>WHERE ("value" = 'KADASTER') AND ("key" = 'source'))) <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> AND ("photoId" IN (<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:48.5pt;text-indent:36.0pt'>SELECT DISTINCT "photoId" <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:48.5pt;text-indent:36.0pt'>FROM "photoMetadata" <o:p></o:p></p><div style='margin-left:90.0pt'><p class=MsoNormal><span style='font-family:"Courier New"'>WHERE (key = 'year' AND ( cast(value as int ) >= 1866 AND cast ( value as int ) <= 1981))<br>))<o:p></o:p></span></p></div><p class=MsoNormal><span style='font-family:"Courier New"'>ORDER BY "filename" LIMIT 36<o:p></o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Courier New"'><o:p> </o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-family:"Courier New"'>here is the query to force use of geometric index :<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>WITH photos_spatialy_close AS (<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> SELECT id AS photoId<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> FROM photo<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> WHERE <b>ST_DWithin</b>("geometry", st_GeomFromText( 'POINT(4.5063099203616 51.923602970634)', 4326)<b>,your_precision</b>)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> LIMIT 36<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>, photo_with_correct_metatadata AS (<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> <br> SELECT DISTINCT "photoId" <br> FROM "photoMetadata" <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> WHERE ("value" = 'KADASTER') <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> AND ("key" = 'source'))) <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> AND ("photoId" IN (<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>SELECT DISTINCT "photoId" <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>FROM "photoMetadata" <o:p></o:p></p><p class=MsoNormal><span style='font-family:"Courier New"'> WHERE (key = 'year' <br> AND ( cast(value as int ) >= 1866 <br> AND cast ( value as int ) <= 1981))<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>, keeping_photo_id_in_both_set AS (<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> SELECT photoId<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> FROM photos_spatialy_close<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> INTERSECTS<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> SELECT photoId<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'> FROM photo_with_correct_metatadata<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>)<o:p></o:p></span></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>SELECT <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>"id", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>"filename", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>ST_AsText("geometry") AS "geometry", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>ST_AsText("center") AS "center", <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;text-indent:36.0pt'>"angle"<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>FROM keeping_photo_id_in_both_set LEFT OUTER JOIN photo ON ( photoId = id) <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>LIMIT ...<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> </o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> </o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><o:p> </o:p></p></div><div><p class=MsoNormal><span style='font-family:"Courier New"'>Cheers<br>Remi-C<o:p></o:p></span></p></div></div><div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>2016-12-16 17:48 GMT+01:00 Sandro Santilli <<a href="mailto:strk@kbt.io" target="_blank">strk@kbt.io</a>>:<o:p></o:p></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><p class=MsoNormal>On Fri, Dec 16, 2016 at 03:33:32PM +0100, Arjen Haayman wrote:<br>> QUERY PLAN<br><br>[...]<br><br>> What does this tell you?<br><br>That your query is too complex ?<br>Check out <a href="http://explain.depesz.com" target="_blank">http://explain.depesz.com</a><o:p></o:p></p><div><div><p class=MsoNormal><br>--strk;<br>_______________________________________________<br>postgis-users mailing list<br><a href="mailto:postgis-users@lists.osgeo.org">postgis-users@lists.osgeo.org</a><br><a href="http://lists.osgeo.org/mailman/listinfo/postgis-users" target="_blank">http://lists.osgeo.org/mailman/listinfo/postgis-users</a><o:p></o:p></p></div></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>