<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7653.38">
<TITLE>RE: [postgis-users] Help with Bad Query Plan</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<P><FONT SIZE=2>Oliver,<BR>
<BR>
Disregard my last comment - I see you did try plain &&. So as expected it is faster<BR>
than ST_Intersects and given your geometries are so large, this is not surprising. Yes it && should be slower than doing a btree search too.<BR>
<BR>
I think Paul is on to something. for those large geometries, I think you will need a lot of memory and with non-prepared (pre geos 3.1) that asia would be copied for each loop. So if you are still at your default postgresql settings, then its no wonder you have problems. You could try using ST_DWithin instead of ST_Intersects.<BR>
<BR>
Somethink like ST_DWithin(a.the_geom, b.the_geom, 0.001)<BR>
<BR>
ST_DWithin pre 3.1 tended to perform better with large multipolygons than ST_Intersects. 3.1 I would say ST_Intersects would always win out. <BR>
<BR>
Now why having all indexes utilized actually makes things worse sometimes (hmm I think that is what I saw) is a bit mysterious except as Paul suggests -- the spatial index is making the spatial search more attractive looking to the planner than it should and looking at the plan (at least for &&, it is being applied first). Also trying to utilize more indexes would require more memory -- so could be your memory settings on postgresql.conf are just too low.<BR>
<BR>
Yes text Plans are hard to read -- I do much better reading graphical ones and hmm I think I've pretty much forgotten how to read the text ones aside from checking whether an index is used or not, the strategy, actual vs. relative row count, and flipping the plan upside down to get a sense of sequencing.<BR>
<BR>
Did we ever ask the obvious question of which Geos are you using?<BR>
<BR>
SELECT postgis_full_version();<BR>
<BR>
Hope that helps,<BR>
Regina<BR>
<BR>
<BR>
<BR>
-----Original Message-----<BR>
From: postgis-users-bounces@postgis.refractions.net on behalf of Paul Ramsey<BR>
Sent: Fri 1/9/2009 1:54 PM<BR>
To: PostGIS Users Discussion<BR>
Subject: Re: [postgis-users] Help with Bad Query Plan<BR>
<BR>
I just tried to load up the geometry portion of the problem only, and<BR>
I am *not* seeing the same effect. Does the geometry-only part of the<BR>
query also exhibit the long query time for you?<BR>
<BR>
select count(*) from gdors_geography g1, gdors_geography g2 where<BR>
st_intersects(g1.the_geom, g2.the_geom) and g2.gid = 3;<BR>
<BR>
With an index this takes me 9s and without it takes 15s (hooray,<BR>
prepared geometry, this would probably have been a multi-minute<BR>
problem in the good old days).<BR>
<BR>
If you're I/O bound that *could* be part of the problem, however, for<BR>
something like Asia. It's a big geometry, it's stored in toast tuples,<BR>
if it's too big to cache, it would be a lot of work to haul it out<BR>
over and over and over again. Still, that should pertain for the<BR>
non-indexed cases too. Is the index forcing a full spatial table join<BR>
to happen before the individual record is pulled (I can't read plans,<BR>
it's tragic)?<BR>
<BR>
P.<BR>
<BR>
On Fri, Jan 9, 2009 at 10:45 AM, Paragon Corporation <lr@pcorp.us> wrote:<BR>
> Oliver,<BR>
><BR>
> Did you try the && instead of ST_Intersects. That would help confirm if its<BR>
> an issue with && or _ST_Intersects.<BR>
><BR>
> What is strange is that in all the plans, it looks like its doing the right<BR>
> thing.<BR>
><BR>
> Did you change your postgresql.conf or is it still at its default settings.<BR>
> If still at its default you could just be IO bound and need to up those<BR>
> settings.<BR>
><BR>
> -----Original Message-----<BR>
> From: postgis-users-bounces@postgis.refractions.net<BR>
> [<A HREF="mailto:postgis-users-bounces@postgis.refractions.net">mailto:postgis-users-bounces@postgis.refractions.net</A>] On Behalf Of Oliver<BR>
> Snowden<BR>
> Sent: Friday, January 09, 2009 12:44 PM<BR>
> To: postgis-users@postgis.refractions.net<BR>
> Subject: RE: [postgis-users] Help with Bad Query Plan<BR>
><BR>
> Hi Mark/Regina, I have installed PostgreSQL 8.2. Unfortunately that is<BR>
> still slow. I am not sure how difficult it is for you to recreate the<BR>
> database but I have embedded some Java code to create some sample report<BR>
> data, should you want to/have time. 10000 refers to the report entries to<BR>
> make, [21619] + 1 refers to the number of geometries.<BR>
><BR>
> All the best, Oliver.<BR>
><BR>
> -- Installed PostgreSQL 8.2.9-1<BR>
><BR>
> -- Query did not finish<BR>
> SELECT geolink.report_id, geography.gid<BR>
> FROM gdors_geolink geolink, gdors_geography geography, gdors_geography<BR>
> selected_geography WHERE geolink.temp_report_date BETWEEN '2008-01-06' AND<BR>
> '2009-01-06'<BR>
> AND geolink.geom_id = geography.gid<BR>
> AND selected_geography.gid=3<BR>
> AND ST_Intersects(selected_geography.the_geom, geography.the_geom); "Nested<BR>
> Loop (cost=0.00..91.85 rows=1 width=8)"<BR>
> " -> Nested Loop (cost=0.00..16.56 rows=1 width=4)"<BR>
> " Join Filter: _st_intersects(selected_geography.the_geom,<BR>
> geography.the_geom)"<BR>
> " -> Index Scan using gdors_geography_pkey on gdors_geography<BR>
> selected_geography (cost=0.00..8.27 rows=1 width=3470)"<BR>
> " Index Cond: (gid = 3)"<BR>
> " -> Index Scan using gdors_geography_the_geom on gdors_geography<BR>
> geography (cost=0.00..8.27 rows=1 width=3474)"<BR>
> " Index Cond: (selected_geography.the_geom &&<BR>
> geography.the_geom)"<BR>
> " Filter: (selected_geography.the_geom && geography.the_geom)"<BR>
> " -> Index Scan using gdors_geolink_pkey on gdors_geolink geolink<BR>
> (cost=0.00..75.28 rows=1 width=8)"<BR>
> " Index Cond: (geolink.geom_id = geography.gid)"<BR>
> " Filter: ((temp_report_date >= '2008-01-06'::date) AND<BR>
> (temp_report_date <= '2009-01-06'::date))"<BR>
><BR>
> -- Query with && - at least we get a result...although slower than without<BR>
> the spatial index.<BR>
> -- 81 rows, ~21000ms.<BR>
> SELECT geolink.report_id, geography.gid<BR>
> FROM gdors_geolink geolink, gdors_geography geography, gdors_geography<BR>
> selected_geography WHERE geolink.temp_report_date BETWEEN '2008-01-06' AND<BR>
> '2009-01-06'<BR>
> AND geolink.geom_id = geography.gid<BR>
> AND selected_geography.gid=3<BR>
> AND selected_geography.the_geom && geography.the_geom; "Hash Join<BR>
> (cost=16.61..231.58 rows=1 width=8)"<BR>
> " Hash Cond: (geolink.geom_id = geography.gid)"<BR>
> " -> Seq Scan on gdors_geolink geolink (cost=0.00..214.00 rows=257<BR>
> width=8)"<BR>
> " Filter: ((temp_report_date >= '2008-01-06'::date) AND<BR>
> (temp_report_date <= '2009-01-06'::date))"<BR>
> " -> Hash (cost=16.56..16.56 rows=4 width=4)"<BR>
> " -> Nested Loop (cost=0.00..16.56 rows=4 width=4)"<BR>
> " -> Index Scan using gdors_geography_pkey on gdors_geography<BR>
> selected_geography (cost=0.00..8.27 rows=1 width=3470)"<BR>
> " Index Cond: (gid = 3)"<BR>
> " -> Index Scan using gdors_geography_the_geom on<BR>
> gdors_geography geography (cost=0.00..8.27 rows=1 width=3474)"<BR>
> " Index Cond: (selected_geography.the_geom &&<BR>
> geography.the_geom)"<BR>
> " Filter: (selected_geography.the_geom &&<BR>
> geography.the_geom)"<BR>
><BR>
> Sample report data:<BR>
><BR>
> package samplereportdata;<BR>
><BR>
> import java.text.SimpleDateFormat;<BR>
> import java.util.Date;<BR>
><BR>
> // quick hack<BR>
> public class Main {<BR>
><BR>
> private static void getSample() {<BR>
> java.util.Random r = new java.util.Random();<BR>
> long timeNow = java.util.Calendar.getInstance().getTimeInMillis();<BR>
><BR>
> Date date = new Date();<BR>
> SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd");<BR>
> String sDate = "";<BR>
><BR>
> for (int i = 0; i < 10000; i++) {<BR>
> int j = r.nextInt(21619) + 1;<BR>
> r.nextLong();<BR>
> date.setTime(new Float(timeNow * r.nextFloat()).longValue());<BR>
> sDate = sdf.format(date);<BR>
><BR>
> System.out.println("INSERT INTO gdors_geolink(report_id,<BR>
> geom_id, lastupdated, temp_report_date) VALUES<BR>
> ("+(i+1)+","+j+",'2006-06-01', '"+sDate +"');");<BR>
> }<BR>
> }<BR>
><BR>
> /**<BR>
> * @param args the command line arguments<BR>
> */<BR>
> public static void main(String[] args) {<BR>
> getSample();<BR>
> }<BR>
> }<BR>
><BR>
> _______________________________________________<BR>
> postgis-users mailing list<BR>
> postgis-users@postgis.refractions.net<BR>
> <A HREF="http://postgis.refractions.net/mailman/listinfo/postgis-users">http://postgis.refractions.net/mailman/listinfo/postgis-users</A><BR>
><BR>
><BR>
><BR>
> _______________________________________________<BR>
> postgis-users mailing list<BR>
> postgis-users@postgis.refractions.net<BR>
> <A HREF="http://postgis.refractions.net/mailman/listinfo/postgis-users">http://postgis.refractions.net/mailman/listinfo/postgis-users</A><BR>
><BR>
_______________________________________________<BR>
postgis-users mailing list<BR>
postgis-users@postgis.refractions.net<BR>
<A HREF="http://postgis.refractions.net/mailman/listinfo/postgis-users">http://postgis.refractions.net/mailman/listinfo/postgis-users</A><BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
</FONT>
</P>
</BODY>
</HTML>
<HTML><BODY><P><hr size=1></P>
<P><STRONG>
The substance of this message, including any attachments, may be confidential, legally privileged and/or exempt from disclosure pursuant to Massachusetts law. It is intended solely for the addressee. If you received this in error, please contact the sender and delete the material from any computer.
</STRONG></P></BODY></HTML>
<P><hr size=1></P>
<P><STRONG><font size="2" color="339900"> Help make the earth a greener place. If at all possible resist printing this email and join us in saving paper. </p> <p> </font></STRONG></P>