<html><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8"
data-hsystem="true"></head><body><style>p{margin: 0;padding: 0;}</style><p>yes,
true.. but with a thorough read you might notice that the gdal_retile.py
experiment was largely ineffective, </p><p>but if you click on the link at the
top to the *next post* </p><h2 class="posttitle"> Variable Buffers in
PostGIS</h2><p>you will find the one that really worked well.. in fact, we used
that 2nd post in production for months, to great effect.</p><p>The trick on one
machine was to split to work by some constant, and then make psycopg2
connections for each "bucket."</p><p><br></p><p>This worked very well..
</p><p><br></p><p>Since then I have experimented only a tiny bit with SPARK from
the Berkeley Amp Lab for a distributed work load on a Hadoop file system, but
that world has no GEOS (yet) </p><p><br></p><p>--</p><p>Brian M
Hamlin</p><p>OSGeo California
Chapter</p><p>blog.light42.com</p><p><br></p><p><br><br>On Wed, 16 Oct 2013
17:28:27 -0700, Bborie Park <dustymugs@gmail.com>
wrote:<br></p><blockquote dir="ltr" style="border-left: 2px solid rgb(0, 0, 0);
padding-right: 0px; padding-left: 5px; margin-left: 5px; margin-right: 0px;"
_mce_style="border-left: 2px solid #000000; padding-right: 0px; padding-left:
5px; margin-left: 5px; margin-right: 0px;"><div id="html-message"><div
dir="ltr">Your best bet is to consider splitting the workload among several
postgresql connections.<div><br></div><div>darkblueb had a blog post about
this...</div><div><br></div><div><a
href="http://blog.light42.com/wordpress/?p=23"
target="">http://blog.light42.com/wordpress/?p=23</a><br></div></div><div
class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div
class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 16, 2013 at 5:21
PM, Brent Wood <span dir="ltr"><<a href="mailto:pcreso@pcreso.com"
target="_blank">pcreso@pcreso.com</a>></span> wrote:<br><blockquote
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex" _mce_style="margin: 0 0 0 .8ex; border-left: 1px #ccc
solid; padding-left: 1ex;"><div><div style="font-size:12pt;font-family:times new
roman,new york,times,serif" _mce_style="font-size: 12pt; font-family: times new
roman,new york,times,serif;"><div>Hi, <br></div><div><br></div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;">
Any advice appreciated!!</div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;"><br></div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;">
I'm undertaking a spatial analysis using Postgis (what else would I use!!!). The
first part works well.<br></div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;">
<br></div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;">I take a large number (potentially millions) of lines defined
by start & end points & buffer them to create polygons. (I'm working in
lat/long EPSG:4326 but transforming to a custom equal area projection for the
buffering operation).</div>
<div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;"><br></div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;">
I generate a grid of 5x5km cells (polygons) covering the region of
interest.</div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;"><br></div><div
style="font-style:normal;font-size:16px;background-color:transparent;font-family:times
new roman,new york,times,serif" _mce_style="font-style: normal; font-size: 16px;
background-color: transparent; font-family: times new roman,new
york,times,serif;">
I clip the line based polygons to the grid, so I can generate statistics for
each cell describing the lines that intersect with it, various quantitative
measures such as ST_Union() the clipped line polygons to generate a footprint in
each cell to work out how much is/is not covered, or sum the ST_Area() of the
clipped polygons grouped by cell to calculate an aggregate cover, which can be
several times the actual cell area.<br></div><div style="font-family:times new
roman,new york,times,serif;font-size:12pt" _mce_style="font-family: times new
roman,new york,times,serif; font-size: 12pt;"><div style="font-family:times new
roman,new york,times,serif;font-size:12pt" _mce_style="font-family: times new
roman,new york,times,serif; font-size: 12pt;"><br><br>So far so good, it works
well, the code is clear & transparent & provides a good result. At least
as good as any commercial software can do. My test
data subset is processed from scratch in about 30 minutes.<br></div> </div>
<span><br>Now I want to ST_Union() all the cell based polygons into an overall
single multipolygon representing the footprint. The code is simple. The
performance, even with my subset, is a problem.<br><br>I have thousands of
cell based footprint multipolygons, each potentially with thousands of vertices
to be ST_Union()ed. Runtime is weeks for an iteration. If I need separate total
footprints for 20 different species annually for 5 years, that is 100
iterations. Memory & I/O use is minimal - it is totally cpu bound.<br><br>I
am looking at trying to simplify the polygons to be unioned to reduce the number
of vertices (& hence processing) involved, but to achieve any significant
benefit I'm having to change the shape of the polygons to ST_Union() too much.
<br><br><br><br>Does anyone have any suggestions as to how this could be made
significantly faster? <br>If I
had $$ to throw at developers to work on the codebase (presumably GEOS?) could
performance be significantly improved?<br><br><br>Thanks,<br><br> Brent
Wood</span><div></div></div></div><br>_______________________________________________<br>
postgis-devel mailing list<br><a href="mailto:postgis-devel@lists.osgeo.org"
target="">postgis-devel@lists.osgeo.org</a><br><a
href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel"
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br></blockquote></div><br></div>
</div>
<br><hr><br>
_______________________________________________<br>
postgis-devel mailing list<br>
postgis-devel@lists.osgeo.org<br><a
href="http://webmail.light42.com/hwebmail/services/go.php?url=http%3A%2F%2Flists.osgeo.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpostgis-devel"
_mce_href="../hwebmail/services/go.php?url=http%3A%2F%2Flists.osgeo.org%2Fcgi-bin%2Fmailman%2Flistinfo%2Fpostgis-devel"
target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a></blockquote><p><br><br></p><p><br></p><p><br></p></body></html>