<html><head><style>body{font-family:Helvetica,Arial;font-size:13px}</style></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">Basically, if your queries don’t need to find tiles that neighbor each other, you can distribute them “randomly” around the cluster, which means any given roll-up query can heat up every node at once, for maximum use of the cluster. That’s the way you want it to work. If you’re going to have multiple layers of rasters and do interactions between them (basic map algebra, etc) you can still get good efficiency just by ensuring that they are tiled on exactly the same basis, so any given tile will line up with all its friends from other layers. If, however, you’re going to need to run operations that need to find neighboring tiles (cost surfaces, slope calculations) then having the tiles randomly spread around won’t work so well.</div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;"><br></div><div id="bloop_customfont" style="font-family:Helvetica,Arial;font-size:13px; color: rgba(0,0,0,1.0); margin: 0px; line-height: auto;">For an OLAP server, being able to shard to that every node gets used during a query is optimal, particularly since, if you shard based on spatial location you’ll end up naturally with bottlenecks on hot nodes. (Like, you will tend to roll up on spatial areas, which means only the node that serves that area will end up hot during the query, and the rest of the nodes will be idle: the absolute worst utilization situation.)</div> <div id="bloop_sign_1411580084566222080" class="bloop_sign">
<title></title>
<div>
<br>
</div><div>P</div><div><br></div>
-- <br>
Paul Ramsey<br>
http://cleverelephant.ca<div>http://postgis.net
</div>
<script type="application/ld+json">
{
"@context": "http://schema.org",
"@type": "em",
"name": "John Doe",
"jobTitle": "Graduate research assistant",
"affiliation": "University of Dreams",
"additionalName": "Johnny",
"url": "http://www.example.com",
"address": {
"@type": "PostalAddress",
"streetAddress": "1234 Peach Drive",
"addressLocality": "Wonderland",
"addressRegion": "Georgia"
}
}
</script>
</div> <br><p style="color:#000;">On September 24, 2014 at 6:26:26 AM, David Haynes II (<a href="mailto:dahaynes@umn.edu">dahaynes@umn.edu</a>) wrote:</p> <blockquote type="cite" class="clean_bq"><span><div><div></div><div>
<title></title>
<div dir="ltr">
<div>Not sure if I understand exactly what you mean by "spatial
correlation"</div>
All of the queries will be using spatial defined queries like
ST_Intersects. The strategy that I am thinking of employing,
basically divides the world into N zones, similar to UTM projection
grids. Each zone will be exist have a defined spatial extent that
will use "inheritance" / hashkey to identify this spatial extent. I
am very new to this area as this is outside my expertise. If there
is some literature you can point me to that might explain it better
I would appreciate that. It would allow me to present the problem
is a better way.
<div><br></div>
<div><a href="https://wiki.postgresql.org/wiki/HashTable">https://wiki.postgresql.org/wiki/HashTable</a><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Sep 23, 2014 at 4:14 PM, Paul
Ramsey <span dir="ltr"><<a href="mailto:pramsey@cleverelephant.ca" target="_blank">pramsey@cleverelephant.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
If you’re just doing queries that don’t take advantage of spatial
correlation at all, using any hashkey on the contents as the
sharding key should work just great. Then you can use pl/proxy to
run roll-ups against spatial polygons, etc, easily and know you
won’t end up with hot nodes.</div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
<br></div>
<div style="font-family:Helvetica,Arial;font-size:13px;color:rgba(0,0,0,1.0);margin:0px;line-height:auto">
P</div>
<div>
<div><br></div>
-- <br>
Paul Ramsey<br>
<a href="http://cleverelephant.ca" target="_blank">http://cleverelephant.ca</a>
<div><a href="http://postgis.net" target="_blank">http://postgis.net</a></div>
</div>
<br>
<p style="color:#000">On September 23, 2014 at 2:11:18 PM, David
Haynes II (<a href="mailto:dahaynes@umn.edu" target="_blank">dahaynes@umn.edu</a>) wrote:</p>
<blockquote type="cite">
<div>
<div>
<div dir="ltr"><span>Yes, we are going to use the more finished
product Postgres-xl, but even that is not production
ready. </span>
<div><span>So I would think the most logical solution is to
generate the process ourselves.</span></div>
<div><span><br></span></div>
<div><span>Mostly wondering how to distribute the tables to various
locations, once I do the difficulty part of aligning tiles to exist
with a defined coordinate space.</span></div>
</div>
<div class="gmail_extra"><span><br></span>
<div class="gmail_quote"><span>On Mon, Sep 22, 2014 at 11:12 AM,
Rémi Cura <span dir="ltr"><<a href="mailto:remi.cura@gmail.com" target="_blank">remi.cura@gmail.com</a>></span>
wrote:<br></span>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Do you mean somethiong like postgres-xc :<a href="http://postgresxc.wikia.com/wiki/Postgres-XC_Wiki" target="_blank">http://postgresxc.wikia.com/wiki/Postgres-XC_Wiki</a><br>
<br></div>
Cheers,<br>
Rémi-C<br></div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2014-09-22 17:04 GMT+02:00 David Haynes II
<span dir="ltr"><<a href="mailto:dahaynes@umn.edu" target="_blank">dahaynes@umn.edu</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hello,
<div><br></div>
<div>I was wondering if there any helpful examples for distributing
rasters tables in postgis. Are there other items that I need to
consider? Will the PostGIS functions need to be re-written if we
want to parallelize the processing, using something like
pg_proxy? </div>
<div><br></div>
<div>I came an example using a different software package, (slide
16)</div>
<div><a href="http://spark-summit.org/wp-content/uploads/2014/07/Geotrellis-Adding-Geospatial-Capabilities-to-Spark-Ameet-Kini-Rob-Emanuele.pdf" target="_blank">http://spark-summit.org/wp-content/uploads/2014/07/Geotrellis-Adding-Geospatial-Capabilities-to-Spark-Ameet-Kini-Rob-Emanuele.pdf</a></div>
<div>I would prefer to stay within postgresql
community.<span><font color="#888888"><span><font color="#888888"><br clear="all"></font></span></font></span>
<div><span><font color="#888888"><font color="#888888"><br></font></font></span></div>
<span><font color="#888888"><font color="#888888">--<br></font></font></span>
<div dir="ltr"><span><font color="#888888"><font color="#888888">David Haynes, Ph.D.</font></font></span>
<div><span><font color="#888888"><font color="#888888">Research
Associate Terra Populous</font></font></span></div>
<div><span><font color="#888888"><font color="#888888">Minnesota
Population Center</font></font></span></div>
</div>
</div>
</div>
<span><font color="#888888"><br>
_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org" target="_blank">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br>
</font></span></blockquote>
</div>
<br></div>
<br>
_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org" target="_blank">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><span class="HOEnZb"><font color="#888888"><br></font></span></div>
<span class="HOEnZb"><font color="#888888">--<br></font></span>
<div dir="ltr"><span class="HOEnZb"><font color="#888888">David
Haynes, Ph.D.</font></span>
<div><span class="HOEnZb"><font color="#888888">Research Associate
Terra Populous</font></span></div>
<div><span class="HOEnZb"><font color="#888888">Minnesota
Population Center</font></span></div>
</div>
</div>
<span class="HOEnZb"><font color="#888888">_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org" target="_blank">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a></font></span></div>
</div>
</blockquote>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<div><br></div>
--<br>
<div dir="ltr">David Haynes, Ph.D.
<div>Research Associate Terra Populous</div>
<div>Minnesota Population Center</div>
</div>
</div>
</div></div></span></blockquote></body></html>