hey thanks both for the answers and insights.<div>S3 seems cool, and is on my personal roadmap for GWC.</div><div>But I'm still looking at/experimenting with alternate storage mechanisms for the tiles on a regular disk so it's not that hard to manage, and am planning also to deal with the ocean is blue issue. So far it doesn't look like computing a SHA1 hash on the blob adds too much overhead, and certainly adds less than the overhead (GWC) imposes on a couple things that I'm working on too (like 1-1 FileNotFoundException thrown when reseeding, and optimizing the computation of the tile location)</div>
<div>But yeah, I'm mixing topics, it's 7:30 am and still working on improvements.</div><div><br></div><div>Cheers,</div><div>Gabriel<br><div class="gmail_quote">On Fri, Feb 18, 2011 at 6:56 AM, Arne Kepp <span dir="ltr"><<a href="mailto:arne@tiledmarble.org">arne@tiledmarble.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="im">On 2/18/11 10:41 AM, Frederic Junod wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It worked but we realized that the tiles directories where impossible<br>
to manage: it took more than 2 days to get the dir size with 'du' and<br>
rsync never finish (> 3 days to compute the files to transfer).<br>
</blockquote></div>
A couple of tips:<br>
<br>
Old versions of rsync try to build a file list in memory, which is hopeless. If you have rsync 3.0.0 or better (on both sides) it should be fine and start transfering almost immediately. But netcat'ing tar is more efficient.<br>
<br>
Some filsystems do accounting on the directories, so du can be fast. But that multiplies the overhead when writing new tiles, since 4-5 directories have to be updated. Better to just have tiles (or layers, even) on a dedicated partition and use df -h.<br>
<font color="#888888">
<br>
-Arne</font><div><div></div><div class="h5"><br>
<br>
_______________________________________________<br>
Tiling mailing list<br>
<a href="mailto:Tiling@lists.osgeo.org" target="_blank">Tiling@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/mailman/listinfo/tiling" target="_blank">http://lists.osgeo.org/mailman/listinfo/tiling</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><span style="font-family:arial, sans-serif;font-size:13px;border-collapse:collapse;color:rgb(136, 136, 136)">Gabriel Roldan<br>OpenGeo - <a href="http://opengeo.org" style="color:rgb(42, 93, 176)" target="_blank">http://opengeo.org</a><br>
Expert service straight from the developers.</span><br>
</div>