<div dir="ltr"><div class="gmail_default" style="font-family:monospace,monospace">Hey Oscar,<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">I'm a really big fan of Lidar for archeological use, and integrating time into it is especially trendy and challenging. Registetring all point cloud together from different sources must have been really difficult.<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br> <br></div><div class="gmail_default" style="font-family:monospace,monospace">I contacted Potree developper one year ago to ask him if it was possible to modify it to read points in a DBMS (actually patch with LOD).<br></div><div class="gmail_default" style="font-family:monospace,monospace">He said it was possible and not too difficult.<br></div><div class="gmail_default" style="font-family:monospace,monospace">I don't know how much point output you get, but we demonstrated around 20kpts/s streaming to browser (with a lot of serialization/deserialization). Currently the upper limit for such output would be in the few hundred kpts/s if you send points, and in the few Million pts/s if you stream compressed patches.<br><br></div><div class="gmail_default" style="font-family:monospace,monospace">Cheers<br></div><div class="gmail_default" style="font-family:monospace,monospace"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-04-20 11:58 GMT+02:00 Oscar Martinez Rubi <span dir="ltr"><<a href="mailto:o.martinezrubi@tudelft.nl" target="_blank">o.martinezrubi@tudelft.nl</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Whoa!<br>
<br>
Thanks guys for all the material! I am now busy reading it all!<br>
<br>
Remi: I had to read your mail a few times ;) Great slides (I
actually looked at all of them, very well done!) Very interesting
topics you are researching!<br>
<br>
Howard: About the Greyhoud+s3, but what storage solution do you use,
it is not clear...mongodb? I mean where are the points stored?
file-based, dbms? <br>
<br>
Paul+Nouri: The geohash tree that Noury mentions is the ght
compression in pgpointclouid, right? I tried it once but there was
some limitation with the type of coordinates, they had to be long
and lat so I guess there need to be a reference system
transformation in between, right? Any place where I can find an
example on how to use this?<br>
<br>
<br>
At NLeSC for the visualization of data we are using a system based
on the potree visualization (so, file-based) but I am very very
interested on the stuff you are guys doing and I would love to be
convinced that DBMS solutions can be really efficient for
visualization as well (i think it is close now!). We choose
file-based and potree because of the initial lack of LoD support in
DBMS, the speed the file-based approach and the super compressed LAZ
storage.<br>
<br>
To see what we have done so far:<br>
<br>
<a href="https://github.com/NLeSC/PattyVis" target="_blank">https://github.com/NLeSC/PattyVis</a><br>
<a href="https://www.esciencecenter.nl/project/mapping-the-via-appia-in-3d" target="_blank">https://www.esciencecenter.nl/project/mapping-the-via-appia-in-3d</a><br>
(see the video from 1:40 for the potree based visualization)<br>
<br>
One of the many reasons I would loved to be convinced that DBMS is
that now we are considering how to visualize the 640B AHN2 dataset,
and in a pure file-based solution (like the potree) I fear that when
restructuring the data to octree we would need a number of octree
nodes/files probably larger than what ext4 can handle!. We will try,
I let you know how that goes ;), but it would be really nice to have
a efficient and fast DBMS-based alternative!<br>
<br>
I am very happy though with all the different work you are all doing
and excited to see how fast things improve and evolve!! <br>
<br>
Keep on like this guys!<br>
<br>
Regards,<br>
<br>
O.<div><div class="h5"><br>
<br>
<br>
<div>On 17-04-15 19:01, Sabo, Nouri wrote:<br>
</div>
<blockquote type="cite">
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Thank
you for sharing these ideas. Many of the ideas can make
improvements. In the prototype we have developed at RNCan
and that we mentioned in the paper in attachment we have
implemented some of these concepts. For example, in the
prototype we are sorting points according to the Morton
pattern before creating blocks.
</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"" lang="EN-CA">And each block is composed only of points that
are spatially close, thereby improving the level of
compression. We also use the properties of the Morton curve
(Z pattern) to do spatial queries using Geohash as BBox. </span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Usually,
in Geohash based system the more the Geohash prefixes for
two points resemble one another, the more they are spatially
close to each other. Unfortunately, this property is not
always complied with two points located on either side of a
subdivision line.
</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"" lang="EN-CA">For this reason we implemented a neighbourhood
based strategy to allow spatial query based on the hash
string.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"" lang="EN-CA">Also to improve the compression and performance
we can change the encoding of Geohash.
</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Currently,
the hashes are encoded as base 32 strings, which causes a
lot of overhead (5 bits are inflated in 8 bits of
character).
</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"" lang="EN-CA">Unfortunately, the current libght does not
include all the concepts of GeoHashTree.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"" lang="EN-CA">Oscar, I will read your paper and get you back
so we could continue to exchange.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Kind
regards!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Nouri,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d" lang="EN-CA"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-CA"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"" lang="EN-US"> Paul Ramsey
[<a href="mailto:pramsey@cleverelephant.ca" target="_blank">mailto:pramsey@cleverelephant.ca</a>]
<br>
<b>Sent:</b> 17 avril 2015 06:56<br>
<b>To:</b> <a href="mailto:pgpointcloud@lists.osgeo.org" target="_blank">pgpointcloud@lists.osgeo.org</a>; Peter van
Oosterom; Oscar Martinez Rubi; Howard Butler; Rémi Cura<br>
<b>Cc:</b> Sabo, Nouri<br>
<b>Subject:</b> Re: [pgpointcloud] RLE and SIGBITS
heuristics<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Hi
Oscar, <u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">This
sounds like a slightly more sophisticated version of the
work done at Natural Resources Canada for what they call
“geohash tree”. They did find that they got pretty good
compression (even with the simple ascii-based key!) using
the scheme, and it did allow easy random access to subsets
of the data.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a href="http://2013.foss4g.org/conf/programme/presentations/60/" target="_blank">http://2013.foss4g.org/conf/programme/presentations/60/</a><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">The
downside was of course the cost of sorting things in the
first place, but for a one-time cost on frequently
accessed data, it’s not a bad thing. The “libght” soft
dependency in pgpointcloud is to a (not so great)
implementation of the scheme that I did for them a couple
years ago. As a scheme, I think it cuts against the idea
of having small patches that is core to the pgpointcloud
concept. It makes more and more sense the larger your file
is, in that it gets greater and greater leverage for
random access.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">ATB,<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">P.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">-- <br>
Paul Ramsey<br>
<a href="http://cleverelephant.ca" target="_blank">http://cleverelephant.ca</a><u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><a href="http://postgis.net" target="_blank">http://postgis.net</a>
<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
<p><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif";color:black">On
April 17, 2015 at 11:02:47 AM, Oscar Martinez Rubi (<a href="mailto:o.martinezrubi@tudelft.nl" target="_blank">o.martinezrubi@tudelft.nl</a>)
wrote:<u></u><u></u></span></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">Hi,<br>
<br>
About the XYZ binding for better compression. In our
research in the NL escience center and TU Delft we
have been thinking (not testing yet though) about one
possible approach for this.<br>
<br>
It is based on using space filling curves. So, once
you have the points that go in a block you could
compute the morton/hilbert code of the XYZ. Since all
the points are close together such codes will be
extremely similar, so one could store only the
increments which could fit in many few bits. We have
not tested or compared this with any of the other
compressions but we just wanted to share it with you
just in case you find it useful!<br>
<br>
An additional improvement would be to sort the points
within the blocks according to the morton code. Then,
when doing crop/filter operations in the blocks one
can use the morton codes for the queries similarly to
what we presented in our papers with the flat table
(without blocks), I attach one of them (see section
5.2). In a nutshell: You convert the query region into
a set of quadtree/octree nodes which can be also
converted to morton code ranges (thanks to relation
between morton/hilbert curve and a quadtree/octree).
You scale down the ranges to increments (like you did
when storing the point of the block) and then you
simply do range queries in sorted data with a binary
algorithm. In this way you avoid the decompression of
the morton code for most of the block. This filtering
is equivalent to a bbox filter so it still requires a
point in polygon check for some of the points.<br>
<br>
Kind Regards,<br>
<br>
Oscar.<br>
<br>
<u></u><u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">On
16-04-15 18:15, Rémi Cura wrote:<u></u><u></u></span></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span>epic fail ! I had avoided html just
for you</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span><br>
Dataset |subset size | compressing |
decompressing |<br>
|(Million pts)|(Million
pts/s)|(Million pts/s)|<br>
Lidar | 473.3 | 4,49 |
4,67 |</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span>21-atributes | 105.7 | 1,11
| 2,62 |</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span>Stereo | 70 |
2,44 | 7,38 |</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span>Cheers</span><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u><u></u></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">2015-04-16
17:42 GMT+02:00 Sandro Santilli <<a href="mailto:strk@keybit.net" target="_blank">strk@keybit.net</a>>:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">On
Thu, Apr 16, 2015 at 05:30:12PM +0200, Rémi Cura
wrote:<br>
> OUps<br>
><br>
> Dataset | subset size(Million pts)
| compressing (Million pts/s) |<br>
> decompressing (Million pts/s)<br>
> Lidar | 473.3
| 4,49<br>
> | __4,67__<br>
> 21 attributes | 105.7
|<br>
> 1,11 | 2,62<br>
> Stereo | 70
| 2,44<br>
> | 7,38<br>
<br>
These tables aren't really readable here.<br>
Could you make sure to use a fixed-width font to
write those tables<br>
and to keep lines within 70 columns at most ?<br>
<br>
--strk;<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><br>
<br>
<br>
<u></u><u></u></span></p>
<pre>_______________________________________________<u></u><u></u></pre>
<pre>pgpointcloud mailing list<u></u><u></u></pre>
<pre><a href="mailto:pgpointcloud@lists.osgeo.org" target="_blank">pgpointcloud@lists.osgeo.org</a><u></u><u></u></pre>
<pre><a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/pgpointcloud" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/pgpointcloud</a><u></u><u></u></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif""><u></u> <u></u></span></p>
<div class="MsoNormal" style="text-align:center" align="center"><span style="font-size:10.0pt;font-family:"Helvetica","sans-serif"">
<hr align="center" size="2" width="100%">
</span></div>
</div>
</div>
</blockquote>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>