<div dir="ltr"><div>Maybe you would be interested in paper from lastool about how they compress the data<br></div>[the paper](<a href="http://lastools.org/download/laszip.pdf">http://lastools.org/download/laszip.pdf</a>)<br></div><div class="gmail_extra"><br><div class="gmail_quote">2015-04-15 13:34 GMT+02:00 Sandro Santilli <span dir="ltr"><<a href="mailto:strk@keybit.net" target="_blank">strk@keybit.net</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Thu, Apr 09, 2015 at 12:41:32PM +0200, Sandro Santilli wrote:<br>
> Reading the code I found that SIGBITS is used when it gains<br>
> a compression ratio of 1.6:1 while RLE is required to get 4:1<br>
> ratio (but the comment talk about a 4:1 for both).<br>
><br>
> How were the ratios decided ?<br>
><br>
> <a href="https://github.com/pgpointcloud/pointcloud/blob/v0.1.0/lib/pc_dimstats.c#L121-L137" target="_blank">https://github.com/pgpointcloud/pointcloud/blob/v0.1.0/lib/pc_dimstats.c#L121-L137</a><br>
<br>
</span>As an experiment, I created a patch containing 260,000 points organized<br>
so that the data in each dimension is layed out in a way to make the heuristic<br>
pick one of the 3 different encodings:<br>
<br>
All dimensions are of type int16_t<br>
- 1st dimension alternates values 0 and 1<br>
- 2nd dimension has value -32768 for the first 130k points, then 32767<br>
- 3rd dimension alternates values -32768 and 32767<br>
<br>
Then I checked the size of the patch after applying different compression<br>
schemes, and here's what I got:<br>
<br>
size | compression<br>
---------+---------------------------<br>
1680 | {zlib,zlib,zlib}<br>
4209 | {zlib,auto,auto} <-- zlib much better than sigbits !<br>
33656 | {auto,zlib,auto} <-- zlib better than rle<br>
36185 | {auto,auto,auto} <----- DEFAULT, effectively {sigbits,rle,zlib}<br>
36185 | {auto,auto,zlib}<br>
1072606 | {sigbits,sigbits,sigbits}<br>
1560073 | {uncompressed} <------ UNCOMPRESSED size<br>
1563148 | {rle,rle,rle}<br>
<br>
Interesting enough, "zlib" results in a better compression than<br>
both "sigbits" and "rle", and we're supposedly talking about their<br>
best performances (only 2 runs for rle, 15 bits over 16 in common for<br>
sigbits).<br>
<br>
It might be a particularly lucky case for zlib too, given the very regular<br>
pattern of values distributions, but now I'm wondering... how would zlib<br>
perform on real world datasets out there ?<br>
<br>
If you're running the code from current master branch you could test this<br>
yourself with a query like this:<br>
<br>
\set c mycol -- set to name of your column<br>
\set t mytab -- set to name or your table<br>
SELECT sum(pc_memsize(<br>
pc_compress(:c, 'dimensional', array_to_string(<br>
array_fill('zlib'::text,ARRAY[100]), ','<br>
))))::float8 /<br>
sum(pc_memsize(:c))<br>
FROM :t;<br>
<br>
It will tell how much smaller could your dataset get by compressing<br>
it all with dimensional/zlib schema.<br>
<br>
I get 0.046 with my test dataset above (260k points in patch),<br>
while it is 1.01 (bigger) on a dataset where patches have 1000 points each<br>
and 3 dimensions over 12 are already compressed with zlib, other 3 with<br>
sigbits and the remaining 6 with rle.<br>
<br>
How about you ?<br>
<div class="HOEnZb"><div class="h5"><br>
--strk;<br>
_______________________________________________<br>
pgpointcloud mailing list<br>
<a href="mailto:pgpointcloud@lists.osgeo.org">pgpointcloud@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/pgpointcloud" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/pgpointcloud</a><br>
</div></div></blockquote></div><br></div>