[postgis-users] PostGIS heatmap generation

[Rémi Cura]

Hey,
you can do it with pointcloud (https://github.com/pramsey/pointcloud)
Currently the database I use with 600 millions points is doing fine.

In your case the problem is not : i want to render 500 millions points, but
more : i want to render the adapted number of points given a scale (the
problem is called generalization).

In my case using pointcloud you're about 100k pts/second streaming. with
parallel processing you could increase a tad to 500k points/sec.
If your points are truly only xyz+data, I guess you could make it to 1
million pts/sec.

So for your usage I would say your goal would be
"which are the 1 million (max!) points in the area that best represent my
dataset?"

To answer efficiently, you have to use Level Of Detail, which I started to
add to pointcloud.
The idea is the same as for pyramidal raster : each patch (group of points)
is ordered with a quad tree order,
so if you read the first 5 (1+4) points you have a representative subset,
if you read 21 (1+4+16) first point you have a more detailed representative
subset, etc.

The process is illustrated by pictures attached :
_create patches (group) of points regularly spaced (1m3 here).
image here<https://github.com/Remi-C/LOD_ordering_for_patches_of_points/blob/master/screen/2014-02-10%2015_18_32-CloudCompare%20-%20%5B3D%20View%201%5D.png>
_then, display points from max details to less details.
Level 6
<https://github.com/Remi-C/LOD_ordering_for_patches_of_points/blob/master/screen/2014-02-10%2015_22_24-CloudCompare%20-%20%5B3D%20View%201%5D.png>
Level 5<https://github.com/Remi-C/LOD_ordering_for_patches_of_points/blob/master/screen/2014-02-10%2015_24_50-CloudCompare%20-%20%5B3D%20View%201%5D.png>
Level 4<https://github.com/Remi-C/LOD_ordering_for_patches_of_points/blob/master/screen/2014-02-10%2015_25_37-CloudCompare%20-%20%5B3D%20View%201%5D.png>
Level 3<https://github.com/Remi-C/LOD_ordering_for_patches_of_points/blob/master/screen/2014-02-10%2015_26_44-CloudCompare%20-%20%5B3D%20View%201%5D.png>
Level 2<https://github.com/Remi-C/LOD_ordering_for_patches_of_points/blob/master/screen/2014-02-10%2015_27_36-CloudCompare%20-%20%5B3D%20View%201%5D.png>
Level 1 <http://2014-02-10 15_28_43-CloudCompare - [3D View 1].png>

The level of detail is just selected by reading more or less points. (so
after computing LOD for whole database once, no computing).

Then about rendering : if you want to render interactively about 1 million
points, you should switch to WebGL (
http://potree.org/demo/pompei/pompei.html)

Cheers,
Rémi-C


2014-03-18 8:40 GMT+01:00 Sandro Santilli <strk at keybit.net>:

> On Mon, Mar 17, 2014 at 03:41:03PM -0700, Paul Norman wrote:
>
> > - I have a dataset of 100 million to 500 million points
> >   across the US
>
> [...]
>
> > Has anyone got any suggestions for how to efficiently render heatmaps
> > from datasets this large?  I was thinking of two tables, one with the
> > raw data for high zooms and one pre-aggregated into space and time
> > bins (e.g. requests in a given box in a given time range) for use
> > in lower zooms. The aggregation operation would obviously be slow, but
> > I'd only have to do it once.
>
> Pre-aggregation would surely help, big problem being amount
> of records that need to be read from disk. You could also look
> at postgis raster as the aggregation type.
>
> --strk;
>
>  ()  ASCII ribbon campaign  --  Keep it simple !
>  /\  http://strk.keybit.net/rants/ascii_mails.txt
> _______________________________________________
> postgis-users mailing list
> postgis-users at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20140318/bab36130/attachment.html>