[postgis-users] Partitionning using geometry

[Mark Wynter]

Remi, sounds like you're close to finding a pathway that matches your needs.

there are always several ways to solve every problem. It seems yours is about storage and fast access to the bits that are relevant. You mention circa 1 thousand tiles.

The only reason I'm pursuing this topic is because I'm sensing a broader solution to an emerging set of challenges. Particularly with the "internet of things", where we will collect seriously BIG spatio-temporal data - eg from sensors and we can't afford to pull everything into PostGIS.

iOT requires solutions that can deal with datasets many orders of magnitude larger, but can also scale down to suit small projects.

> using Big Data /noSQL solutions to do some of the simple heavy lifting


> mongo db has got spatial capabilities And you can use the mongo FDW to return query results into PG.

Mongo db when I last looked at this only has 2d spatial indexing.  Not ideal, but still useful for point cloud storage... I also think the attached article is handy because the author lays out the business problem and how he's tackled it.

http://rs.tudelft.nl/~rlindenbergh/workshop/BoehmIQmulus.pdf

I'm not advocating that people jump on the NoSQL bandwagon - I love Postgis and prefer to do as much of my work in postgis - but it's good to know each DBs respective strengths and why..

IMHO, the quicker we become conversant inPolyglot DB design,  the better.

If we think every problem is a nail and therefore postgis is the hammer / answer, then innovation ( which is about using and combining existing technologies in new ways) will leave us behind.

Ok, time to park NoSQL and apply some left field thinking..


http://rs.tudelft.nl/~rlindenbergh/workshop/BoehmIQmulus.pdf

I'm not advocating that people jump on the NoSQL bandwagon - I love Postgis and prefer to do as much of my work in postgis - but it's good to know each DBs respective strengths and why..

> IMHO, the quicker we become conversant inPolyglot DB design,  the better.
> 
> If we think every problem is a nail and therefore postgis is the hammer / answer, then innovation ( which is about using and combining existing technologies in new ways) will leave us behind.

Ok, time to park NoSQL and apply some left field thinking..

take one of Paul's recent innovations...

https://github.com/pramsey/pgsql-ogr-fdw

What if we could read straight from file... Or "document" into PG...?  And I'm not talking every document - just the ones relevant to us... On an as needs basis.

This is what the mongo_fdw does... In fact we can write back too.

FDW's go to heart of polyglot design.

The common theme for IOT is the need to leverage a cheap distributed file / document storage system to enable fast search and retrieval... With a basic level of spatial awareness that can inform and feed downstream applications incl, PostGIS.

The idea is that stuff not needed or of no business value downstream gets left behind on the cheapest h/w, s/w before discarding.

I'm keen to hear from anyone about how they are using postgis in combination with NoSQL DBs for their big data pipelines ?

Or if there's another mailing list that covers spatial IT / spatial - temporal big data processing, I'd love to hear about it. 

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20150404/03493788/attachment.html>