[postgis-users] Better storage strategy for 1 million tile of 50*50 pixels, 24 bands?

[guido lemoine]

So what do you use instead? Your data set is not exactly common (400+ Gb, 24 band doubles (complex data?), overlapping tiles, which are half empty). Is this LiDAR data? What exactly are you trying to visualise?


We have little issue with postgis raster speed using multi-temporal, multi-frame, all bands Landsat-8, either in-db or out-db. 
Over 100 Gb by now. We prefer out-of-db, because it keeps the raster db of manageable size. We expect it to work up to Terrabyte limits (I am told I should use SciDB beyond that). This works with mounted network disk as well, which is even nicer.


GL






On 04/17/14, Rémi Cura  <remi.cura at gmail.com> wrote:
> 
> 
> 
> 
> 
> 
> 
> Hey,
> from what I tried PostGis Raster is (relatively) slow and not adapted to multi bands
> (for instance, no way to set multiple bands at once. For my use case this would mean about 300k*0.1sec i.e. about 8 hours at best).
> 
> 
> The GDAL driver + QGIS should be considered an experimental function at the moment, 
> because it is still easily broken (and I'm speaking cutting edge gdal + qgis v 2, 2.2, 2.3 on win and linux).
> 
> 
> More generaly using QGIS with postgis is easily broken (i restart qgis several dozen times a day)
> 
> 
> 
> Taking that into consideration, 
> My conclusion is that for the moment there is no incentive to use __in base__ postgis raster, 
> 
> 
> because there is no stable way to access raster from outside base.
> 
> Sadly in my general use case, this translate to no incentive to use postgis raster at all :-(
> 
> Cheers,
> Rémi-C
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 2014-04-17 17:43 GMT+02:00 Tumasgiu Rossini <rossini.t at gmail.com <rossini.t at gmail.com>>:
> 
> 
> > 
> > 
> > 
> > Remi,
> > 
> > I'm interested in the way you handled this problem.
> > 
> > 
> > In my opinion, the problem is on the duality of your needs.
> > 
> > 
> > Out-db is known to be faster for visualisation.
> > 
> > 
> > In-db is better for analysis.
> > 
> > 
> > So, a compromise should be made Maybe storing each tile per "band type" ? So accessing data from qgis would be less painful ?
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > 2014-03-10 10:54 GMT+01:00 Rémi Cura <remi.cura at gmail.com <remi.cura at gmail.com>>:
> > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > Hey Dear List,
> > > 
> > > I would appreciate some advice about the best way to store my raster :
> > > 
> > > 1 million tiles,
> > > 50*50 pixels each (1 m2 or less in real world), around 24 bands (mostly doubles)
> > > 
> > > 
> > > 
> > > 
> > > ,in db.
> > > 
> > > 
> > > 
> > > About half the pixels are empty, some tiles overlaps, but most are regularly spaced.
> > > 
> > > 
> > > I would query it mainly by localisation (intersects), and also based on id of the tile.
> > > 
> > > 
> > > 
> > > 
> > > 
> > > The use would be fast visualisation with qgis (and latest gdal), interpolation, classification, matching and so.
> > > 
> > > 
> > > What is the best strategy?
> > > 1 table with many lines and indexes, indb,
> > > 
> > > 
> > > 
> > > 1 table, out db
> > > 
> > > 1 table, 1 line
> > > multiple tables, heritage?
> > > 
> > > 
> > > 
> > > Thanks for inputs!
> > > Cheers,
> > > 
> > > Rémi-C
> > > 
> > > 
> > > _______________________________________________
> > > 
> > > postgis-users mailing list
> > > 
> > > postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org>
> > > 
> > > http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
> > > 
> > 
> > 
> > 
> > _______________________________________________
> > 
> > postgis-users mailing list
> > 
> > postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org>
> > 
> > http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
> > 
> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20140417/91b442cf/attachment.html>