[postgis-users] Better storage strategy for 1 million tile of 50*50 pixels, 24 bands?
guido lemoine
guido.lemoine at jrc.ec.europa.eu
Fri Apr 18 04:50:45 PDT 2014
Remi,
I access postgis raster data with Java, using JDBC and ST_AsTiff for raster results and then JAI (Java Advanced Imaging). JAI allows me to do stuff which can now be done mostly with ST_MapAlgebra as well. Since I am using OpenCV (which has most functions accessible through Java wrappers) in other approaches, I don't expect to have any issue with using that with postgis. In particular, I expect to use GPU accelerated OpenCV routines (template matching, segmentation, classification) to work on chunks I pull out of the Landsat-8 data base.
I will definitely try both rasdaman and SciDB at some stage, but for now, the MVC possibilities provided by the postgis/java/grails suite are pretty solid.
Guido
On 04/18/14, Rémi Cura <remi.cura at gmail.com> wrote:
>
>
>
>
>
>
> Hey, thanks for sharing your experience !
>
>
>
> 2014-04-18 0:54 GMT+02:00 Peter Baumann <p.baumann at jacobs-university.de <p.baumann at jacobs-university.de>>:
>
>
> >
> >
> >
> >
> >
> >
> >
> >
> > On 04/17/2014 08:11 PM, guido lemoine
> > wrote:
> >
> >
> >
> > >
> > >
> > > So what do you use instead? Your data set is not exactly common
> > > (400+ Gb, 24 band doubles (complex data?), overlapping tiles,
> > > which are half empty). Is this LiDAR data? What exactly are you
> > > trying to visualise?
> > >
> > >
> >
>
> Yep , good guess.
> Postgres + pointcloud seems to work good (I have a 5.4 billion points test database, no problem) to handle lidar,
> but visualisation within GIS software is an issue. Having a rasterized view could be usefull for this.
>
>
> How course importing millions of points in QGIS is not really an option (it works up to a million if style is not too complex).
>
> >
> >
> >
> >
> > >
> > >
> > >
> > > We have little issue with postgis raster speed using
> > > multi-temporal, multi-frame, all bands Landsat-8, either in-db
> > > or out-db.
> > >
> > > Over 100 Gb by now. We prefer out-of-db, because it keeps
> > > the raster db of manageable size. We expect it to work up to
> > > Terrabyte limits (I am told I should use SciDB
> > >
> > >
> >
> >
> >
> > when doing that, give rasdaman a try and compare then. It's
> > operational on n-D multi-TB objects :)
> >
> > -Peter
> >
> >
> >
>
> I know PostGIS raster is a powerful tool,
>
> I just say that in-base multi-band raster creation is slow
>
> ,any raster processing is slow (need to rewrite entire raster)
>
>
> ,image processing options are limited (it could be so cool to plug a good image processing lib like itk or otb)
>
> ,accessing in-base raster is still fragile
>
> ,Level of Detail strategy are not fully integrated.
>
>
>
> >
> >
> >
> > >
> > >
> > >
> > >
> > >
> > >
> > > beyond that). This works with mounted network disk as well,
> > > which is even nicer.
> > >
> > >
> > >
> >
>
> The whole postgres data folder can be put on network disk (with caveat), which is very usefull !
>
>
> >
> >
> >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > GL
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On 04/17/14, Rémi Cura
> > > <remi.cura at gmail.com> <remi.cura at gmail.com> wrote:
> > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Hey,
> > > >
> > > > from what I tried PostGis Raster is
> > > > (relatively) slow and not adapted to multi
> > > > bands
> > > >
> > > >
> > > > (for instance, no way to set multiple bands at
> > > > once. For my use case this would mean about
> > > > 300k*0.1sec i.e. about 8 hours at best).
> > > >
> > > >
> > > >
> > > >
> > > > The GDAL driver + QGIS should be considered an
> > > > experimental function at the moment,
> > > >
> > > > because it is still easily broken (and I'm
> > > > speaking cutting edge gdal + qgis v 2, 2.2, 2.3
> > > > on win and linux).
> > > >
> > > >
> > > >
> > > > More generaly using QGIS with postgis is
> > > > easily broken (i restart qgis several dozen
> > > > times a day)
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > Taking that into consideration,
> > > >
> > > >
> > > > My conclusion is that for the moment there is no
> > > > incentive to use __in base__ postgis raster,
> > > >
> > > >
> > > >
> > > > because there is no stable way to access raster
> > > > from outside base.
> > > >
> > > >
> > > >
> > > >
> > > > Sadly in my general use case, this translate to no
> > > > incentive to use postgis raster at all :-(
> > > >
> > > >
> > > >
> > > >
> > > > Cheers,
> > > >
> > > > Rémi-C
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2014-04-17 17:43 GMT+02:00
> > > > Tumasgiu Rossini <rossini.t at gmail.com <rossini.t at gmail.com>
> > > > <rossini.t at gmail.com> <rossini.t at gmail.com>>:
> > > >
> > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Remi,
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > I'm interested in the way you handled this
> > > > > problem.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > In my opinion, the problem is on the duality
> > > > > of your needs.
> > > > >
> > > > >
> > > > >
> > > > > Out-db is known to be faster for
> > > > > visualisation.
> > > > >
> > > > >
> > > > >
> > > > > In-db is better for analysis.
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > So, a compromise should be made Maybe
> > > > > storing each tile per "band type" ? So
> > > > > accessing data from qgis would be less painful
> > > > > ?
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > 2014-03-10 10:54
> > > > > GMT+01:00 Rémi Cura <remi.cura at gmail.com <remi.cura at gmail.com>
> > > > > <remi.cura at gmail.com> <remi.cura at gmail.com>>:
> > > > >
> > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hey Dear List,
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > I would appreciate
> > > > > > some advice about the
> > > > > > best way to store my
> > > > > > raster :
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1 million tiles,
> > > > > >
> > > > > >
> > > > > > 50*50 pixels each (1 m2 or
> > > > > > less in real world),
> > > > > > around 24 bands (mostly
> > > > > > doubles)
> > > > > >
> > > > > >
> > > > > >
> > > > > > ,in db.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > About half the pixels
> > > > > > are empty, some tiles
> > > > > > overlaps, but most are
> > > > > > regularly spaced.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > I would query it mainly by
> > > > > > localisation (intersects),
> > > > > > and also based on id of the
> > > > > > tile.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > The use would be fast
> > > > > > visualisation with qgis (and
> > > > > > latest gdal), interpolation,
> > > > > > classification, matching and
> > > > > > so.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > What is the best strategy?
> > > > > >
> > > > > >
> > > > > > 1 table with many lines and
> > > > > > indexes, indb,
> > > > > >
> > > > > >
> > > > > > 1 table, out db
> > > > > >
> > > > > >
> > > > > >
> > > > > > 1 table, 1 line
> > > > > >
> > > > > >
> > > > > > multiple tables, heritage?
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks for inputs!
> > > > > >
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > >
> > > > > >
> > > > > > Rémi-C
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > >
> > > > > > postgis-users mailing list
> > > > > >
> > > > > > postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org>
> > > > > > <postgis-users at lists.osgeo.org> <postgis-users at lists.osgeo.org>
> > > > > >
> > > > > > http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > >
> > > > > postgis-users mailing list
> > > > >
> > > > > postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org>
> > > > > <postgis-users at lists.osgeo.org> <postgis-users at lists.osgeo.org>
> > > > >
> > > > > http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > postgis-users mailing list
> > > postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org>
> > > http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
> > >
> > >
> >
> >
> >
> >
> > --
> > Dr. Peter Baumann
> > - Professor of Computer Science, Jacobs University Bremen
> > www.faculty.jacobs-university.de/pbaumann(http://www.faculty.jacobs-university.de/pbaumann)
> > mail: p.baumann at jacobs-university.de <p.baumann at jacobs-university.de>
> > tel: +49-421-200-3178(tel:%2B49-421-200-3178), fax: +49-421-200-493178(tel:%2B49-421-200-493178)
> > - Executive Director, rasdaman GmbH Bremen (HRB 26793)
> > www.rasdaman.com(http://www.rasdaman.com), mail: baumann at rasdaman.com <baumann at rasdaman.com>
> > tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882(tel:%2B49-173-5837882)
> > "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> >
> > postgis-users mailing list
> >
> > postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org>
> >
> > http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
> >
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20140418/bccefac3/attachment.html>
More information about the postgis-users
mailing list