[postgis-users] Better storage strategy for 1 million tile of 50*50 pixels, 24 bands?

Fri Apr 18 03:35:22 PDT 2014

2014-04-18 10:06 GMT+02:00 Rémi Cura <remi.cura at gmail.com>:

>
>
> Hey, thanks for sharing your experience !
>
> 2014-04-18 0:54 GMT+02:00 Peter Baumann <p.baumann at jacobs-university.de>:
>
>  On 04/17/2014 08:11 PM, guido lemoine wrote:
>>
>> So what do you use instead? Your data set is not exactly common (400+ Gb,
>> 24 band doubles (complex data?), overlapping tiles, which are half empty).
>> Is this LiDAR data? What exactly are you trying to visualise?
>>
>> Yep , good guess.
> Postgres + pointcloud seems to work good (I have a 5.4 billion points test
> database, no problem) to handle lidar,
> but visualisation within GIS software is an issue. Having a rasterized
> view could be usefull for this.
> How course importing millions of points in QGIS is not really an option
> (it works up to a million if style is not too complex).
>
>>  We have little issue with postgis raster speed using multi-temporal,
>> multi-frame, all bands Landsat-8, either in-db or out-db.
>> Over 100 Gb by now. We prefer out-of-db, because it keeps the raster db
>> of manageable size. We expect it to work up to Terrabyte limits (I am told
>> I should use SciDB
>>
>>
>> when doing that, give rasdaman a try and compare then. It's operational
>> on n-D multi-TB objects :)
>> -Peter
>>
>>
> I know PostGIS raster is a powerful tool,
> I just say that in-base multi-band raster creation is slow
> ,any raster processing is slow (need to rewrite entire raster)
> ,image processing options are limited (it could be so cool to plug a good
> image processing lib like itk or otb)
> ,accessing in-base raster is still fragile
> ,Level of Detail strategy are not fully integrated.
>
>  beyond that). This works with mounted network disk as well, which is
>> even nicer.
>>
>> The whole postgres data folder can be put on network disk (with caveat),
> which is very usefull !
>
>
>>  GL
>>
>>
>>  On 04/17/14, *Rémi Cura * <remi.cura at gmail.com> <remi.cura at gmail.com>wrote:
>>
>>    Hey,
>> from what I tried PostGis Raster is (relatively) slow and not adapted to
>> multi bands
>>  (for instance, no way to set multiple bands at once. For my use case
>> this would mean about 300k*0.1sec i.e. about 8 hours at best).
>>
>>  The GDAL driver + QGIS should be considered an experimental function at
>> the moment,
>> because it is still easily broken (and I'm speaking cutting edge gdal +
>> qgis v 2, 2.2, 2.3 on win and linux).
>>  More generaly using QGIS with postgis is easily broken (i restart qgis
>> several dozen times a day)
>>
>>  Taking that into consideration,
>>  My conclusion is that for the moment there is no incentive to use __in
>> base__ postgis raster,
>>  because there is no stable way to access raster from outside base.
>>
>>  Sadly in my general use case, this translate to no incentive to use
>> postgis raster at all :-(
>>
>>  Cheers,
>> Rémi-C
>>
>>
>>
>>
>> 2014-04-17 17:43 GMT+02:00 Tumasgiu Rossini <rossini.t at gmail.com
>> <rossini.t at gmail.com> <rossini.t at gmail.com>>:
>>
>>>  Remi,
>>>
>>>  I'm interested in the way you handled this problem.
>>>
>>> In my opinion, the problem is on the duality of your needs.
>>>  Out-db is known to be faster for visualisation.
>>>  In-db is better for analysis.
>>>
>>>  So, a compromise should be made Maybe storing each tile per "band
>>> type" ? So accessing data from qgis would be less painful ?
>>>
>>>
>>>
>>>
>>> 2014-03-10 10:54 GMT+01:00 Rémi Cura <remi.cura at gmail.com
>>> <remi.cura at gmail.com> <remi.cura at gmail.com>>:
>>>
>>>>       Hey Dear List,
>>>>
>>>>  I would appreciate some advice about the best way to store my raster :
>>>>
>>>>  1 million tiles,
>>>>  50*50 pixels each (1 m2 or less in real world), around 24 bands
>>>> (mostly doubles)
>>>>  ,in db.
>>>>
>>>>  About half the pixels are empty, some tiles overlaps, but most are
>>>> regularly spaced.
>>>>
>>>>  I would query it mainly by localisation (intersects), and also based
>>>> on id of the tile.
>>>>
>>>>  The use would be fast visualisation with qgis (and latest gdal),
>>>> interpolation, classification, matching and so.
>>>>
>>>>  What is the best strategy?
>>>>  1 table with many lines and indexes, indb,
>>>>  1 table, out db
>>>>  1 table, 1 line
>>>>  multiple tables, heritage?
>>>>
>>>>
>>>>  Thanks for inputs!
>>>>  Cheers,
>>>>
>>>> Rémi-C
>>>>
>>>>  _______________________________________________
>>>> postgis-users mailing list
>>>> postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org><postgis-users at lists.osgeo.org>
>>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> postgis-users mailing list
>>> postgis-users at lists.osgeo.org <postgis-users at lists.osgeo.org><postgis-users at lists.osgeo.org>
>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
>>>
>>
>>
>>
>> _______________________________________________
>> postgis-users mailing listpostgis-users at lists.osgeo.org
>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
>>
>>
>> --
>> Dr. Peter Baumann
>>  - Professor of Computer Science, Jacobs University Bremen
>>    www.faculty.jacobs-university.de/pbaumann
>>    mail: p.baumann at jacobs-university.de
>>    tel: +49-421-200-3178, fax: +49-421-200-493178
>>  - Executive Director, rasdaman GmbH Bremen (HRB 26793)
>>    www.rasdaman.com, mail: baumann at rasdaman.com
>>    tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
>> "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat quisquam non sibi parata." (mail disclaimer, AD 1083)
>>
>>
>>
>>
>> _______________________________________________
>> postgis-users mailing list
>> postgis-users at lists.osgeo.org
>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-users
>>
>
>
I must correct myself :
there is a way to export in-base rater efficiently :
http://geeohspatial.blogspot.fr/2013/07/exporting-postgis-rasters-to-other.html
Cheers,
Rémi-C
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20140418/30560575/attachment.html>