[OSGeo-Discuss] Raster catalog ideas [SEC=UNCLASSIFIED]

[Lucena, Ivan]

Hi Mike, Bruce,

We went to that discussion a couple of year ago, and by that time there was some strong opinion 
against storing raster data on databases, but it look like time has changed things a little bit. Am 
I right to say that? I can see basically three reasons for that.

1. Data volume.

Raster images are getting more and more accessible and whoever deals with big volume of those are 
starting to feel the pain of managing it on files. I bet that people have started to think about why 
spending so much money buying and maintaining huge Network Attached Storage (NAS) and end up making 
a person do the job of a database server. They must realize that shared folders accessible by end 
users is a problem not a solution. Yes, the systematical and disciplinary use of file based storage 
is possible too, adding to that some sort of raster catalog, but adding to that also the trouble of 
managing permission access on the NAS, backup procedures, temporary files management, and the result 
is a full time position or two.

2. Performance.

The myth says: "the is always a overhead to access your raster data from a database in comparison to 
accessing it directly from the file system". If you are talking about a single computer that might 
be the case since fopen(), read(), write() doesn't need to make connections, check permission, 
parse SQL statement to finally figure out where to read BLOBs with the same C functions. But as the 
complexity of your storage grows the file based options will start to experience overhead too and 
there will be not RDBMS server to do connection pooling or caching for you. Again, you can go to the 
whole trouble of managing overviews and tile cache yourself. That means overhead for a file based 
raster data server and more cost of maintaining it. Are we tied?

3. Examples

Satellite companies, data provider, government agency and consulting firms are giving good testimony 
of their use of raster on database, so it might be true for the FOSS options too. The two examples 
provided by Bruce are pretty good. Rasdaman only works with PostgreSQL while Terralib works with 
directly with PostgreSQL, MySQL and commercial RDBMS systems. Both have some big projects to 
showcase their muscles.

Please keep us posted.

Good luck.

Ivan



Bruce Bannerman wrote:
> Hi Mike,
> 
> 
> A product for managing (muti-dimensional) raster data within a database that I've been monitoring since around 2002 has just been released under GPL. 
> 
> They have also applied for OSGeo Incubation.
> 
> This product, Rasdaman, can be used with Postgres as its data store.
> 
> Rasdaman has its lineage going back to around 1995 (I think). They are claiming the French government as a client using the product to manage TB image mosaics.
> 
> I understand that there is a sister product that will soon be merged in with Rasdaman that offers good OGC support (WCS 1.1 and 2.0, WMS and WCPS).
> 
> We intend investigating this product as a potential tool for managing grid, model and multi/hyperspectral remotely sensed data relating to the Climate domain. 
> 
> If you look at the product, let me know. I'll be happy to share experiences.
> 
> 
> 
> See [1] for Rasdaman product information and [2] for example implementations.
> 
> 
> ======
> 
> Another option is the Brazillian Terralib Project [3].
> 
> They have developed an integrated suite of server and framework tools for developing integrated applications based around the use of imagery and the management of imagery within a database (Postgres from memory). 
> 
> I'm not doing them justice with this brief description. They have done some very impressive work. I'd recommend Gilberto Camara's intro paper at [4] for an overview of Terralib.
> 
> 
> =======
> 
> 
> [1] http://www.rasdaman.org/
> 
> [2] www.earthlook.org
> 
> [3] http://www.terralib.org/index.php
> 
> [4] http://www.terralib.org/docs/papers/TerraLib-OSBook-versionJanuary2008.pdf
> 
> 
> 
> 
> Bruce Bannerman
> 
> 
> 
> ________________________________________
> From: discuss-bounces at lists.osgeo.org [discuss-bounces at lists.osgeo.org] On Behalf Of Mike Toews [mwtoews at gmail.com]
> Sent: Tuesday, 4 May 2010 6:11 AM
> To: discuss at lists.osgeo.org
> Subject: [OSGeo-Discuss] Raster catalog ideas
> 
> Hi All,
> 
> I'm wondering what existing open source options are available to store rasters with attributes.
> 
> For example, I have several hundred air photos from northern Canada spanning from the 1940s to now. They have different projections (UTM zones), cell sizes, etc. This data store needs to be accessed from both web and desktop GIS software, but needs to have a definition query for the "year" attribute (so I can take all air photos from 1962, or between 1973 to 1978, or whatever is required).
> 
> Our (my company) present solution is to use ESRI's raster catalog on a geodatabase. We've had a mixed range of problems on File/Personal/SDE Geodatabases. We've experienced corruption on all levels of storage options, so we keep our file path attributes to the original GeoTIFFs so the raster catalogs can be restored, if required.
> 
> I'm a bit lost for the available options. The future PostGIS with raster capabilities sounds promising, but I need something that already exists. I don't think a WMS service will work, since it cannot use a query definition (e.g., I don't want to make layers for each year).
> 
> What are other people doing for large stores of air photos? Thanks in advance.
> 
> -Mike
> _______________________________________________
> Discuss mailing list
> Discuss at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/discuss
>