[postgis-users] PostGIS WKT Raster

[muhammad imran]

Hi Raster People,
Racine proposed model has very much potential to discuss from open source raster community.
As a raster student I have some comments and I would like have further comments...
Image manipulation is beyond just storing the band information, pyramid information and BBOX in a well known text format field of a relation however this structure of raster dataset is exactly similar to that of vector dataset inside PostGIS and can be useful and might be required for overlay of vector and raster data sets for one application domain that is integrated analyses of the vector and raster data however there are more applications and open source libraries that can better work directly with image stores as BLOB. So restricting the DBMS with WKTRaster will constrain the developer. SQL programming interface return cursor to BLOB (also defined in OGC coverage specifications) and the developer can traverse through get and set methods at atomic level of the image for local, focal and zonal operations more freely and independently. Almost all format conversion libraries also deals with binary BLOB data not WKT. Although the idea is novel but it
 comes with reinventing new structures and it would be very difficult to integrate the previous work.
 
Futher image data has contents in addition to bounding boxes and retrieval models incorporate contents of image like colour, shape, texture, time and there are many characteristic associated directly with the image data like sensor type, image type, image quality, and those involved from image capturing to image processing at base stations. Further, images contents are semantic oriented and one image may have different semantics and interpretation for different users.
 
In this context, an important point to consider the true integration is much more than just overlaying the raster and vector data and it must make the relations between the individual objects in the different data sets explicit. This requires the realization of a true integration of spatial data. Corresponding objects of the different data sets have to be identified so they can be connected by explicit links. The links have to be set up by automatic matching algorithms (Mattias Butenuth, 2007). Integrating images from different sensor platforms and vector data with different accuracies is not trivial. This may involve the segmentation of original image stored in database and then these common objects should obviously coincide geometrically  
 
This requires to store image itself as BLOB and allow to access and manipulate it by extending the DBMS with ADTs and user defined functions  UDFs. There are many applications and open source libraries (like Terralib, GDAL) that can better work  directly with image stores as BLOB. So restricting the DBMS with WKTRaster will constrain the developer.
 
Further, all the DBMS supports to handle intensive image data in the form of BOLB, special buffers in memory cache and handlers are allocated for BLOB operations therefore if we use BLOB storage structure the queries will use the built in performance options for image operations. Further, BLOB also provides atomic level access control that can be adjusted by both database administrators and the application developers. I think this fine control over the storage both for performance and access through database parameters and through application itself according to varying conditions and nature of application is the beauty of databases that should not snatched by defining only one closed raster type like proposed by Racine however this is specific application requirement for integrated vector raster analyses that could be fulfilled by transforming the image data in BLOB to raster type on fly or physically. 
 
Conclusion:
 
In order to capture the diversity of data, ad diversity of applications the data type should be more flexible 
 
Metadata should be separate to original raster image data. In most cases only metadata is processed in this case why we process the image data itself. Secondly, the metadata has its own standard formats and contains such an information both collection level and atomic level like name, size, process input. Ancillary data, quality, platform, instrument, sensor, Model, spatial coverage, temporal coverage all this information can be used both by database optimizers and applications outside without processing image so it should be maintained in separate metadata table and data itself as BLOB in raster table.
 
Storing only tiles (as WKTRaster ) and not image (as BLOB) will limit the scope of raster operation and developers could not integrate many open source image processing algorithms like from Terralib that operates on the image it self.
 
The WKTraster proposed by Racine should be one of the format that can be generated from conversion on requirement for integrated analyses on demand through PGrasterToWKTRaster
Userdefind function  like rasterTogeotiff.
Modeling only the Relational structure used by oracle will not voilate the patent (this can be the result by following the standered normalization and design principle of databases by any one) . It is also adopted by open source Terralib and until now the best archtecture in broader scope both satisfying the image data administration and processing. however, copying the whole paradigm may be patent voilation like (Xing Lin).   (..... need comments on this)
Best Regards,
--Imran



----- Original Message ----
From: Pierre Racine <Pierre.Racine at sbf.ulaval.ca>
To: PostGIS Users Discussion <postgis-users at postgis.refractions.net>
Sent: Friday, July 25, 2008 4:11:41 PM
Subject: RE: [postgis-users] PostGIS WKT Raster

Hi Imran, thanks for your interest,

>However for building GIST indexes he is also worried 
>from the embedded BBOX information and proposes to store it 
>separately like metadata information in georaster and 
>similarly I can imagine he have to drag out more information 
>out of raster type for query optimization, hierarchal  storage 
>structure etc

Actually, in the PPT presentation, I just ask if it is possible to embed the raster BBOX in a new WKT/WKB type and still build the Gist index with it. My first guess is "yes" it is possible... So I don't really plan yet to store the BBOX outside of the tiles as metadata. Unless someone says it not possible... I really don't like the GeoRaster relational schema. PostGIS geometries are simple to use and successfull because they are self sufficient. I think raster tiles should be viewed the same way.

>1-     Whether this new structure will be according to OGC 
>specification for coverage?

As explained in slide 42, the OGC specification for coverage do not speak about how a coverage should be stored. It speak about how it should be queried. A coverage can also be a vector coverage as stored in PostGIS right now. Unfortunalety this specification does not really propose a seamless integration of topological operator between a raster coverage and a vector coverage. This said, however we store a coverage, the integration of the operators I propose does not exclude the possible addition of queries like the one proposed in the coverage specs. My point of view is "lets see if/how people will implement the coverage specs for a vector coverage and then we could try to do the same for a raster coverage"...

>2-     While adopting this structure are we not limiting 
>ourselves to support only the integrated vector raster 
>operations on the cast there are hundreds of image (raster) 
>processing libraries/algorithms from open source that are 
>supporting Xing Lin (oracle) georaster type  and direct 
>support from DBMS like indexes etc if we have Lin.s georaster 
>type we can extend it directly with hundreds of aforesaid 
>libraries/algorithms available from open source community

First, like I said I don´t really like this design mainly for three reasons:

-it is a copy of Oracle Georaster which (I think) was not originally designed to store georeferenced images... There is also these patents stuff.
-it does not propose any integration with vector,
-it splits the object into two separated tables. I think this is too complicated for what we need. I prefer the PostGIS way of storing a geometry as a self containing object.

Second, I don't know yet any other package/libraries that supports it. Maybe I'm wrong...

Pierre

>best regards
> 
>--imran 
> 
>
>
>
>----- Original Message ----
>From: Paul Ramsey <pramsey at cleverelephant.ca>
>To: Pierre Racine <Pierre.Racine at sbf.ulaval.ca>
>Cc: warmerdam at pobox.com; nicolas.gignac at msp.gouv.qc.ca; 
>smarshall at wsi.com; PostGIS Users Discussion 
><postgis-users at postgis.refractions.net>
>Sent: Monday, July 14, 2008 10:11:05 PM
>Subject: Re: [postgis-users] PostGIS WKT Raster
>
>Pierre,
>
>Firstly, let me commend you for your understanding of the open source
>community process! Not being shy, not taking "no" for an answer, and
>needling the people you need to move are all excellent ways to keep
>the wheels turning, particularly when done with good humor as you have
>shown.  Other first timers could/should learn from your example!
>
>So now I owe you a reply, no doubt:
>
>You have gotten over my first hurdle, in that you have an actual use
>case for raster, that is more involved than "I want to my images,
>because I hear database are faster".
>
>I have long thought that analysis, in particular a fusing of vector
>and raster analysis was the only really compelling use case for raster
>in database, so again, you're on the side of the angels (me).
>
>So, is your *design* good?
>
>Probably as good as possible, but let me point out places of concern
>and see what you think:
>
>- You propose to do this because their is a "great demand for it", but
>the great demand is generally the stupid demand for images-in-database
>"just because". You must either somehow *stop* those people, or ensure
>your solution is capable of meeting their needs.  Since you propose to
>meet the demand, presumably you aim for the latter.  This *will*
>involve a good deal of non-analytical pre-processing infrastructure to
>convert people's multi-resolution, overlapping/underlapping file
>libraries into a uniform resolution coverage.  I suggest you ignore
>these people.
>- You are going to carry a certain amount of information into the
>database and process data into bands, potentially for external
>storage.  Why not store external data in a format like TIFF that can
>hold all the bands and pyramids, etc?
>- Once you have the idea of external storage, you could as easily move
>to internal storage with the BLOB interface.  Not that I recommend
>this, it's easier to back up and restore a filesystem of files than a
>huge database full of BLOBs.
>- If you step back a bit and don't even bother splitting up the
>rasters into tiles, you can use an existing raster access library like
>GDAL to work with serialized data.
>- Or you could use GRASS as your serialization format and hook into
>that.  Added bonus: free algorithms.
>- Basically the less you muck with creating a "disk format for raster"
>(solved problem) and the more you muck with "integrating raster and
>vector analysis in SQL" (unsolved problem) the more leverage you will
>get.
>- I like your external storage idea. What are the implications for
>CREATE TABLE foo AS SELECT...?
>
>Summary: your proposal is better than any I have seen, addresses
>solving problems that if solved will provide actual new functionality
>and benefit to users, and clearly you've thought this through over
>some time.
>
>So, I look forward to seeing your initial plan of attack for 
>development :)
>
>As you begin looking at the code base, you'll find a few stubs of
>raster support that Sandro built at my request for rasterizing vectors
>into CHIPs in the database... basically the first tip toe down the
>path you have written up so completely.
>
>Go with God.
>
>Paul
>
>
>On Mon, Jul 14, 2008 at 12:43 PM, Pierre Racine
><Pierre.Racine at sbf.ulaval.ca> wrote:
>> Paul Ramsey hasn't yet said my design was:
>>
>> -a "meme" (http://blog.cleverelephant.ca/2008/06/x-my-l.html),
>> -a "waste of time" 
>(http://postgis.refractions.net/pipermail/postgis-devel/2007-Ju
>ly/002653.html)
>> -"useless" 
>(http://postgis.refractions.net/pipermail/postgis-users/2007-Ma
>y/015578.html)
>> -that it was "pointless" 
>(http://postgis.refractions.net/pipermail/postgis-users/2007-Oc
>tober/017250.html)
>> -or that I was a "fool" 
>(http://postgis.refractions.net/pipermail/postgis-users/2007-Oc
>tober/017239.html)
>>
>> So I conclude it's a good design!!! :-)
>>
>> He asked for a good design some time ago for raster in the 
>database 
>(http://postgis.refractions.net/pipermail/postgis-users/2006-Oc
>tober/013628.html)
>>
>> Where is the clever elephant?
>>
>> Pierre
>>
>>>-----Message d'origine-----
>>>De : postgis-users-bounces at postgis.refractions.net
>>>[mailto:postgis-users-bounces at postgis.refractions.net] De la
>>>part de Pierre Racine
>>>Envoyé : 7 juillet 2008 11:30
>>>À : postgis-users at postgis.refractions.net
>>>Cc : warmerdam at pobox.com; nicolas.gignac at msp.gouv.qc.ca;
>>>smarshall at wsi.com
>>>Objet : [postgis-users] PostGIS WKT Raster
>>>
>>>Hi raster people,
>>>
>>>I'm starting a 2-3 year project to develop a web-based application to
>>>automate certain GIS tasks commonly needed in ecological 
>research. The
>>>tool will support queries over VERY large extent based on a 
>Canada-wide
>>>assemblages of raster and vector data layers. The queries will
>>>typically
>>>involve intersecting those layers with buffers defined 
>around points or
>>>transects.
>>>
>>>We would like to implement this system with PostGIS, but it currently
>>>does not to support raster data. We could convert all our data to a
>>>single format (raster or vector) and use other tools, however PostGIS
>>>seems to us the best and most powerful vector analysis tool available
>>>and we would prefer to use it. We would like to develop a unified
>>>toolkit so that the application mostly need not worry about
>>>whether base
>>>layers are in vector or raster format. We are then strong 
>proponents of
>>>having raster functionality in PostGIS. I have reviewed 
>every thread on
>>>this list on the subject. I have analysed them and I have 
>compiled them
>>>in the wiki
>>>(http://postgis.refractions.net/support/wiki/index.php?RasterNotes).
>>>
>>>The argument goes like this: The geodatabase paradigm has 
>been a major
>>>recent enhancement in GIS technology, I feel that the seamless
>>>integration of raster and vector analysis should be one of the next.
>>>Spatial analysis is emerging, beside the making and 
>publishing of maps,
>>>as one of the main desktop and web-GIS applications. Rastor/vector
>>>seamless integration is already done for display (in most 
>GIS and with
>>>MapServer or ArcIMS on the web) but definitely not for
>>>analysis. Desktop
>>>analysts must still learn to use two distinct toolsets within most
>>>(all?) GIS packages: one toolset for raster and another for
>>>vector data.
>>>Would not it be easier to build and use applications if we 
>had a unique
>>>data query and analysis toolset independent of the data 
>model? I don't
>>>know any tool having this approach right now. A PostGIS 
>foundation that
>>>addressed this problem would offer better directions to application
>>>developers than the dichotomic one proposed by ESRI since their
>>>beginning. It would provide the necessary abstraction to develop GIS
>>>with ONE set of analysis tools. I feel this is one good reason to
>>>integrate raster in PostGIS. There is "GIS" in the word "PostGIS".
>>>PostGIS should then provide a COMPLETE GIS data foundation (read
>>>"base"!) for geoapplication developers. I think Steve 
>Marshall's design
>>>is a good step in this direction
>>>(http://postgis.refractions.net/pipermail/postgis-users/2006-De
>>>cember/01
>>>4059.html).
>>>
>>>At the same time, we would have a chance to reconsider the 
>raster model
>>>as a coverage instead of a series of images. Spatial 
>databases got rid
>>>of map sheets by allowing complete vector coverages to be stored as
>>>single table (e.g. the entire United States) This approach 
>should works
>>>for raster data as well. Isn't this the approach taken by ArcSDE?
>>>
>>>In summary, I think we must stop thinking about PostGIS as a
>>>mere vector
>>>data repository to support mapping applications. This is the way most
>>>objectors to raster integration seem to view it. We must think about
>>>PostGIS as a powerfull indexed data analysis tool. Seamless
>>>raster/vector analysis in the database could lead to a major
>>>simplification of geoapplications.
>>>
>>>I prepared a PowerPoint presentation with a complete specification of
>>>raster in PostGIS and an analysis of what should be the result of
>>>overlay operation between raster and vector layers. You can download
>>>this PPT at: 
>http://www.cef-cfr.ca/uploads/Membres/WKTRasterPostGIS.ppt
>>>Please have a look at it, feel free to answer questions I ask and
>>>comment.
>>>
>>>Here are the main propositions of this design:
>>>
>>>-Like a vector feature, rasters are stored as a subtype of 
>"geometry"(a
>>>new WKB/WKT geometry type called RASTER instead of POINT or POLYGON)
>>>-There is no distinction between a raster and a tile. A tile
>>>is a raster
>>>and a table of rasters can be seen as a tile coverage. 
>Hence, contrary
>>>to Oracle GeoRaster, only one table is needed to store a coverage and
>>>there is only one type.
>>>-It support raster in the database AND raster out of the 
>database. This
>>>mailing list has shown that both approaches have their pros and cons.
>>>Let's not impose one approach over the other.
>>>-It supports: multi-band, pyramids and variable nodata value, raster
>>>size, pixel size and pixel type for each row.
>>>-It supports import/export from/to Tiff and JPEG.
>>>
>>>But moreover seamless raster/vector integration is materialized by
>>>theses propositions:
>>>
>>>-A lot of existing vector-only functions are adapted to work
>>>with raster
>>>also.
>>>-Most new raster functions are also implemented to work with vector.
>>>Only basic raster derivation functions are proposed.
>>>-Most vector/vector existing functions are adapted to work seamlessly
>>>with raster/vector or raster/raster
>>>
>>>I also want to ask:
>>>
>>>-What is the status of PGRaster? Is any development now underway?
>>>
>>>We have some resources to devote to this project over the next
>>>few years
>>>and are very interested in forming collaborations to move this work
>>>forward.
>>>
>>>Pierre Racine
>>>GIS/Programmer Analyst
>>>University Laval
>>>http://www.cef-cfr.ca/index.php?n=Membres.PierreRacine
>>>_______________________________________________
>>>postgis-users mailing list
>>>postgis-users at postgis.refractions.net
>>>http://postgis.refractions.net/mailman/listinfo/postgis-users
>>>
>>
>_______________________________________________
>postgis-users mailing list
>postgis-users at postgis.refractions.net
>http://postgis.refractions.net/mailman/listinfo/postgis-users
>
>
>
>      
>_______________________________________________
>postgis-users mailing list
>postgis-users at postgis.refractions.net
>http://postgis.refractions.net/mailman/listinfo/postgis-users
>
_______________________________________________
postgis-users mailing list
postgis-users at postgis.refractions.net
http://postgis.refractions.net/mailman/listinfo/postgis-users