[postgis-users] PostGIS WKT Raster

[Paul Ramsey]

Pierre,

Firstly, let me commend you for your understanding of the open source
community process! Not being shy, not taking "no" for an answer, and
needling the people you need to move are all excellent ways to keep
the wheels turning, particularly when done with good humor as you have
shown.  Other first timers could/should learn from your example!

So now I owe you a reply, no doubt:

You have gotten over my first hurdle, in that you have an actual use
case for raster, that is more involved than "I want to my images,
because I hear database are faster".

I have long thought that analysis, in particular a fusing of vector
and raster analysis was the only really compelling use case for raster
in database, so again, you're on the side of the angels (me).

So, is your *design* good?

Probably as good as possible, but let me point out places of concern
and see what you think:

- You propose to do this because their is a "great demand for it", but
the great demand is generally the stupid demand for images-in-database
"just because". You must either somehow *stop* those people, or ensure
your solution is capable of meeting their needs.  Since you propose to
meet the demand, presumably you aim for the latter.  This *will*
involve a good deal of non-analytical pre-processing infrastructure to
convert people's multi-resolution, overlapping/underlapping file
libraries into a uniform resolution coverage.  I suggest you ignore
these people.
- You are going to carry a certain amount of information into the
database and process data into bands, potentially for external
storage.  Why not store external data in a format like TIFF that can
hold all the bands and pyramids, etc?
- Once you have the idea of external storage, you could as easily move
to internal storage with the BLOB interface.  Not that I recommend
this, it's easier to back up and restore a filesystem of files than a
huge database full of BLOBs.
- If you step back a bit and don't even bother splitting up the
rasters into tiles, you can use an existing raster access library like
GDAL to work with serialized data.
- Or you could use GRASS as your serialization format and hook into
that.  Added bonus: free algorithms.
- Basically the less you muck with creating a "disk format for raster"
(solved problem) and the more you muck with "integrating raster and
vector analysis in SQL" (unsolved problem) the more leverage you will
get.
- I like your external storage idea. What are the implications for
CREATE TABLE foo AS SELECT...?

Summary: your proposal is better than any I have seen, addresses
solving problems that if solved will provide actual new functionality
and benefit to users, and clearly you've thought this through over
some time.

So, I look forward to seeing your initial plan of attack for development :)

As you begin looking at the code base, you'll find a few stubs of
raster support that Sandro built at my request for rasterizing vectors
into CHIPs in the database... basically the first tip toe down the
path you have written up so completely.

Go with God.

Paul


On Mon, Jul 14, 2008 at 12:43 PM, Pierre Racine
<Pierre.Racine at sbf.ulaval.ca> wrote:
> Paul Ramsey hasn't yet said my design was:
>
> -a "meme" (http://blog.cleverelephant.ca/2008/06/x-my-l.html),
> -a "waste of time" (http://postgis.refractions.net/pipermail/postgis-devel/2007-July/002653.html)
> -"useless" (http://postgis.refractions.net/pipermail/postgis-users/2007-May/015578.html)
> -that it was "pointless" (http://postgis.refractions.net/pipermail/postgis-users/2007-October/017250.html)
> -or that I was a "fool" (http://postgis.refractions.net/pipermail/postgis-users/2007-October/017239.html)
>
> So I conclude it's a good design!!! :-)
>
> He asked for a good design some time ago for raster in the database (http://postgis.refractions.net/pipermail/postgis-users/2006-October/013628.html)
>
> Where is the clever elephant?
>
> Pierre
>
>>-----Message d'origine-----
>>De : postgis-users-bounces at postgis.refractions.net
>>[mailto:postgis-users-bounces at postgis.refractions.net] De la
>>part de Pierre Racine
>>Envoyé : 7 juillet 2008 11:30
>>À : postgis-users at postgis.refractions.net
>>Cc : warmerdam at pobox.com; nicolas.gignac at msp.gouv.qc.ca;
>>smarshall at wsi.com
>>Objet : [postgis-users] PostGIS WKT Raster
>>
>>Hi raster people,
>>
>>I'm starting a 2-3 year project to develop a web-based application to
>>automate certain GIS tasks commonly needed in ecological research. The
>>tool will support queries over VERY large extent based on a Canada-wide
>>assemblages of raster and vector data layers. The queries will
>>typically
>>involve intersecting those layers with buffers defined around points or
>>transects.
>>
>>We would like to implement this system with PostGIS, but it currently
>>does not to support raster data. We could convert all our data to a
>>single format (raster or vector) and use other tools, however PostGIS
>>seems to us the best and most powerful vector analysis tool available
>>and we would prefer to use it. We would like to develop a unified
>>toolkit so that the application mostly need not worry about
>>whether base
>>layers are in vector or raster format. We are then strong proponents of
>>having raster functionality in PostGIS. I have reviewed every thread on
>>this list on the subject. I have analysed them and I have compiled them
>>in the wiki
>>(http://postgis.refractions.net/support/wiki/index.php?RasterNotes).
>>
>>The argument goes like this: The geodatabase paradigm has been a major
>>recent enhancement in GIS technology, I feel that the seamless
>>integration of raster and vector analysis should be one of the next.
>>Spatial analysis is emerging, beside the making and publishing of maps,
>>as one of the main desktop and web-GIS applications. Rastor/vector
>>seamless integration is already done for display (in most GIS and with
>>MapServer or ArcIMS on the web) but definitely not for
>>analysis. Desktop
>>analysts must still learn to use two distinct toolsets within most
>>(all?) GIS packages: one toolset for raster and another for
>>vector data.
>>Would not it be easier to build and use applications if we had a unique
>>data query and analysis toolset independent of the data model? I don't
>>know any tool having this approach right now. A PostGIS foundation that
>>addressed this problem would offer better directions to application
>>developers than the dichotomic one proposed by ESRI since their
>>beginning. It would provide the necessary abstraction to develop GIS
>>with ONE set of analysis tools. I feel this is one good reason to
>>integrate raster in PostGIS. There is "GIS" in the word "PostGIS".
>>PostGIS should then provide a COMPLETE GIS data foundation (read
>>"base"!) for geoapplication developers. I think Steve Marshall's design
>>is a good step in this direction
>>(http://postgis.refractions.net/pipermail/postgis-users/2006-De
>>cember/01
>>4059.html).
>>
>>At the same time, we would have a chance to reconsider the raster model
>>as a coverage instead of a series of images. Spatial databases got rid
>>of map sheets by allowing complete vector coverages to be stored as
>>single table (e.g. the entire United States) This approach should works
>>for raster data as well. Isn't this the approach taken by ArcSDE?
>>
>>In summary, I think we must stop thinking about PostGIS as a
>>mere vector
>>data repository to support mapping applications. This is the way most
>>objectors to raster integration seem to view it. We must think about
>>PostGIS as a powerfull indexed data analysis tool. Seamless
>>raster/vector analysis in the database could lead to a major
>>simplification of geoapplications.
>>
>>I prepared a PowerPoint presentation with a complete specification of
>>raster in PostGIS and an analysis of what should be the result of
>>overlay operation between raster and vector layers. You can download
>>this PPT at: http://www.cef-cfr.ca/uploads/Membres/WKTRasterPostGIS.ppt
>>Please have a look at it, feel free to answer questions I ask and
>>comment.
>>
>>Here are the main propositions of this design:
>>
>>-Like a vector feature, rasters are stored as a subtype of "geometry"(a
>>new WKB/WKT geometry type called RASTER instead of POINT or POLYGON)
>>-There is no distinction between a raster and a tile. A tile
>>is a raster
>>and a table of rasters can be seen as a tile coverage. Hence, contrary
>>to Oracle GeoRaster, only one table is needed to store a coverage and
>>there is only one type.
>>-It support raster in the database AND raster out of the database. This
>>mailing list has shown that both approaches have their pros and cons.
>>Let's not impose one approach over the other.
>>-It supports: multi-band, pyramids and variable nodata value, raster
>>size, pixel size and pixel type for each row.
>>-It supports import/export from/to Tiff and JPEG.
>>
>>But moreover seamless raster/vector integration is materialized by
>>theses propositions:
>>
>>-A lot of existing vector-only functions are adapted to work
>>with raster
>>also.
>>-Most new raster functions are also implemented to work with vector.
>>Only basic raster derivation functions are proposed.
>>-Most vector/vector existing functions are adapted to work seamlessly
>>with raster/vector or raster/raster
>>
>>I also want to ask:
>>
>>-What is the status of PGRaster? Is any development now underway?
>>
>>We have some resources to devote to this project over the next
>>few years
>>and are very interested in forming collaborations to move this work
>>forward.
>>
>>Pierre Racine
>>GIS/Programmer Analyst
>>University Laval
>>http://www.cef-cfr.ca/index.php?n=Membres.PierreRacine
>>_______________________________________________
>>postgis-users mailing list
>>postgis-users at postgis.refractions.net
>>http://postgis.refractions.net/mailman/listinfo/postgis-users
>>
>