[postgis-devel] Final section, raster architecture document

[Bryce L Nordgren]

On Tue, Jun 28, 2011 at 10:43 PM, Chris Hodgson <chodgson at refractions.net>wrote:

> Maybe I'm taking this out of context, but this seems like a pretty serious
> drawback, with the expectation being that any raster big enough to be worth
> putting in the database is quite likely to be larger than can fit in memory
> (or in 1/3 of memory even, given similarly sized inputs and output). Or are
> the rasters that we are talking about being loaded, just the smaller
> "chunks"? Regardless I don't think it can be assumed that any arbitrary
> raster we wish to operate on will necessarily be chunked/gridded into an
> appropriate size, and it seems to me that a "Grid iteration engine" for the
> purposes of pgraster, would need to able to read chunks of input and write
> chunks of output, of a reasonable size.
>
> Again, sorry if I've misunderstood you.
>
>
Nope, you got it exactly right: but it's not a problem specific to the "grid
iteration engine". Currently, all raster functions except the "raster header
functions" (get/set width/height, etc.) load the entire raster into memory
(all spatial areas, all bands). Anything that accesses bands will load the
entire raster, even if a single band's metadata is to be changed/examined
(e.g., SetBandNodataValue). Getting or setting a single pixel loads the
entire raster (and stores it too, in the case of "setting"). If the
non-raster-header function has two rasters in the method signature, the
function loads two complete rasters into memory, etc.

The fix, if one is even possible, is currently unknown, according to Mark's
message:
http://postgis.refractions.net/pipermail/postgis-devel/2011-June/013960.html
However, it seems worth having someone look at it.

Note this applies to "in-database" rasters, and the loading occurs with the
DETOAST_DATUM, according to Paul's message:
http://postgis.refractions.net/pipermail/postgis-devel/2011-June/013882.html
Out-db rasters should stand to benefit from GDAL's "blocking" code.  The
moral of the story seems to be that large rasters cannot be stored
in-database until this is fixed.

If this is fixed for rasters in general, the same fix can be applied to the
grid iteration engine. I don't know enough about this to even take a stab at
a fix...but I my code can wait for one along with the rest of the codebase.
;)

Bryce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20110629/7a58ae8a/attachment.html>