[postgis-devel] [raster] Memory management and IO concerns
Bryce L Nordgren
bnordgren at gmail.com
Tue Jun 28 10:50:24 PDT 2011
On Tue, Jun 28, 2011 at 10:08 AM, Mark Cave-Ayland <
mark.cave-ayland at siriusit.co.uk> wrote:
> On 24/06/11 00:04, Bryce L Nordgren wrote:
> Is there a way to encapsulate "DETOAST" logic such that only the raster
>> header, a specific band header, and the corresponding band data are
>> loaded? I'd hate to have to write that logic in more than one function.
>> And the correlary, can you DETOAST different slices as they're needed or
>> do you have to load everything upfront? (e.g., load image blocks?)
> Unfortunately I don't think so; I think you'd have to disable TOAST on the
> raster column and then implement some kind of custom grid compression scheme
> to avoid having to decompress an entire raster as part of the
> PG_DETOAST_DATUM() call. But someone else who knows better may be able to
> correct me here.
If this is correct, the size limitation on in-database rasters is not the
full 32Tb (or whatever) of the Postgresql backend, but the amount of memory
(and swap) present on the server, minus whatever is being used by other
processes (including other raster operations) on the same server. Truly
large images will have to be out-database rasters which GDAL understands how
to "chunk"/"block"/"tile". Unless I'm missing something.
If there is a way to "DETOAST" only part of a raster, it may be possible to
develop a GDAL driver which understands the Postgresql server-side
Actually, I've already hit something else which may be a more stringent
limit: when trying to load a single band of a Landsat image (not tiled,
loaded as a single raster), I got an out of memory error. I assumed this was
in the parser, as the SQL file generated by raster2pgsql.py used a single
"INSERT" statement to plop in the entire 64Mb binary image. I hadn't thought
of this before, but large rasters may need to be loaded via many
"ST_SetRasterDataBlock" calls rather than one INSERT.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the postgis-devel