[postgis-devel] [raster] Memory management and IO concerns

[Bryce L Nordgren]

On Wed, Jun 29, 2011 at 2:23 PM, Pierre Racine
<Pierre.Racine at sbf.ulaval.ca>wrote:

>
> What you're missing is that big rasters should be tiled. Period. The limits
> are: 1GB per tile and 32TB per table (per image if you store one image per
> table).
>

I agree. Tiling api's are typically an abstraction layer which do not
necessarily depend on explicit tiling support from the backing store. Tiles
are almost never explicitly managed by the end user, and are usually
transparent. There needs to be a way to go from the raster's coordinates to
the (tile ID, tile coordinates) and back; retrieve and store pixels based on
the raster (not tile) coordinates; a tile cache should exist, should not
require explicit management, etc. Rows are not tiles because they are not
related in any particular way. Rows may even store data from different
images! Tiles need to offer a guarantee that they are tiles from a specific
image. Rows don't do that. Using rows in place of tiles is somewhat analgous
to dividing one big image into many little images with random filenames
(unsorted rows), and dumping them all in a directory (table).

In any case, even with a little 64Mb row ("tile") there needs to be a way to
load only part of the tile. I'm not sure it makes sense to load (and save)
all 64Mb just to set one band's Nodata value, for instance.

To put it another way, the SQL version of MapAlgebra is currently O(N^4). (N
is the raster length along one side). Being able to tile individual rows
could knock that down to O(N^2). Real tiling is probably the single biggest
performance enhancement postgis raster could have.

Bryce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20110629/23a51ade/attachment.html>