[postgis-users] A PostGIS-Raster data proposal

[John Novak]

>From what I've been able to find re toasted columns; they're stored
somewhere in the database as a separate table, outside the referencing row,
and limited to 1GB.  The 1GB size may be significant in that some raster
formats can be larger.  Not a show-stopper, but it would require repackaging
of very large jpeg2000s, for example.

Given that raster archives can be many TB in size, what are the
ramifications for backup and restore of a database containing toasted
rasters ?

Since toasted values are in a separate table, can I backup and restore the
primary table rows sans toasted columns, and manage the toast in slices ?
What happens if my raster table gets, um, toasted  ?  Hopefully, a full
multiTB  restore/reorg would not be necessary before I'm running again.

John Novak



On 10/24/06, Frank Warmerdam <warmerdam at pobox.com> wrote:
>
>
> Steve,
>
> Excellent summary of the usual sequence of these conversations.
>
> Marshall, Steve wrote:
> > The code that really seems well-adapted to this problem is the Kakadu
> > package written by David Taubman, one of the originators of JPEG 2000.
> > Unfortunately, this library is not open source.  Kakadu has been
> > included as an optional component of other open source projects, like
> > GDAL.  However, I think Kakadu's license would come into conflict with
> > the GPL used by PostGIS.   I'm not a lawyer, but I think this
> > conflict could be overcome if PostGIS could be release under dual
> > licenses, such as GPL or LGPL.  Whether or not its desirable to include
> > of Kakadu in a PostGIS extension is another question.
>
> As mentioned by others I think the licensing conflicts may be more
> fundamental then you suggest.
>
> By the way, I've implemented "virtual" jpeg2000 access in GDAL for
> both the Kakadu library and the ECW library.  The ECW library supports
> jpeg2000 and ecw by the way.  I suspect it is also possible for the
> MrSID library by I haven't done it myself.
>
> > If anyone has knowledge of other JPEG 2000 codecs that have these
> > low-level access capabilities, I'd be very happy to hear about them.
> > Also, if I've mischaracterized any of the codecs, I'd love to be
> corrected.
> >
> > In any event, I'm curious to see if there is significant interest in an
> > implementation of JPEG2000 raster data type within PostGIS.  If so, I
> > think I could dedicate a significant amount of my time over the next
> > several months, as well as perhaps some funding from my employer,
> > depending upon whether some of the issues I raised above can be
> resolved.
>
> I'm not sure I entirely buy into the idea of using jpeg2000 as a special
> native raster data type.  There are a variety of limitations, and odd
> performance characteristics with jpeg2000 which would make me nervous.
>
> I would suggest instead the same general approach of treating BLOBs
> in the database as seekable objects from which components could be
> extracted, but instead of restricting things to jpeg2000, allow any
> GDAL supported raster format to be in the database, and use GDAL to
> access them in the database.
>
> Actually, when I say "any" format, I really mean any format in GDAL
> that supports the virtualization interface where all IO is routed
> through VSIFOpenL() and related functions.  This is a lot of the
> drivers, but by no means all.  This would also allow use of formats
> like tiled GeoTIFF with overviews, from which chunks or overviews
> can be efficiently extracted with much better understood performance
> characteristics.
>
> If you have the time to work on a prototype, then I'd say go for it.
> If you are interested in using GDAL, then I'd be willing to provide
> advice and answers on GDAL issues.
>
> PS. I'd encourage you do some sort of performance testing to see
> if there is a lot of overhead in doing random seeks in large
> toasted blobs in postgres.  I don't understand the mechanism well,
> but it seems like it could be an issue, and better to find out
> sooner rather than later.
>
> Best regards,
> --
>
> ---------------------------------------+--------------------------------------
> I set the clouds in motion - turn up   | Frank Warmerdam,
> warmerdam at pobox.com
> light and sound - activate the windows | http://pobox.com/~warmerdam
> and watch the world go round - Rush    | President OSGeo, http://osgeo.org
>
> _______________________________________________
> postgis-users mailing list
> postgis-users at postgis.refractions.net
> http://postgis.refractions.net/mailman/listinfo/postgis-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-users/attachments/20061025/90ac337b/attachment.html>