[postgis-devel] Postgis Raster JPEG 2000 (openjpeg)

Tue Apr 8 08:21:31 PDT 2014

Have you given some thought in how this could be implemented in PostgreSQL?
You stated previously that you'd ideally need nested tables, which is
supported in the manner you require.

You still haven't answered my prior question regarding...

"No, each of the packets would be stored in a row in the packet table, and
the raster tile object would have the FK of each of the packets related to
that tile and maintain a ordered lists of the packet tables FK's of each
progression order."

So, how would a raster query look? Would you have pass in metadata into the
query itself informing the postgis methods where the raster tile's packets
are?

Personally, I can't see how JPEG2000 fits into the PostgreSQL paradigms for
storage and data-access...

My biggest pain points when it comes to PostGIS raster are:

1. TOAST compression/decompression: Beyond just the overhead, this prevents
any performance gains possible from PG_DETOAST_DATUM_SLICE() as slicing
still requires the entire TOAST object to be decompressed.

2. Operational Performance: We need to run an op in parallel on a single
raster tile. PostgreSQL 9.4 introduces dynamic background workers that may
provide the answer to this.

3. Overviews: Easily create, manage, update overviews in the database. I
think a good chuck of this could be done with SQL functions and triggers

4. Metadata: I can't decide if it's better for this to be in the raster
itself or another column in the table. Having the metadata in the raster
seems nice as it travels with the raster. Having the metadata as a column
on the same row is nice as we can index and use standard SQL to query it.
Another request that I occasionally get is the ability to have aliases for
band numbers.

5. Out-db: More out-db capabilities to consume what information is
available in the out-db raster file (metadata, overviews, etc).

As for addressing the pain points...

1. Disable compression in TOAST. Instead the raster format would
selectively compress data (primarily band data) while leaving raster and
band headers alone. Compression/decompression should be loss-less and fast
without an emphasis on compressed size. I'm leaning towards LZ4.

2. Explore dynamic background workers. Before anyone asks, threads are a no
go as per many core PostgreSQL developers.

3. Creating a new overview in the database isn't easy (mostly due to
probable memory issues). Keeping existing overviews in sync with base
rasters can be done with triggers deployed with some maintenance functions.

4. I could see some metadata (band aliases) being in the raster itself but
not others (color-maps, band descriptors).

5. This is a hodge-podge of different efforts such as reading out-db
metadata, registering out-db overview, etc.

-bborie

On Mon, Apr 7, 2014 at 12:54 PM, Nathaniel Clay <clay.nathaniel at gmail.com>wrote:

> Bborie,
>
> There are two JPEG 2000 specifications that apply that are applicable to
> our needs and fulfill some of the requirements that I came up with.
>
> 1. JPIP or JPEG 2000 Photo Interactive Protocol (
> https://www.dropbox.com/s/ikb9ml7vuftvjgn/T-REC-T.808-200501-I%21%21PDF-E.pdf)
> This would give us a well thought out index format that would allow us to
> treat our raster blob as a embedded table. With the ability to insert,
> update and union the rasters without decompressing the tiles in their
> entirety and in most cases decompression would not be necessary at all. (
> https://www.dropbox.com/s/59sdg4yo5rpw0a8/10.1.1.136.7762.pdf )
> 2. OSGEO JPEG 2000 GML (
> https://www.dropbox.com/s/r6i985z6uzabve0/05-047r3_GML_in_JPEG_2000_for_Geographic_Imagery%20%281%29.pdf)
> GML would allow us to annotate raster with vectors and provide spatial and
> temporal reference of the embedded code streams ( tiles).
>
> PS.  Have you collected your thoughts yet on the current short comings?
>
> Thanks,
>
> Nathaniel Hunter Clay
>
>
>
>
> On Mon, Mar 24, 2014 at 3:08 PM, Nathaniel Clay <clay.nathaniel at gmail.com>wrote:
>
>> Bborie,
>>
>> What I really want is Nested Tables, as far as I can tell Postrgresql
>> doesn't fully support it.....
>>
>> thanks,
>>
>> Nathaniel Hunter Clay
>>
>>
>> On Mon, Mar 24, 2014 at 1:37 PM, Bborie Park <dustymugs at gmail.com> wrote:
>>
>>> So not self-contained? For one coverage table containing raster tiles,
>>> you'd have a second hidden table of the packets? How would a function know
>>> where the packet table is?
>>>
>>> -bborie
>>>
>>>
>>> On Mon, Mar 24, 2014 at 9:41 AM, Nathaniel Clay <
>>> clay.nathaniel at gmail.com> wrote:
>>>
>>>> Bborie,
>>>>
>>>> No, each of the packets would be stored in a row in the packet table,
>>>> and the raster tile object would have the FK of each of the packets related
>>>> to that tile and maintain a ordered lists of the packet tables FK's of each
>>>> progression order.
>>>>
>>>> Thanks,
>>>>
>>>> Nathaniel Hunter Clay
>>>>
>>>>
>>>> _______________________________________________
>>>> postgis-devel mailing list
>>>> postgis-devel at lists.osgeo.org
>>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> postgis-devel mailing list
>>> postgis-devel at lists.osgeo.org
>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>>
>>
>>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20140408/aedbb33e/attachment.html>