<div dir="ltr">Have you given some thought in how this could be implemented in PostgreSQL? You stated previously that you'd ideally need nested tables, which is supported in the manner you require.<div><br></div><div>You still haven't answered my prior question regarding...</div>
<div><br></div><div>"<span style="font-family:arial,sans-serif;font-size:13px">No, each of the packets would be stored in a row in the packet table, and the raster tile object would have the FK of each of the packets related to that tile and maintain a ordered lists of the packet tables FK's of each progression order."</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">So, how would a raster query look? Would you have pass in metadata into the query itself informing the postgis methods where the raster tile's packets are?</span></div>
<div><br></div><div>Personally, I can't see how JPEG2000 fits into the PostgreSQL paradigms for storage and data-access...</div><div><br></div><div>My biggest pain points when it comes to PostGIS raster are:</div><div>
<br></div><div>1. TOAST compression/decompression: Beyond just the overhead, this prevents any performance gains possible from PG_DETOAST_DATUM_SLICE() as slicing still requires the entire TOAST object to be decompressed.</div>
<div><br></div><div>2. Operational Performance: We need to run an op in parallel on a single raster tile. PostgreSQL 9.4 introduces dynamic background workers that may provide the answer to this.</div><div><br></div><div>
3. Overviews: Easily create, manage, update overviews in the database. I think a good chuck of this could be done with SQL functions and triggers</div><div><br></div><div>4. Metadata: I can't decide if it's better for this to be in the raster itself or another column in the table. Having the metadata in the raster seems nice as it travels with the raster. Having the metadata as a column on the same row is nice as we can index and use standard SQL to query it. Another request that I occasionally get is the ability to have aliases for band numbers.</div>
<div><br></div><div>5. Out-db: More out-db capabilities to consume what information is available in the out-db raster file (metadata, overviews, etc).</div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div>As for addressing the pain points...</div><div><br></div><div>1. Disable compression in TOAST. Instead the raster format would selectively compress data (primarily band data) while leaving raster and band headers alone. Compression/decompression should be loss-less and fast without an emphasis on compressed size. I'm leaning towards LZ4.</div>
<div><br></div><div>2. Explore dynamic background workers. Before anyone asks, threads are a no go as per many core PostgreSQL developers.</div><div><br></div><div>3. Creating a new overview in the database isn't easy (mostly due to probable memory issues). Keeping existing overviews in sync with base rasters can be done with triggers deployed with some maintenance functions.</div>
<div><br></div><div>4. I could see some metadata (band aliases) being in the raster itself but not others (color-maps, band descriptors).</div><div><br></div><div>5. This is a hodge-podge of different efforts such as reading out-db metadata, registering out-db overview, etc.</div>
<div><br></div><div><span style="font-family:arial,sans-serif;font-size:13px">-bborie</span></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Apr 7, 2014 at 12:54 PM, Nathaniel Clay <span dir="ltr"><<a href="mailto:clay.nathaniel@gmail.com" target="_blank">clay.nathaniel@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Bborie,<div><br></div><div>There are two JPEG 2000 specifications that apply that are applicable to our needs and fulfill some of the requirements that I came up with. </div>
<div><br></div><div>1. JPIP or JPEG 2000 Photo Interactive Protocol ( <a href="https://www.dropbox.com/s/ikb9ml7vuftvjgn/T-REC-T.808-200501-I%21%21PDF-E.pdf" target="_blank">https://www.dropbox.com/s/ikb9ml7vuftvjgn/T-REC-T.808-200501-I%21%21PDF-E.pdf</a> )</div>
<div>This would give us a well thought out index format that would allow us to treat our raster blob as a embedded table. With the ability to insert, update and union the rasters without decompressing the tiles in their entirety and in most cases decompression would not be necessary at all. ( <a href="https://www.dropbox.com/s/59sdg4yo5rpw0a8/10.1.1.136.7762.pdf" target="_blank">https://www.dropbox.com/s/59sdg4yo5rpw0a8/10.1.1.136.7762.pdf</a> )</div>
<div>2. OSGEO JPEG 2000 GML ( <a href="https://www.dropbox.com/s/r6i985z6uzabve0/05-047r3_GML_in_JPEG_2000_for_Geographic_Imagery%20%281%29.pdf" target="_blank">https://www.dropbox.com/s/r6i985z6uzabve0/05-047r3_GML_in_JPEG_2000_for_Geographic_Imagery%20%281%29.pdf</a> )</div>
<div>GML would allow us to annotate raster with vectors and provide spatial and temporal reference of the embedded code streams ( tiles). </div><div><br></div><div>PS. Have you collected your thoughts yet on the current short comings?</div>
<div><br></div><div>Thanks,</div><div><br></div><div>Nathaniel Hunter Clay</div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 24, 2014 at 3:08 PM, Nathaniel Clay <span dir="ltr"><<a href="mailto:clay.nathaniel@gmail.com" target="_blank">clay.nathaniel@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Bborie,<br><br></div>What I really want is Nested Tables, as far as I can tell Postrgresql doesn’t fully support it.....<br>
<br></div>thanks,<br><br></div>Nathaniel Hunter Clay<br></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Mon, Mar 24, 2014 at 1:37 PM, Bborie Park <span dir="ltr"><<a href="mailto:dustymugs@gmail.com" target="_blank">dustymugs@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">So not self-contained? For one coverage table containing raster tiles, you'd have a second hidden table of the packets? How would a function know where the packet table is?<span><font color="#888888"><div>
<br></div><div>-bborie</div>
</font></span></div><div class="gmail_extra"><br><br><div class="gmail_quote"><div><div>On Mon, Mar 24, 2014 at 9:41 AM, Nathaniel Clay <span dir="ltr"><<a href="mailto:clay.nathaniel@gmail.com" target="_blank">clay.nathaniel@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div><div><div>Bborie,<br><br></div>No, each of the packets would be stored in a row in the packet table, and the raster tile object would have the FK of each of the packets related to that tile and maintain a ordered lists of the packet tables FK's of each progression order.<br>
<br></div>Thanks,<br><br></div>Nathaniel Hunter Clay<br><br></div>
<br></div></div><div>_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org" target="_blank">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br></div></blockquote></div><br></div>
<br>_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org" target="_blank">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
postgis-devel mailing list<br>
<a href="mailto:postgis-devel@lists.osgeo.org">postgis-devel@lists.osgeo.org</a><br>
<a href="http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel" target="_blank">http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel</a><br></blockquote></div><br></div>