[postgis-devel] Postgis Raster JPEG 2000 (openjpeg)
Rémi Cura
remi.cura at gmail.com
Thu Feb 20 00:58:50 PST 2014
You may add some advantages for Jpeg 2000.
_It is the defacto standard for satellite raster.
_powerfull open source tools exist (parallelized and memory safe :
http://orfeo-toolbox.org/otb/)
_it is way more advanced than other compression type (wavelet vs fourier).
About TOAST table, I think one can desactivate the compression done by
postgres in toast table (
http://www.postgresql.org/docs/9.3/static/storage-toast.html , PLAIN).
Maybe if you want to store it directly in filesystem, instead of Blob or
Toast you could look into the direction of Foregin Data Wrapper (
http://www.postgresql.org/docs/9.3/static/postgres-fdw.html).
It would have the disadvantage of being outside DB (so non of the ACID
guarantees), but the advantage of being possibly faster to read (without DB
IO overhead), and possibly read by several other tools (like geographic
server). It would be cleaner than abuse Toast (small) api.
It is quite new and I don't know how it performs though.
Cheers,
Rémi-C
Cheers,
Rémi-C
2014-02-18 18:59 GMT+01:00 Paul Ramsey <pramsey at cleverelephant.ca>:
> Note that PgSQL will be applying libz compression to over-sized tuples
> automatically in the TOAST machinery, so applying it yourself won’t
> radically change anything, unless it’s being done carefully within a
> blocking scheme, perhaps.
>
> Also note that no matter what you do, in-database data is going to be cut
> up into page-size hunks and dropped into tables, it’s never stored directly
> on the file system. That’s true for blobs, and true for toasted tuples too.
>
> P.
>
> --
> Paul Ramsey
> http://cleverelephant.ca
> http://postgis.net
>
> On February 18, 2014 at 9:56:02 AM, Nathaniel Clay (
> clay.nathaniel at gmail.com <//clay.nathaniel at gmail.com>) wrote:
>
>
>
>
> On Tue, Feb 18, 2014 at 11:24 AM, Bborie Park <dustymugs at gmail.com> wrote:
>
>> How does OpenJPEG 2000 compare to other formats, such as GeoTIFF, HDF5 or
>> NetCDF?
>>
> All of the mentioned, formats use libz compression, a loss less
> compression scheme. With the exception that GTiff offers plain jpeg
> compression along with a fax format??
> I think it may be a good idea to also ofter libz (deflate) and or LZMA
> compression.
>
> Before I move on to the next questions a little background on my thought
> process.
> I see three different use types:
> 1. Active Server
>
> Before deciding on a replacement format, we should identify if a
>> replacement is required? What are the current short-comings? Where are the
>> performance limitations? What are the wish-list items for a replacement
>> format?
>>
>> I have spent some time at poking at what I considered short-comings,
>> where I think performance is suffering, and what features are missing. I'm
>> not ready to publicly share my list though.
>>
>> I wasn't involved with the original PostGIS raster binary format but from
>> my understanding while working it with, it appears as though the goal was
>> to keep things simple as all input and output goes through PostgreSQL,
>> which can (should?) be considered a black-box.
>>
>> -bborie
>>
>>
>> On Mon, Feb 17, 2014 at 6:32 PM, Nathaniel Clay <
>> clay.nathaniel at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>>
>>> This is just some food for thought and discussion. I've take a cursory
>>> look at JPEG 2000 openjpeg. I think it may hold promise for a possible
>>> future format or co-format for Postgis Raster in database storage.
>>>
>>> - Supports both Loss less and Lossful Compression approaching the
>>> performance of ERDAS ( uncompressed Gtif 200M, openjpeg JP2 25M Lossfull
>>> and 80M Lossless, ERDAS JP2 15M)
>>> -
>>> - I don't have a ERADAS encoder license, but this image was of
>>> the same original
>>> - Openjpeg lossfull compression in my opinion visually identical
>>> to the original with default settings
>>> - Supports N band imagery
>>> - Multi-Resolution Packing
>>> -
>>> - eg more than one raster resolution with in the "code stream"
>>> - can be ordered by user specification, default smallest to
>>> largest
>>> - This would allow us to get rid of the overview tables or make
>>> them views.
>>> -
>>> http://ericportis.com/posts/2014/resolution-progressive-jpeg2000-performance/
>>> - Extensible header
>>> -
>>> - SRID, IPX,IPY, etc
>>> - Area of interest specification
>>> -
>>> - Allows you to specify a region with in the same raster to have
>>> a higher bitrate, etc
>>> - Error Detection and Correction
>>> - Openjpeg project seems to be alive last commit was a few days a go.
>>> - BSD License
>>>
>>>
>>> Possible Concerns identified so far:
>>>
>>> - Writes require a recode of the entire tile, however the tile size
>>> is user specifiable
>>> - takes a long time to encode image, ~10 sec on my system with the
>>> test image ( 3067x3777 8bit 4bands)
>>> -
>>> - I dont have anything to compare it to, so I dont know if this
>>> is a good or bad
>>> - I havent tested on small image size and other possible
>>> scenarios.
>>> - Memory Usage seemed high during decode and encode, ~2G, I
>>> believe this is due to the decoder and encoder preallocate memory for all
>>> of the "Pyramids" and bands at once. (but have not investigated thoroughly)
>>> ( Image Res 3067x 3777 8bit 4 bands)
>>> - Its included decoder application has weird issues with tif, other
>>> formats are fine. ( Dont think it has anything to do with the openjpeg lib)
>>> -
>>> - May be indective of other problems
>>>
>>> If I were to start on this project it would be some time in the future
>>> and I would most likely require help and lots of input, but I thought I it
>>> would be a good idea to bring, it up as a discussion point.
>>>
>>> Any thoughts,
>>>
>>>
>>> Nathaniel Hunter Clay
>>>
>>> _______________________________________________
>>> postgis-devel mailing list
>>> postgis-devel at lists.osgeo.org
>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>>
>>
>>
>> _______________________________________________
>> postgis-devel mailing list
>> postgis-devel at lists.osgeo.org
>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20140220/d602fdbd/attachment.html>
More information about the postgis-devel
mailing list