[postgis-devel] Postgis Raster JPEG 2000 (openjpeg)
Bborie Park
dustymugs at gmail.com
Thu Feb 20 06:48:36 PST 2014
Storing directly in the filesystem is essentially the out-db mechanism that
already exists. And since that goes through GDAL, any format that was
compiled into the local GDAL library is supported.
-bborie
On Thu, Feb 20, 2014 at 12:58 AM, Rémi Cura <remi.cura at gmail.com> wrote:
> You may add some advantages for Jpeg 2000.
> _It is the defacto standard for satellite raster.
> _powerfull open source tools exist (parallelized and memory safe :
> http://orfeo-toolbox.org/otb/)
> _it is way more advanced than other compression type (wavelet vs fourier).
>
> About TOAST table, I think one can desactivate the compression done by
> postgres in toast table (
> http://www.postgresql.org/docs/9.3/static/storage-toast.html , PLAIN).
>
> Maybe if you want to store it directly in filesystem, instead of Blob or
> Toast you could look into the direction of Foregin Data Wrapper (
> http://www.postgresql.org/docs/9.3/static/postgres-fdw.html).
> It would have the disadvantage of being outside DB (so non of the ACID
> guarantees), but the advantage of being possibly faster to read (without DB
> IO overhead), and possibly read by several other tools (like geographic
> server). It would be cleaner than abuse Toast (small) api.
> It is quite new and I don't know how it performs though.
>
> Cheers,
>
> Rémi-C
>
>
> Cheers,
> Rémi-C
>
>
>
>
> 2014-02-18 18:59 GMT+01:00 Paul Ramsey <pramsey at cleverelephant.ca>:
>
> Note that PgSQL will be applying libz compression to over-sized tuples
>> automatically in the TOAST machinery, so applying it yourself won't
>> radically change anything, unless it's being done carefully within a
>> blocking scheme, perhaps.
>>
>> Also note that no matter what you do, in-database data is going to be cut
>> up into page-size hunks and dropped into tables, it's never stored directly
>> on the file system. That's true for blobs, and true for toasted tuples too.
>>
>> P.
>>
>> --
>> Paul Ramsey
>> http://cleverelephant.ca
>> http://postgis.net
>>
>> On February 18, 2014 at 9:56:02 AM, Nathaniel Clay (
>> clay.nathaniel at gmail.com <//clay.nathaniel at gmail.com>) wrote:
>>
>>
>>
>>
>> On Tue, Feb 18, 2014 at 11:24 AM, Bborie Park <dustymugs at gmail.com>wrote:
>>
>>> How does OpenJPEG 2000 compare to other formats, such as GeoTIFF, HDF5
>>> or NetCDF?
>>>
>> All of the mentioned, formats use libz compression, a loss less
>> compression scheme. With the exception that GTiff offers plain jpeg
>> compression along with a fax format??
>> I think it may be a good idea to also ofter libz (deflate) and or LZMA
>> compression.
>>
>> Before I move on to the next questions a little background on my thought
>> process.
>> I see three different use types:
>> 1. Active Server
>>
>> Before deciding on a replacement format, we should identify if a
>>> replacement is required? What are the current short-comings? Where are the
>>> performance limitations? What are the wish-list items for a replacement
>>> format?
>>>
>>> I have spent some time at poking at what I considered short-comings,
>>> where I think performance is suffering, and what features are missing. I'm
>>> not ready to publicly share my list though.
>>>
>>> I wasn't involved with the original PostGIS raster binary format but
>>> from my understanding while working it with, it appears as though the goal
>>> was to keep things simple as all input and output goes through PostgreSQL,
>>> which can (should?) be considered a black-box.
>>>
>>> -bborie
>>>
>>>
>>> On Mon, Feb 17, 2014 at 6:32 PM, Nathaniel Clay <
>>> clay.nathaniel at gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>>
>>>> This is just some food for thought and discussion. I've take a cursory
>>>> look at JPEG 2000 openjpeg. I think it may hold promise for a possible
>>>> future format or co-format for Postgis Raster in database storage.
>>>>
>>>> - Supports both Loss less and Lossful Compression approaching the
>>>> performance of ERDAS ( uncompressed Gtif 200M, openjpeg JP2 25M Lossfull
>>>> and 80M Lossless, ERDAS JP2 15M)
>>>> -
>>>> - I don't have a ERADAS encoder license, but this image was of
>>>> the same original
>>>> - Openjpeg lossfull compression in my opinion visually identical
>>>> to the original with default settings
>>>> - Supports N band imagery
>>>> - Multi-Resolution Packing
>>>> -
>>>> - eg more than one raster resolution with in the "code stream"
>>>> - can be ordered by user specification, default smallest to
>>>> largest
>>>> - This would allow us to get rid of the overview tables or make
>>>> them views.
>>>> -
>>>> http://ericportis.com/posts/2014/resolution-progressive-jpeg2000-performance/
>>>> - Extensible header
>>>> -
>>>> - SRID, IPX,IPY, etc
>>>> - Area of interest specification
>>>> -
>>>> - Allows you to specify a region with in the same raster to have
>>>> a higher bitrate, etc
>>>> - Error Detection and Correction
>>>> - Openjpeg project seems to be alive last commit was a few days a
>>>> go.
>>>> - BSD License
>>>>
>>>>
>>>> Possible Concerns identified so far:
>>>>
>>>> - Writes require a recode of the entire tile, however the tile size
>>>> is user specifiable
>>>> - takes a long time to encode image, ~10 sec on my system with the
>>>> test image ( 3067x3777 8bit 4bands)
>>>> -
>>>> - I dont have anything to compare it to, so I dont know if this
>>>> is a good or bad
>>>> - I havent tested on small image size and other possible
>>>> scenarios.
>>>> - Memory Usage seemed high during decode and encode, ~2G, I
>>>> believe this is due to the decoder and encoder preallocate memory for all
>>>> of the "Pyramids" and bands at once. (but have not investigated thoroughly)
>>>> ( Image Res 3067x 3777 8bit 4 bands)
>>>> - Its included decoder application has weird issues with tif, other
>>>> formats are fine. ( Dont think it has anything to do with the openjpeg lib)
>>>> -
>>>> - May be indective of other problems
>>>>
>>>> If I were to start on this project it would be some time in the future
>>>> and I would most likely require help and lots of input, but I thought I it
>>>> would be a good idea to bring, it up as a discussion point.
>>>>
>>>> Any thoughts,
>>>>
>>>>
>>>> Nathaniel Hunter Clay
>>>>
>>>> _______________________________________________
>>>> postgis-devel mailing list
>>>> postgis-devel at lists.osgeo.org
>>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>>>
>>>
>>>
>>> _______________________________________________
>>> postgis-devel mailing list
>>> postgis-devel at lists.osgeo.org
>>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>>
>>
>> _______________________________________________
>> postgis-devel mailing list
>> postgis-devel at lists.osgeo.org
>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>
>>
>> _______________________________________________
>> postgis-devel mailing list
>> postgis-devel at lists.osgeo.org
>> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>>
>
>
> _______________________________________________
> postgis-devel mailing list
> postgis-devel at lists.osgeo.org
> http://lists.osgeo.org/cgi-bin/mailman/listinfo/postgis-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/postgis-devel/attachments/20140220/470a302d/attachment.html>
More information about the postgis-devel
mailing list