[gdal-dev] OpenJPEG

[Julien Malik]

Hi Aaron & Even,

Decoding at preccints level in OpenJPEG would definitely be a great 
addition.

I believe OpenJPEG performance could also be improved if multi-threading 
was managed directly inside the lib, since in the current situation 
OpenJPEG is purely monothreaded.
 From my experience, Kakadu does a great job in keeping all your cores 
occupied close to 100% during decoding, even when decoding a single tile 
(so this is probably linked to the capability to manage independent 
decoding at the precinct level).
In 3 geospatial software I know leverage on OpenJPEG (gdal, orfeo 
toolbox, and snap) to decode big datasets (think tens or hundreds of 
thousand pixels in each dimension), multithreading is handled from the 
user side, by creating a number of independent contexts and run them in 
separate threads to improve performance.
This means we are all interpreting the codestreams and seeking into them 
independently, which clearly sounds suboptimal.
I suspect managing multi-threading at a lower level directly inside 
OpenJPEG could open the doors to much better resource management (cores 
occupation, reduced I/O, etc...).

I certainly concur with Even with the "as fast as it can be" :)
Any percent you gain has a direct visible impact when (de)coding huge 
datasets.

Regards,
Julien

On 20/12/2015 11:00, Even Rouault wrote:
> Aaron,
>
>> I am working on some performance enhancements for OpenJPEG,
>> and I am curious about people's experience using the OpenJPEG driver with
>> GDAL:
>>
>> Does it have all the features you need, if not what is missing?
> Being the developer of the driver, I believe it must be close to using the
> available capabilities of OpenJPEG as best as possible. I may have overlooked
> some features, so other eyes checking the code are always welcome.
>
>  From my point of view what is mainly missing in OpenJPEG (the lib):
> - ability to decode efficiently sub-windows inside a tile (I guess this means
> using precincts). This is particuarly true for single-tiled big files
> - reduce memory consumption. I've found that to decode a 2048x2048 tile of a
> Sentinel2 product, it takes 600 MB of memory whereas the uncompressed size of
> the tile is 8 MB (2048x2048*sizeof(int16)).
> - ability to open files who have more than 2 gigapixels (or 4, I'm not sure
> where the limit is)
> - currently the driver closes and reopen the file each time it reads a tile,
> because there are(were?) some instabilities in seeking through arbitrary tiles
> (ie reading potentially not in ascending tile order). Would be nice to be able
> to avoid that
>
>> Given that it is the slowest J2K library around, what would be acceptable
>> performance in the geo-spatial world?
> As fast as possible :-)
>
>> These changes apply to the current master version of OpenJPEG, which will
>> be released shortly.
> Were you refering to this PR : https://github.com/uclouvain/openjpeg/pull/668
> ?
>
> Even
>