[gdal-dev] Fwd: OpenJPEG

[Aaron Boxer]

---------- Forwarded message ----------
From: Aaron Boxer <boxerab at gmail.com>
Date: Mon, Dec 21, 2015 at 10:27 AM
Subject: Re: [gdal-dev] OpenJPEG
To: Julien Malik <julien.malik at c-s.fr>


Hi Julien,


> Decoding at preccints level in OpenJPEG would definitely be a great
> addition.
>

Yes, sounds useful.


>
> I believe OpenJPEG performance could also be improved if multi-threading
> was managed directly inside the lib, since in the current situation
> OpenJPEG is purely monothreaded.
> From my experience, Kakadu does a great job in keeping all your cores
> occupied close to 100% during decoding, even when decoding a single tile
> (so this is probably linked to the capability to manage independent
> decoding at the precinct level).
> In 3 geospatial software I know leverage on OpenJPEG (gdal, orfeo toolbox,
> and snap) to decode big datasets (think tens or hundreds of thousand pixels
> in each dimension), multithreading is handled from the user side, by
> creating a number of independent contexts and run them in separate threads
> to improve performance.
> This means we are all interpreting the codestreams and seeking into them
> independently, which clearly sounds suboptimal.
> I suspect managing multi-threading at a lower level directly inside
> OpenJPEG could open the doors to much better resource management (cores
> occupation, reduced I/O, etc...).
>


The pull request I mentioned does use all cores for the computationally
intensive part of decoding. Reading through the file is still single
threaded. And this can take a good chunk
of time.


> I certainly concur with Even with the "as fast as it can be" :)
> Any percent you gain has a direct visible impact when (de)coding huge
> datasets.
>
>
Thanks for the feedback. Good news for OpenJPEG performance : they will be
*hiring* a developer next year to focus solely on performance.
If you guys are interested, please check out the OpenJPEG mailing list and
you can apply.

Also, I would encourage submitting all of these great ideas as issues on
the openjpeg github site:

https://github.com/uclouvain/openjpeg/issues


Cheers,
Aaron


On 20/12/2015 11:00, Even Rouault wrote:

> Aaron,
>
> I am working on some performance enhancements for OpenJPEG,
>> and I am curious about people's experience using the OpenJPEG driver with
>> GDAL:
>>
>> Does it have all the features you need, if not what is missing?
>>
> Being the developer of the driver, I believe it must be close to using the
> available capabilities of OpenJPEG as best as possible. I may have
> overlooked
> some features, so other eyes checking the code are always welcome.
>
>  From my point of view what is mainly missing in OpenJPEG (the lib):
> - ability to decode efficiently sub-windows inside a tile (I guess this
> means
> using precincts). This is particuarly true for single-tiled big files
> - reduce memory consumption. I've found that to decode a 2048x2048 tile of
> a
> Sentinel2 product, it takes 600 MB of memory whereas the uncompressed size
> of
> the tile is 8 MB (2048x2048*sizeof(int16)).
> - ability to open files who have more than 2 gigapixels (or 4, I'm not sure
> where the limit is)
> - currently the driver closes and reopen the file each time it reads a
> tile,
> because there are(were?) some instabilities in seeking through arbitrary
> tiles
> (ie reading potentially not in ascending tile order). Would be nice to be
> able
> to avoid that
>
> Given that it is the slowest J2K library around, what would be acceptable
>> performance in the geo-spatial world?
>>
> As fast as possible :-)
>
> These changes apply to the current master version of OpenJPEG, which will
>> be released shortly.
>>
> Were you refering to this PR :
> https://github.com/uclouvain/openjpeg/pull/668
> ?
>
> Even
>
>
_______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20151221/44eebe94/attachment-0001.html>