[gdal-dev] Excessive range requests for JP2 files with openjpeg

[Jérémy Anger]

Hello,

Trying to optimize the reading speed of JP2 files from object storages,
I've stumbled upon this PR https://github.com/uclouvain/openjpeg/pull/514.
It introduced a behaviour particularly harmful for network cropping JP2
files, as it leads to many excessive requests (time, bandwidth and cost
wasted).
As described in the PR, it is possible to disable the fix through the
compilation flag -DOPJ_DISABLE_TPSOT_FIX=ON.

Here is a small experiment to compare with and without the fix:

Setup:
    $ gdalinfo --version
    GDAL 3.9.2, released 2024/08/13
    $ git clone https://github.com/uclouvain/openjpeg --branch v2.5.2
--depth 1
    $ cd openjpeg

    # build with the fix (default)
    $ mkdir build-enabled-fix && cd build-enabled-fix && cmake ..
-DCMAKE_BUILD_TYPE=Release -DOPJ_DISABLE_TPSOT_FIX=OFF && make -j && cd ..
    # build without the fix
    $ mkdir build-disabled-fix && cd build-disabled-fix && cmake ..
-DCMAKE_BUILD_TYPE=Release -DOPJ_DISABLE_TPSOT_FIX=ON && make -j && cd ..

Benchmark:
    export GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
    export CPL_CURL_VERBOSE=ON
    # it is possible to tweak CPL_VSIL_CURL_CHUNK_SIZE to reduce the number
of requests, but this is not studied here

    # this file is composed of 121 tiles of size 1024x1024
    url=/vsicurl/
https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/31/T/CJ/S2B_MSIL1C_20230209T105109_N0509_R051_T31TCJ_20230209T124951.SAFE/GRANULE/L1C_T31TCJ_A030965_20230209T105200/IMG_DATA/T31TCJ_20230209T105109_B02.jp2

    # tile at the beginning
    $ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin gdal_translate
-srcwin 0 0 500 500 "$url" o.tif 2>&1 | grep ^Range: | wc -l
    122
    $ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin gdal_translate
-srcwin 0 0 500 500 "$url" o.tif 2>&1 | grep ^Range: | wc -l
    2

    # tile in the middle
    $ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin gdal_translate
-srcwin 5120 5120 500 500 "$url" o.tif 2>&1 | grep ^Range: | wc -l
    122
    $ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin gdal_translate
-srcwin 5120 5120 500 500 "$url" o.tif 2>&1 | grep ^Range: | wc -l
    62

    #.tile at the end
    $ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin gdal_translate
-srcwin 10240 10240 500 500 "$url" o.tif 2>&1 | grep ^Range: | wc -l
    122
    $ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin gdal_translate
-srcwin 10240 10240 500 500 "$url" o.tif 2>&1 | grep ^Range: | wc -l
    122

Note that disabling the fix is particularly important when TLM markers are
present (on the master branch of openjpeg), but this is not shown here. The
tests above are from openjpeg v2.5.2 and the Sentinel-2 rasters do not have
TLM markers.

It might be worth converting the compilation flag into a runtime flag on
openjpeg's side, and expose it through a GDAL option at least (or disabling
the fix from GDAL by default).
What do you think?

Best regards,
Jérémy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.osgeo.org/pipermail/gdal-dev/attachments/20241113/3125e82f/attachment.htm>