[gdal-dev] Excessive range requests for JP2 files with openjpeg

[Even Rouault]

Jérémy,

good point. Adressed per https://github.com/uclouvain/openjpeg/pull/1560 
and https://github.com/OSGeo/gdal/pull/11270

Even

Le 13/11/2024 à 15:27, Jérémy Anger via gdal-dev a écrit :
> Hello,
>
> Trying to optimize the reading speed of JP2 files from object 
> storages, I've stumbled upon this PR 
> https://github.com/uclouvain/openjpeg/pull/514.
> It introduced a behaviour particularly harmful for network cropping 
> JP2 files, as it leads to many excessive requests (time, bandwidth and 
> cost wasted).
> As described in the PR, it is possible to disable the fix through the 
> compilation flag -DOPJ_DISABLE_TPSOT_FIX=ON.
>
> Here is a small experiment to compare with and without the fix:
>
> Setup:
>     $ gdalinfo --version
>     GDAL 3.9.2, released 2024/08/13
>     $ git clone https://github.com/uclouvain/openjpeg --branch v2.5.2 
> --depth 1
>     $ cd openjpeg
>
>     # build with the fix (default)
>     $ mkdir build-enabled-fix && cd build-enabled-fix && cmake .. 
> -DCMAKE_BUILD_TYPE=Release -DOPJ_DISABLE_TPSOT_FIX=OFF && make -j && cd ..
>     # build without the fix
>     $ mkdir build-disabled-fix && cd build-disabled-fix && cmake .. 
> -DCMAKE_BUILD_TYPE=Release -DOPJ_DISABLE_TPSOT_FIX=ON && make -j && cd ..
>
> Benchmark:
>     export GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR
>     export CPL_CURL_VERBOSE=ON
>     # it is possible to tweak CPL_VSIL_CURL_CHUNK_SIZE to reduce the 
> number of requests, but this is not studied here
>
>     # this file is composed of 121 tiles of size 1024x1024
>     
> url=/vsicurl/https://storage.googleapis.com/gcp-public-data-sentinel-2/tiles/31/T/CJ/S2B_MSIL1C_20230209T105109_N0509_R051_T31TCJ_20230209T124951.SAFE/GRANULE/L1C_T31TCJ_A030965_20230209T105200/IMG_DATA/T31TCJ_20230209T105109_B02.jp2
>
>     # tile at the beginning
>     $ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin 
> gdal_translate -srcwin 0 0 500 500 "$url" o.tif 2>&1 | grep ^Range: | 
> wc -l
>     122
>     $ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin 
> gdal_translate -srcwin 0 0 500 500 "$url" o.tif 2>&1 | grep ^Range: | 
> wc -l
>     2
>
>     # tile in the middle
>     $ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin 
> gdal_translate -srcwin 5120 5120 500 500 "$url" o.tif 2>&1 | grep 
> ^Range: | wc -l
>     122
>     $ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin 
> gdal_translate -srcwin 5120 5120 500 500 "$url" o.tif 2>&1 | grep 
> ^Range: | wc -l
>     62
>
>     #.tile at the end
>     $ LD_LIBRARY_PATH=$PWD/openjpeg/build-enabled-fix/bin 
> gdal_translate -srcwin 10240 10240 500 500 "$url" o.tif 2>&1 | grep 
> ^Range: | wc -l
>     122
>     $ LD_LIBRARY_PATH=$PWD/openjpeg/build-disabled-fix/bin 
> gdal_translate -srcwin 10240 10240 500 500 "$url" o.tif 2>&1 | grep 
> ^Range: | wc -l
>     122
>
> Note that disabling the fix is particularly important when TLM markers 
> are present (on the master branch of openjpeg), but this is not shown 
> here. The tests above are from openjpeg v2.5.2 and the Sentinel-2 
> rasters do not have TLM markers.
>
> It might be worth converting the compilation flag into a runtime flag 
> on openjpeg's side, and expose it through a GDAL option at least (or 
> disabling the fix from GDAL by default).
> What do you think?
>
> Best regards,
> Jérémy
>
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev

-- 
http://www.spatialys.com
My software is free, but my time generally not.
Butcher of all kinds of standards, open or closed formats. At the end, this is just about bytes.