[GRASS-user] importing Mars DEM takes too long

[Even Rouault]

On dimanche 20 novembre 2016 17:38:19 CET Anna Petrášová wrote:
> On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky <hellik at web.de> wrote:
> >>11 seconds can be pretty long depending on the size of the file. Try
> >>to convert it to tif and load again, it will be probably much faster.
> >>
> > the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
> > 
> > the import takes about 7 seconds.
> > 
> > there is some difference in importing time ; maybe cause of compressed vs.
> > uncompressed.
> > 
> > it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
> > driver.
> 
> I just tested it on in wingrass both standalone and from OSGeo4W and
> it's very slow with ~50MB JP2 file. I tested this one:
> http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_02
> 0528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2
> 
> How could you import it in 11 s? Maybe the problem appears only with
> certain files?

When enabling GDAL debug messages (CPL_DEBUG=ON), one can see:

OPENJPEG: nX0 = 0
OPENJPEG: nY0 = 0
OPENJPEG: nTileW = 7552
OPENJPEG: nTileH = 19113
OPENJPEG: nTilesX = 1
OPENJPEG: nTilesY = 1
OPENJPEG: mct = 0
OPENJPEG: psImage->x0 = 0
OPENJPEG: psImage->y0 = 0
OPENJPEG: psImage->x1 = 7552
OPENJPEG: psImage->y1 = 19113
OPENJPEG: psImage->numcomps = 1

The tile dimensions are the same as the image, meaning the image is single 
tiled.
The OpenJPEG library must thus decompress the whole time into memory since it 
doesn't support decompressing a region of interest within a tile. To avoid 
exposing a block size so large, GDAL exposes a 1024x1024 artificial blocking, 
which will in fact the image to be decompressed each time one of those blocks 
is read.

I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES 
open option that can help with whole image conversion"
thats add the open option :

USE_TILE_AS_BLOCK=YES/NO</b>: (GDAL > 2.2) Whether to always use the 
JPEG-2000 block size as the GDAL block size
Defaults to NO. Setting this option can be useful when doing whole image
decompression and the image is single-tiled. Note however that the tile size
must not exceed 2 GB since that's the limit supported by GDAL.</li>

(note: this limitation of 2GB is reached unfortunately with other images as 
ESP_020528_1750_IRB_A_01_ORTHO.JP2)

$ time GDAL_CACHEMAX=150 gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2  
out.tif -co TILED=YES -oo USE_TILE_AS_BLOCK=YES -co COMPRESS=DEFLATE
real	0m22.981s
user	0m22.756s
sys	0m0.220s

GDAL_CACHEMAX must be larger than width * height * data_type_size * 
number_of_bands = 7552 * 19113 * 1 * 1 = 144 MB, so as to avoid the block 
corresponding to the JPEG-2000 image to be evicted from the GDAL block cache.

Even

> 
> Anna
> 
> > -----
> > best regards
> > Helmut
> > --
> > View this message in context:
> > http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5
> > 296530p5296685.html Sent from the Grass - Users mailing list archive at
> > Nabble.com.
> > _______________________________________________
> > grass-user mailing list
> > grass-user at lists.osgeo.org
> > http://lists.osgeo.org/mailman/listinfo/grass-user
> 
> _______________________________________________
> grass-user mailing list
> grass-user at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/grass-user


-- 
Spatialys - Geospatial professional services
http://www.spatialys.com