[GRASS-user] importing Mars DEM takes too long

[Helmut Kudrnovsky]

Even Rouault-2 wrote
> On dimanche 20 novembre 2016 17:38:19 CET Anna Petrášová wrote:
>> On Sun, Nov 20, 2016 at 5:02 PM, Helmut Kudrnovsky <

> hellik@

> > wrote:
>> >>11 seconds can be pretty long depending on the size of the file. Try
>> >>to convert it to tif and load again, it will be probably much faster.
>> >>
>> > the gdal_translated uncompressed geotif has about ~150MB (jp2 ~48 MB).
>> > 
>> > the import takes about 7 seconds.
>> > 
>> > there is some difference in importing time ; maybe cause of compressed
>> vs.
>> > uncompressed.
>> > 
>> > it's worth to ask on the GDAL ML about performance of GDAL'S jpeg2000
>> > driver.
>> 
>> I just tested it on in wingrass both standalone and from OSGeo4W and
>> it's very slow with ~50MB JP2 file. I tested this one:
>> http://www.uahirise.org/PDS/DTM/ESP/ORB_020600_020699/ESP_020673_1750_ESP_02
>> 0528_1750/ESP_020528_1750_RED_C_01_ORTHO.JP2
>> 
>> How could you import it in 11 s? Maybe the problem appears only with
>> certain files?
> 
> When enabling GDAL debug messages (CPL_DEBUG=ON), one can see:
> 
> OPENJPEG: nX0 = 0
> OPENJPEG: nY0 = 0
> OPENJPEG: nTileW = 7552
> OPENJPEG: nTileH = 19113
> OPENJPEG: nTilesX = 1
> OPENJPEG: nTilesY = 1
> OPENJPEG: mct = 0
> OPENJPEG: psImage->x0 = 0
> OPENJPEG: psImage->y0 = 0
> OPENJPEG: psImage->x1 = 7552
> OPENJPEG: psImage->y1 = 19113
> OPENJPEG: psImage->numcomps = 1
> 
> The tile dimensions are the same as the image, meaning the image is single 
> tiled.
> The OpenJPEG library must thus decompress the whole time into memory since
> it 
> doesn't support decompressing a region of interest within a tile. To avoid 
> exposing a block size so large, GDAL exposes a 1024x1024 artificial
> blocking, 
> which will in fact the image to be decompressed each time one of those
> blocks 
> is read.
> 
> I've committed in GDAL trunk r36357 "OpenJPEG: add a USE_TILE_AS_BLOCK=YES 
> open option that can help with whole image conversion"
> thats add the open option :
> 
> USE_TILE_AS_BLOCK=YES/NO
*
> : (GDAL > 2.2) Whether to always use the 
> JPEG-2000 block size as the GDAL block size
> Defaults to NO. Setting this option can be useful when doing whole image
> decompression and the image is single-tiled. Note however that the tile
> size
> must not exceed 2 GB since that's the limit supported by GDAL.
> </li>
> (note: this limitation of 2GB is reached unfortunately with other images
> as 
> ESP_020528_1750_IRB_A_01_ORTHO.JP2)
> 
> $ time GDAL_CACHEMAX=150 gdal_translate ESP_020528_1750_RED_C_01_ORTHO.JP2  
> out.tif -co TILED=YES -oo USE_TILE_AS_BLOCK=YES -co COMPRESS=DEFLATE
> real	0m22.981s
> user	0m22.756s
> sys	0m0.220s
> 
> GDAL_CACHEMAX must be larger than width * height * data_type_size * 
> number_of_bands = 7552 * 19113 * 1 * 1 = 144 MB, so as to avoid the block 
> corresponding to the JPEG-2000 image to be evicted from the GDAL block
> cache.

Even, thanks for the insight and improvements on GDAL's level!



-----
best regards
Helmut
--
View this message in context: http://osgeo-org.1560.x6.nabble.com/importing-Mars-DEM-takes-too-long-tp5296530p5296689.html
Sent from the Grass - Users mailing list archive at Nabble.com.