Gdal Bug? was: Re: [gdal-dev] JP2MRSID Issues

[Stephen Woodbridge]

Frank Warmerdam wrote:
> Stephen Woodbridge wrote:
>> Frank, anyone,
>>
>> Thoughts on this issue?
>> Thoughts on how to better diagnose the issue?
>> I can talk to the client about making an image available to someone if 
>> that would help.
>>
>> I'm totally stuck on this one, converting 300+ images to tif first at 
>> 2-3 hours per image is not going to hack it.
> ...
>>>> gdalwarp: ../compressed/compressed.cpp:1724: bool 
>>>> kd_tile::read_tile_part_header(): Assertion `tpart_body_length >= 0' 
>>>> failed.
> 
> Stephen,
> 
> As I understand it, the problem you see is that under some access
> arrangements you are getting this kd_tile assertion from the JP2MRSID
> driver.  Is that right?  I'm not familiar with this error, and it is coming
> from deep inside the MrSID SDK (in fact within the subcomponent that is
> derived from the Kakadu library as we can see from the names).
> 
> My impression is that to make any progress you would need to supply
> a sample that demonstrates the problem.  At that point I could try and
> diagnose if I'm doing something boneheaded.  If nothing is apparently
> someone would need to bounce the issue (hopefully with a narrow example
> program operating directly against the MrSID SDK) to Lizardtech for further
> evaluation.
> 
> No offense to Lizardtech - we have vaguely similar issues with ECW,
> and Kakadu too - but it is just hellish to diagnose some of these problems
> deep in proprietary and/or complicated sub-libraries.  I find it hard to
> bring myself to even dig into such issues unless a paying client makes me
> do it, though I'd be willing to make a quick try for you if there was an
> easy way for me to try.

Frank,

Thank you for your response. I know you are busy and I really appreciate 
your help and suggestions.

I understand your reluctance on diving into vendor code issues, I feel 
pretty much the same way, but I also feel like I need to offer some 
options to the client. I will discuss having them talk to you about 
funding this if there are no other options.

> PS. My daily download limit via my satellite ISP is 500MB and I don't want
> to use it all up on a single jp2 file.  Please try to reproduce the issue
> with a small file.

I'm still looking for a smaller file that I can reproduce this issue on. 
The smallest file I have at 92MB worked fine, so I need to try a few 
more. The file below at 342MB exhibits the problem. Looking for 
something smaller.

> PPS.  I don't understand your point where you seem to suggest that 
> converting
> to tiff first is slower than other mechanisms.  Are you suggesting it takes
> 2-3 hours to convert a 500MB jpeg2000 image to GeoTIFF?  Perhaps you need
> to up your GDAL block cache size?

Yes, this is exactly my problem it is taking 2-3 hours to run:

GDAL_SKIP=JPEG2000 gdal_translate my.jp2 -b 1 -b 2 -b 3 my.tif

The above was a replacement to for creating a VRT file which was very 
fast using:

GDAL_SKIP=JPEG2000 gdal_translate -of VRT my.jp2 -b 1 -b 2 -b 3 my.vrt

This is a pretty beefy system with 6GB of memory and fast disks, etc.

How do I up the GDAL block cache size? Please give me a hint or point me 
to the right page(s).

I have also tried:

[swoodbridge at booboo ~]$ ls -lh 
/var/data/raw_data/minnesota/ortho_1-1_1m_j_mn161_2008_1.jp2
-rw-rw-r--  1 swoodbridgeusers 342M Oct 20 16:49 
/var/data/raw_data/minnesota/ortho_1-1_1m_j_mn161_2008_1.jp2

[swoodbridge at booboo ~]$ GDAL_SKIP=JPEG2000 gdal_translate --config 
GDAL_CACHEMAX 500 
/var/data/raw_data/minnesota/ortho_1-1_1m_j_mn161_2008_1.jp2 -b 1 -b 2 
-b 3 my.tif
Input file size is 36010, 49447
0...

and it does not seem to be any faster, but I confess to not having 
waited very long 2-5mins max.

I also tried the following which was a little faster at 43 mins:

GDAL_SKIP=JPEG2000 gdalwarp --config GDAL_CACHEMAX 500 -wm 500 -co 
"TILED=YES" my.vrt my.tif
Creating output file that is 36010P x 49447L.
Processing input file my.vrt.
0...10...20...30...40...50...60...70...80...90...100 - done.

Thanks,
   -Steve