Gdal Bug? was: Re: [gdal-dev] JP2MRSID Issues

[Ivan Lucena]

Stephen,

It seams like the suggestion to use VRT would be working all right if it 
wasn't for that that problem with MrSID.

Well, you can increase the value on GDAL_CACHEMAX to 500 (mb) or more if 
you have that amount of free memory.

By the way, since the crash on the MrSID driver(s) only happens when you 
select some bands so what if you use "gdal_translate -of MEM", then 
"gdal_translate -of VRT -b 1 -b 2 -b 3" and then "gdalwarp". That will 
put the problem away, since you will be extracting all the bands and the 
access to a MEM should be way faster than a Geotiff file.

Just an idea,

Regards,

Ivan

Stephen Woodbridge wrote:
> Frank Warmerdam wrote:
>> Stephen Woodbridge wrote:
>>> Frank, anyone,
>>>
>>> Thoughts on this issue?
>>> Thoughts on how to better diagnose the issue?
>>> I can talk to the client about making an image available to someone 
>>> if that would help.
>>>
>>> I'm totally stuck on this one, converting 300+ images to tif first 
>>> at 2-3 hours per image is not going to hack it.
>> ...
>>>>> gdalwarp: ../compressed/compressed.cpp:1724: bool 
>>>>> kd_tile::read_tile_part_header(): Assertion `tpart_body_length >= 
>>>>> 0' failed.
>>
>> Stephen,
>>
>> As I understand it, the problem you see is that under some access
>> arrangements you are getting this kd_tile assertion from the JP2MRSID
>> driver.  Is that right?  I'm not familiar with this error, and it is 
>> coming
>> from deep inside the MrSID SDK (in fact within the subcomponent that is
>> derived from the Kakadu library as we can see from the names).
>>
>> My impression is that to make any progress you would need to supply
>> a sample that demonstrates the problem.  At that point I could try and
>> diagnose if I'm doing something boneheaded.  If nothing is apparently
>> someone would need to bounce the issue (hopefully with a narrow example
>> program operating directly against the MrSID SDK) to Lizardtech for 
>> further
>> evaluation.
>>
>> No offense to Lizardtech - we have vaguely similar issues with ECW,
>> and Kakadu too - but it is just hellish to diagnose some of these 
>> problems
>> deep in proprietary and/or complicated sub-libraries.  I find it hard to
>> bring myself to even dig into such issues unless a paying client 
>> makes me
>> do it, though I'd be willing to make a quick try for you if there was an
>> easy way for me to try.
>
> Frank,
>
> Thank you for your response. I know you are busy and I really 
> appreciate your help and suggestions.
>
> I understand your reluctance on diving into vendor code issues, I feel 
> pretty much the same way, but I also feel like I need to offer some 
> options to the client. I will discuss having them talk to you about 
> funding this if there are no other options.
>
>> PS. My daily download limit via my satellite ISP is 500MB and I don't 
>> want
>> to use it all up on a single jp2 file.  Please try to reproduce the 
>> issue
>> with a small file.
>
> I'm still looking for a smaller file that I can reproduce this issue 
> on. The smallest file I have at 92MB worked fine, so I need to try a 
> few more. The file below at 342MB exhibits the problem. Looking for 
> something smaller.
>
>> PPS.  I don't understand your point where you seem to suggest that 
>> converting
>> to tiff first is slower than other mechanisms.  Are you suggesting it 
>> takes
>> 2-3 hours to convert a 500MB jpeg2000 image to GeoTIFF?  Perhaps you 
>> need
>> to up your GDAL block cache size?
>
> Yes, this is exactly my problem it is taking 2-3 hours to run:
>
> GDAL_SKIP=JPEG2000 gdal_translate my.jp2 -b 1 -b 2 -b 3 my.tif
>
> The above was a replacement to for creating a VRT file which was very 
> fast using:
>
> GDAL_SKIP=JPEG2000 gdal_translate -of VRT my.jp2 -b 1 -b 2 -b 3 my.vrt
>
> This is a pretty beefy system with 6GB of memory and fast disks, etc.
>
> How do I up the GDAL block cache size? Please give me a hint or point 
> me to the right page(s).
>
> I have also tried:
>
> [swoodbridge at booboo ~]$ ls -lh 
> /var/data/raw_data/minnesota/ortho_1-1_1m_j_mn161_2008_1.jp2
> -rw-rw-r--  1 swoodbridgeusers 342M Oct 20 16:49 
> /var/data/raw_data/minnesota/ortho_1-1_1m_j_mn161_2008_1.jp2
>
> [swoodbridge at booboo ~]$ GDAL_SKIP=JPEG2000 gdal_translate --config 
> GDAL_CACHEMAX 500 
> /var/data/raw_data/minnesota/ortho_1-1_1m_j_mn161_2008_1.jp2 -b 1 -b 2 
> -b 3 my.tif
> Input file size is 36010, 49447
> 0...
>
> and it does not seem to be any faster, but I confess to not having 
> waited very long 2-5mins max.
>
> I also tried the following which was a little faster at 43 mins:
>
> GDAL_SKIP=JPEG2000 gdalwarp --config GDAL_CACHEMAX 500 -wm 500 -co 
> "TILED=YES" my.vrt my.tif
> Creating output file that is 36010P x 49447L.
> Processing input file my.vrt.
> 0...10...20...30...40...50...60...70...80...90...100 - done.
>
> Thanks,
>   -Steve
> _______________________________________________
> gdal-dev mailing list
> gdal-dev at lists.osgeo.org
> http://lists.osgeo.org/mailman/listinfo/gdal-dev
>
>